Planet Library SOA

A blog entry by Paul Walk - An infrastructure service anti-pattern - drew some attention a while ago. He argues against a model in which a service provider independently develops APIs and a user interface, and in which, accordingly, the APIs are developed in advance of actual use or explicit external requirements. He claims that this model underlies aspects of historic UK higher education thinking, where application to application interoperability was seen as a good in itself and was often built into initiatives from the start in mistaken anticipation of hypothetical future use.

His alternative suggestion is that an API should be developed in tandem with real requirements generated by the user-facing application supporting user interfaces. This motivates the API with real requirements and strengthens the chances of being able to support third party application developers who want to write to the API. He suggests a refinement of this approach as follows:

An interesting alternative to this is the approach of combining the user-facing web pages and the machine-actionable API into one interface, through embedded RDFa for example:

It remains to be seen how this approach is going to work out over time, but we have seen hints of simpler approaches to combining user and machine interfaces in the past, such as RSS being styled to give a decent human-readable interface, or earlier attempts to do interesting things with XHTML. [An infrastructure service anti-pattern]

I left a note suggesting that VIAF and Worldcat Identities were architected along the lines of this last picture. For VIAF details, for example, see Thom's posts here and here. Several interfaces - OpenSearch, linked data, user-oriented - are supported by an underlying SRU interface. Here is Paul's response, noted here with his permission:

I’m fascinated to see that VIAF merges the human/machine interfaces in two ways:

a machine-centric, but with a human-readably styled version if accessed from a browser

and

a human-centric, get the machine view with a little addition to the URL approach here

the second of which is, in turn, also styled for human consumption if accessed from a browser.

I just finished reading What Technology? Reflections on Evolving Services, a report from Sharon Collins and the 2009 EDUCAUSE Evolving Technologies Committee. For the first time, Information resource management technologies for libraries were featured as an evolving technology.

From the report:

"The increasing primacy of highly distributed digital resources has brought disruptive change to the way libraries must approach their work to remain relevant to their parent organizations and constituencies."

"Organizing content to support research and learning is at the heart of the library's institutional role. Once limited to applying subject terms, co-locating physical materials, and producing research guides, this role has been changed by the volume and variety of online resources, which require new tools to more effectively meet the needs of users. A growing collection of technologies and tools can be used to more granularly organize, customize, and personalize the online information environment to fit professional, learning, and research activities."

"These technologies are evolving away from being strictly stand-alone tools and resources and are converging into a more interoperable, collaborative, enterprise-level information management environment — one more closely integrated with teaching, learning, research, and administrative systems. Underlying system architectures are focusing more on providing discrete services (service-oriented architecture) rather than monolithic systems, enabling more interoperable and customizable workflows."

"By combining discrete services with cloud storage and cloud-enabled applications, institutions can build collaborative work environments between libraries as well as between libraries and non-library units, both on and off their home campuses, for discovering, acquiring, describing, and managing all types of resources. Layered over this enterprise-level resource management environment, information discovery and management tools are providing individuals and workgroups with much more intuitive and productive ways to discover, manipulate, incorporate, and share information for teaching, learning, and research, allowing users to shift time from the mechanics of managing specific resources to a focus on analyzing the information itself."

There is an announcement on the OLE Project site that links to the final report as submitted to the Mellon Foundation. This version of the report has minor corrections in the text and now includes information about the group of libraries that have committed to the build phase of the project. Those libraries are:

• Indiana University (lead)
• Florida Consortium (University of Florida, Florida International University, Florida State University, New College of Florida, Rollins College, University of Central Florida, University of Miami, University of South Florida, and the Florida Center for Library Automation)
• Lehigh University
• Triangle Research Libraries Network (Duke University and North Carolina State University participating)
• University of Chicago
• University of Maryland
• University of Michigan
• University of Pennsylvania

The project is also listed as being in the incubation phase on the Kuali website.

Post from: Disruptive Library Technology Jester

OLE Project Design Phase Final Report

Carl Grant, president of Ex Libris North America, posted a pair of messages on his corporate blog that it is worth calling attention to regarding the OLE Project final report, if you haven’t already run into them: OLE; The unanswered questions and Library Software Solutions – We need a higher level of discourse... Equally important is the comment on the first by Brad Wheeler, Vice President for Information Technology and Chief Information Officer at Indiana University. The whole thread should take about five minutes to read; five minutes well spent if you are interested in the intersection of community source software development with proprietary, closed-source software development. It is even more important if you are looking for a case study of governance issues surrounding community source software development. Go ahead…I’ll wait.

It is important to clear up one misconception. Brad Wheeler, at Indiana University, was not involved in the design phase of the OLE Project. Heavens — I hope someone who is VP of IT and CIO of a major university would have better things to do than to slog through discussions of workflow decomposition of back-room library automation functions. No, Brad is a consumer of the final report draft, just as many library directors and CIOs are the intended audience of the draft. In fact, Brad is a special category of report readers because he is at the forefront of community source in higher education; he believes in the community source model because he is witness to how it has and is working. From his biography:

Dr. Wheeler has been a pioneer in leveraged models for university collaboration. He serves in leadership roles for over $50M of shared university investments in open source software and was a co-founder of the Sakai Project and co-principal investigator on its $2.7M in grants and the $500K Open Source Portfolio project. He was a co-founder of the original Kuali Project, and now chairs the Kuali Foundation, Inc. board of directors. He has been a co-principal investigator on $4M of Kuali grants and three of the foundation.

Sakai (learning management system), Open Source Portfolio (electronic portfolio, since folded into Sakai), and Kuali (a suite of project including student records, research tracking, university financials, as well as other systems) are all well-regarded open source projects that are “leveraged models for university collaboration” — projects where universities have pooled their resources to develop and own the systems that are at the core of their institutions. The OLE Project is looking to join the Kuali Foundation and follow its proven and evolving patterns for community source development. From that perspective, it might be valuable to go back and reread Brad Wheeler’s response to Carl’s first post.

As someone embedded deeply in the OLE Project design phase, I can only speak for myself. Given the depth my head was in the design documents — and there were many others deeper than I was — it is difficult to separate reality from my perhaps biased impression of the report. I’m not sure a point-for-point debate on the merits of the report are useful as public discourse. Rather, I for one am listening to what others have to say and assimilating that into my version of reality. I’m grateful to Carl for taking the time to post his observations, just as I am to Brad and the other commenters on Carl’s post.

Post from: Disruptive Library Technology Jester

A Thread of Comments on the OLE Project Draft Report

It doesn’t happen often, but it is really nice when when you receive something produced for one purpose to find that it has been produced so well that it is good for so much more.  Let me explain….

My colleague Andy Latham has been pulling together a white paper Opening the Walls of the Library – SOA and web services at Talis[pdf].  It’s main purpose is to support the marketing effort behind Talis Keystone, our SOA platform that underpins Talis Library Integration Services.  To help explain those services, to the not necessarily technical people in library and other departments considering integration, he needed to explore the history, principles, and practical considerations of this approach.  It is in this explanation, I believe that he has produced a document that is a great introduction to the application of SOA and library web services in general.

Because of it’s original purpose, and the fact that for obvious reasons the examples and case studies come from Talis products and customers, the document could be considered by some as being a bit marketingy.  Nevertheless, if you want an overview of real-world issues (many of which are to do with people not technology), or business models, or web service functions, or why choose REST in favour of SOAP, in library SOA I can recommend this White Paper as an informative easy way in.

As Andy says in the conclusion:

SOA is not all about technology; SOA is a business journey that needs to follow a path with small commercial and technical steps towards a known vision of business maturity. Commercial and Open Source technology has paved a way for businesses to begin introducing an SOA strategy. Introducing an SOA strategy is as much of a technical challenge as it is an operational challenge as the technology will break down silos between teams, departments and organisations and conflicting business processes which worked well in the silo will need to be redeveloped to meet the new needs of the more agile business.

The release of the OLE’s report, which I commented upon previously, plus vendor initiatives such as OCLC’s Web Services and Ex Libris’ URM, have served to raise the prominence of web services in the world of libraries.  On a recent Library 2.0 Gang show about the OLE project it was clear, in the discussions between Andy, OLE’s Tim McGeary, Marshall Breeding and Ex Libris’ Oren Beit-Arie, that there is much more to integration than just technology.

I think it is fair to say that Libraries as a sector have not been at the leading edge of the SOA/web services debate.  It is also fair to say that for whatever reason the UK seems to been a few years ahead of some areas in reaping the benefits of such integration in libraries.  As Andy’s document shows, there is the potential for significant financial and organisational benefits when undertaking integration in this way.

“The 25,000 students at one of the largest Universities in the UK are now able to pay their library charges online using either debit or credit cards, enabling further efficiency savings for library staff and improving student services.”

“Getting relevant information from Voyager into personalised portal sites has been a key requirement for the University for some time…..  By building a SharePoint integration we are maximising the positive impact of our new VLE and enhancing elements of the Library service.”

“The University of Salford is in the process of transforming the way that the identities of its entire user population are managed across all key systems in the organisation. An essential part of the solution employed (using Sun Microsystems’ IdM suite) is the transition and management of up to 23,000 Talis LMS borrower identities via Talis Keystone.”

To reap these sort of benefits in a sustainable way a library has to be aware of, and have, a SOA strategy.  There is much in this white paper that can help those new to the subject to understand the issues.  As someone who thinks he knows about these things, I also found it very useful for checking and clarifying my assumptions.

So as I say, a recommended read….

Over the weekend, the folks at Duke University coordinating the development of the OLE Project Design Final Report released a draft for public comment. Weighing in at 100 pages (don’t let that put you off — there are lots of pictures), it represents the best thinking of a couple dozen individuals listening to hundreds of professionals working in libraries. Participants were challenged to consider not only their existing environments and workflows, but also how things could be put together differently. And “differently” — in this context — means thinking about tighter integration with information systems and processes at the host institution.

I’m proud to have worked on and been associated with the project to this point. My future participation will depend on whether OhioLINK finds itself in the governance structure being set up to oversee the build process. I’m excited about the direction this project is going, and hope that enough build partners can be found to propel it to the next phase.

This is a project of and for the community. Comments are welcome on the draft.

Post from: Disruptive Library Technology Jester

Open Library Environment Final Report Draft Released

In a service-oriented architecture, dynamic data structures and service design allow for greater specificity or "tailoring" of data. The structural difference in SOA-generated data creates new opportunities and challenges for data warehousing and business intelligence. Learn about the strategies and outcomes our consortium of institutions has discovered.

Designing the next-generation student information system based on service-oriented architecture technologies requires a complete change in SIS and SOA design (SOAD) methodologies. Service design creates an SIS framework that can be shaped to each institution's needs, not a point solution. Come see how we design services.

There will be two programs at the ALA Annual Conference in Chicago where aspects of the Open Library Environment Project will be discussed. The participants in the design phase of the project encourage you to attend one or both of them to learn about the design phase deliverables and the plans for the build phase.

Post from: Disruptive Library Technology Jester

Two ways to learn about the OLE Project at ALA

The Research Information Network in the UK has released a timely report: Creating catalogues: bibliographic records in a networked world [Splash page; pdf]. It is concise and has a useful Summary and Key Findings section.

I found it an interesting read, in no small part because it rehearses various key themes of these pages. Critically, it discusses systemwide reconfiguration of library services, a far-reaching and critical issue for libraries in a network environment.

There are two overarching issues. One is that library processes are inefficient because resources are consumed in redundant activity that do not create distinctive local value. The second is that a fragmented presence on the web reduces impact and visibility. Of course each of these is related to the institutional-scale nature of much current work, which is poorly aligned with emerging network organization, where we have become accustomed to institutions externalizing activities. The network allows routine work to be consolidated (think payroll) and it favors concentrated user hubs (think Amazon and Google).

So, from a processing point of view it is not surprising to see recommendations which point to better sharing of the burden of data creation across the book/journal world and between libraries (and in this context, there is a discussion of record re-use and innovation). And from an impact point of view, there are recommendations about consolidating the web presence of libraries in order to better focus user attention.

Perhaps the recommendation which will receive the most attention is the following:

Libraries are therefore spending significant resources in editing the records they receive, as well as adding data to meet their own local needs. Sustaining and developing individual catalogues for the more than 160 university libraries in the UK demands considerable resources. A shared catalogue for the whole UK higher education (HE) sector, with dynamic links to local holdings, could bring enormous benefits, in terms of reduced costs, of a more comprehensive coverage of both national and local holdings with better-quality records. It would also provide the potential for developing new user-focused services allowing them to remain relevant to their users and to compete with Amazon, Google and others.

I guess I would not emphasise competition in this way, as one of the functions of such a concentration (noted elsewhere in the report) would be to act as a switch and syndication point on the network.

The report has sensible recommendations about better provision of e-book data; it argues for better dissemination of article level data; and it supports various consensus making activities. I was surprised not to see more discussion of knowledgebase/ERM activity, although this may have been out of scope. This is especially because this is also an area with some potential for consolidation and network concentration.

I will watch with interest how this report is received and discussed. I think that it is inevitable that libraries will externalize more of what they do to third parties and shared services. This is both to increase efficiency and to increase impact. This does not remove the local role, rather it represents a natural evolution in which time and attention can be devoted to changing local needs and interests.

Update: The report does not provide a blueprint for action. It does put a stake in the ground which may shift the conversation.

The new prototype discovery service from the National Library of Australia caused a ripple of interest the other week when it was released. One reason for the interest is that it brings together access to a range of NLA resources (Picture Australia, Libraries Australia, and Pandora, among others) as well as to external resources (Oaister, for example). The rationale for this approach was outlined in an NLA report from two years ago. I will do an entry about the new service, but in the interim here is what I wrote about the report which argued for bringing the systems together in this way .....

The National Library of Australia has made an interesting report available, National Library of Australia IT Architecture Project Report, March 2007. [pdf] Here is the declared purpose:

The aim of this report is to define the IT architecture that will be needed to support the management, discovery and delivery of the National Library of Australia's collections over the next three years. The current architecture has allowed the library to develop a significant digital library capability over the last decade. Now the burden of maintaining and supporting existing systems and services is increasingly hindering us from bringing new services online, improving the user experience, exploring new ideas or responding to technological change. In the meantime, enormous changes are occurring in the broader environment.

The report identifies three major responses within the context of a new framework for digital library services (I talk about them in a different order than the one in which they are presented). One, it recommends a move to a service-oriented architecture. The grounds for this are clear, and clearly made in the report. They include the ability to share common services across applications, to be able to respond to change effectively, and to reduce over time the redundancy, cost and complexity of development.

Two, it argues for using open source solutions where they are 'functional and robust'. It notes an amendment to prior policy which favored a buy over a build policy. The Library will now consider open source solutions based on function and cost comparisons. The assessment of cost will not only include consideration of the direct costs of additional development but also the benefits of contributing code to the community and, interestingly, the opportunity costs of using commercial software whose development path is not aligned with library direction and need. The report notes the possibility of collaboratively sourcing some functionality with partners.

And three, the report talks about a 'single business' approach. This was the most interesting aspect of the document to me, because it underscores a major issue for libraries and the systems they deploy. This is that applications have developed in a piecemeal fashion over recent years, so that library operations are now supported by many applications, in different stages of maturity, and with different levels of process standardization. However, this ensemble of applications does not support efficient working across the range of library requirements, and inhibits flexible service development. Indeed, boundaries between these applications seem increasingly arbitrary, and to owe more to historic circumstance, and to the structure of the industry that has developed over time, than to current needs. Simply managing this diversity is a major task in itself. The ERAMS (electronic resource access and management services) discussion I mentioned a while ago is one symptom of a growing sense that the library systems landscape needs to be redefined.

The 'single business' approach is a recommendation that the library think in terms of a single 'business' and a single data corpus as part of its planning process, rather than in terms of separate planning for each service line or resource type (e.g. images, books, music). And that technical solutions be designed in ways that minimize the number of separate business applications that need to be developed. Of course, the service-oriented approach would facilitate the latter goal. In practice this would mean trying to streamline workflow across management environments for different resource types; using common delivery, rights management and other solutions; and developing a single integrated discovery environment across collections and resource types, which can be accessed through different views.

The report is well structured, and is worth reading as much as for its discussion of some general issues as it is for the particular National Library of Australia situation. .

[Original post]

Last Friday, Andrew Pace (Executive Director of Networked Library Services for OCLC) was interviewed by Richard Wallis of Talis on OCLC’s recent announcement of a cloud-based library management service. As part of that conversation, Richard and Andrew touched on the ongoing debate on the OCLC record use policy. Below is a transcript from that part of the interview (with time markers from the start of the interview).

Richard Wallis (27:00)

What about [libraries'] local data? By providing data up onto the OCLC platform, will that data be restricted in its use — how they can use it — or will it be totally open for them to use it in any way that they want to?

Andrew Pace (27:17)

That data is the library’s data.

Richard Wallis (27:21)

One of the reasons I ask that question is obviously we’re aware of the issues about bibliographic record reuse licensing that is going on at the moment. Do you see that conversation having any impact on the back-end data or the usage statistics data or anything like that?

Andrew Pace (27:41)

I imagine there will be service-level agreements we’ll build for the data that are going into library management services, but I am reluctant to combine what is going there with the record use policy discussions. I think as the record use policies are under revision — they are under discussion right now — I think libraries are cognizant of all of the data issues, but I am reluctant to tie the two together completely.

Richard Wallis (28:11)

So you probably see the bibliographic conversation separate from the raw data type conversation.

Andrew Pace (28:20)

Yeah, I think they are related to each other but I think they are separate conversations.

I think this is absolutely the right answer, and I’m glad to see the distinction between the shared bibliographic data and the holdings/circulation-transaction data so cleanly separated. They are related, but in the case of the former it is truly the library’s data. Earlier on in the interview, Andrew addressed the issue of how OCLC would respond to disclosure requests from law enforcement agencies.

Richard Wallis (21:35)

How would OCLC handle an inquiry under the [USA] Patriot Act or something like that?

Andrew Pace (21:42)

I might beg off on that as being a legal question, but it is one that we have asked about what it means for that data. I’m not sure it is going to be entirely different than how libraries would have to deal with it on a local system.

Richard Wallis (22:02)

I suppose the only concern is if you have the records for transactions in a signifcant number of libraries, it may actually be somewhere that government star people might want to wander and ask questions. I suppose that is where it is different in this environment.

Andrew Pace (22:20)

Yeah, and what I’m arguing is that it would be similar situation to software as a service or other hosted applications as well. But I’m not going to attempt any kind of legal answer since I’m not a lawyer.

Richard Wallis (22:37)

Ah, that’s disappointing. I could have quoted you back to yourself in a year’s time, but never mind, I understand why you ducked that question. I would have done so as well.

For me, this is further evidence that OCLC would consider the transaction data to be owned by the library. I’m not a lawyer either, but this would seem to push the responsibility for responding to a law enforcement agent request to the member library. Hopefully, there is legal precedent to make that stick.

Overall, it is a good interview that really puts some added definition to the plans for WorldCat Local Library Management Services.

Post from: Disruptive Library Technology Jester

At the Intersection of the OCLC Records Use Policy and the WorldCat Local Cloud-based Library Management Service

Richard: A non-technical librarian would read what is out there at the moment as OCLC is offering me a library system. Is that what it is? Andrew (12:46) - I would say that it is a cloud-based library management service. If it is meant to include the functionality that you have in a local system, the answer is yes. But I am reluctant to describe it as an integrated library system because I think this is not meant to duplicate what is currently available in that market -- that is not the goal. The goal is to start with the cooperative WorldCat and use that as the basis for managing local inventory. Richard (13:49) - Could I envisage being an OCLC member, for instance in three or so years time, imagining that I wouldn't need a local ILS -- I could do it all on OCLC's cloud. Andrew (14:00) - Yes.

Numerous new toolsets and products are available to facilitate the transition to an SOA-based architecture. Examples are IDEs for the rapid deployment of web services, middleware infrastructure such as enterprise service bus solutions, and new application servers such as GlassFish by Sun. This panel presentation will feature practical applications of SOA concepts in higher education, including the Kuali Project and other multi-institution software implementations that require standardized, flexible, reusable interfaces. Panelists will discuss the strategies behind their SOA efforts, as well as their experiences in using the first wave of open-source and vendor-supported "pro-SOA" offerings in the marketplace.

The OLE Project, with funding from the Andrew W. Mellon Foundation, is designing an Open Library Environment (OLE), a reconceptualized alternative to the current model of vendor-driven, largely self-contained, integrated library systems. The goal of this multinational group of academic, research, and national libraries is to produce a design document for a service oriented architecture (SOA)-compliant system positioned as a component of an institutional enterprise architecture. A secondary goal is to inform community-source library system development efforts regarding the tools and techniques of business process modeling and SOA architectures.

by Karen Coyle

The executive director for networked library services at the world’s largest library consortium paints the big picture regarding OCLC’s web-platform initiative in an April 24 conversation with American Libraries Editor in Chief Leonard Kniffel.

What more important problem could we solve than helping students make intelligent decisions in their course selections? The South Orange County CCD created My Academic Plan (MAP), a new system dedicated to helping students define, refine, and implement their personal educational goals. Learn how we developed this system utilizing a user-driven design team, a service-oriented architecture, the latest technology, and a passion for serving students.

Participants in the design phase of the OLE Project met in Lawrence, Kansas, earlier this month for a week-long work session. Coming out of the session are several documents that form the foundational elements of the report to be published and delivered to Mellon in July. Interested parties are invited and encouraged to sign up for the project update webinar to be held on March 31st from 3:00pm to 4:30pm (Eastern time). There will be a project update at the Coalition for Networked Information (CNI) Task Force Meeting on April 7th. Those in the midwest might also be interested in the Indianapolis OLE Workshop on April 22nd.

One document is the updated Frequently Asked Questions; this has answers to the common questions received during the previous regional workshops and webcasts. This is backed up by two other documents: first the assumptions underlying the design of OLE and the OLE project scope. The latter is notable in particular because it was updated to take on the perspective of the OLE Project in general, not specifically the design phase of the project (as it was originally).

For the most details, see the OLE Project Meeting Notes from Kansas, March 15-20, 2009.

Post from: Disruptive Library Technology Jester

Open Library Environment Project Picks Up the Pace

Organizations and information system architectures are both embedded with a set of values, beliefs, and assumptions about what they should do and about how they should go about getting things done. This presentation will offer a conceptual framework for understanding the culture of an organization and the cultural differences between data-oriented and service-oriented architectures.