Audacity
Gnome Wave Cleaner
icecast.org

and many more software links at Wikipedia: List of Linux audio software

Then this evening I ran across this, by the incomparable Dorothea Salo:

I heard yesterday that one such corpus, while of impressive quality and very highly regarded in the discipline, was all but invisible on its home campus, according to the corpus’s own staff. Basically, these projects are what I have previously called fiefdoms. (If you don’t like that word, you may wish to substitute “research lab.” Most of what I’ll say applies to them as well.)

Sustainability is the crucial flaw in any sort of fiefdom model for data management. Most fiefdoms get the ball rolling with grant money. This may commit the institution to a certain amount of financial or in-kind support (depending on what the grant spells out), or it may not. If it does, that institutional support lasts only as long as the grant does. No one in this cycle—not the researchers in the fiefdom, not the institution, not the grant agency, no one—takes responsibility for the post-grant existence of anything the fiefdom produces.

Sustainability: the institutional fiefdom

any suggestions for libraries for demonstration projects?
– most common place to start is a thesis/dissertation IR; above that, more generally for scholarly works by faculty
– beyond that, far less standardized: storing data for faculty, e-science projects
– where are you at as an institution, do you have the resources in the library to support more ambitious efforts?
– danger is “if we build it, they will come” doesn’t work (ref Dorothea Salo: formerly here, now there)
– look at the software tools and staff resources, then look at faculty interests on campus

what are some ways in which rapid change can be encouraged?
– rapid, transformative change almost always comes from outside the organization
– simply let people use technology in unexpected ways
– can lower the resistance by providing the rights and the infrastructure to do interesting stuff
– training for librarians: semantic web camps, data linking camps etc – standards are still being built, aren’t many experts; in the short term, simply being a part of this conversation is a step
– pay attention, get training, build collaborations with faculty: change will come from users, not institutions

what might be strategies for realigning fiscal imperatives?
– open question
– institutions and funders are part of the same system
– can begin to make change by looking at past successes; e.g. of genomics and proteomics: funders have created a standardized data library through NLM, requirements that publications reflect data in same standardized ways
– when we fund the creation of data, it needs to live somewhere on the internet, with standard identifiers, in ways that others can access it
– yet to be decided who will do this, and how; but important role for libraries to provide storage facilities, naming standards

with respect to involving faculty, how can librarians most effectively bring faculty onside?
– faculty aren’t standardized
– try to find people who already agree with us, rather than expend the energy to convince people we’re right
– build understanding of what their role is, what they can achieve by working with librarians, will be different how to do this with different faculty

are IRs generally under the library?
– yes, don’t know of cases where it’s not

what are your hopes for Open Access Week?
– good way to get general knowledge built
– outside the open access movement, most people are completely unaware
– students get involved, librarians reach out to faculty, we’re all in this together: broader understanding that we can change in the way we communicate knowledge; get the message out beyond the niche

genome project mechanisms?
– big, international project
– data went into NLM site, by default entered public domain legally; raw facts, not creative works, so from a publication perspective they were public domain from the outset
– in practice, the process was slower, many of the sequencing labs held on to their data waiting for publication: US government met with the scientists and developed a process to drop all sequencing data into the website within 24 hours, but retain the right to first publication on the sequences they had included
– also required all publication on genomics to reference database identifiers to sequences being used: became habitual that everyone simply started depositing all their sequencing in the database, and open data access became normal: a set of interlocking initiatives that created the ecosystem for open data

how might librarians acquire necessary skills?
– technologies are relatively difficult to pick up, but there are semantic web conferences, get-togethers, mailing lists: lots of chatter on the web where standards are being developed
– best way to get involved is volunteer and get going
– begin looking at your library’s metadata and see whether there are pieces that can be exposed, make your stuff usable and findable

do other institutions play a similar role to libraries?
– libraries are important infrastructure to the data network: stability over the long term is difficult for institutions to achieve, libraries/universities tend to stick around longer.
– libraries providing storage infrastructure, metadata
– researchers must see a value in marking up their data, in reaching out to the library
– lastly, companies will come to see these open data sources as resources for creating value-added services that might be profitable
– libraries need to create the most usable, open layer; think of your content as a web 2.0 platform on which others can build applications

interoperability best practices and norms?
– differ widely, difficult to make a general statement
– even within the life sciences, very different between those who deal with people and those who don’t (privacy, etc.)
– need data that can be reused, funders should require reusable data
– universities need to look beyond the impact factor of journals towards the accessibility of data/informaton when evaluating faculty
– creators need to be rewarded for interoperable and reusable data

how to manage conflict between collaboration and competition?
– some people just like to share, others don’t
– by promoting to people who do like to share, over time universities who provide sharing tools will out-compete universities who don’t provide that platform
– expects a new set of impact factors that will reward scientists whose data leads to more data and more publications
– institutions who are ready with that platform will be in a position to recruit the best faculty: your job is not just to harvest data for publication, but to promote reuse in a web context, and we can support that better

are open access economic models useful for open data?
– big differences
– legally, it’s universally accepted that copyright applies to journal articles, copyright is comparatively harmonized internationally re articles; don’t have that with data: public domain in US, in UK there’s a “sweat of the brow” protection, crown copyright applies to govt data in commonwealth
– economically, without the harmonious legal framework, there are disincentives to overcome the issues of data accessibility: data needs to be processed to be understood, so it’s not just about publishing and dealing with copyrights, there’s an enormous cost to formatting data for human or machine reading
– no established industry of data promulgation: no entrenched industry fighting change, but also no established ways of doing things; have to bootstrap a data publishing system: registration, certification, dissemination, preservation
– no peer review system for certification, difficult to disseminate, how do we decided what to preserve and for how long….
– add additional layers specific to data, and it becomes really hard to envision open data

how to get large, commercial non-profit journals to switch to open access?
– easiest way to make this transition is for journal to allow authors to retain copyright
– economic issue is bigger, because commercial non-profit journals rely on subscription fees
– education around support fees, etc.
– a lot depends on the sector, the journal, the finances
– key is to remember that it’s usually an economic rather than a legal problem

some of the more successful approaches to encouraging researchers to follow good data management principles?
– astronomy; open, standardized system where experts and non-experts can post and share data
– incredible number of standards developed by the discipline
– model we should be looking to
– the people who cared about sharing data put in the time and effort to make their data open and interoperable
– researchers are motivated by solving problems they have: must work to create a culture in which the solution to your problem is an open solution
– incentive is to publish; must develop an infrastructure that makes shared data usable, useful, and valued; there is no scholarly punishment for bad data management

[missed one question, but the gist was that the web becomes the infrastructure for humanities and social science applications as well
– the expertise will get paid for in the sciences before it gets into the humanities

how to address learning curve for researchers?
– starting to have more skills, rapid web prototyping changing the game
– outreach and training should be the universities job: wouldn’t it be nice if everytime some did powerpoint, instead they could run a simple demo program?
– waiting: generational changes
– 15 years ago writing web pages was an arcane exercise
– today, prototyping systems make it easier for people to write code that acts on data

neurocommons project?
– initial focus in neuroscience, but it has forked into these three things:
1. a distribution of integrated, public domain data which will soon have a name, becomes the nucleus for distributing data in a package model similar to linux distros
2. standards development effort: web ontology not far enough along
3. problem with persistent web identifiers: proposed semantically empty URLs that are persistent and allow mapping of names, binding them under unique identifiers; shared names can’t be taken private or changed, the community owns the names and the people who use the names are in charge of them
– future goals: RDF distribution adopted as standard communication protocol for data
– need to get more people involved and engaged
– really want shared name thing to become equivalent to DNS for data, prefer to have discussions around that in the community rather than just in Science Commons

Turkel’s web page

Historians and others working in a monographic mode tend to present the past as a monolith, supported by an internal structure of references; but you try to hide that fact from others, don’t let the pieces show.

Footnotes really plug our work into everyone else’s. Historians train to analyze other peoples footnotes to criticize their work at the same time as they learn to hide their own away.

Works are built from other works, a commonplace of our era. Works are made from pieces of others, in dialogue with one another: remix, remake, celebrate. The audience wants to talk back. Few-to-few model gave way to broadcasting model, giving way to many-to-many communication. Access Conference has a rich back channel, IRC, twitter, etc. Can we incorporate that into our scholarship?

How do we design scholarly works so that they can be hacked? Requires open source, open access. What would it be like if history were written by anonymous people (missed citation name); Wikipedia as example: 1.5M articles in English alone–authors with no vested interest (in principal). W is the tip of the iceberg for a new way of thinking about how we create knowledge, give it authority. Traditional forms of scholarship might offer accreditation and peer review as networked goods. W is a deeply phililogical exercise. Collaborative editing software keeps track of and makes available every edit anybody ever does.

IBM research project “History Flow” a visualization for W edits. Can see consensus building, failure thereof, bursts of interest. UC Davis project “Code Swarm” visualizes tracking history of edits to an open source software project–networked individuals making lots of changes. A role for the future of historians to sort out how these things came to be. How would you write the history of Apache, for example. Must think about the activities of thousands of people working in real time in collaborative environments: making sense of what we’re doing as human beings.

Data and object creation going on constantly and incredible rate. P.W. Anderson “More is Different” (paper), arguing that at different scales, different laws kick in. Can’t infer qualities of larger structure looking at the properties of smaller structures. Differences between properties of paper sources and data sources begin to show some of the possibilities. Data can be replicated almost without cost!

Networks allow scholars to collaborate in new ways, but in many disciplines (History) no thought of doing anything more than monographs. Big difference: can do computational analysis on ANY source. Plebeian Lives project: may be able to follow lives of up to 80% of individuals living in 18th century London. Data mining, visualization, machine learning tools will be necessary. Turkel created a machine learning program to analyze Old Bailey records. Uses statistical techniques, becomes very good at looking through documents a researcher hasn’t looked at yet and sorting probably interesting records to the fore. Old Bailey is building this into their website. Can potentially share learning machines, leading to the development of open source pre-trained learning machines.

Compression-based clustering: can use compression techniques to build a general purpose yardstick to evaluate similarity of files and then cluster related texts. E.g. of Cdn Dict of Biog, see clusters of related biographies form quickly.

Interlinking between digital world and physical world. Started with handheld computers with GIS support, historical maps–MA student walking tour with elementary students. Initially all by hand, now use semapedia.org for grassroots geotagging: layers of info over everyday places. iPhone 3Gs and Android phones include digital compasses and accelerometers; can tell which direction you are pointing it, link to existing annotations about physical environment. Huge potential to transform how we understand places and the past. Every single place is an archive of sources, everyone and everything is a time-traveller from the past. Talking about the ability to attach virtual sources to physical places and objects. The authority of people who formerly had to interpret stuff becomes less, world is perhaps more democratic. Barcodes and RFID tags can make objects knowledgeable about themselves. Bruce Sterling: every single object can become the protagonist of it’s own history.

What happens to the curated object when every object becomes a site of contestation? Students believe the future of public history looks like the holodeck. Star Trek often uses the holodeck to simulate a usable past. Ok, now students must brainstorm tools, toys, devices that will magically dispense history. What would a history appliance look like, BUT you’re not allowed to reinvent the holodeck. One group said, what about a pair of knitting needles that would remember everything they ever made? Or a reverse babel-fish: would immediately make everyone around you sound like they were speaking an ancient language? Or tangible spray: comes in an aerosol, creates a cloud you can reach into and feel what the past was like until the cloud dissolves? Students haven’t yet suggested the Star Trek replicator. Notice how people in Star Trek don’t have pockets?

A cradle to cradle universe: everything exists only temporarily, until recycled into something else. What THEN happens to the curated object? E.g. of Buddhist temples that are routinely burned and rebuilt every 20 years. Can’t get heritage status because even though the ritual is thousands of years old, but building is less than two decades. Existing sci-fi tech: immersive virtual reality, augmented reality, tangible computing, ubiquitous computing, desktop fabrication.

Project with students to create an exhibit to show the work of William Harvey. Works justly held up as a paragon of scientific writing; but some of his techniques would now be ethically repugnant. How to give people a hands-on feel for what he did, but don’t make the simulations stomach-turning? Matter is the new medium. Affordable printers made it possible to materialize documents and pictures; now reaching a price point where other kinds of objects can be created as well. Used a scanner to create 3D anatomical representations. Digital data are extremely plastic. Editing work can be done at the digital level, can then print out physical objects, moulds. Interested in embedding electronic devices into physical objects to enhance interactions (Arduino).

Virtual/tangible cycle. The Harvey project by itself not that interesting, but the new modes of working are what’s really interesting. A tight loop between the tangible stuff and the virtual representations thereof. Naturally invite open source communities to come along and play.

Ability to make stuff changes your understanding of the stuff and the research process. “Real Humanists Make Tools” t-shirt…. Most of T’s colleagues don’t really get this yet. It is hard to get stuff to work! Need new spaces to work in, new kinds of tools. Practically every kindergarten in NA is better equipped for making than every humanities department. Easy to buy tools, much harder to set up proper, safe workspaces. If we don’t have a hands on engagement with this pool of knowledge, can’t learn what it has to teach. Until recently, hobbyists are excluded from history. There’s a new respect for tinkerers, makers, doers, who can remake the world in a generation or two (or now, faster). Scholarly engagement must extend to citizen technologies. Reprap machine, a desktop fabricator that can replicate anything including itself.

So what happens to our sense of the created object when we can start turning those back into atoms? E.g. of plastics used in 3D printers: specifically inexpensive to purchase and recycle. Conceivable to have a scanner, a printer, and a recycler on your desktop. What happens if you could make anything, and give it away as open source plans? Thingiverse; Instructibles another example of people sharing how to do stuff in a free and open way. Sensor data shared through pachube, the ability to mash up things.

Possible to imagine new histories of things, of the body, of environments.

How do students organize the information they’ve collected so they know exactly what they have and where they got it? Bibliographics tools: Endnote, ProCite, RefWorks, Zotero. Show of hands, RefWorks #1 in this crowd. In GM’s experience at Dalhousie RefWorks has very little uptake by students. Too many clicks, file folders, formats, etc. Students get bogged down, want something more intuitive.

Zotero a “personal research assistant inside your browser” (quoted whom?). Z has a growing list of significant institutions that support.

Used 10 Comp sci grad student volunteers, already using Firefox, Zotero a simple plug in. Sent them to the quick start guide, then gave them a research activity: pick a topic, find and zotero 4 relevant resources: scholarly article, PDF doc, blog post, and YouTube vid. Demonstrated the activity to them individually hoping to ensure consistency.

Zotero has “iTunes style interface”: Students had a completely different reaction to this compared to RefWorks. Collections > Items > Details. If desired, can take a snapshot of a page, in case you wish to consult later after it’s changed. Zotero will pull subject headings from indexes into tags. Final stage of the process was to generate APA style works list and analyse how well Zotero did the job. How well does the scraping work? GM felt not that well, but student’s felt results were good enough–no interest in editing field contents to make them right.

What do students do?
– save a lot of PDFs, creating a personal digital library, usually spread over many folders, but not really logically organized. The one thing they all saw as being of value was Zotero’s ability to store and identify PDFs in a logical way
– initial comments: easy to use, better way to manage and interact with PDF, liked that it was open source
– three months later: really glad to have the snapshots, one said they were using it for ALL research assignments, other nine said using it informally but didn’t see it as a useful research tool. Several commented that if they had an IEEE output they would use it more (IEEE has since been added in V2).

Who owns style files? They’re not reducing in number! Students go crazy trying to match styles, why do we continue to support them? Endnote supports 3000+ styles; many students don’t even know what they are. A movement is needed to reduce the number of styles out there!
– a lawsuit over the ownership of style files was launched. Endnote .ens files are proprietary conversion files; Zotero uses .csl (citation style language). George Mason University encouraged students, if Endnote wasn’t meeting they’re needs, to export their citations to Zotero. Thomson-Reuters sued: the E site license contains clauses that prohibit the reverse-engineering of software, basically claiming style files as intellectual property. At the end of 2008 GMU didn’t renew it’s site license. The case was dismissed in June 2009 [due to a lack of jurisdiction]

Meanwhile, V2 beta was released in May 2009 with so many improvements GM feels her research needs to start all over again. Strongly encourages us to try using it. Zotero encourages the same type of OS community involvement as others we have looked at. Zotero is different because we’re not in the middle: if a student wants to use it, they just plug it in (but they have to do it themselves). Librarian colleagues seem less interested in learning about Z because it’s not a university-licensed product. So GM’s world includes two products: RefWorks, and Zotero; like two worlds that don’t overlap.

Z is free, computer-based, firefox only; RW licensed, web-based, works in all browserw
Z works well with flickr, youtube, factiva (but nto ISI), web sites, OpenOffice (MS Word is there, will improve); RW scholarly journals, ISI (but not factiva) web sites, MS Word

V2 of Z has a USB portable option, which facilitates training sessions, can use on imaged public machines, can take library from one computer to another.

Vertov and BiblioBouts examples of library-oriented development that sits on top of Z. BB is a game; Vertov enhances description of media sources.

Began mid 90s by pulling in a small, orphaned website: Advisory Commission on Intergovernmental Relations. Agency had 50+ years of publishing; became a burden for reference, docs, and ILL staff. Got some funding to outsource scanning of their periodical publications and started mounting them online as PDFs. This was the beginning of what migrated into the Portal project.

Wanted to help Texas libraries, museums, and cultural heritage organizations.
– Many small organizations wanted to put stuff online, but didn’t have the skills our resources.
– build online collections connecting like materials from many libraries/museums/archives.
– provide resources for educators: lesson plans and resources built on digitized history collection.
– provide single interface to heterogeneous collections.
– text and photographic materials, some video and audio

Knew external funding would be needed. Easier to obtain funding for site creation/improvement and content building when providing services to many organizations. Federal, State, and foundation funding was received.

More than 100 partners, 3 models:
– UNT does all the work (costs more!)
– Partners scan, create metadata; UNT puts online
– UNT scans, partners create metadata, UNT puts online
UNT then does quality assessment for models 2 and 3. Creating metadata is the most expensive part.

Infrastructure: IOGENE Project, an IMLS funded project. Rapid development framework for digi lib interfaces with genealogists as the target users. Focus on user centred interface design.

Wanted a lightweight public access system to digital content; easily scalable in content, number of requests, collections, partners, types of content. Using tools that have well established communities, using other people’s code as much as possible writing only the library stuff. All digital content in one big pot (no silos) using different interfaces to brand and manipulate sub sets.

METS, DC (locally qualified), Pairtree, BagIt, ARKs identifiers for public access metadata management. Archive backend a different set of tools.

Defined a digital object model, serialize to METS, use same object model for all document types, works for everything so far. Object structure is mapped to URLs. ARK identifiers map well to beautiful URLs. URLs become the API. Designers can make significant changes to interface without interacting with developers. Metadata editor adds ‘edit’ subdomain, easy click to fix errors when discovered.

68K objects, 60K more in the queue, possible 700K records from State agencies, even more from possible newspaper sources.

Adding partner services
– brandable interfaces
– partner’s domain names
– SRU/OpenSearch target for each partner/collection
– OAI-PMH repository for each
– new services are developed, added to the stack, benefit everyone

UNT benefits
– all “digital library” content in the same system
– build rich research collections for students, researchers, the community

Working on interface overhaul, beta is here, somewhat cleaner interface, focus on usability improvements; help guides available to reroute questions if needed.

Islandora Takes Shape

Islandora is the combination of “Island” and “Dora the Explorer”.

Anticipating the death and rebirth of the repository. Drupal/Fedora combination: default alterior motive to build capacity in the Library/Campus, keep the library at the forefront of all campus activities: teaching/learning, research, admin. Data stewardship a critical concept. Must store, transform, provide access, mutate, migrate data. Longevity and usefulness.

Stewardship is a must, curation is just one small part. It is a deluge, researchers are freaking that they have to manage this stuff. Recent reaction from scientists at conference: This is the first time someone has responded to the data challenges I am facing. Typical, IT draws the line at hardware. There is no more significant opportunity for academic libraries in the next few decades. Will also stimulate library’s research development, IR development, etc.

Two years later: vision is the same, have some experience, a good evolution of tools in the community, and the library is the foundation for data management in all three landscapes. Research is the core driver, but admin is a significant driver as well. Staff feels enabled in finding solutions to challenges.

It’s all about the local: Google can’t do local like we can. Every research project is multidisciplinary and multinational, so your local becomes global and international.

Focus is on OS and open data. Have about 7 staff 50% or more committed to the project, soon to be 12+. Research, have received about $150K in hardware, faculty are encouraged to leverage research grants to by hardware add-ons to increase capacity rather than get standalone servers. Also about $200K recent funding for staff. Research fund tech staff grants rerouted to the library. Leveraging multiple pools of resources to build shared capacity both in hardware and staffing. Interest in using Islandora increasing at Admin level, e.g. Senate document management pilot: immediate storage and stewardship for institutional documents. Rather than throw money at adopting a document management system from scratch, are investing in building document management capabilities into Islandora.

The learning environment is the least active, e.g. learning object repositories haven’t really taken off, but focus is on plugging Islandora into learning management system. Currently over 50 research VREs from a broad range of disciplines, a number of others dedicated to admin areas. Probably looking at giving each major committee its own VRE. The look is pretty basic, starting to look more at improving; focus has been content.

Islandora external: 1st external contract by end of Sept. Implementors include U of North Texas, Georgia Inst of Tech, UNB, Carleton, UGuelph; like all OS projects, there may be more. Sloan-Kettering is interested. Working on DuraSpace partnership. Also working of FESL.

All data is stored in Fedora rather than Drupal: data, metadata, workflow, authentication all stored and maintained in the repository. Drupal is the collaborative layer. Three or four Drupal multi-site installations. Keep public installations separate from higher

Islandora is not in the Drupal contrib because some clean up is needed. It’s the glue that ties Fedora and Drupal together: D module, php and java apps, rule engine for flexible workflows, drop-in support for modules. Use a senior level comp sci class for development of components (e.g. of TEI editor shown earlier). Plug-in capability.

Ability to define and integrate complex digital workflows. A lot of science is drudge work, similar to the drudgery in digitizing books. Looking at integrating Taverna.

Solution packs: policies, disseminators, worklows, apps, data. First solution pack will be IRs.

Sun partnership: rapidly evolving to provide support to Sun resellers for selling hardware platform with Islandora pre-installed. Goal is to provide development and support contracts to the community generally.

Goal is to have a rich, fully defined community framework. There is effort that goes into customizing data management for different research groups. There will be a quarterly roadmap for code changes, each will have a new solution pack starting with IRs in January 2010. Planning Islandora/RIRI institute for Summer 2010.

Jon Jiras, Rochester Institute of Tech

eXtensible Catalog

XC is a modular set of OS tools, can use any part that’s useful. Facilitates the resource discovery piece, but also has core metadata management piece. V1 scheduled for January 2010 release.

faceted, frbrized user interface; customizable, a web app framework for library data. metadata tools. connectivity tools: OAI and NCIP

interface will be a Drupal module. Can include traditional, digital, and web resources. Can use interface as a platform for library website.

Metadata toolkit can aggregate data from many sources, normalize MARC and DC data for indexing using XC schema (RDA); there will be a web interface for staff to tweak, run reports, configure, etc.

Connectivity: XC OAI toolkit, can maintain a synchronised copy of repository metadata. XC NCIP toolkit allows ILS circulation features in XC and access authentication through ILS or LDAP.

Bess Sadler, U Virginia

Blacklight: Findability for your whole collection

Blacklight is a discovery layer, about creating a great user interface, increase serendipity: Dan Rubin “If your interface requires instructions, it needs to be redesigned”

Lack of relevance ranking, lack of permanent URLs, no RSS, siloing of collections, lack of object type appropriate behaviours (e.g. of DVD which doesn’t have an author), inability to respond to user requests and suggestions; these things are broken and we need to fix them ourselves.

One problem Blacklight is trying to solve is the variety of different siloed data sources: catalogue, IR, dissertations, Google books project, local digitization projects, licensed journals and databases, etc.

Solr is the anti-silo. Easy to use: download it tonight! Indexing is under our control, adding a new collection is as simple as adding a new xml output. We determine how to index our own data. E.g. of music collection, meticulously catalogued; number one reference question: flute/violin duets? Hasn’t been in our power to define indexing in order to answer that question. Can have different metadata profiles for different kinds of objects.

But how do you get good results? Because we index it ourselves, we have control over our relevance algorithms. In practice, a testing language called Cucumber which plugs in to Ruby. Looks like English. Can further resolve conflicts by developing lightweight interfaces for specific user groups. E.g. music people get a difference relevance algorithm: most relevant thing is rarely an exact title match.

Solr, Ruby on Rails; plugin structure allows for local customizations without forking.

Data should be stable, enduring; applications should be kept lightweight. Control of the app must be as close to the user as possible.

Becoming widely adopted: Stanford, Johns Hopkins, US Natl Agriculture Lib, U Wisconsin, others; working on commercial ILS integration, GIS stuff, and more.

Roy works in research and is somewhat apologetic about yesterday’s sales pitch.

The Hathi Trust is a shared digital repository that grew out of the Google digitization project. U Mich leads the effort, OCLC works on the service side. Lots of partners, but 82% of contributions from UM, 12% from UCal (ramping up quickly), Indiana and Wisconsin have small pieces as well.

When Hathi Trust web site was set up, also allowed download of all metadata describing volumes in the project; 13 elements mostly of little use outside the environment. Roy, for fun, grabbed the file, parsed it in XML (no standards, just because). Indexed it and created a search utility.

Then a colleague (Constance Malpas) at OCLC came up with a “cloud library” project. Shared digital and print repositories to create new operational efficiencies for research institutions. Requires new infrastructure for managing, monitoring, consuming shared services.

[insert Stan Rogers, "The White Collar Holler", here]

Downloaded HT metadata, enhanced with OCLC numbers, explode the data into millions of tiny xml files, indexed it, extracted unique OCLC numbers and sent to JT (a person) who extracts the WorldCat records. Then merge HT data into WC records, index, and extract info to simplify reporting. Perl/XML/Swish-e, XSLT, xsltproc

OCLC doesn’t really use MARC internally. Rather, have their own CDF (common data format?) which allows data to be extracted in more usable ways (e.g. of dates). Also inserted HT metadata. Also plan to insert metadata from libraries involved (holdings? I kinda missed that).

Some interesting reports. Murky buckets, a Lorcan Dempsy term. E.g. of weird dates. Roughly 16% as of Sept 2009 are public domain: 600K volumes, mostly pre-1922 and government documents. UM actually does proactive review of copyright status (unlike Google), trying to open up as much as possible. Subject distribution dominated by arts and humanities, esp. literature and history.

Now working with NYU: considering impact of collections overlaps among NYU, Hathi Trust, and ReCAP (a shared storage facility in NY).

Lessons
– identifiers are essential: OCLC number Roy thinks is best
– standards are great until they get in your way; have ignored both internal and external standards but it gets effective work done
– never underestimate the power of a prototype