Image via Wikipedia

The Open Knowledge Foundation (OKFN) promotes the creation, dissemination and use of ‘open knowledge.’ As part of this activity OKFN developed a data repository called CKAN, and has seen this become increasingly important to a range of data dissemination activities such as data.gov.uk and publicdata.eu.

In this podcast I talk with OKFN Director Rufus Pollock and CKAN Product Owner Irina Bolychevsky, to learn more about CKAN, its use in the context of open data, and the wider implications for dissemination of any data (whether open or closed).

Following up on a blog post that I wrote at the start of 2012, this is the tenth in an ongoing series of podcasts with key stakeholders in the emerging category of Data Markets.

Related articles

• Open Knowledge Releases Open Data Handbook 1.0 (readwriteweb.com)
• Data Market Chat: Leigh Dodds discusses Kasabi (cloudofdata.com)
• Data for the public good (radar.oreilly.com)
• Data Market Chat: Stephen O’Grady of RedMonk examines the bigger picture (cloudofdata.com)
• Data Market Chat: Nick Edouard discusses BuzzData (cloudofdata.com)

Image via Wikipedia

Openness is undeniably cool right now, at least if you move in the slightly odd circles that I do. Openly available scientific papers are disrupting the world of scholarly publishing (which may not be all good, but that’s a post for another day). Openly available university courses are finally beginning to work out how to offer meaningful accreditation to students. Openly accessible data from government agencies around the world bulks out almost every data marketplace, and anchors many an analysis. Openly available code for cloud infrastructure or networking is challenging the hold of the tech world’s giants. Everywhere you look, ‘incumbents’ are apparently being ‘challenged’ and ‘disrupted’ by the power of open.

The truth, of course, is a little more complex and a lot more nuanced, as business models shift and evolve just like they always have. In sustainable systems, some people still need to be rewarded (often through being paid) for their effort. And in sustainable systems, paying someone can often be a pretty straightforward means of ensuring that you have a throat to choke if something breaks; big companies adopting open source often seek a proper financial relationship with someone who installs and maintains the ‘free’ software or hardware they’re depending upon.

One area of openness that I’ve been involved with for about ten years is that of open licensing for both creative works and data. And it’s come a very long way.

Here in Europe, for example, the (badly flawed) 2003 Public Sector Information Directive is under review, and there’s every likelihood that the replacement will make a number of sensible moves toward greater openness, transparency, and reusability for publicly funded data. As the EPSI Platform site notes today, Andrés Nin proposes going a step further than the European Commission is currently contemplating, by instituting a common open license across Europe;

“The creation of a single public information re-use space in Europe requires much more, it requires a common European OpenData license applicable to all data generated by European public administrations.”

I would certainly welcome a model license that European member states might be enabled to use. I’d also welcome — and support — vigorous efforts to dissuade individual member states or ministries from their usual practice of tweaking and otherwise modifying perfectly good documents in order to demonstrate how ‘special’ or ‘different’ their circumstances apparently are. When will they all realise that they are neither as special nor as different as they like to think?

But — and it’s a big but — it seems unwise, premature, and unhelpful to even begin to suggest that such a license might be mandated across Europe. It isn’t required, and attempts to develop a single document that everyone could accept would be an unhelpful distraction that would result in something so bureaucratic, so ringed in opt-outs and prevarications, as to be utterly worthless. It would also, in all likelihood, be one of those exercises in which the process very quickly subsumed the point. A prime candidate for, in the words of an old boss, being too busy to be effective.

Related articles

• Branding academic publishers ‘enemies of science’ is offensive and wrong (guardian.co.uk)
• Open Data: Europe Starts to Get It (opendotdotdot.blogspot.com)
• Open Data in Europe gets a huge boost from new EU rules (thenextweb.com)

Image by Kevin Krejci via Flickr

The CloudCamp unconference returned to London for the 14th time this evening, regaling a capacity crowd in the Crypt below Clerkenwell’s St James Church with several hours of discussion and debate on the somewhat elusive topic of ‘Big Data’.

Rather rough notes of the proceedings follow, after the break.

LEF‘s Simon Wardley kicked proceedings off as usual, once again managing to pepper an on-topic canter through the topic with a seemingly never-ending stream of Flickr images of cats… and analogies to electricity. You possibly had to be there? His core message, though? There’s nothing new under the sun… and the cycles of change just keep on coming.

Next, Peter Matthews from CA Labs, on “is big data mutually compatible with the cloud?” Erm, yes. Data volumes with big data are so large that it’s difficult to move it around… which creates opportunities for lock-in that vendors may wish to seize. And then he was out of time.

Next, Fujitsu’s Mark Wilson on ‘Structuring Big Data.’ He’s actually talking about Linked Data, a topic I’ve dug into before here and over on semanticweb.com – Linked Data could be/ might be the effective realisation of the decade-old Semantic Web dream. Big Data means masses of unstructured or semi-structured content, presenting a management headache of previously unanticipated proportions. Linked Data, he argues, creates the mechanism to link all of this data together from across disparate sources. Yes, but it’s easier to say than to do… And in 5 minutes he really couldn’t explain enough to persuade the audience. Linked Data should be “the optimal reference source,” he said. It should be “a broker for all data sources,” and we should “think about integration, not duplication.” Yeeeeees… But.

Next, Canonical’s Nick Barcet, talking around scalability, Ubuntu, package management, configuration management, etc. Not wholly sure what the point was, I’m afraid.

Next, Chris Swan from UBS – big data and security. “If you’ve got security controls that aren’t properly monitored, then they don’t matter.”

Next, Tom Leyden of Amplidata – Big “Unstructured” Data in the Cloud. Data storage to increase 30x over the next decade, but staff will only increase 50% over the same period. Challenge in the 90s, as existing storage and analysis technologies struggled to cope with new data volumes. Seeing similar problems today with data streaming from sensor web, etc. Traditional file systems cannot cope. Object Storage the way forward ?

Next, Alex Farquhar – “Cloud v Big Data.” Not really versus… but intersection of the two. Too much discussion of his company, Forward. Just talking about how his company uses cloud to provision IT resources. Might work as a conference presentation or case study – not sure it fits as a 5 minute lightning chat. Around 60TB of data at Forward. Diverse and vital. Using Hadoop cluster – 24 nodes on-premise. Rationale (proximity to the cluster) seemed odd. That can be true, but not clear that it really needs to be the case here?

Next, Alaric Snell-Pym, on Scaling Hadoop. Trying to overcome Hadoop’s I/O bottleneck. Explaining basics of Hadoop and Map/Reduce – no one else has. Explains use of HDFS and ‘selective reading’ to manage lots of small tables and overcome the problems of I/O.

Next, Matt Wood from Amazon. Talking about genetics and the human genome. It’s an analogy. Human Genome Project took years and millions of dollars. Development of gene sequencing machines led to a step change – dramatic drop in cost of sequencing DNA. Like the cloud, anyone? But… the machines create an analysis challenge, because they generate so much data. Cloud offers “collection of productivity tools” to help scientists work with this data collaboratively and (relatively) affordably. A perfect example of a lightning presentation, unlike most of those who preceded him.

And finally, an impromptu slot from HP’s Joe Weinman. A quick overview of current thinking behind his latest book. This one could have gone for much longer… Good stuff.

And that’s the lightning talks finished. Now, the panel, and Simon Wardley’s search for “experts” and “volunteers.”

…and unfortunately, your scribe was ‘volunteered’ as an ‘expert’ by Mr Wardley… and here end the notes. It was great to have Amazon’s Werner Vogels sneak in, and lob comments into the panel, though…

Great event, though with the usual mix of people you wish could have talked for longer… and people you wish wouldn’t have spoken.

Related articles

• The brave new world of big data & Hadoop (venturebeat.com)
• Big VCs Invest In Big Data Startup Continuuity (techcrunch.com)

Image of Stephen Wolfram via Wikipedia

British-born computer scientist Stephen Wolfram sees ongoing efforts to extend the Internet’s top-level domains (TLDs) beyond the familiar .com, .org, .uk etc as an opportunity to raise the profile of machine-readable data. In a blog post published yesterday, he argues that a new .data domain would increase “exposure of data on the internet—and [provide] added impetus for organizations to expose data in a way that can efficiently be found and accessed.” Whilst wholly in favour of Wolfram’s stated aim, I can’t help feeling that his suggested solution is at best unnecessary and at worst a worrying segregration of data from the ‘proper’ web that everyone else will continue to exploit.

Back in June of last year, the body responsible for coordinating the global domain name system approved a plan to permit new top-level domains (the letters after the final dot in an internet address — the .com in cloudofdata.com, the .uk in bbc.co.uk, the .edu in harvard.edu). Until recently, these top-level domains have been tightly controlled, with a small set of generic domains (.edu, .gov, .mil, .org, etc), a larger set of country domains (.uk, .fi, .nz, etc) and one or two others such as .eu. From tomorrow, anyone with $185,000 will be able to submit a proposal to create and manage a new top level domain, and it’s possible that there could eventually be thousands of them. Wolfram is keen to ensure that data doesn’t miss out on the ‘opportunity.’

As Wolfram himself recognises, there is already an awful lot of machine-readable data on the web. Some of it sits embedded within the web pages that humans read, with specially formatted code waiting to be triggered by the calendars, the address books, or the browser plugins of site visitors. Some of it is packaged up in data files, offered for download. And some of it waits inside a database, ready to be delivered in response to an API call or a query typed into a web form.

There is a growing enthusiasm for exposing this data for reuse. Government transparency agendas have driven public sector data sites like data.gov.uk and data.gov. Similarly, efforts such as data.open.ac.uk and data.southampton.ac.uk see universities beginning to consciously collect data sets together and offer them up for reuse. Similar efforts in the commercial world are less easy to point to, but that reticence has nothing whatsoever to do with the lack of a ford.data, boeing.data, ge.data or astrazeneca.data domain!

In some ways, the convention for gathering significant chunks of data on a data.xxx.yyy site echoes Wolfram’s intention, but with a number of advantages. Data without context is far less valuable than data with context. Much of that context may be inferred from the domain in which the data lives, with data delivered from a .gov or .edu (or .gov.uk or .ac.uk) site perhaps interpreted differently to data hosted on .com, .biz, or .xxx. Southampton University, the Open University, and the US Federal Government are able to gather data up and make it available for download via their existing data. sites if they choose. This offers human visitors to their sites a degree of convenience, whilst retaining the power and brand attributes of their existing domain. Gov.data, gov.uk.data, open.ac.uk.data, southampton.ac.uk.data, though? All are messy, in ways that Wolfram’s own wolfram.data would admittedly not be, and all are simply additional registrations that the institutions would have to pay for in order to stop someone else grabbing the domain.

At the end of the day, the machines don’t actually care. The existing data.open.ac.uk-type sites are human conveniences, not machine enablers. The computers, and the software they run, are quite capable of crawling the public web and finding accessible data wherever it lies on a site. There are plenty of reasons to continue embedding little snippets of data inside human readable web pages, regardless of whether you have a data.wolfram.com or a wolfram.data site. Content negotiation is becoming increasingly capable, such that there really is no need for what Wolfram calls a ‘parallel construct to the ordinary web’ at all. A human being arriving at a web site sees human readable content, whilst various software tools would automatically be presented with very different data or functions, optimised to their capabilities and requirements.

By all means, let us show the curious some of the existing techniques that work in making data more easily accessible. By all means, let us identify the gaps, the issues, the problems (none of which a new TLD even begins to address). Yes, let us definitely and unambiguously set about “highlighting the exposure of data on the internet—and providing added impetus for organizations to expose data in a way that can efficiently be found and accessed.”

But please, let us not be distracted by the false hope that adding yet another TLD to the babel that ICANN is about to unleash can do anything more than consign data to some online ghetto, wallowing unwanted, unloved and unused as companies and their customers lavish love, attention, and clicks upon the .com domain over on the ‘proper’ web.

Thanks to Raphaël Troncy, whose tweet first drew the story to my attention.

Related articles

• Is It Time For Computers To Have Their Own .Data Domains? (techcrunch.com)
• ICANN Pushes Ahead With January 12 Launch For New Top-Level Domains (wired.com)
• The biggest change in DNS since Dot-Com (wired.com)

• My latest monthly column on SemanticWeb.com; Big Data Presents a Big Opportunity?
• My latest weekly wrap-up on GigaOMPro; Rosslyn Analytics, Microsoft Finding Value in Data Aggregation
• The teaser piece on GigaOM’s public Cloud site; In Exploiting the Data Market, Trust Is Key

I spot a theme building…

There are some great sessions on the Programme, and I look forward to comparing the diverse ways in which Big Data concepts and methods are being put to work across a range of market segments. I also look forward to exploring answers to some of the questions I posed back in November.

As usual, the diary is filling up with meetings, briefings, and the odd conference session, but there are still some gaps to fill. If you’re at the event – or in the area – and want a chat, why not grab one of the slots over on Tungle?

Related articles

• Six months after “What is data science?” (radar.oreilly.com)
• O’Reilly Strata Conference: Only a few days left for early registration + reader discount (flowingdata.com)
• Big Data Examined At Inaugural O’Reilly Strata Conference (lockergnome.com)

‘Big Data‘ is currently capturing the imagination, attracting hype, investment and ambitious startups in almost equal measure. Kim and Eric Norlin’s excellent Defrag and Glue events have gained big-name company, with O’Reilly‘s Strata and GigaOM‘s Structure both set to arrive in the first quarter of 2011. Venture firms like IA Ventures have emerged, specifically targeted at finding, funding, and profiting from the big Big Data idea. Giants of the web from Yahoo! and Amazon to Twitter and Facebook solve their own Big Data problems in very different ways, contributing valuable code and experience to the community whilst simultaneously diluting focus and adding to the cacophony.

Flippantly reckoned by many to be ‘anything that requires more than a single machine to run,’ the Big Data reality remains somewhat harder to pin down. To those seeking routine business insight, that mammoth Excel spreadsheet they laboriously query overnight at the end of each month might quite justifiably be thought of as ‘Big.’ At the other end of the scale, data wizards scorn anything that doesn’t require a room full of servers, a mountain of empty pizza boxes, and the careful construction of a bespoke data ingest, management and querying system atop the most bare-bones version of the Linux kernel they can find. Somewhere between the two, a growing mass of cheaply gathered data holds out the promise of invaluable insight. Remote sensors, web clickstreams, social graph interactions, purchaser (and non-purchaser) behaviours. All these, and more, have much to tell planners, builders, makers, sellers, and buyers. If only we could formulate the right questions. If only we could devise the right sampling strategies. If only we had big enough machines to ask lots of questions using lots of sampling strategies. If only we had big enough machines to not bother sampling at all.

On the hardware side of things, even humble domestic laptops typically ship with at least two cores these days; two separate little computers ready to do the data processor’s bidding. Four, eight, sixteen and more cores are not far behind, but mainstream software products typically fail to exploit anything more than a single core. Push Excel as hard as you like, and it won’t do more than take one of your computer’s multiple cores to the max. On that 12-core Mac Pro you persuaded the boss to buy, only one core will be hard at work on your data. Twitter, Mail, YouTube, and ripping DVDs  will each be giving other cores a little light exercise whilst others sit idly by, waiting for the arrival of operating systems and applications capable of exploiting multi-core power. The same is true as jobs grow and move to run across multiple machines, whether under your desk, in your data centre, or out in the Cloud. Those big datasets need to be carved up and shared amongst the available computers before any analysis takes place. You’re typically not accessing a ‘big computer in the Cloud’ at all… but lots of relatively small (commodity) computers, and it takes careful planning and smart software to manage the division and recombination of those jobs in a cost-effective manner. Projects such as Joseph Hellerstein‘s Berkeley Orders of Magnitude (BOOM) begin to demonstrate some of the potential for working natively with multiple processors, but there’s a long way to go before those advances reach the mainstream.

Hadoop, Cassandra, MapReduce, Dynamo, Voldemort. These, and more, are solutions developed by the likes of Yahoo!, Facebook, Google, Amazon and LinkedIn to tackle the influx of data that each faced – and for which each had failed to find an existing solution. Hadoop, with the addition of Cloudera‘s commercial polish, is rapidly emerging as the front runner for an off the shelf Big Data solution, but all of these tools remain rather narrow in their abilities. Find the type of data or the nature of query for which each of these was built and its performance will be unbeatable, but we are a very long way from Big Data’s equivalent of the jack-of-all-trades SQL-powered relational database of old.

And there, for many enterprises, lies the problem. Useful Google searches require the crawler, index and UI to do a relatively small number of essentially similar tasks, very quickly, very cost-effectively, and at massive scale. Focus on that finite set of problems, and you build a solution that delivers the experience we’ve all come to know. Each type of data manipulation or analysis requires a different tool, differently optimised, with the inevitable result that a typically diverse organisation may require a plethora of Big Data tools to get their work done. Or they might just continue to muddle along with Oracle or mySQL, churning inefficiently through their data analysis jobs for interminably long periods of time. These relational database tools are understood, they are mature, and they get the job done. Except in the most data-intensive industries, they have a market presence that will be difficult to disrupt.

The Big Data space is seeing remarkable innovation, but there is a long way to go in order to lift it out of the domain of the technically proficient specialist and place it on desktops across the organisation. As IA Ventures’ Brad Gillespie notes, “Excel is where the world’s data lives… [and] Big Data has to get to that place… so that a CMO can leverage it directly.”

And in all of this fervent of innovation, to return to the title of the post, it strikes me that Big Data is becoming disconnected from the fabric of the web itself. Oh, much of the data certainly comes from the Web, and a lot of it might even be queried on the Web after processing. But, somewhere along the line, the linkedness of the Web has either been forgotten or ignored. That rich set of connections, interconnections and associations has been reduced to a table, an index, or a (large) set of key-value pairs. And in the process, something fundamental has gone away.

This is enough for now, though. Looking more closely at different Big Data approaches, and exploring the potential for re-introducing the Web must wait for future posts.

Related articles

• Cloudera raises $25M to help deal with the enterprise data deluge (venturebeat.com)
• Strata Week: Building data startups (radar.oreilly.com)
• Big Data and a Critique of Geek Culture (readwriteweb.com)
• Big Data and NoSQL March to the Enterprise (nytimes.com)
• Does ‘big data’ equal big opportunity for storage vendors? (news.cnet.com)
• New York Times Event Shows the Promise of Big Data (programmableweb.com)
• Overwhelmed Executives Still Crave Big Data, Says Survey (readwriteweb.com)

A short report that I was commissioned to write for the European Public Sector Information Platform has just been published.

The rise of the App: a PSI opportunity? introduces (smartphone) apps and app stores to those in European governments responsible for meeting their obligations under the 2003 Public Sector Information (PSI) Directive.

Unfortunately somewhat tangential to the more recent (and cooler?) enthusiasm for Open Data, governments’ compliance with the PSI Directive has largely failed to engage the community of active and enthusiastic developers who might build compelling tools atop all that data.

Although elected officials might love showing their friends a council-branded iPhone app that knows where all the publicly-funded swimming pools within a single local government area are located, is that really a useful tool for anybody? Would it not be more useful to see the data made available in forms, formats and locations regularly frequented by communities of third party developers? Then you might see all the swimming pools, you might cross (mostly meaningless) local government boundaries, and you might pull in other leisure activities, so that a real user can ask ‘where can I swim?’ or ‘where can I go and have some fun?,’ instead of the rather unlikely ‘where can I swim in a council swimming pool?’ If you care that much about swimming in the pools of your local council, won’t you know where they are? And if you don’t, how likely are you to download an app to answer the question? Once you’ve answered it once (surely a Google query, rather than an app download) the app is useless.

As well as being formally released via council, region, agency and national web sites, should freely reusable public sector data not be actively contributed to Factual, Infochimps and the like? So long as a free copy is available somewhere, is it really a problem if someone else can take that data, add value to it, and make a little bit of money?

Public Sector Information is wide-ranging, comprehensive, and authoritative. It is truly insane for these rich resources not to underpin a wealth of applications originating in both the public and private sectors.

All we need to do is abolish some of the weirder licensing restrictions, disabuse some governments of the idea that PSI will make them rich, and make the data easy to find, easy to select, easy to get, easy to integrate, and easy to keep current. Easy, huh? Let’s do it.

The European Commission recognises that the PSI Directive is due a refresh, and is currently consulting on next steps in this area.

Related articles

• Europe begins revamp of rules on re-use of public info (go.theregister.com)
• Transport for London locks app users out of online travel data feed (guardian.co.uk)
• “Shiny app syndrome” and Gov 2.0 (radar.oreilly.com)

Towards the end of George Orwell’s allegorical take on the Stalinist Revolution, the pigs of Animal Farm take on the trappings of the humans they supplanted, shifting ideologically from ‘Four Legs Good, Two Legs Bad’ to declare ‘Four Legs Good, Two Legs Better!’ as they rise to stand on their hind legs.

The pigs’ dogmatism forces a series of increasingly convoluted rationalisations, until they end up professing exactly the opposite of their original position. Black, it seems, really can be white… but there’s absolutely no room for grey.

With data, current moves toward ‘open’ are certainly to be lauded, and we should continue to demonstrate the benefits of more equitable access in persuading those who have yet to realise the opportunities for rethinking their business.

However, I’ve been in too many situations recently where persuasion, encouragement and demonstration have been cast aside in favour of brow beating, castigation and vitriol. Anyone who fails to immediately throw open the doors to their data vaults is, the argument increasingly seems to go, cruelly, wantonly, and entirely unreasonably standing in the path of truth, justice, and the {insert name of country} way. The language is intemperate, and the unspoken undercurrent of feeling seems almost to lump these evil data hoarders with the most vile underminers of social cohesion.

Nonsense.

Open Data is a good thing, and we could benefit from an awful lot more of it. But the arguments surely shouldn’t be religious (‘Open’ is better than ‘Closed’) or so polarised that compromise typically becomes impossible. Instead, we need collectively to demonstrate the value of change, and we need to understand and respect the positions of the market’s incumbents. Current practice should never be accepted as an excuse for lack of change, but all too often it may actually mask quite a good set of reasons.

Where data are currently sold, can we (as was to some extent done for the Ordnance Survey) calculate the costs of data collection, curation and sale, and demonstrate convincingly that more money could be made by removing that initial barrier to access?

Where a data holder participates in an existing data sharing arrangement with their peers, surely we can gather the evidence to demonstrate the likely effect of opening parts of that value chain… without destabilising an otherwise useful set of collaborations?

Where large quantities of low value data (such as a customer’s address) are stored and managed alongside highly valuable business data (the facts of a customer relationship), we can certainly set about demonstrating the ways in which a more open approach could pay dividends; instead of managing that postcode yourself, share a little in order to benefit from the work done by others on tracking past, current, and future changes of address.

Then again, maybe we should just scream and swear at all those data-hoarding dinosaurs, without trying to understand them or engage with their fears, concerns, and counter-arguments. It’s much easier that way.

Four Legs Good. Two Legs Often Quite Good, Too!

Related articles

• The black market for data (radar.oreilly.com)
• GTEC 2010: David Eaves on Open Data: ‘Just Do It!’ (techvibes.com)
• The Linked Open Data and Pavlova (blogs.talis.com)
• Nottingham University offers masterclasses in dealing with open data – for free of course (guardian.co.uk)
• Talking with Richard Stirling about progress with data.gov.uk (cloudofdata.com)
• ICO launches consultation on data sharing code of practice (v3.co.uk)
• How Open Data is Used Against the Poor (readwriteweb.com)

Furthermore, ‘Open Access‘ and ‘Repository’ typically go hand in hand. If you support Open Access you need a repository, and if you question the role of repositories you’re in the pocket of evil publishers who want to lock up everything ever written and lease reading rights back to the employers of those who wrote the stuff in the first place.

Nonsense.

Open Access is an important component of today’s scholarly ecosystem. It’s not the only answer, and it’s not perfect, but it does have a significant part to play. Institutions have a role in preserving, disseminating and exploiting the work of their employees, but these are very different tasks that may benefit from different solutions. In too many cases, the repository is by default seen as a preservation mechanism and a dissemination vehicle, and as such it may fail to cost-effectively achieve either aim.

There are some large, well known, and research-intensive institutions where it might be possible to make a compelling argument for projecting a strong institutional image around a single ‘home’ for all of that research output. Never mind, for a moment, that so much research today is the result of inter-institutional collaboration, or that the eminent researcher might wish to take ‘their’ research publications with them as they move from Oxford to Harvard to York during their glittering career.

Alongside those institutions sit a plethora of others where research of equal quality is also being conducted; there just, maybe, isn’t quite as much of it. Bombarded by ‘advice’ and funding, and desperate to keep up with the Russell Group, ever-more institutions blindly join the repository cult and wonder why their new toys do not fill to overflowing with the jewels of scholarly erudition.

As research becomes increasingly data-rich, the whole cycle looks set to repeat. The recently released Panton Principles for Open Data in Science are to be welcomed, but I’ll bet the institutional response will all too often be the commissioning of a ‘data repository’ to sit alongside the ‘publication repository’ they already don’t use.

All of which is a rather long-winded way of introducing the fact that Eduserv’s Andy Powell has asked me to facilitate a breakout afternoon on ‘Policy Issues’ at the Repositories in the Cloud event Eduserv and JISC are holding in London on Tuesday.

“This free event, organised jointly by Eduserv and the JISC, will bring together software developers, repository managers, service providers, funding and advisory bodies to discuss the potential policy and technical issues associated with cloud computing and the delivery of repository services in UK HEIs.”

In a post on 11 February, Andy invited participants to share some of their views ahead of the meeting, and on 19 February he wrote about some of his own thoughts.

Like Andy, I struggled somewhat to nail down a coherent set of thoughts about the issue of pushing today’s repositories into the Cloud. On one level, I wonder whether the vast majority of institutions with small (and relatively low traffic) repositories would see much of a tangible efficiency gain or cost saving by moving off an in-house computer to rent an equivalent Virtual Machine from Amazon, Rackspace, or any of their competitors. If we’re talking about IT systems within a typical university, there are others (email, calendaring, pools of compute resource for research jobs, etc) that appear more immediately compelling for the shift Cloud-ward. Which is not to say that there isn’t a clear opportunity for someone trusted to step into this space and offer a SaaS repository to which institutions might affordably subscribe. Eduserv? Mimas? Edina? The British Library? The National Archives? Duraspace? Any could, and if we’re not ready for something more then at least one probably should.

However, a bolder reconsideration of what repositories are and what they’re for might very well lead to something interesting, sustainable, and perfectly suited for benefitting from Cloud Computing’s strengths.

Why does a paper have to be ‘deposited’ in a repository? Why does a single paper with three authors from three institutions have to be deposited in three separate institutional repositories? Why does that same paper have to be deposited – separately – in the subject repository favoured by scholars in the relevant discipline? Why does the institution’s very reasonable desire to protect, preserve, promote and disseminate its excellence mean that it has to run systems in perpetuity that preserve and permit access? Why do we address the fundamentally different (perhaps even contradictory) problems of access and preservation in the same system? Why can’t the individual researcher easily assemble a view across their publication history, regardless of the institution within which they happened to reside as they wrote each paper? Why don’t the assemblages of papers reflect personal, professional and disciplinary relationships, alongside (or instead of) the contractual accident of employee-employer relationships? Why isn’t the wealth of metadata implicit to any publication (authors, subjects, dates, citations, and more) available and actionable, both inside the repository and far beyond it across the Web? Why isn’t there a tight and active association between the paper and the data from which its findings were derived (something for which Internet Archaeology was demonstrating utility a very long time ago)?

Scholarly papers principally comprise text, augmented by the occasional static image. They’re not big, and they don’t tend to change very fast. In many ways, they represent a fairly easy problem set with which to work. As more and more data becomes key to research in a growing number of subject areas, the problems are set to become far larger and far more difficult. For individual universities to even consider replicating the process by which they all ended up with their repositories of text surely seems madness in this data-rich environment. Even with levels of uptake as low as those seen in too many text repositories, the issues of data management, curation, access and dissemination are too great to be sensibly solved in the institutional machine room. Services like InfoChimps and Amazon’s own Public Data Sets offering show some of the ways that we might begin to work with data at scale. Might we, for example, come to recognise as Amazon has that it’s actually cheaper and quicker to entrust large data sets to FedEx rather than transmit them over the Internet?

‘The answer’ might be some central service for the community, funded by JISC like the Arts & Humanities Data Service (AHDS) of old. Or it might be something different, something nimbler, more responsive, more flexible to individual, institutional, and disciplinary requirements, and something more scalable to new disciplines; institutional support for and use of existing Cloud infrastructures extending far beyond UK Higher Education, aligned with a clear understanding of the separation between preservation and access.

I certainly don’t have all the answers, but I do believe that simply asking whether or not we should move existing repositories to the Cloud is to miss the point. Rather, we should ask what role the Cloud might play in addressing the business requirements to which the institutional repository was our initial – faltering – response. The answer might very well be ‘None,’ but I doubt it.

I look forward to Tuesday’s discussion. I’m not going there to push my personal view that individual institutions frequently shouldn’t be building, running or populating their own repositories at all. I’m going there to facilitate the discussion those in the room want to have, and to learn from their experiences and their perspectives.

Related articles by Zemanta

• Does a Citation Advantage Exist for Mandated Open Access Articles? (scholarlykitchen.sspnet.org)
• Scholarly content and the cliff edge: the place of subject ‘repositories’ (hangingtogether.org)
• Scholarly Communications must be Scalable (downes.ca)
• Beyond Open Access: Open Publishing (opendotdotdot.blogspot.com)
• 57 college presidents declare support for public access to publicly funded research in the US (scienceblog.com)
• Mandelson says academics are ‘set in aspic’ (guardian.co.uk)