I don’t think I’ve ever really been convinced by the argument that everything will end up in the data centres of Amazon.

The straightforward provision of commodity Cloud Computing is an important – and growing – area, and one that will continue to expand as interfaces become simpler, FUD is challenged, and prices maintain their relentless march towards the bottom. Everyone has something they could usefully, sensibly, and cost-effectively run in a commodity Cloud such as those offered by Amazon, Rackspace, Flexiant, and others. In this space, basic stability, security and reliability combine with a compelling – and diminishing – pricing proposition to create commodity services targeted squarely to lowest common denominator functionality. Here, market forces may (inevitably?) lead to an eventual reduction in the number of providers. Cost, although not the only consideration, is both important and compelling. Although markets like competition, there may even be a single winner here, one day.

Layered all around the basic, routine, grunt-work computation that these commodity public clouds handle so well, many organisations find themselves having to cope with a wide range of other use cases and data sets. Some require specialist hardware (like the GPUs that Amazon has recently begun selling access to). Some demand particular regulatory and legislative hoops to be jumped through. Some have quirky requirements around latency in data transfer or speed of in-CPU processing. Some have lots of data, and issues with regard to getting the stuff from one location to another with a sensible balance between transfer cost and time.

All of these are certainly capable of being addressed in the Cloud, but the economics and the business rationale begin to shift. For the data owner, cost may no longer be quite so significant a factor. Reliability may matter more, or speed, or the audit trail. For the Cloud provider, these requirements no longer look like the lowest common denominator. It’s not cost-effective to provide these capabilities to everyone and still keep the price low. It becomes more sensible to segment, to divide, and to create bespoke offerings of various kinds. Some of these services require such specific things in terms of network topology, physical building layout, and staff expertise that it may even become counter-productive to have these services in the same building as the commodity Cloud. Here, there’s plenty of room for new entrants, plenty of scope for competition, and plenty of opportunity to differentiate in terms of price, location, support, and a host of other factors. This segment of the Cloud is only just getting started.

In these contexts, we see compelling arguments made for on-premise private clouds, off-premise private clouds, hybrid clouds, community clouds and the rest. Some of the arguments made in favour of private and hybrid certainly are part of the FUD we see in this space, but beneath the noise, the security scares, and the vested interests of SysAdmins and sellers of data centre components, there lies a grain of truth. Not everything is most sensibly run on a cheap VM, rented from Amazon (or Rackspace, or whoever) with your credit card, and physically located half way round the planet.

Unfortunately, it can be difficult to make sensible decisions about which type of cloud works best in each situation, and large swathes of the market are doing everything in their power to add to the confusion.

Having accepted that the basic offering from a public cloud provider is not the solution for my particular requirements, where do I turn next?

Do I listen to the (convincing) pitch from a vendor of ‘community cloud’ solutions for my domain? If I’m in Healthcare, they come with HIPAA and European Data Protection Directive, and all sorts of other accreditations. For dealing with sensitive patient data, this may be just what I need… but does the wily salesman also persuade me to run staff email and the hospital volleyball club website on this over-specified (and expensive) infrastructure?

Do I listen to the (convincing) pitch from a vendor of virtualisation software? If I’ve got a reasonably sized data centre with some life left in it, I may see the value of virtualising all of that expensive hardware, and running current applications in house more efficiently. But instead of gradually reducing my in-house costs, do I continue to add more machines as current ones reach end of life, or as new requirements come along?

Do I listen to the (convincing) pitch from my co-location facility, which happily sells me a ‘private cloud’ that may fail to deliver some of the economies of scale so central to the main Cloud proposition?

Do I listen to the horror stories, stick my head in the sand, and simply keep ordering servers until every single one of my competitors undercuts my costs and I go out of business?

These, and more, are certainly possible. But let’s return to that UK project I mentioned right at the start.

Flexible Services for the Support of Research (FleSSR) is

“a new cloud pilot project looking at utilising hybrid private-public IaaS cloud infrastructure to provide computational and data services to the academic research community. The project is a collaboration between the Oxford e-Research Center, IT Service @ University or Reading, e-Science Centre @ STFC, Eduserv, EoverI, Eucalyptus INC and Canonical Ltd.”

The ten month project is funded by the Joint Information Systems Committee (JISC), an organisation that supports the innovative use of IT across UK universities.

Now, to a degree, the project’s mindset must be influenced by its partners. IT staff at Reading and STFC are incumbents with turf to protect (or new vistas to discover, map, and claim). Eduserv has a new data centre that they’d like to fill with willing clients. It would be easy to be cynical, but knowing some of the people involved, I see no real reason to be. It is perfectly reasonable to suggest that a ‘community’ the size of UK Higher Education would realise value in replicating less (not nothing) at every university campus across the country, and bringing much of that together in some sort of Cloud. That Cloud might use public infrastructure, or it might be served up from an organisation such as Eduserv, which is known to the community, aware of the community’s requirements, quirks and foibles, and (importantly) not-for profit (and therefore cheaper?).

Personally, I’d always rather presumed that an organisation like Eduserv (or JISC itself) would act on behalf of the community to procure a competitive price on access to the resources of Amazon, Rackspace, or one of the others. I’m not convinced that most UK research computation needs any sort of special treatment that couldn’t be met from Amazon’s Dublin data centre… unless the community itself can somehow beat – and continue to beat – Amazon on price. Somewhat surprisingly, that’s exactly what some calculations in two posts by Eduserv’s Andy Powell suggest could happen. By including network costs and other charges over and above the basic storage cost, Andy finds Amazon, Rackspace and Dropbox to be more expensive than anticipated, and posits that Eduserv (connected to every UK university free of charge via JISC’s high speed JANET service, and constrained in the ways it can generate profit from services sold to universities by its charitable status) might actually work out cheaper.

There’s a lot of work to do in terms of fleshing out the assumptions behind some of Andy’s figures, but the whole industry certainly benefits when people conduct exercises like these out in the open, for all to see. If Andy has made mistakes, the vendors should be quick to jump in and correct them. If his assumptions miss the mark, public debate can redress the balance.

The Cloud is not all about price. But more transparency around the true cost of computing in the Cloud – and in your data centre – means that we can all make more informed decisions.

Thanks for sharing, Andy – and hopefully readers will be willing and able to look over your calculations and share their own views.

Note: this post was conceived and written in the United Kingdom. By reading this post you agree to comply with UK usage, and will henceforth pronounce the word ‘niche’ from the title as ‘neesh,’ not ‘nitch.’ Or maybe not.

Related articles

• Rackspace launches managed cloud services (zdnet.com)
• Are hybrid clouds the path to cloud-computing nirvana? (venturebeat.com)
• We’ll Take Care of Your Cloud, While You Manage Your Business (rackspacecloud.com)
• Trust Is Key For Cloud Success And What Can We Do About It? (cloudave.com)

Furthermore, ‘Open Access‘ and ‘Repository’ typically go hand in hand. If you support Open Access you need a repository, and if you question the role of repositories you’re in the pocket of evil publishers who want to lock up everything ever written and lease reading rights back to the employers of those who wrote the stuff in the first place.

Nonsense.

Open Access is an important component of today’s scholarly ecosystem. It’s not the only answer, and it’s not perfect, but it does have a significant part to play. Institutions have a role in preserving, disseminating and exploiting the work of their employees, but these are very different tasks that may benefit from different solutions. In too many cases, the repository is by default seen as a preservation mechanism and a dissemination vehicle, and as such it may fail to cost-effectively achieve either aim.

There are some large, well known, and research-intensive institutions where it might be possible to make a compelling argument for projecting a strong institutional image around a single ‘home’ for all of that research output. Never mind, for a moment, that so much research today is the result of inter-institutional collaboration, or that the eminent researcher might wish to take ‘their’ research publications with them as they move from Oxford to Harvard to York during their glittering career.

Alongside those institutions sit a plethora of others where research of equal quality is also being conducted; there just, maybe, isn’t quite as much of it. Bombarded by ‘advice’ and funding, and desperate to keep up with the Russell Group, ever-more institutions blindly join the repository cult and wonder why their new toys do not fill to overflowing with the jewels of scholarly erudition.

As research becomes increasingly data-rich, the whole cycle looks set to repeat. The recently released Panton Principles for Open Data in Science are to be welcomed, but I’ll bet the institutional response will all too often be the commissioning of a ‘data repository’ to sit alongside the ‘publication repository’ they already don’t use.

All of which is a rather long-winded way of introducing the fact that Eduserv’s Andy Powell has asked me to facilitate a breakout afternoon on ‘Policy Issues’ at the Repositories in the Cloud event Eduserv and JISC are holding in London on Tuesday.

“This free event, organised jointly by Eduserv and the JISC, will bring together software developers, repository managers, service providers, funding and advisory bodies to discuss the potential policy and technical issues associated with cloud computing and the delivery of repository services in UK HEIs.”

In a post on 11 February, Andy invited participants to share some of their views ahead of the meeting, and on 19 February he wrote about some of his own thoughts.

Like Andy, I struggled somewhat to nail down a coherent set of thoughts about the issue of pushing today’s repositories into the Cloud. On one level, I wonder whether the vast majority of institutions with small (and relatively low traffic) repositories would see much of a tangible efficiency gain or cost saving by moving off an in-house computer to rent an equivalent Virtual Machine from Amazon, Rackspace, or any of their competitors. If we’re talking about IT systems within a typical university, there are others (email, calendaring, pools of compute resource for research jobs, etc) that appear more immediately compelling for the shift Cloud-ward. Which is not to say that there isn’t a clear opportunity for someone trusted to step into this space and offer a SaaS repository to which institutions might affordably subscribe. Eduserv? Mimas? Edina? The British Library? The National Archives? Duraspace? Any could, and if we’re not ready for something more then at least one probably should.

However, a bolder reconsideration of what repositories are and what they’re for might very well lead to something interesting, sustainable, and perfectly suited for benefitting from Cloud Computing’s strengths.

Why does a paper have to be ‘deposited’ in a repository? Why does a single paper with three authors from three institutions have to be deposited in three separate institutional repositories? Why does that same paper have to be deposited – separately – in the subject repository favoured by scholars in the relevant discipline? Why does the institution’s very reasonable desire to protect, preserve, promote and disseminate its excellence mean that it has to run systems in perpetuity that preserve and permit access? Why do we address the fundamentally different (perhaps even contradictory) problems of access and preservation in the same system? Why can’t the individual researcher easily assemble a view across their publication history, regardless of the institution within which they happened to reside as they wrote each paper? Why don’t the assemblages of papers reflect personal, professional and disciplinary relationships, alongside (or instead of) the contractual accident of employee-employer relationships? Why isn’t the wealth of metadata implicit to any publication (authors, subjects, dates, citations, and more) available and actionable, both inside the repository and far beyond it across the Web? Why isn’t there a tight and active association between the paper and the data from which its findings were derived (something for which Internet Archaeology was demonstrating utility a very long time ago)?

Scholarly papers principally comprise text, augmented by the occasional static image. They’re not big, and they don’t tend to change very fast. In many ways, they represent a fairly easy problem set with which to work. As more and more data becomes key to research in a growing number of subject areas, the problems are set to become far larger and far more difficult. For individual universities to even consider replicating the process by which they all ended up with their repositories of text surely seems madness in this data-rich environment. Even with levels of uptake as low as those seen in too many text repositories, the issues of data management, curation, access and dissemination are too great to be sensibly solved in the institutional machine room. Services like InfoChimps and Amazon’s own Public Data Sets offering show some of the ways that we might begin to work with data at scale. Might we, for example, come to recognise as Amazon has that it’s actually cheaper and quicker to entrust large data sets to FedEx rather than transmit them over the Internet?

‘The answer’ might be some central service for the community, funded by JISC like the Arts & Humanities Data Service (AHDS) of old. Or it might be something different, something nimbler, more responsive, more flexible to individual, institutional, and disciplinary requirements, and something more scalable to new disciplines; institutional support for and use of existing Cloud infrastructures extending far beyond UK Higher Education, aligned with a clear understanding of the separation between preservation and access.

I certainly don’t have all the answers, but I do believe that simply asking whether or not we should move existing repositories to the Cloud is to miss the point. Rather, we should ask what role the Cloud might play in addressing the business requirements to which the institutional repository was our initial – faltering – response. The answer might very well be ‘None,’ but I doubt it.

I look forward to Tuesday’s discussion. I’m not going there to push my personal view that individual institutions frequently shouldn’t be building, running or populating their own repositories at all. I’m going there to facilitate the discussion those in the room want to have, and to learn from their experiences and their perspectives.

Related articles by Zemanta

• Does a Citation Advantage Exist for Mandated Open Access Articles? (scholarlykitchen.sspnet.org)
• Scholarly content and the cliff edge: the place of subject ‘repositories’ (hangingtogether.org)
• Scholarly Communications must be Scalable (downes.ca)
• Beyond Open Access: Open Publishing (opendotdotdot.blogspot.com)
• 57 college presidents declare support for public access to publicly funded research in the US (scienceblog.com)
• Mandelson says academics are ‘set in aspic’ (guardian.co.uk)

Before going any further, let’s get a few things crystal clear;

1. The recent success of the Linked Data meme is long overdue, very welcome, and entirely capable of carrying the Web of Data far beyond its current niche adherents. A lot of my current work involves arguing that more organisations should adopt this approach;
2. The Resource Description Framework, RDF, is a key — and powerful — piece in W3C‘s Semantic Web Architecture. Since its earliest days, I have played various parts in advocating the potential of RDF and will continue to do so;
3. RDF is an obvious means of publishing — and consuming — Linked Data powerfully, flexibly, and interoperably. I will continue to argue this, and to advocate its wider adoption.

So far, so good.

The problem, I contend, comes when well-meaning and knowledgeable advocates of both Linked Data and RDF conflate the two and infer, imply or assert that ‘Linked Data’ can only be Linked Data if expressed in RDF.

This dogmatism makes me deeply uncomfortable, and I find myself unable to agree with the underlying premise.

The rest of this post attempts to explain why, hopefully more lucidly than I or those with whom I was debating managed on Friday evening via the largely unsuitable medium of the 140 character tweet.

Andy Powell started things off lucidly enough on Friday, asking;

“is there an agreed name for an approach that adopts the 4 principles of #linkeddata minus the phrase, ‘using the standards (RDF, SPARQL)’ ??”

I was amongst those to respond, suggesting as I usually do that;

“well, personally, I’d argue that Linked Data does NOT require that phrase. But I know others disagree…  ”

Other pieces of that conversation can be extracted from the stream; start by scrolling to the bottom, find Andy’s tweet, and work back toward the top.

It’s worth noting that two of those arguing most vehemently against me were former colleagues Ian Davis and Leigh Dodds. I have massive respect for the technical prowess of both (which is certainly greater than my own), and have learned a great deal from Ian in particular over the years that we have known one another. This issue, though, is one on which we have long disagreed, and it was interesting to see the subject of many a difference of opinion in the bars of various conference hotels spill into this public arena.

Anyway, now let me try to explain what I meant.

Perhaps the most commonly cited definition for Linked Data is the one to which Andy was referring; Sir Tim Berners-Lee‘s Linked Data – Design Issues document. It’s worth noting that this document is clearly flagged (in the current version amended on 18 June 2009, at least) as being both a ‘personal view only’ and ‘imperfect but published.’ So a very long way from being a ‘standard,’ ‘specification,’ or ‘definition,’ but certainly still a pretty good starting point, and one to which I often direct clients and others.

Berners-Lee begins,

“The Semantic Web isn’t just about putting data on the web. It is about making links, so that a person or machine can explore the web of data.  With linked data, when you have some of it, you can find other, related, data.”

(my emphasis)

That sounds good, doesn’t it? Indeed, we talked about that on the Linked Data panel I moderated at the recent Semantic Technology Conference, and I’ve embedded the video here.

It is the next section of Berners-Lee’s document that is used to validate the view that Linked Data needs RDF;

“1. Use URIs as names for things

2. Use HTTP URIs so that people can look up those names

3. When someone looks up a URI, provide useful information, using the standards (RDF, SPARQL)

4. Include links to other URIs. so that they can discover more things.

(my emphasis)”

On one reading, an unambiguous validation of the view with which I disagree. On another, a suggestion of best practice, expressed as part of a ‘personal view’ with which we are perfectly entitled to take issue.

Would the zealots be calmed by the simple insertion of ‘preferably’ or ‘ideally,’ immediately after point three’s second comma? Maybe. Or perhaps the fires of Linked Data’s self-appointed Inquisition would be stoked for Berners-Lee himself.

Talk of Linked Data, Open Data, the Web of Data and related concepts in recent years have led to a quite remarkable shift in attitude amongst individuals, public bodies and private corporations. Almost everywhere my work takes me, clever people are seriously grappling with the implications of consuming from or contributing to these emerging ecosystems. Not all of their questions have good answers, and not all of the technological, strategic and business implications have necessarily been fully worked through. But these people are asking the questions, and they are asking them in all seriousness.That is a dramatic and welcome shift.

Some, such as the BBC, Thomson Reuters and the UK Government’s Central Office of Information are sufficiently persuaded of the benefits to take risks and to open the previously closed in taking a lead. Others will follow, as fears are assuaged, doubts eased, and benefits realised.

Despite this undoubted progress, the green shoots of a Linked Data ecology remain delicate. By moving from a message that stresses the value of unambiguous and web-addressable naming (HTTP URIs), providing ‘useful information,’ and enabling people to ‘discover more things’ by linking toward a message that elevates one of the best mechanisms (RDF) for achieving this to become the only permissible approach, we do the broader aims great harm.

Yes, those already in the club will probably be very pleased with the purity and functionality of the toys in their playground. But they will have barred a far larger group with data to share, a willingness to learn, and an enthusiasm to engage. At best, they will have slowed the growth of the pool of Linked Data quite dramatically. At worst, they will have created an increasingly irrelevant backwater that more pragmatic people will simply route around. Perhaps, in their pragmatism, those people will now never look seriously at RDF and its power, scared away by the fervour of those who sought to elevate it too high, and too fast.

What are we after? More Linked Data, or more RDF? I sincerely hope it’s the former.

So let’s see loads more Linked Data, and plenty of evangelism as to why RDF could be the best way to do it. But let’s not ostracise the vast majority of potential participants, contributors and beneficiaries in the world of Linked Data, just because they haven’t wholeheartedly embraced RDF yet.

Related articles by Zemanta

• Linked Data is Blooming: Why You Should Care (readwriteweb.com)
• Nodalities (Talis): Ivan Herman talks about the Semantic Web and W3C (blogs.talis.com)
• ReadWriteWeb Interview With Tim Berners-Lee, Part 1: Linked Data (readwriteweb.com)
• Linking bbc.co.uk to the Linked Data cloud (derivadow.com)
• Tim Berners-Lee: “We need data on the Web to work better together” (semantic-web.at)
• What does the history of the web tell us about its future? (derivadow.com)
• Tim Berners-Lee’s Eloquent Ted Speech on Linked Data (byronmiller.typepad.com)
• Sir Tim, the web and silos (bbc.co.uk)
• Talis Connected Commons: Linked open data repository opens up shop (mndoci.com)
• Jeff Pollock talks about his new book, The Semantic Web for Dummies (blogs.talis.com)
• CNET Partners with Thomson Reuters on Linked Data Initiative (readwriteweb.com)
• ReadWriteWeb Interview With Tim Berners-Lee, Part 2: Search Engines, User Interfaces for Data, Wolfram Alpha, And More… (readwriteweb.com)
• How Open is ‘Open’ ? (cloudofdata.com)
• Crossref, OpenURL and more Linked Data Heresy (go-to-hellman.blogspot.com)