I don’t think I’ve ever really been convinced by the argument that everything will end up in the data centres of Amazon.

The straightforward provision of commodity Cloud Computing is an important – and growing – area, and one that will continue to expand as interfaces become simpler, FUD is challenged, and prices maintain their relentless march towards the bottom. Everyone has something they could usefully, sensibly, and cost-effectively run in a commodity Cloud such as those offered by Amazon, Rackspace, Flexiant, and others. In this space, basic stability, security and reliability combine with a compelling – and diminishing – pricing proposition to create commodity services targeted squarely to lowest common denominator functionality. Here, market forces may (inevitably?) lead to an eventual reduction in the number of providers. Cost, although not the only consideration, is both important and compelling. Although markets like competition, there may even be a single winner here, one day.

Layered all around the basic, routine, grunt-work computation that these commodity public clouds handle so well, many organisations find themselves having to cope with a wide range of other use cases and data sets. Some require specialist hardware (like the GPUs that Amazon has recently begun selling access to). Some demand particular regulatory and legislative hoops to be jumped through. Some have quirky requirements around latency in data transfer or speed of in-CPU processing. Some have lots of data, and issues with regard to getting the stuff from one location to another with a sensible balance between transfer cost and time.

All of these are certainly capable of being addressed in the Cloud, but the economics and the business rationale begin to shift. For the data owner, cost may no longer be quite so significant a factor. Reliability may matter more, or speed, or the audit trail. For the Cloud provider, these requirements no longer look like the lowest common denominator. It’s not cost-effective to provide these capabilities to everyone and still keep the price low. It becomes more sensible to segment, to divide, and to create bespoke offerings of various kinds. Some of these services require such specific things in terms of network topology, physical building layout, and staff expertise that it may even become counter-productive to have these services in the same building as the commodity Cloud. Here, there’s plenty of room for new entrants, plenty of scope for competition, and plenty of opportunity to differentiate in terms of price, location, support, and a host of other factors. This segment of the Cloud is only just getting started.

In these contexts, we see compelling arguments made for on-premise private clouds, off-premise private clouds, hybrid clouds, community clouds and the rest. Some of the arguments made in favour of private and hybrid certainly are part of the FUD we see in this space, but beneath the noise, the security scares, and the vested interests of SysAdmins and sellers of data centre components, there lies a grain of truth. Not everything is most sensibly run on a cheap VM, rented from Amazon (or Rackspace, or whoever) with your credit card, and physically located half way round the planet.

Unfortunately, it can be difficult to make sensible decisions about which type of cloud works best in each situation, and large swathes of the market are doing everything in their power to add to the confusion.

Having accepted that the basic offering from a public cloud provider is not the solution for my particular requirements, where do I turn next?

Do I listen to the (convincing) pitch from a vendor of ‘community cloud’ solutions for my domain? If I’m in Healthcare, they come with HIPAA and European Data Protection Directive, and all sorts of other accreditations. For dealing with sensitive patient data, this may be just what I need… but does the wily salesman also persuade me to run staff email and the hospital volleyball club website on this over-specified (and expensive) infrastructure?

Do I listen to the (convincing) pitch from a vendor of virtualisation software? If I’ve got a reasonably sized data centre with some life left in it, I may see the value of virtualising all of that expensive hardware, and running current applications in house more efficiently. But instead of gradually reducing my in-house costs, do I continue to add more machines as current ones reach end of life, or as new requirements come along?

Do I listen to the (convincing) pitch from my co-location facility, which happily sells me a ‘private cloud’ that may fail to deliver some of the economies of scale so central to the main Cloud proposition?

Do I listen to the horror stories, stick my head in the sand, and simply keep ordering servers until every single one of my competitors undercuts my costs and I go out of business?

These, and more, are certainly possible. But let’s return to that UK project I mentioned right at the start.

Flexible Services for the Support of Research (FleSSR) is

“a new cloud pilot project looking at utilising hybrid private-public IaaS cloud infrastructure to provide computational and data services to the academic research community. The project is a collaboration between the Oxford e-Research Center, IT Service @ University or Reading, e-Science Centre @ STFC, Eduserv, EoverI, Eucalyptus INC and Canonical Ltd.”

The ten month project is funded by the Joint Information Systems Committee (JISC), an organisation that supports the innovative use of IT across UK universities.

Now, to a degree, the project’s mindset must be influenced by its partners. IT staff at Reading and STFC are incumbents with turf to protect (or new vistas to discover, map, and claim). Eduserv has a new data centre that they’d like to fill with willing clients. It would be easy to be cynical, but knowing some of the people involved, I see no real reason to be. It is perfectly reasonable to suggest that a ‘community’ the size of UK Higher Education would realise value in replicating less (not nothing) at every university campus across the country, and bringing much of that together in some sort of Cloud. That Cloud might use public infrastructure, or it might be served up from an organisation such as Eduserv, which is known to the community, aware of the community’s requirements, quirks and foibles, and (importantly) not-for profit (and therefore cheaper?).

Personally, I’d always rather presumed that an organisation like Eduserv (or JISC itself) would act on behalf of the community to procure a competitive price on access to the resources of Amazon, Rackspace, or one of the others. I’m not convinced that most UK research computation needs any sort of special treatment that couldn’t be met from Amazon’s Dublin data centre… unless the community itself can somehow beat – and continue to beat – Amazon on price. Somewhat surprisingly, that’s exactly what some calculations in two posts by Eduserv’s Andy Powell suggest could happen. By including network costs and other charges over and above the basic storage cost, Andy finds Amazon, Rackspace and Dropbox to be more expensive than anticipated, and posits that Eduserv (connected to every UK university free of charge via JISC’s high speed JANET service, and constrained in the ways it can generate profit from services sold to universities by its charitable status) might actually work out cheaper.

There’s a lot of work to do in terms of fleshing out the assumptions behind some of Andy’s figures, but the whole industry certainly benefits when people conduct exercises like these out in the open, for all to see. If Andy has made mistakes, the vendors should be quick to jump in and correct them. If his assumptions miss the mark, public debate can redress the balance.

The Cloud is not all about price. But more transparency around the true cost of computing in the Cloud – and in your data centre – means that we can all make more informed decisions.

Thanks for sharing, Andy – and hopefully readers will be willing and able to look over your calculations and share their own views.

Note: this post was conceived and written in the United Kingdom. By reading this post you agree to comply with UK usage, and will henceforth pronounce the word ‘niche’ from the title as ‘neesh,’ not ‘nitch.’ Or maybe not.

Related articles

• Rackspace launches managed cloud services (zdnet.com)
• Are hybrid clouds the path to cloud-computing nirvana? (venturebeat.com)
• We’ll Take Care of Your Cloud, While You Manage Your Business (rackspacecloud.com)
• Trust Is Key For Cloud Success And What Can We Do About It? (cloudave.com)

As with so many words, registry is one that the technology sector coopted and quickly began to render almost meaningless by adding layer upon layer of obscure nuance. For Windows users (at least back when I last used Windows), the Registry was the place you went to disable whatever was responsible for slowing your machine to a painful crawl. On the web, registry shares some of the attributes of library, directory, repository, and similar concepts; it is typically a place into which some resource, function, or package is registered for later discovery and use by others. Personally, I’ve always tended to presume that registries are differentiated from those similar terms by being intended wholly or predominantly for use by machines rather than people; a registry a place for software to find the data it needs, isn’t it?

Registries crop up regularly in discussion of curated online resources, typically when someone says “We need a registry for x” or (increasingly), “The Web is the registry for y.” To synthesise some of the different perspectives here, and to really look at whether or not the Web could be the registry, the UK’s Joint Information Systems Committee (JISC) has asked me to conduct a short desk-based project which is now well underway.

In the education/ cultural heritage/ government contexts within which JISC operates, there are plenty of ‘registries’ to look at; the venerable Information Environment Service Registry, the CKAN instance sitting behind data.gov.uk, America’s new National Learning Registry, Australia’s Registry of Research Collections and more. But what do they have in common, how are they differentiated, and where are the similar examples from the business world? Since the decline in enthusiasm for standards like UDDI, are formal registries in decline?

I’m certainly not seeing mainstream uses of the term that seem ‘wrong’, but there’s clearly scope to develop some sort of simple taxonomy to capture the various functions for which someone might propose a registry-based solution. Having done that, I can then finish by looking at the extent to which the architecture of the Web can reliably and sustainably support those functions.

So… what do you think a registry is, and do you have any great examples of that use case? Is it best done in some centralised and organised fashion, or can resources spread out across the web simply self-assemble as required?

Related articles

• FUSION Semantic Registry (seerc.org)

The final version of that report is available today, both in commentable form via JISC’s JISCPress tool and for download as a PDF.

JISC’s associated call to fund Linked Data projects will be available imminently, and £750,000 is available to share between successful bidders. Watch the JISC funding pages, and get ready to bid. If you’re not inside a UK university yourself, then find a friendly partner who is… as projects have to be led by a University. And good luck!

Related articles by Zemanta

• DRAFT report explores Linked Data potential in UK Universities (cloudofdata.com)

Furthermore, ‘Open Access‘ and ‘Repository’ typically go hand in hand. If you support Open Access you need a repository, and if you question the role of repositories you’re in the pocket of evil publishers who want to lock up everything ever written and lease reading rights back to the employers of those who wrote the stuff in the first place.

Nonsense.

Open Access is an important component of today’s scholarly ecosystem. It’s not the only answer, and it’s not perfect, but it does have a significant part to play. Institutions have a role in preserving, disseminating and exploiting the work of their employees, but these are very different tasks that may benefit from different solutions. In too many cases, the repository is by default seen as a preservation mechanism and a dissemination vehicle, and as such it may fail to cost-effectively achieve either aim.

There are some large, well known, and research-intensive institutions where it might be possible to make a compelling argument for projecting a strong institutional image around a single ‘home’ for all of that research output. Never mind, for a moment, that so much research today is the result of inter-institutional collaboration, or that the eminent researcher might wish to take ‘their’ research publications with them as they move from Oxford to Harvard to York during their glittering career.

Alongside those institutions sit a plethora of others where research of equal quality is also being conducted; there just, maybe, isn’t quite as much of it. Bombarded by ‘advice’ and funding, and desperate to keep up with the Russell Group, ever-more institutions blindly join the repository cult and wonder why their new toys do not fill to overflowing with the jewels of scholarly erudition.

As research becomes increasingly data-rich, the whole cycle looks set to repeat. The recently released Panton Principles for Open Data in Science are to be welcomed, but I’ll bet the institutional response will all too often be the commissioning of a ‘data repository’ to sit alongside the ‘publication repository’ they already don’t use.

All of which is a rather long-winded way of introducing the fact that Eduserv’s Andy Powell has asked me to facilitate a breakout afternoon on ‘Policy Issues’ at the Repositories in the Cloud event Eduserv and JISC are holding in London on Tuesday.

“This free event, organised jointly by Eduserv and the JISC, will bring together software developers, repository managers, service providers, funding and advisory bodies to discuss the potential policy and technical issues associated with cloud computing and the delivery of repository services in UK HEIs.”

In a post on 11 February, Andy invited participants to share some of their views ahead of the meeting, and on 19 February he wrote about some of his own thoughts.

Like Andy, I struggled somewhat to nail down a coherent set of thoughts about the issue of pushing today’s repositories into the Cloud. On one level, I wonder whether the vast majority of institutions with small (and relatively low traffic) repositories would see much of a tangible efficiency gain or cost saving by moving off an in-house computer to rent an equivalent Virtual Machine from Amazon, Rackspace, or any of their competitors. If we’re talking about IT systems within a typical university, there are others (email, calendaring, pools of compute resource for research jobs, etc) that appear more immediately compelling for the shift Cloud-ward. Which is not to say that there isn’t a clear opportunity for someone trusted to step into this space and offer a SaaS repository to which institutions might affordably subscribe. Eduserv? Mimas? Edina? The British Library? The National Archives? Duraspace? Any could, and if we’re not ready for something more then at least one probably should.

However, a bolder reconsideration of what repositories are and what they’re for might very well lead to something interesting, sustainable, and perfectly suited for benefitting from Cloud Computing’s strengths.

Why does a paper have to be ‘deposited’ in a repository? Why does a single paper with three authors from three institutions have to be deposited in three separate institutional repositories? Why does that same paper have to be deposited – separately – in the subject repository favoured by scholars in the relevant discipline? Why does the institution’s very reasonable desire to protect, preserve, promote and disseminate its excellence mean that it has to run systems in perpetuity that preserve and permit access? Why do we address the fundamentally different (perhaps even contradictory) problems of access and preservation in the same system? Why can’t the individual researcher easily assemble a view across their publication history, regardless of the institution within which they happened to reside as they wrote each paper? Why don’t the assemblages of papers reflect personal, professional and disciplinary relationships, alongside (or instead of) the contractual accident of employee-employer relationships? Why isn’t the wealth of metadata implicit to any publication (authors, subjects, dates, citations, and more) available and actionable, both inside the repository and far beyond it across the Web? Why isn’t there a tight and active association between the paper and the data from which its findings were derived (something for which Internet Archaeology was demonstrating utility a very long time ago)?

Scholarly papers principally comprise text, augmented by the occasional static image. They’re not big, and they don’t tend to change very fast. In many ways, they represent a fairly easy problem set with which to work. As more and more data becomes key to research in a growing number of subject areas, the problems are set to become far larger and far more difficult. For individual universities to even consider replicating the process by which they all ended up with their repositories of text surely seems madness in this data-rich environment. Even with levels of uptake as low as those seen in too many text repositories, the issues of data management, curation, access and dissemination are too great to be sensibly solved in the institutional machine room. Services like InfoChimps and Amazon’s own Public Data Sets offering show some of the ways that we might begin to work with data at scale. Might we, for example, come to recognise as Amazon has that it’s actually cheaper and quicker to entrust large data sets to FedEx rather than transmit them over the Internet?

‘The answer’ might be some central service for the community, funded by JISC like the Arts & Humanities Data Service (AHDS) of old. Or it might be something different, something nimbler, more responsive, more flexible to individual, institutional, and disciplinary requirements, and something more scalable to new disciplines; institutional support for and use of existing Cloud infrastructures extending far beyond UK Higher Education, aligned with a clear understanding of the separation between preservation and access.

I certainly don’t have all the answers, but I do believe that simply asking whether or not we should move existing repositories to the Cloud is to miss the point. Rather, we should ask what role the Cloud might play in addressing the business requirements to which the institutional repository was our initial – faltering – response. The answer might very well be ‘None,’ but I doubt it.

I look forward to Tuesday’s discussion. I’m not going there to push my personal view that individual institutions frequently shouldn’t be building, running or populating their own repositories at all. I’m going there to facilitate the discussion those in the room want to have, and to learn from their experiences and their perspectives.

Related articles by Zemanta

• Does a Citation Advantage Exist for Mandated Open Access Articles? (scholarlykitchen.sspnet.org)
• Scholarly content and the cliff edge: the place of subject ‘repositories’ (hangingtogether.org)
• Scholarly Communications must be Scalable (downes.ca)
• Beyond Open Access: Open Publishing (opendotdotdot.blogspot.com)
• 57 college presidents declare support for public access to publicly funded research in the US (scienceblog.com)
• Mandelson says academics are ‘set in aspic’ (guardian.co.uk)

Regardless of where you stand on some of the questions of detail with respect to the Linked Data meme, it’s clear that significant enthusiasm is being marshalled behind both the concept and the opportunities that it promises.

Dion Hinchcliffe looks at some of the means by which enterprise data can be more visible on (and useful to) the Web in a ZDNet post this week. The ‘Semantic Web & Linked Data’ are included, and Dion writes;

“By far the most sophisticated and complex of the three approaches to open data presented here, [Linked Data is] highly suitable for certain applications that have rich data sets that need powerful means of processing and consumption. In particular, scientific, technical, medical, mapping, and certain government domains are highly suitable for this approach. It remains unclear if Linked Data will finally trigger the boom in the Semantic Web so use with care. However, definite consideration should be applied, given the potential of the approach to create data sets with extraordinarily high function. Businesses already managing their data with Semantic Web technologies will be the most likely candidates for adoption.”

We’re certainly seeing plenty of talk — and some interesting beginnings — in the Government domain, and organisations such as Thomson Reuters and the BBC are also taking compelling steps around the periphery of their core businesses.

Education offers another interesting set of opportunities, and Jason Ohler’s piece in Educause Quarterly (and a related podcast I recorded with him whilst I was still at Talis) illustrates one view of that opportunity.

Here in the UK, the Joint Information Systems Committee (JISC) is beginning to take note. They first funded a review of ‘Semantic Web Technologies‘ back in 2005, then revisited the topic with ‘Semantic Technologies in Learning and Teaching‘ (and a related workshop in the south of France later this year). I’ll be recording a podcast with the manager of that project, Thanassis Tiropanis, later this month.

JISC have also asked me to conduct a short piece of work to look specifically at the opportunity presented to the Higher Education community by Linked Data, and this work will run over the next few months. I’m certainly keen to learn about concrete examples, and to hear reasoned arguments for and against in order to submit comprehensive findings and recommendations. So if you have something to say, please do get in touch.

Related articles by Zemanta

• Web3 and Enterprise Linked Data. The Middleware Revolution (aldobucchi.com)
• Open government and Linked Data; now it’s time to draft… (johnbreslin.com)
• ReadWriteWeb Interview With Tim Berners-Lee, Part 1: Linked Data (readwriteweb.com)
• Linked Data is Blooming: Why You Should Care (readwriteweb.com)