Image by Kevin Krejci via Flickr

The CloudCamp unconference returned to London for the 14th time this evening, regaling a capacity crowd in the Crypt below Clerkenwell’s St James Church with several hours of discussion and debate on the somewhat elusive topic of ‘Big Data’.

Rather rough notes of the proceedings follow, after the break.

LEF‘s Simon Wardley kicked proceedings off as usual, once again managing to pepper an on-topic canter through the topic with a seemingly never-ending stream of Flickr images of cats… and analogies to electricity. You possibly had to be there? His core message, though? There’s nothing new under the sun… and the cycles of change just keep on coming.

Next, Peter Matthews from CA Labs, on “is big data mutually compatible with the cloud?” Erm, yes. Data volumes with big data are so large that it’s difficult to move it around… which creates opportunities for lock-in that vendors may wish to seize. And then he was out of time.

Next, Fujitsu’s Mark Wilson on ‘Structuring Big Data.’ He’s actually talking about Linked Data, a topic I’ve dug into before here and over on semanticweb.com – Linked Data could be/ might be the effective realisation of the decade-old Semantic Web dream. Big Data means masses of unstructured or semi-structured content, presenting a management headache of previously unanticipated proportions. Linked Data, he argues, creates the mechanism to link all of this data together from across disparate sources. Yes, but it’s easier to say than to do… And in 5 minutes he really couldn’t explain enough to persuade the audience. Linked Data should be “the optimal reference source,” he said. It should be “a broker for all data sources,” and we should “think about integration, not duplication.” Yeeeeees… But.

Next, Canonical’s Nick Barcet, talking around scalability, Ubuntu, package management, configuration management, etc. Not wholly sure what the point was, I’m afraid.

Next, Chris Swan from UBS – big data and security. “If you’ve got security controls that aren’t properly monitored, then they don’t matter.”

Next, Tom Leyden of Amplidata – Big “Unstructured” Data in the Cloud. Data storage to increase 30x over the next decade, but staff will only increase 50% over the same period. Challenge in the 90s, as existing storage and analysis technologies struggled to cope with new data volumes. Seeing similar problems today with data streaming from sensor web, etc. Traditional file systems cannot cope. Object Storage the way forward ?

Next, Alex Farquhar – “Cloud v Big Data.” Not really versus… but intersection of the two. Too much discussion of his company, Forward. Just talking about how his company uses cloud to provision IT resources. Might work as a conference presentation or case study – not sure it fits as a 5 minute lightning chat. Around 60TB of data at Forward. Diverse and vital. Using Hadoop cluster – 24 nodes on-premise. Rationale (proximity to the cluster) seemed odd. That can be true, but not clear that it really needs to be the case here?

Next, Alaric Snell-Pym, on Scaling Hadoop. Trying to overcome Hadoop’s I/O bottleneck. Explaining basics of Hadoop and Map/Reduce – no one else has. Explains use of HDFS and ‘selective reading’ to manage lots of small tables and overcome the problems of I/O.

Next, Matt Wood from Amazon. Talking about genetics and the human genome. It’s an analogy. Human Genome Project took years and millions of dollars. Development of gene sequencing machines led to a step change – dramatic drop in cost of sequencing DNA. Like the cloud, anyone? But… the machines create an analysis challenge, because they generate so much data. Cloud offers “collection of productivity tools” to help scientists work with this data collaboratively and (relatively) affordably. A perfect example of a lightning presentation, unlike most of those who preceded him.

And finally, an impromptu slot from HP’s Joe Weinman. A quick overview of current thinking behind his latest book. This one could have gone for much longer… Good stuff.

And that’s the lightning talks finished. Now, the panel, and Simon Wardley’s search for “experts” and “volunteers.”

…and unfortunately, your scribe was ‘volunteered’ as an ‘expert’ by Mr Wardley… and here end the notes. It was great to have Amazon’s Werner Vogels sneak in, and lob comments into the panel, though…

Great event, though with the usual mix of people you wish could have talked for longer… and people you wish wouldn’t have spoken.

Related articles

• The brave new world of big data & Hadoop (venturebeat.com)
• Big VCs Invest In Big Data Startup Continuuity (techcrunch.com)

Image via Wikipedia

For real-world applications of Linked Data and the Semantic Web, the long-running Semantic Technology Conference is hard to beat. For getting a real handle on the Cloud Computing landscape, GigaOM‘s Structure Conference is also a leading light.

Working across both areas as I do, these events tend to figure prominently in my calendar for the year*. Last year, both took place in San Francisco during the same week. I tried to attend both, and therefore succeeded in spending most of my week in cabs, shuttling between meetings at the two venues. I saw very few sessions that I wasn’t personally involved in, and the experience wasn’t a huge success.

This year the conference organisers have taken pity on me, and moved their events to opposite ends of June. The Semantic Technology Conference is up first, back at the Hilton Union Square from 5-9 June. Structure follows, returning to the Mission Bay Conference Centre on 22 and 23 June.

I’ll be attending both, and probably doing various official things during each event. At the moment, the only thing we’ve definitely nailed down is a special live appearance by The Semantic Link crew on the evening of 5 June; we’ll be taking a look at the highlights expected for the conference, offering some tips for those new to the event and its multitude of parallel sessions, and generally bringing our usual podcast chatter to the stage.

So if you’re in town around the time of either event, do get in touch; it promises to be an interesting month.

* Disclosure: WebMediaBrands pay me to host the monthly Semantic Link podcast, and to write a monthly column on SemanticWeb.com. GigaOM pay me to curate the Infrastructure/ Cloud Computing channel on their Pro site. I attended and participated in both of these events before that was the case, and still would today without the contractual relationship.

Related articles

• GigaOM Structure 2011 (datacenterknowledge.com)
• Semantic Technology Conference (fakeiitian.com)

• My latest monthly column on SemanticWeb.com; Big Data Presents a Big Opportunity?
• My latest weekly wrap-up on GigaOMPro; Rosslyn Analytics, Microsoft Finding Value in Data Aggregation
• The teaser piece on GigaOM’s public Cloud site; In Exploiting the Data Market, Trust Is Key

I spot a theme building…

Simon Rogers, The Guardian Data Store

What we do with data. Guardian started in Manchester, first issue 4 pages. That first issue included a table of data about schools in Manchester; number of pupils, cost, etc.

Guardian now… aggregates data published by Government departments in PDF, extract data values and present in useful manner.

guardian.co.uk/data – Data Store. Make data available from all over the place, do some stuff with it, but invite others to engage, enhance, correct, use.

“Every time we publish government data, we get 2-3 calls from Gov departments, asking if they can have a copy.”

Making data available to end users means they follow their interests… sometimes results in stories for the news.

Wikileaks… “without Guardian data journalism, we wouldn’t have got the stories from the data.”

Journalists had the data… but didn’t know what to do with it in order to find and tell stories. Needed to provide tools for journalists so that they could navigate the data.

Wikileaks first story – IED cases in Afghanistan; flat visualisation for the paper, plus an interactive tool on the site.

Google Fusion Tables – “an absolutely brilliant tool for mapping lots of data very quickly. Thank you, Google.”

Data exploration tools prove more popular than canned visualisations. People want to explore and understand the data.

Embassy Cables – latest release. As with the others, the paper chooses not to release some of the data. “4 million people can see these ‘secret’ cables.”

The power of stories; didn’t kill journalism. Instead it enhanced it. Provide good journalists with the tools to enrich their stories.

Guardian journalist James Cameron; “the only questions left will be answered by computers because only computers will know the questions to ask.”

Next, a panel… Amber Case, Brad Cross (formerly Flightcaster), Toby Siegler (Metaweb/ Google)

“How dependent are we going to get on data to tell us how to lead our lives?”

Amber – “Very. It will become not only convenient, but also the thing to do.”

Toby – “We’re already dependent on data… we just need to get better at deciding what to ignore.”

“Will we stop asking questions, and wait for the computer to tell us what to do? Will we reach The Shallows?”

Brad – “not clear that people distracted by twitter etc were the deep thinkers before…”

Amber – “large proportion of the population may just consume the channels they consumed before…”

Brad – “right now we only really have search… over time we’ll have more systems that do more for us… but they’re not too far away.”

“We put knowledge into a data store… sales figures, etc. With Big Data we don’t necessarily know what we know or want to know… so should we just keep everything?”

Amber – yes.

Toby – “what people say about themselves is a lot less useful than what they do. If you want to predict what people would do, you should collect the data exhaust and look at what they have done.”

Ethically… if we’ve captured all of our data… can we then subpoena data to show where people were, what they were doing, etc. How will law catch up to an expectation that data is available for all of our actions?

Brad – “not new. The problem will get worse, but it’s just something we have to deal with.”

How will this change government? Representative democracy is a hack… we can’t afford to send everyone to Washington/Ottawa. Can digital take us to a real democracy?

not sure they’re sure…

Amber – recording more data to greater resolution is a good thing and will deliver benefits… so long as it’s not always personally identifiable.

What’s the next technology to unlock big data…?

Amber – “not brain implants… Better location-based data, that knows where you’re going and pushes relevant data to you in real-time.”

Brad – “a mix of what’s happening on the internet now, and on mobile now. When you connect information about you on the server with information about you on your mobile device, there’s some really neat stuff that happens there… an agent.” Sounds like Siri?

Toby – “tried an experiment… connected phone records, sms etc to my social network… it was really cool… Integration of your personal data in this way, just for you to look at, is really useful.”

Next, Ed Boyajian from EnterpriseDB. Legacy Databases and the Data Deluge. Sell commercial version of free Postgres database. Enterprise customers coming to them, asking to solve problems with big data.

“Compare growth of IT budget to growth of data… and business demands to use that data more effectively… Gap between growth of data (and demands on it) and IT budget creates real problems.”

“By moving to open source database solutions, you can save money.”

“Mission for our company is to bring low cost high performance solutions to the market.”

Barry Devlin. 25 years ago this month, the first paper describing a data warehouse architecture was published.

Next, DJ Patil from LinkedIn. Analytics, and building teams around data/ analytics.

‘To connect the world’s professionals and help make them more productive and successful’… requires data. Insights on activity today, plus insights on possible career directions.

Showing graph of data science from LinkedIn data… demand is taking off. See the job board here, too.

529 of 790 LinkedIn-using attendees are connected to 1 other.

30 to 20 or more

85 to 10 or more

189 to 5 or more.

Is that good? Is it what we’d expect?

“Data scientists are frustrated by an inability to ship product.” “LinkedIn tried something different – made data science a top-level product team.”

good talk… but then I needed to slip out to a meeting…

The notes that follow are a rather raw summary of some of the things I’m hearing. Later in the week, I’ll try to come back and extract the main issues in a rather more polished form.

Up first, Mike Driscoll of Metamarkets, talking about Mining the Tar Sands of Big Data.

Trying to frame conversation; why are we all here? Why is now the time that we’re all beginning to focus on ‘Big Data’ ? Drawing analogy with the tar sands in Alberta; lots of oil there, but it’s been expensive to extract.

If ‘information is the oil of the 21st century [as Gartner has suggested], then Big Data are the tar sands;’ lots of data, sitting in our data centres, waiting for us to invest in extracting usable data. Expensive, painful, but ultimately valuable.

‘Attack of the exponentials;’ cost of storage, bandwidth and compute falling exponentially. Number of nodes on the network rising exponentially. Intersection creates ‘data singularity.’

Unlike oil, data abundant and renewable. Like oil, extraction of data creates value. Cheaper and easier to extract value from data than ever before.

Three forces reshaping data landscape; sensor networks, cloud computing, and machine learning.

Sensor networks; now prevalent, all-pervasive, ubiquitous, and typically connected. Generating vast amounts of data.

Cloud computing; ‘used to mean everything and nothing.’ But real advantage is that it turns CapEx into OpEx.

Machine learning; gives us the capabilities to process the flood of data, intelligently. Smart Planet, Smart Grid, Smart Business, etc. Driverless cars, spam filters, recommendation engines all drawing upon Machine Learning ideas. All iterating and improving with increasing rapidity. Unlike Cloud and sensors (which are becoming commodities), machine learning algorithms are – and may remain – a competitive advantage.

Four consequences of all this; battle for finite number of good data scientists, changes in the way that data is published (and valued), the end of privacy (?), the rise of data startups.

Battle for data scientists; difficult to hire people who can munge, interpret, and tell stories with data. And everyone last night at bigdatacamp was ‘hiring’.

Retailers, banks, online publishers, etc have tended to hand over the keys of data management to third parties. Seeing pendulum shift the other way, as companies recognise the value of their data and seek to control it – and realise the value. Tension with ‘open data,’ data to the cloud, etc?

Privacy – not about shifting access to data, but about more accurately defining the ways in which it may be used.

Q&A

‘What sort of educational background does a data scientist need?’

knowing some stats helps. knowing some programming helps. But curiosity is key. Not sure there’s a degree out there. Pick up the skills if you have the right mindset. ‘You have to be a bit of a hacker,’ says Mike.

‘What are the three big problems that data science will solve?’

Making sense of the world around you; Freakonomics, for example. Taking data and making sense of how the world is working. Scaling up decision making, so that a data-powered story can be presented in a way that lets people make intelligent decisions.

Next, Barry Devlin from 9sight Consulting talks about The Data-Driven Business and Other Lessons from History.

‘the old guy who has been brought along to talk about history,’ and ’illegitimate grandfather of data warehousing’

Address Past, Present, and Future.

Past – the origins of Data Warehousing

Data Warehouse architecture work at IBM in Europe in the mid-80s.

‘Big Data’ (a couple of hundred MB, at the time) created need to structure the Enterprise Data Warehouse in a particular way. Led to silos. ‘Hard information’ only, at the time. Warehouse designed in a well-architected fashion. Ensures that data flows in a single direction. Possibly too regimented for the 21st century?

Lessons?

Information quality and reliability are key; Master Data Management, etc. This is unlikely to change.

Data volumes and variety have presented big challenges over the years. Expectations and business demands outstrip technological capabilities. Organisational and political issues hamper progress. This is unlikely to change.

Exploration and analysis drives innovation.

Present – Business and technical challenges

3 key trends in business are driving rapid change; closed loop business, massive information volumes, collaboration driving innovation.

Really important to stop talking about ‘unstructured information;’ that’s just noise. Information has structure. Instead, hard information is data; tables, structure, computer-oriented. Meaning and values have been separated. Metadata explicit, and formally modelled. Soft information is not well defined, it is by and for people, it mixes meanings and values. Metadata is implicit, tacit, or non-existent.

Moving from information we understand – and control – to information outside the enterprise that we don’t. Implications for quality, meaning, etc.

Future – a new architecture?

current architecture 25 years old – time for a change?

Next, Bob Page from eBay, talking about Building the Data-Driven Organisation.

eBay “fascinated with numbers;” early 1999 screenshot of homepage, showing lots of stats.

What role does data play in the business? Using Analytics focussed on big buckets; velocity, efficiency, trust, etc.

Efficiency drive – lower insertion fees to list/sell new products by 99%. Decision based on analysis of data?

Trust – top-rated sellers constantly monitored to ensure algorithm is reflecting reality. 22% of sales from trusted sellers in 2009. Now 32%. Actually surprised it’s not higher…

eBay handled $2Bn of sales on mobile devices last year.

Analytics and continual data analysis drives all the apps, trust metrics, etc.

Some figures… 50 TB/day of new data, etc. Lots of other numbers on slide, but it was only up for seconds…

Analytics work in marketing, sales, product dev, and all areas of the business.

More than 85% of eBay’s analytial workload is new and unknown; design for the unknown. Enable exploration of the data, rather than just reporting of established metrics.

Machine Learning: Data trumps Algorithms. This is the promise of Big Data; existing algorithms get better as you throw more data at them. It’s cheaper to throw more data at an algorithm than to invest in developing new algorithms.

Enterprise Data Warehouse for transactional data; purchase history, etc. Behavioural data to track clickstreams, impulse purchases, etc… much larger than the amount of transactional data in traditional systems. No technology silver bullet for the behavioural data; optimise for concurrency, or TCO, or CPU usage, or flexibility, or storage, or governance? Those priorities change the tool you should use.

eBay built a 500-node Hadoop cluster in June 2010. Now they have a much bigger cluster.

Data Marts; ‘a reality for many of us,’ because businesses need to give control to the user. Don’t want infrastructure team/ data scientists as bottle neck. Totally opposite to attitude expressed by previous speaker. But Data Marts end up being very expensive and inefficient. eBay have built a virtual data mart; views onto a single pool of data. Far more efficient, in theory.

And then I had to slip away, and miss the final session before lunch…  More later…

There are some great sessions on the Programme, and I look forward to comparing the diverse ways in which Big Data concepts and methods are being put to work across a range of market segments. I also look forward to exploring answers to some of the questions I posed back in November.

As usual, the diary is filling up with meetings, briefings, and the odd conference session, but there are still some gaps to fill. If you’re at the event – or in the area – and want a chat, why not grab one of the slots over on Tungle?

Related articles

• Six months after “What is data science?” (radar.oreilly.com)
• O’Reilly Strata Conference: Only a few days left for early registration + reader discount (flowingdata.com)
• Big Data Examined At Inaugural O’Reilly Strata Conference (lockergnome.com)

San Jose-based Cirtas emerged from stealth back in September 2010 with a $10 Million (€7.86 Million then) Series A funding round, their novel Bluejet hardware appliance, and the backing of Amazon.

Today they’re back, with a new CEO and another $22.5 Million (€16.6 Million) in the bank. The Series A investors — New Enterprise Associates, Lightspeed Venture Partners and, unusually, Amazon — are joined by Shasta Ventures and Bessemer Venture Partners for a Series B round that positions the company for some rapid growth.

Cirtas’ Bluejet Cloud Storage Controller is a hardware appliance, deployed in the data centres of medium and large enterprises to simplify the task of integrating existing on-premise Tier 2 and Tier 3 storage with disparate Cloud-based solutions such as Amazon’s S3, EMC Atmos, Iron Mountain and AT&T’s Synaptic Storage as a Service.

Talking ahead of today’s announcements, new CEO Gary Messiana suggested that Cirtas’ decision to deliver a hardware appliance rather than a software-based solution reflects their deep understanding of both their customers and the sales channel. Messiana is not the first to suggest that buyers of enterprise storage are a conservative bunch, and he’s certainly not criticising that conservatism; when business continuity depends upon the decisions you make and the systems you buy, you’re hardly going to take unnecessary risks, now are you? A piece of physical hardware that you can see, touch (and even hug) delivers an element of familiarity that appears to appeal to enterprise-class customers taking the first steps to leverage Cloud-based storage within their existing solutions portfolio. With Bluejet, Messiana suggests, control continues to reside inside the data centre. The Cloud provider(s) to which the appliance directs data are simply (dumb?) utilities upon which the enterprise may choose to draw in a manner abstracted by Cirtas’ technology. A hardware solution also suits the channel-based (rather than direct sales) model by which these companies tend to buy. As Messiana notes,

“[much of] the money changes hands in the channel… and [channel partners] know how to sell tin.”

Formerly an entrepreneur in residence at Series B participant Bessemer, Messiana talks persuasively about the clarity of proposition and go to market strategy that drew him to Cirtas. Drawing upon pre-Bessemer experiences as CEO of traffic optimising Netli (acquired by Akamai in 2007), Messiana argues that the shift off-premise makes expertise in moving data just as critical as the data storage skills of incumbents. Might Akamai, Limelight and their ilk make inroads into this market, at the expense of EMC, HDS, 3Par et al, I wonder?

In storage as in so much else, big incumbent enterprises are very different from smaller or younger companies. Whilst startups and SMEs might be quick to embrace entirely virtual solutions — often starting in the Cloud rather than migrating to it from elsewhere — the ‘multi-billion dollar corporations’ served by Cirtas will almost inevitably follow a very different path. Across Pharma, HR, manufacturing, publishing, insurance and finance, Messiana reports that customers with market caps of $500 Million – $10 Billion and more are flocking to the company.

So why take more VC money, so soon, and dilute the company? Messiana insists that “plenty” of the initial $10 Million is still in the bank, and that VCs were falling over one another in their enthusiasm to invest. The deal was apparently closed rapidly, with an aggressive valuation that sees “minimal dilution” whilst giving Messiana the cash to expand sales, support, and other areas of the company.

Cirtas would appear to be off to a good start, but it would be dangerous to be complacent. The company is not alone in seeing hardware as a way to encourage enterprises toward the Cloud, and there are plenty of software-based cloud storage gateways waiting for the opportunity to demonstrate their capabilities as those conservative CIOs become more willing to trust the Cloud. And then there’s Amazon. What might they do next?

Related articles

• Nasuni grabs cloud development cash (go.theregister.com)
• Enterprise Poll: Do You Use Cloud Storage? (readwriteweb.com)
• F5 Friday: F5 ARX Cloud Extender Opens Cloud Storage (devcentral.f5.com)
• The $20 Million Club: 10 Well-Funded Cloud Startups (nytimes.com)

I’ve been a fan of Om Malik‘s boutique analyst site, GigaOM Pro, pretty much from the outset, and happily renew my subscription each year. The site covers a wide range of industry topics, and those Quarterly Wrap-ups are worth the fee all by themselves.

I’ve written a few reports for them in the past, but was delighted when Mike Wolf got in touch to see if I fancied trying my hand at curation on their Infrastructure/Cloud channel.

So next week (from 31 January) I’m going to be gathering and commenting upon links from around the web, writing a daily ‘Today in Infrastructure,’ and finishing off with a Weekly Update. If you’re not (yet!) a subscriber, why not sign up for a free seven day trial and join me for the start of my little adventure?

If there’s something you think I should be covering, do let me know.

Related articles

• Free GigaOM Pro Webinar: The Scalable Cloud (gigaom.com)
• GigaOm raises $2.5M, claims 10,000 Pro subscribers (venturebeat.com)
• The Cloud has a place, even inside heavily regulated industries (cloudofdata.com)
• Talking Scalable Clouds with GigaOM Pro and Limelight Networks (cloudofdata.com)
• GigaOM Structure 2011 (datacenterknowledge.com)
• Join GigaOM at Big Data on March 23 in New York City (gigaom.com)

Image via Wikipedia

As part of a restructuring of the organisation’s US-based data centres, the acronym-laden NATO (North Atlantic Treaty Organisation) today announced that its ACT (Allied Command Transformation) will be deploying IBM‘s PCS (Private Cloud Solution*) inside its HQ SACT (Headquarters, Supreme Allied Commander Transformation) in Norfolk, VA.

According to IBM’s E.J. Herold, the company is delivering the software component of their ‘Cloud in a Box’ solution. Normally sold alongside racks full of IBM blades, the NATO installation will see IBM’s software running on the data centre’s existing (mixed) hardware. A second extant data centre – in San Diego, CA – will also be shut down as part of the transformation.

Comprising 28 members from Albania to the United States, NATO constantly grapples with issues of interoperability. 28 members, plus NATO itself, means (at least!) 29 different IT infrastructures, (at least) 29 different communications protocols, and (at least) 29 different views on the standards and specifications to underpin any one IT deployment. In operational environments, whether military or peace keeping, limitations in interoperability of IT inevitably lead to expense, confusion, delay… and perhaps loss of life. Following NATO’s summit in Lisbon last month, the organisation reaffirmed the importance of greater cooperation between partners.

Today’s announcement aligns well with this new ‘Strategic Concept,’ delivering a cost-effective test and development environment to the ACT. According to IBM’s press release,

“The on-premise cloud will be used to test and develop network solutions for command, control, intelligence, surveillance, and reconnaissance projects. The goal is to demonstrate how recent developments in cloud computing can reduce ramp-up time for enhanced technology capabilities, while improving important operational functions, such as increasing situational awareness and faster decision-making.”

In this initial test/dev deployment, it is anticipated that IBM’s solution will enable NATO to more rapidly explore a range of scenarios for collaboration and data exchange across the Alliance. Broader roll-out of the Cloud solution itself may follow in due course, initially to other NATO sites, but ultimately into the military infrastructure of NATO’s members. IBM’s Herold suggests that, with this project, NATO ACT is taking a leadership role in developing techniques and solutions that will make NATO itself more effective… and eventually trickle down into the internal practices of NATO member states.

Here, as in so many other markets, the relative safety of test&dev is an obvious place to begin exploring the potential – and pitfalls – of a Cloud solution. As more data centres – and more partners – become involved in future deployments, the ugly spectre of security will of course raise its head. But if NATO and its members can (mostly) keep their existing infrastructure secure, a private cloud should be no more risky…

Today’s announcement is relatively modest, and many purists would argue that it’s not even a Cloud. But it’s a tentative first step for a security-conscious organisation made up of (at least) 28 frequently paranoid military hierarchies, and will become far more interesting – and challenging – as it expands beyond the (relatively) safe confines of a single data centre inside a US Naval Base. Definitely one to keep an eye on.

* ok, I made that one up.

Related articles

• NATO, IBM team up on cloud, collaboration project (zdnet.com)

I don’t think I’ve ever really been convinced by the argument that everything will end up in the data centres of Amazon.

The straightforward provision of commodity Cloud Computing is an important – and growing – area, and one that will continue to expand as interfaces become simpler, FUD is challenged, and prices maintain their relentless march towards the bottom. Everyone has something they could usefully, sensibly, and cost-effectively run in a commodity Cloud such as those offered by Amazon, Rackspace, Flexiant, and others. In this space, basic stability, security and reliability combine with a compelling – and diminishing – pricing proposition to create commodity services targeted squarely to lowest common denominator functionality. Here, market forces may (inevitably?) lead to an eventual reduction in the number of providers. Cost, although not the only consideration, is both important and compelling. Although markets like competition, there may even be a single winner here, one day.

Layered all around the basic, routine, grunt-work computation that these commodity public clouds handle so well, many organisations find themselves having to cope with a wide range of other use cases and data sets. Some require specialist hardware (like the GPUs that Amazon has recently begun selling access to). Some demand particular regulatory and legislative hoops to be jumped through. Some have quirky requirements around latency in data transfer or speed of in-CPU processing. Some have lots of data, and issues with regard to getting the stuff from one location to another with a sensible balance between transfer cost and time.

All of these are certainly capable of being addressed in the Cloud, but the economics and the business rationale begin to shift. For the data owner, cost may no longer be quite so significant a factor. Reliability may matter more, or speed, or the audit trail. For the Cloud provider, these requirements no longer look like the lowest common denominator. It’s not cost-effective to provide these capabilities to everyone and still keep the price low. It becomes more sensible to segment, to divide, and to create bespoke offerings of various kinds. Some of these services require such specific things in terms of network topology, physical building layout, and staff expertise that it may even become counter-productive to have these services in the same building as the commodity Cloud. Here, there’s plenty of room for new entrants, plenty of scope for competition, and plenty of opportunity to differentiate in terms of price, location, support, and a host of other factors. This segment of the Cloud is only just getting started.

In these contexts, we see compelling arguments made for on-premise private clouds, off-premise private clouds, hybrid clouds, community clouds and the rest. Some of the arguments made in favour of private and hybrid certainly are part of the FUD we see in this space, but beneath the noise, the security scares, and the vested interests of SysAdmins and sellers of data centre components, there lies a grain of truth. Not everything is most sensibly run on a cheap VM, rented from Amazon (or Rackspace, or whoever) with your credit card, and physically located half way round the planet.

Unfortunately, it can be difficult to make sensible decisions about which type of cloud works best in each situation, and large swathes of the market are doing everything in their power to add to the confusion.

Having accepted that the basic offering from a public cloud provider is not the solution for my particular requirements, where do I turn next?

Do I listen to the (convincing) pitch from a vendor of ‘community cloud’ solutions for my domain? If I’m in Healthcare, they come with HIPAA and European Data Protection Directive, and all sorts of other accreditations. For dealing with sensitive patient data, this may be just what I need… but does the wily salesman also persuade me to run staff email and the hospital volleyball club website on this over-specified (and expensive) infrastructure?

Do I listen to the (convincing) pitch from a vendor of virtualisation software? If I’ve got a reasonably sized data centre with some life left in it, I may see the value of virtualising all of that expensive hardware, and running current applications in house more efficiently. But instead of gradually reducing my in-house costs, do I continue to add more machines as current ones reach end of life, or as new requirements come along?

Do I listen to the (convincing) pitch from my co-location facility, which happily sells me a ‘private cloud’ that may fail to deliver some of the economies of scale so central to the main Cloud proposition?

Do I listen to the horror stories, stick my head in the sand, and simply keep ordering servers until every single one of my competitors undercuts my costs and I go out of business?

These, and more, are certainly possible. But let’s return to that UK project I mentioned right at the start.

Flexible Services for the Support of Research (FleSSR) is

“a new cloud pilot project looking at utilising hybrid private-public IaaS cloud infrastructure to provide computational and data services to the academic research community. The project is a collaboration between the Oxford e-Research Center, IT Service @ University or Reading, e-Science Centre @ STFC, Eduserv, EoverI, Eucalyptus INC and Canonical Ltd.”

The ten month project is funded by the Joint Information Systems Committee (JISC), an organisation that supports the innovative use of IT across UK universities.

Now, to a degree, the project’s mindset must be influenced by its partners. IT staff at Reading and STFC are incumbents with turf to protect (or new vistas to discover, map, and claim). Eduserv has a new data centre that they’d like to fill with willing clients. It would be easy to be cynical, but knowing some of the people involved, I see no real reason to be. It is perfectly reasonable to suggest that a ‘community’ the size of UK Higher Education would realise value in replicating less (not nothing) at every university campus across the country, and bringing much of that together in some sort of Cloud. That Cloud might use public infrastructure, or it might be served up from an organisation such as Eduserv, which is known to the community, aware of the community’s requirements, quirks and foibles, and (importantly) not-for profit (and therefore cheaper?).

Personally, I’d always rather presumed that an organisation like Eduserv (or JISC itself) would act on behalf of the community to procure a competitive price on access to the resources of Amazon, Rackspace, or one of the others. I’m not convinced that most UK research computation needs any sort of special treatment that couldn’t be met from Amazon’s Dublin data centre… unless the community itself can somehow beat – and continue to beat – Amazon on price. Somewhat surprisingly, that’s exactly what some calculations in two posts by Eduserv’s Andy Powell suggest could happen. By including network costs and other charges over and above the basic storage cost, Andy finds Amazon, Rackspace and Dropbox to be more expensive than anticipated, and posits that Eduserv (connected to every UK university free of charge via JISC’s high speed JANET service, and constrained in the ways it can generate profit from services sold to universities by its charitable status) might actually work out cheaper.

There’s a lot of work to do in terms of fleshing out the assumptions behind some of Andy’s figures, but the whole industry certainly benefits when people conduct exercises like these out in the open, for all to see. If Andy has made mistakes, the vendors should be quick to jump in and correct them. If his assumptions miss the mark, public debate can redress the balance.

The Cloud is not all about price. But more transparency around the true cost of computing in the Cloud – and in your data centre – means that we can all make more informed decisions.

Thanks for sharing, Andy – and hopefully readers will be willing and able to look over your calculations and share their own views.

Note: this post was conceived and written in the United Kingdom. By reading this post you agree to comply with UK usage, and will henceforth pronounce the word ‘niche’ from the title as ‘neesh,’ not ‘nitch.’ Or maybe not.

Related articles

• Rackspace launches managed cloud services (zdnet.com)
• Are hybrid clouds the path to cloud-computing nirvana? (venturebeat.com)
• We’ll Take Care of Your Cloud, While You Manage Your Business (rackspacecloud.com)
• Trust Is Key For Cloud Success And What Can We Do About It? (cloudave.com)