Image by Kevin Krejci via Flickr

The CloudCamp unconference returned to London for the 14th time this evening, regaling a capacity crowd in the Crypt below Clerkenwell’s St James Church with several hours of discussion and debate on the somewhat elusive topic of ‘Big Data’.

Rather rough notes of the proceedings follow, after the break.

LEF‘s Simon Wardley kicked proceedings off as usual, once again managing to pepper an on-topic canter through the topic with a seemingly never-ending stream of Flickr images of cats… and analogies to electricity. You possibly had to be there? His core message, though? There’s nothing new under the sun… and the cycles of change just keep on coming.

Next, Peter Matthews from CA Labs, on “is big data mutually compatible with the cloud?” Erm, yes. Data volumes with big data are so large that it’s difficult to move it around… which creates opportunities for lock-in that vendors may wish to seize. And then he was out of time.

Next, Fujitsu’s Mark Wilson on ‘Structuring Big Data.’ He’s actually talking about Linked Data, a topic I’ve dug into before here and over on semanticweb.com – Linked Data could be/ might be the effective realisation of the decade-old Semantic Web dream. Big Data means masses of unstructured or semi-structured content, presenting a management headache of previously unanticipated proportions. Linked Data, he argues, creates the mechanism to link all of this data together from across disparate sources. Yes, but it’s easier to say than to do… And in 5 minutes he really couldn’t explain enough to persuade the audience. Linked Data should be “the optimal reference source,” he said. It should be “a broker for all data sources,” and we should “think about integration, not duplication.” Yeeeeees… But.

Next, Canonical’s Nick Barcet, talking around scalability, Ubuntu, package management, configuration management, etc. Not wholly sure what the point was, I’m afraid.

Next, Chris Swan from UBS – big data and security. “If you’ve got security controls that aren’t properly monitored, then they don’t matter.”

Next, Tom Leyden of Amplidata – Big “Unstructured” Data in the Cloud. Data storage to increase 30x over the next decade, but staff will only increase 50% over the same period. Challenge in the 90s, as existing storage and analysis technologies struggled to cope with new data volumes. Seeing similar problems today with data streaming from sensor web, etc. Traditional file systems cannot cope. Object Storage the way forward ?

Next, Alex Farquhar – “Cloud v Big Data.” Not really versus… but intersection of the two. Too much discussion of his company, Forward. Just talking about how his company uses cloud to provision IT resources. Might work as a conference presentation or case study – not sure it fits as a 5 minute lightning chat. Around 60TB of data at Forward. Diverse and vital. Using Hadoop cluster – 24 nodes on-premise. Rationale (proximity to the cluster) seemed odd. That can be true, but not clear that it really needs to be the case here?

Next, Alaric Snell-Pym, on Scaling Hadoop. Trying to overcome Hadoop’s I/O bottleneck. Explaining basics of Hadoop and Map/Reduce – no one else has. Explains use of HDFS and ‘selective reading’ to manage lots of small tables and overcome the problems of I/O.

Next, Matt Wood from Amazon. Talking about genetics and the human genome. It’s an analogy. Human Genome Project took years and millions of dollars. Development of gene sequencing machines led to a step change – dramatic drop in cost of sequencing DNA. Like the cloud, anyone? But… the machines create an analysis challenge, because they generate so much data. Cloud offers “collection of productivity tools” to help scientists work with this data collaboratively and (relatively) affordably. A perfect example of a lightning presentation, unlike most of those who preceded him.

And finally, an impromptu slot from HP’s Joe Weinman. A quick overview of current thinking behind his latest book. This one could have gone for much longer… Good stuff.

And that’s the lightning talks finished. Now, the panel, and Simon Wardley’s search for “experts” and “volunteers.”

…and unfortunately, your scribe was ‘volunteered’ as an ‘expert’ by Mr Wardley… and here end the notes. It was great to have Amazon’s Werner Vogels sneak in, and lob comments into the panel, though…

Great event, though with the usual mix of people you wish could have talked for longer… and people you wish wouldn’t have spoken.

Related articles

• The brave new world of big data & Hadoop (venturebeat.com)
• Big VCs Invest In Big Data Startup Continuuity (techcrunch.com)

One problem with living somewhere like East Yorkshire (a problem for which there are certainly compensations…) is that it proves effectively impossible to engage with my chosen industry’s current penchant for short evening events (except one) without the added complication of such things as overnight stays in hotels.

Unless these events coincide with other things during the day, it’s not necessarily easy to justify time, travel, and accommodation costs for a couple of hours’ (often) unstructured event.

That said, it does tend to be worthwhile when things come together, and last night’s CloudCamp in Newcastle upon Tyne (not to be confused with last night’s CloudCamp across the water in Boston) was an interesting example of the genre.

More intimate than it’s established cousin away to the south in London, CloudCamp North East makes use of facilities at the University of Newcastle (a previous employer of mine, long ago…) to provide a gathering place for an audience that felt predominantly local.

Ably compered by Phil Huber of symetriQ, the evening kicked off with a series of short presentations.

Amazon Web Services Evangelist, Simone Brunozzi, went first and shared some of his views on security in the Cloud. He suggested that psychology plays a significant part in enterprise concerns; it’s not a real belief that the Cloud is less secure, so much as a fear of loss of control. Simone made the distinction between Amazon-controlled physical security of data centres and real computers, and largely customer-controlled responsibility for running virtual machines responsibly. He noted that Amazon’s virtual machine instances are created by default with every network port closed; a new customer needs to request that a port is opened before they can even log in to their new instance.

How much responsibility – if any – do Infrastructure providers such as Amazon have to ensure that customers don’t subsequently do stupid things? The design of the data centre will tend to mean that attacks on one virtual machine carelessly left vulnerable by its tenant have absolutely no impact on Amazon or its other customers… but in the eyes of the media et al it will still be “Amazon’s Cloud” that failed when some high profile startup is hacked through a port that it (not Amazon) left open.

Next, Flexiscale‘s Gihan Munasinghe suggested that the Cloud is ‘virtualisation at it’s best,’ whilst stressing that effective storage strategies and network management are as important as access to virtual machines.

Sun‘s Stewart Townsend side-stepped jokes about Oracle to repeatedly stress the simplicity of the Cloud, both conceptually and technologically. People, he suggested, were making it hard.

“People get in the way of Change.”

Microsoft‘s Matt Deacon picked out highlights from a 2008 Freeform Dynamics report sponsored by the company, and concluded by suggesting that evidence points to a shrinking internal IT function in many enterprises at a time when the scope of IT is growing ever greater. Uncomfortable as it may be to hear, he suggested, the vast majority of IT functions are perfectly suited to cost-effective outsourcing… so long as the enterprise maintains oversight for the IT architecture in-house.

Arjuna‘s Steve Caughey asked, ‘Where will Cloud Computing take us?,’ and pointed to well-understood trends such as Moore’s Law. More important, he suggested, is Gilder’s Law and its observation that network bandwidth doubles every nine months. Geography, Caughey suggested, is becoming increasingly irrelevant. Economies of scale might suggest an eventual movement of all data and processing to a single, massive, data centre, although latency (the speed of light isn’t getting any faster) and jurisdictional quirks such as the US PATRIOT Act mean that a moving all data to a single location remains impractical. Nevertheless, Caughey suggests that it is perfectly reasonable to expect customers to deal with a single conceptual Cloud that masks the complexity of load balancing, cacheing, replication and the various tricks required to keep data where it’s needed. The Cloud, he implied, should be capable of handling jurisdictional issues (European personal data should never pass through a server physically located in the United States, etc), just as it should be capable of automatically moving frequently changing data closer to application servers and cacheing or otherwise managing the background mass of data that changes far more slowly. We still have some way to go in achieving this vision, but it is certainly one that is compelling.

The final presentation before beer and pizza was from Ross Cooney of Rozmic. Ross spoke about the lessons his team learned in building emailcloud, and praised the flexibility and affordability of Amazon’s Web Services during the product’s development phase. He suggested that the pay as you go model of renting computing and storage at the point of need was a fundamental aspect of the Cloud, and far more important than whatever technical innovations there might have been. With just a credit card, a small team of developers can now compete against both established companies and startups in receipt of significant Venture funding. Ross closed by noting that, as emailcloud moves from the bootstrap phase into production deployment, it is proving more cost effective to transition the majority of computing to more traditional data centres. By his calculations, the optimal split was to have 75% of computing owned and running in a managed data centre, leaving 25% at Amazon to cope with peaks in demand. An interesting model, and one I’d like to understand in more detail.

All of the speakers did a great job of respecting CloudCamp’s ethos, and avoided unnecessary mention of their companies, products, and solutions. One thing I did notice (and this is certainly not intended as a criticism of anyone concerned) was the way in which the different corporate perspectives shone through each presentation. There certainly wasn’t selling, marketing, or anything like that. What there was, I think, was an illustration of the different corporate attitudes to the Cloud and the opportunity that it represents. I’d actually be interested in finding a way to exaggerate those corporate attributes. Sun, Microsoft and Amazon, for example, have very different views as to where the Cloud is headed, and there’s value in understanding each of those equally valid perspectives, rather than pretending they’re not there.

After a break for pizza, the vast majority of attendees reconvened for a panel discussion that I failed to record in as much detail. One point from a panellist (I can’t be sure whom, I’m afraid) that resonated with some of my other interests was the assertion that traditional software approaches such as the Relational Database do not scale effectively to the Cloud. To maximise the value offered by Cloud Computing, he argued, we need to be looking at different ways to manage and manipulate data, such as tuple stores and semantic databases. These reasonably well understood academic exercises, he argued, suddenly make far more sense in the context of the Cloud.

Well, I could hardly disagree, now could I? 

And finally thanks, as always, to the sponsors (arjuna, emailcloud, flexiscale, Microsoft, Sun and symetriQ) who paid for beer, pizza and more… whilst behaving themselves and not ramming corporate messaging into our unwilling ears.

I’m heading down to London tomorrow, for the city’s third CloudCamp. If you’re in the area Thursday afternoon or Friday morning, do say Hi.

Related articles by Zemanta

• London CloudCamp #3 (Mar 12, 2009) (elasticvapor.com)
• Upcoming 2009 CloudCamps (elasticvapor.com)