County Arcade, Leeds

The global CloudCamp movement continues to grow, with events over the next few weeks in Denmark, Germany, Nigeria, Ghana, Kenya, and across the United States. And now, I’m very pleased to announce that the English city of Leeds is joining the party.

CloudCamp events have been taking place in the UK for years, and the London gatherings have picked up real momentum. Outside London, we’ve seen a few events in Warrington, Newcastle, and Edinburgh. We believe that the time is now right for something more regular; a place in which the cloud-building, cloud-using, cloud-interested and cloud-exploring can come together for talk, beer, pizza and more… without having to jump on a train to the deep south.

CloudCamps are interesting events, with a real emphasis on informality. I’ve attended several around the world, and am always impressed by the energy in the room, and by the welcome extended to newcomers. As the main CloudCamp site describes,

“CloudCamp is an unconference where early adopters of Cloud Computing technologies exchange ideas. With the rapid change occurring in the industry, we need a place we can meet to share our experiences, challenges and solutions. At CloudCamp, you are encouraged you to share your thoughts in several open discussions, as we strive for the advancement of Cloud Computing. End users, IT professionals and vendors are all encouraged to participate.”

Jeremy Jarvis at Brightbox and Karyn Fleeting and Joel Turner at Tinderbox Media have been driving this event forward, and they’ve invited me on board to help out. I also get to be MC on the night.

We’ve got speakers and sponsors committed, with more of both to come. If you think you should be one of those doing the speaking or the sponsoring, do let us know.

So, if you like talking Cloud, if your boss has ordered you to learn Cloud, or if you’re just keen to understand a little more about what this Cloud thing can do for you, stick the evening of 14 June in your diary, sign up (for free) on Eventbrite, and come along to the Hilton DoubleTree in Leeds for an evening of fun, learning, beer, and more.

See you there!

Image of the County Arcade in Leeds by Flickr user Francisco Perez

Image: NASA

Amazon’s globe-encircling cloud infrastructure is compelling to many. From Virginia to California, from Ireland to Singapore, and from Japan to Brazil; wherever you find yourself there’s a local instance of the same familiar set of services. And, in all likelihood, Australia will soon be added to the list. For those primarily interested in just serving both Europe and the US, the list of options grows to include Rackspace, GoGrid, CloudSigma and a few others. And yet, despite the buying power and increasing ubiquity of these larger players, there seems to be plenty of space left for smaller entrants. For prospective customers only concerned with a single country or region, for example, the choices are almost too many to count, and choosing between them becomes a complex and multi-faceted affair.

Take Brightbox, for example. As my family know all too well, those pesky timezones mean that many of my evenings are punctuated with calls to or from the States, where so much of the innovation in this sector continues to take root and grow. Either that, or I’m creeping out of a sleeping house to catch early trains for the 150 mile journey south to London. It was therefore refreshing to talk to someone in this industry whose offices are only 50 miles away in the UK city of Leeds.

Established back in 2005 as a Ruby shop capable of hosting apps on dedicated hardware, Brightbox has evolved to place increasing emphasis upon the provision of infrastructure. In 2010, the company began to seriously explore the possibility of offering a generic cloud infrastructure environment. This was in the days before OpenStack, but Eucalyptus existed and was attracting interest. But according to Brightbox co-founder Jeremy Jarvis, 2010′s Eucalyptus lacked some key resilience features (load balancing, multi-data centre capabilities, etc) that the team believed were critical… so they built their own system from the ground up. And, at the end of September last year, Brightbox Cloud entered general availability.

The Brightbox cloud operates out of two UK data centres, with planning underway for a third. Both data centres are now owned and operated by Telecity, which acquired the two independent data centre providers with whom Brightbox had launched. Brightbox owns the racks and (Dell) hardware, and also ensures provision of redundant network access into the data centres. Customers are predominantly drawn from across Europe, but Jarvis says he’s seeing some customers coming from as far away as Australia, New Zealand, and Brazil. The company sees its potential growth through eventual expansion to data centres on the European mainland, but Jarvis says he’s “much less interested in setting up yet another” North American operation. The company is profitable, employs ten staff, and is seeing steady growth in usage.

Brightbox does not (yet) offer a web management console, but Jarvis describes this as a conscious decision and also something of an asset. The company has instead focused their attention upon crafting a rich, capable and intuitive API (and associated command-line interface). According to Jarvis, the developers that the company tends to target have responded favourably to the API, describing it as “nice” and “more consistent” than the various APIs offered by Amazon’s growing suite of services.

A focus upon developers (and the growing Dev/Ops movement) was also a strategic decision, and Jarvis cites examples in which ‘mere developers’ have proved instrumental in securing significant contracts with Brightbox from their employers. Corporate purchasing processes may, finally, be evolving. Despite the success of Platform as a Service (PaaS) offerings such as Heroku and EngineYard, Jarvis also suggests that Brightbox is seeing growing evidence that developers are seeking more fine-grained control over infrastructure than PaaS typically offers. A platform abstracts the underlying complexity of infrastructure, making it easier for application builders to focus upon creating the specific services they wish to provide. But abstractions typically require compromises, with configuration decisions being made for everyone on the platform on the basis of ‘normal’ requirements. For developers with non-normal requirements (and they may actually be the majority of users), IaaS is more likely to offer the fine-grained control that they need.

But the cloud infrastructure world has come a long way since Brightbox began planning their product two years ago. OpenStack has arrived, and (despite a growing body of nay-sayers) is credible. Rackspace, HP, and others are on the cusp of delivering real clouds to real customers on the back of its codeline. Eucalyptus appears to have turned a corner, and pulled back from a brink that I (and others) saw them teetering on the edge of. Canonical’s marriage of Ubuntu to OpenStack is now just one of several ways to get the same cloud code, capabilities and apis onto machines running inside your own data centre. Amazon just keeps on doing what Amazon does, incrementally adding features, cutting prices, and becoming ever-harder to not choose.

In that world, surely a little cloud provider operating their own bespoke solution from the wrong end of the Leeds-London railway line, in a country on the wrong side of both the English Channel and the Atlantic Ocean has no hope? Surely they should just pack up shop, and either adopt OpenStack/Eucalyptus fast… or find a new line of work?

Jeremy Jarvis disagrees, vehemently. And he’s not alone. Look at Edinburgh’s Flexiant, Glasgow’s SymetriQ, and a whole host of other companies that have built their own solutions from nothing. Others, of course, have recognised the value in taking OpenStack, Eucalyptus, Microsoft’s Windows Azure, and similarly established technologies, and making them their own. Look at Eduserv’s Swindon data centre, or the hosted desktops from East Yorkshire’s own GoCloud.

So how can Brightbox (or Flexiant, or SymetriQ, or any of the other non-conformers) compete? How, indeed, can they survive? Jarvis suggests that “Buy British” continues to carry weight here. Even without raising the scary (and often, actually, wholly irrelevant) spectre of PATRIOT Act-powered snooping, a significant proportion of UK (or European) companies like the idea of buying services from UK (or European) suppliers. They like that the documentation is spelled correctly. They like that telephone support is (more or less) in their timezone. They like that the development team shows up at local events, and that it’s a person buying the drinks in the bar, rather than the disembodied marketing budget of some far-off corporation.

Purchasing decisions for something like cloud infrastructure are complicated. Often, they’re probably quite illogical. Price isn’t always the deciding factor (and even if it were, smaller providers like Brightbox aren’t ridiculously expensive in comparison to their larger competitors). Having been the friendly face at a local developer event might swing that big contract. Or having a “nice” API. Or implementing niche features in firewalls, or networking, or port forwarding might each grab the attention — and loyalty — of specific sectors of the long tail. Tesco might sell ‘everything’ to ‘everyone,’ but we still have room in our lives for the SPAR corner shop, and for the upscale deli with the nice cakes.

The same’s true in the cloud, although I can’t help feeling that we’re going to see quite a rapid decline in entirely new cloud infrastructures as the next generation of niche cloud boutiques take OpenStack or Eucalyptus and mould them to their requirements. They probably won’t be making those decisions because (like we Clouderati) they agonise endlessly about interoperability or portability or the de facto standard of the Amazon Web Services stack. For most of their SME customers, those things simply don’t matter. Instead, they’ll be adopting OpenStack or Eucalyptus because the grunt work (and the marketing) has been done. It simply costs less to take something off the shelf than to develop it yourself from scratch. But for companies like Brightbox, where that investment has already been made? Well, for them there may still be plenty of prospective customers out there.

Related articles

• Eucalyptus grabs $30M from IVP to push the open-source private cloud forward (venturebeat.com)
• Announcing Brightbox Cloud – the UK’s first true IaaS platform! (brightbox.co.uk)
• Platform as a service ready to rev up, says Gartner (zdnet.com)
• Hosting providers aim to challenge cloud leaders (zdnet.com)
• Joyent Brings Its Public Cloud To Europe (techweekeurope.co.uk)
• Rackspace launches OpenStack-powered next-gen public cloud (venturebeat.com)
• Ubuntu Cloud Portal: Brightbox 12.04 daily images now available, discounts for testers and Ubuntu members (cloud.ubuntu.com)
• Ubuntu Linux heads to the clouds (zdnet.com)

Image via Wikipedia

Most of us recognise that the Earth is warming and that — despite our planet’s temperatures having dramatically risen and fallen before — we humans must accept some measure of responsibility for the current changes.

Already consuming at least 1.1-1.5% of global power, and only forecast to grow ever-more rapacious, the data centres that power our information economy are surely one area in which we can, should, and must find ways to reduce consumption. And, although by no means perfect, data centre builders, operators and suppliers are paying attention. Individual servers are becoming more efficient. Buildings are being cooled by the wind and by toilets. Waste heat is warming homes. Newer facilities are being sited to save the planet (and money) by drawing power from hydro-electric schemes rather than coal, oil, gas, or the once-again-unpopular-and-scary Nuclear.

And then there’s solar. On the face of it, the ultimate renewable. The sun shines. It does so every day. It will continue to do so every day for millions of years to come. Every watt diverted to a data centre has absolutely no effect whatsoever upon the sunlight that the rest of us receive today, or that we shall receive tomorrow. There’s certainly a location trade-off to be made, as the sunshine rarely reaches the ground for much of the day in locations where the air is cool and damp enough for all that fresh air cooling to work, but there should certainly be a case to be made for both. If you’re in Iceland or Ireland or Oregon, sunlight is not something to rely upon for meeting your ravenous appetite for energy. But if you’re in sunnier climes, surely it’s a no-brainer?

Apple, for one, is constructing a 171 acre (69 hectare) solar farm next to its North Carolina data centre. That’s big. It’s apparently the biggest commercial deployment in the United States and — with some help from WolframAlpha — it’s also equivalent to 1.6 Vatican Cities or 13 Great Pyramids.

More tellingly, it’s also a lot of fields and trees.  If all those mirrors reduce the impact of Apple’s data centre on the environment, then maybe — just maybe — the loss of forest and farmland and open space is a cost worth paying. But Amazon’s James Hamilton does some maths, which he shares in a recent blog post, and he reckons that all the mirrors in that 69 hectares contribute around 3.2 MW to the data centre’s power requirements; requirements that Hamilton calculates to be around 78 MW. That’s 4%, which really is not that much at all. It’s worth noting that the figures for the solar array’s power output and the data centre’s consumption are only Hamilton’s estimates, but he tends to know what’s he’s talking about. In the absence of solid numbers from Apple, you can do a lot worse than believe the maths of someone like Hamilton.

So… why the enthusiasm for solar? Are data centre operators taking reductions where they can get them, and calculating that a 4% contribution to the fuel bill is worth 69 hectares? They might be right. Maybe. Or are they being more callous and calculating than that? Are the ‘savings’ on energy bills and greenhouse emissions essentially negligible, and are trees and fields being turned over to big mirrors simply so that companies can look good? They get a tick in the “green” box. They get a mention from Greenpeace. Customers, consumers and law makers think they care. Whilst actually, they don’t care at all.

Related articles

• The Switch SuperNAP data centre – one of the most impressive I’ve been in (greenmonk.net)
• Apple slaps mega-solar panel field on new ENORMO data centre (go.theregister.com)
• In cooler climates, outdoor air systems can save energy dollars for data centres, retailers, warehouses (cleanbreak.ca)
• Colt Telecom and Verne Global’s dual renewably sourced data centre in Iceland (greenmonk.net)
• Facebook Data Centre Uses As Much Power As The Entire County It’s In (gizmodo.com.au)
• Greenpeace, Facebook in truce over energy (news.smh.com.au)
• Solar power gets public vote (evoenergy.co.uk)
• The revenge of Thomas Edison (business.financialpost.com)
• California’s Military Bases Could Generate 30x More Electricity From Solar Power Than They Consume (treehugger.com)
• Apple’s iCloud Has Solar Lining (wired.com)

Piyush Lumba, Director of Product Management for Azure Data Services at Microsoft, talks about what the Marketplace can do today and shares some of his perspectives on ways that the nascent data market space could evolve.

Following up on a blog post that I wrote at the start of 2012, this is the eighth in an ongoing series of podcasts with key stakeholders in the emerging category of Data Markets.

Related articles

• Data Market Chat: the podcasts are a-coming…(cloudofdata.com)
• Data Market Chat: Tyler Bell discusses Factual (cloudofdata.com)
• Data Market Chat: Hjálmar Gíslason discusses DataMarket.com (cloudofdata.com)
• Data Market Chat: Chris Hathaway discusses AggData (cloudofdata.com)
• Data Market Chat: Flip Kromer discusses Infochimps (cloudofdata.com)
• Data Market Chat: Nick Edouard discusses BuzzData (cloudofdata.com)
• Data Market Chat: Stephen O’Grady of RedMonk examines the bigger picture (cloudofdata.com)
• Data Market Chat: Shion Deysarkar discusses Datafiniti (cloudofdata.com)
• Big data in the cloud (radar.oreilly.com)
• ‘Roswell’: Another key component of Microsoft’s cloud strategy (zdnet.com)

Image by Kevin Krejci via Flickr

The CloudCamp unconference returned to London for the 14th time this evening, regaling a capacity crowd in the Crypt below Clerkenwell’s St James Church with several hours of discussion and debate on the somewhat elusive topic of ‘Big Data’.

Rather rough notes of the proceedings follow, after the break.

LEF‘s Simon Wardley kicked proceedings off as usual, once again managing to pepper an on-topic canter through the topic with a seemingly never-ending stream of Flickr images of cats… and analogies to electricity. You possibly had to be there? His core message, though? There’s nothing new under the sun… and the cycles of change just keep on coming.

Next, Peter Matthews from CA Labs, on “is big data mutually compatible with the cloud?” Erm, yes. Data volumes with big data are so large that it’s difficult to move it around… which creates opportunities for lock-in that vendors may wish to seize. And then he was out of time.

Next, Fujitsu’s Mark Wilson on ‘Structuring Big Data.’ He’s actually talking about Linked Data, a topic I’ve dug into before here and over on semanticweb.com – Linked Data could be/ might be the effective realisation of the decade-old Semantic Web dream. Big Data means masses of unstructured or semi-structured content, presenting a management headache of previously unanticipated proportions. Linked Data, he argues, creates the mechanism to link all of this data together from across disparate sources. Yes, but it’s easier to say than to do… And in 5 minutes he really couldn’t explain enough to persuade the audience. Linked Data should be “the optimal reference source,” he said. It should be “a broker for all data sources,” and we should “think about integration, not duplication.” Yeeeeees… But.

Next, Canonical’s Nick Barcet, talking around scalability, Ubuntu, package management, configuration management, etc. Not wholly sure what the point was, I’m afraid.

Next, Chris Swan from UBS – big data and security. “If you’ve got security controls that aren’t properly monitored, then they don’t matter.”

Next, Tom Leyden of Amplidata – Big “Unstructured” Data in the Cloud. Data storage to increase 30x over the next decade, but staff will only increase 50% over the same period. Challenge in the 90s, as existing storage and analysis technologies struggled to cope with new data volumes. Seeing similar problems today with data streaming from sensor web, etc. Traditional file systems cannot cope. Object Storage the way forward ?

Next, Alex Farquhar – “Cloud v Big Data.” Not really versus… but intersection of the two. Too much discussion of his company, Forward. Just talking about how his company uses cloud to provision IT resources. Might work as a conference presentation or case study – not sure it fits as a 5 minute lightning chat. Around 60TB of data at Forward. Diverse and vital. Using Hadoop cluster – 24 nodes on-premise. Rationale (proximity to the cluster) seemed odd. That can be true, but not clear that it really needs to be the case here?

Next, Alaric Snell-Pym, on Scaling Hadoop. Trying to overcome Hadoop’s I/O bottleneck. Explaining basics of Hadoop and Map/Reduce – no one else has. Explains use of HDFS and ‘selective reading’ to manage lots of small tables and overcome the problems of I/O.

Next, Matt Wood from Amazon. Talking about genetics and the human genome. It’s an analogy. Human Genome Project took years and millions of dollars. Development of gene sequencing machines led to a step change – dramatic drop in cost of sequencing DNA. Like the cloud, anyone? But… the machines create an analysis challenge, because they generate so much data. Cloud offers “collection of productivity tools” to help scientists work with this data collaboratively and (relatively) affordably. A perfect example of a lightning presentation, unlike most of those who preceded him.

And finally, an impromptu slot from HP’s Joe Weinman. A quick overview of current thinking behind his latest book. This one could have gone for much longer… Good stuff.

And that’s the lightning talks finished. Now, the panel, and Simon Wardley’s search for “experts” and “volunteers.”

…and unfortunately, your scribe was ‘volunteered’ as an ‘expert’ by Mr Wardley… and here end the notes. It was great to have Amazon’s Werner Vogels sneak in, and lob comments into the panel, though…

Great event, though with the usual mix of people you wish could have talked for longer… and people you wish wouldn’t have spoken.

Related articles

• The brave new world of big data & Hadoop (venturebeat.com)
• Big VCs Invest In Big Data Startup Continuuity (techcrunch.com)

Image via Wikipedia

For real-world applications of Linked Data and the Semantic Web, the long-running Semantic Technology Conference is hard to beat. For getting a real handle on the Cloud Computing landscape, GigaOM‘s Structure Conference is also a leading light.

Working across both areas as I do, these events tend to figure prominently in my calendar for the year*. Last year, both took place in San Francisco during the same week. I tried to attend both, and therefore succeeded in spending most of my week in cabs, shuttling between meetings at the two venues. I saw very few sessions that I wasn’t personally involved in, and the experience wasn’t a huge success.

This year the conference organisers have taken pity on me, and moved their events to opposite ends of June. The Semantic Technology Conference is up first, back at the Hilton Union Square from 5-9 June. Structure follows, returning to the Mission Bay Conference Centre on 22 and 23 June.

I’ll be attending both, and probably doing various official things during each event. At the moment, the only thing we’ve definitely nailed down is a special live appearance by The Semantic Link crew on the evening of 5 June; we’ll be taking a look at the highlights expected for the conference, offering some tips for those new to the event and its multitude of parallel sessions, and generally bringing our usual podcast chatter to the stage.

So if you’re in town around the time of either event, do get in touch; it promises to be an interesting month.

* Disclosure: WebMediaBrands pay me to host the monthly Semantic Link podcast, and to write a monthly column on SemanticWeb.com. GigaOM pay me to curate the Infrastructure/ Cloud Computing channel on their Pro site. I attended and participated in both of these events before that was the case, and still would today without the contractual relationship.

Related articles

• GigaOM Structure 2011 (datacenterknowledge.com)
• Semantic Technology Conference (fakeiitian.com)

• My latest monthly column on SemanticWeb.com; Big Data Presents a Big Opportunity?
• My latest weekly wrap-up on GigaOMPro; Rosslyn Analytics, Microsoft Finding Value in Data Aggregation
• The teaser piece on GigaOM’s public Cloud site; In Exploiting the Data Market, Trust Is Key

I spot a theme building…

Simon Rogers, The Guardian Data Store

What we do with data. Guardian started in Manchester, first issue 4 pages. That first issue included a table of data about schools in Manchester; number of pupils, cost, etc.

Guardian now… aggregates data published by Government departments in PDF, extract data values and present in useful manner.

guardian.co.uk/data – Data Store. Make data available from all over the place, do some stuff with it, but invite others to engage, enhance, correct, use.

“Every time we publish government data, we get 2-3 calls from Gov departments, asking if they can have a copy.”

Making data available to end users means they follow their interests… sometimes results in stories for the news.

Wikileaks… “without Guardian data journalism, we wouldn’t have got the stories from the data.”

Journalists had the data… but didn’t know what to do with it in order to find and tell stories. Needed to provide tools for journalists so that they could navigate the data.

Wikileaks first story – IED cases in Afghanistan; flat visualisation for the paper, plus an interactive tool on the site.

Google Fusion Tables – “an absolutely brilliant tool for mapping lots of data very quickly. Thank you, Google.”

Data exploration tools prove more popular than canned visualisations. People want to explore and understand the data.

Embassy Cables – latest release. As with the others, the paper chooses not to release some of the data. “4 million people can see these ‘secret’ cables.”

The power of stories; didn’t kill journalism. Instead it enhanced it. Provide good journalists with the tools to enrich their stories.

Guardian journalist James Cameron; “the only questions left will be answered by computers because only computers will know the questions to ask.”

Next, a panel… Amber Case, Brad Cross (formerly Flightcaster), Toby Siegler (Metaweb/ Google)

“How dependent are we going to get on data to tell us how to lead our lives?”

Amber – “Very. It will become not only convenient, but also the thing to do.”

Toby – “We’re already dependent on data… we just need to get better at deciding what to ignore.”

“Will we stop asking questions, and wait for the computer to tell us what to do? Will we reach The Shallows?”

Brad – “not clear that people distracted by twitter etc were the deep thinkers before…”

Amber – “large proportion of the population may just consume the channels they consumed before…”

Brad – “right now we only really have search… over time we’ll have more systems that do more for us… but they’re not too far away.”

“We put knowledge into a data store… sales figures, etc. With Big Data we don’t necessarily know what we know or want to know… so should we just keep everything?”

Amber – yes.

Toby – “what people say about themselves is a lot less useful than what they do. If you want to predict what people would do, you should collect the data exhaust and look at what they have done.”

Ethically… if we’ve captured all of our data… can we then subpoena data to show where people were, what they were doing, etc. How will law catch up to an expectation that data is available for all of our actions?

Brad – “not new. The problem will get worse, but it’s just something we have to deal with.”

How will this change government? Representative democracy is a hack… we can’t afford to send everyone to Washington/Ottawa. Can digital take us to a real democracy?

not sure they’re sure…

Amber – recording more data to greater resolution is a good thing and will deliver benefits… so long as it’s not always personally identifiable.

What’s the next technology to unlock big data…?

Amber – “not brain implants… Better location-based data, that knows where you’re going and pushes relevant data to you in real-time.”

Brad – “a mix of what’s happening on the internet now, and on mobile now. When you connect information about you on the server with information about you on your mobile device, there’s some really neat stuff that happens there… an agent.” Sounds like Siri?

Toby – “tried an experiment… connected phone records, sms etc to my social network… it was really cool… Integration of your personal data in this way, just for you to look at, is really useful.”

Next, Ed Boyajian from EnterpriseDB. Legacy Databases and the Data Deluge. Sell commercial version of free Postgres database. Enterprise customers coming to them, asking to solve problems with big data.

“Compare growth of IT budget to growth of data… and business demands to use that data more effectively… Gap between growth of data (and demands on it) and IT budget creates real problems.”

“By moving to open source database solutions, you can save money.”

“Mission for our company is to bring low cost high performance solutions to the market.”

Barry Devlin. 25 years ago this month, the first paper describing a data warehouse architecture was published.

Next, DJ Patil from LinkedIn. Analytics, and building teams around data/ analytics.

‘To connect the world’s professionals and help make them more productive and successful’… requires data. Insights on activity today, plus insights on possible career directions.

Showing graph of data science from LinkedIn data… demand is taking off. See the job board here, too.

529 of 790 LinkedIn-using attendees are connected to 1 other.

30 to 20 or more

85 to 10 or more

189 to 5 or more.

Is that good? Is it what we’d expect?

“Data scientists are frustrated by an inability to ship product.” “LinkedIn tried something different – made data science a top-level product team.”

good talk… but then I needed to slip out to a meeting…

The notes that follow are a rather raw summary of some of the things I’m hearing. Later in the week, I’ll try to come back and extract the main issues in a rather more polished form.

Up first, Mike Driscoll of Metamarkets, talking about Mining the Tar Sands of Big Data.

Trying to frame conversation; why are we all here? Why is now the time that we’re all beginning to focus on ‘Big Data’ ? Drawing analogy with the tar sands in Alberta; lots of oil there, but it’s been expensive to extract.

If ‘information is the oil of the 21st century [as Gartner has suggested], then Big Data are the tar sands;’ lots of data, sitting in our data centres, waiting for us to invest in extracting usable data. Expensive, painful, but ultimately valuable.

‘Attack of the exponentials;’ cost of storage, bandwidth and compute falling exponentially. Number of nodes on the network rising exponentially. Intersection creates ‘data singularity.’

Unlike oil, data abundant and renewable. Like oil, extraction of data creates value. Cheaper and easier to extract value from data than ever before.

Three forces reshaping data landscape; sensor networks, cloud computing, and machine learning.

Sensor networks; now prevalent, all-pervasive, ubiquitous, and typically connected. Generating vast amounts of data.

Cloud computing; ‘used to mean everything and nothing.’ But real advantage is that it turns CapEx into OpEx.

Machine learning; gives us the capabilities to process the flood of data, intelligently. Smart Planet, Smart Grid, Smart Business, etc. Driverless cars, spam filters, recommendation engines all drawing upon Machine Learning ideas. All iterating and improving with increasing rapidity. Unlike Cloud and sensors (which are becoming commodities), machine learning algorithms are – and may remain – a competitive advantage.

Four consequences of all this; battle for finite number of good data scientists, changes in the way that data is published (and valued), the end of privacy (?), the rise of data startups.

Battle for data scientists; difficult to hire people who can munge, interpret, and tell stories with data. And everyone last night at bigdatacamp was ‘hiring’.

Retailers, banks, online publishers, etc have tended to hand over the keys of data management to third parties. Seeing pendulum shift the other way, as companies recognise the value of their data and seek to control it – and realise the value. Tension with ‘open data,’ data to the cloud, etc?

Privacy – not about shifting access to data, but about more accurately defining the ways in which it may be used.

Q&A

‘What sort of educational background does a data scientist need?’

knowing some stats helps. knowing some programming helps. But curiosity is key. Not sure there’s a degree out there. Pick up the skills if you have the right mindset. ‘You have to be a bit of a hacker,’ says Mike.

‘What are the three big problems that data science will solve?’

Making sense of the world around you; Freakonomics, for example. Taking data and making sense of how the world is working. Scaling up decision making, so that a data-powered story can be presented in a way that lets people make intelligent decisions.

Next, Barry Devlin from 9sight Consulting talks about The Data-Driven Business and Other Lessons from History.

‘the old guy who has been brought along to talk about history,’ and ’illegitimate grandfather of data warehousing’

Address Past, Present, and Future.

Past – the origins of Data Warehousing

Data Warehouse architecture work at IBM in Europe in the mid-80s.

‘Big Data’ (a couple of hundred MB, at the time) created need to structure the Enterprise Data Warehouse in a particular way. Led to silos. ‘Hard information’ only, at the time. Warehouse designed in a well-architected fashion. Ensures that data flows in a single direction. Possibly too regimented for the 21st century?

Lessons?

Information quality and reliability are key; Master Data Management, etc. This is unlikely to change.

Data volumes and variety have presented big challenges over the years. Expectations and business demands outstrip technological capabilities. Organisational and political issues hamper progress. This is unlikely to change.

Exploration and analysis drives innovation.

Present – Business and technical challenges

3 key trends in business are driving rapid change; closed loop business, massive information volumes, collaboration driving innovation.

Really important to stop talking about ‘unstructured information;’ that’s just noise. Information has structure. Instead, hard information is data; tables, structure, computer-oriented. Meaning and values have been separated. Metadata explicit, and formally modelled. Soft information is not well defined, it is by and for people, it mixes meanings and values. Metadata is implicit, tacit, or non-existent.

Moving from information we understand – and control – to information outside the enterprise that we don’t. Implications for quality, meaning, etc.

Future – a new architecture?

current architecture 25 years old – time for a change?

Next, Bob Page from eBay, talking about Building the Data-Driven Organisation.

eBay “fascinated with numbers;” early 1999 screenshot of homepage, showing lots of stats.

What role does data play in the business? Using Analytics focussed on big buckets; velocity, efficiency, trust, etc.

Efficiency drive – lower insertion fees to list/sell new products by 99%. Decision based on analysis of data?

Trust – top-rated sellers constantly monitored to ensure algorithm is reflecting reality. 22% of sales from trusted sellers in 2009. Now 32%. Actually surprised it’s not higher…

eBay handled $2Bn of sales on mobile devices last year.

Analytics and continual data analysis drives all the apps, trust metrics, etc.

Some figures… 50 TB/day of new data, etc. Lots of other numbers on slide, but it was only up for seconds…

Analytics work in marketing, sales, product dev, and all areas of the business.

More than 85% of eBay’s analytial workload is new and unknown; design for the unknown. Enable exploration of the data, rather than just reporting of established metrics.

Machine Learning: Data trumps Algorithms. This is the promise of Big Data; existing algorithms get better as you throw more data at them. It’s cheaper to throw more data at an algorithm than to invest in developing new algorithms.

Enterprise Data Warehouse for transactional data; purchase history, etc. Behavioural data to track clickstreams, impulse purchases, etc… much larger than the amount of transactional data in traditional systems. No technology silver bullet for the behavioural data; optimise for concurrency, or TCO, or CPU usage, or flexibility, or storage, or governance? Those priorities change the tool you should use.

eBay built a 500-node Hadoop cluster in June 2010. Now they have a much bigger cluster.

Data Marts; ‘a reality for many of us,’ because businesses need to give control to the user. Don’t want infrastructure team/ data scientists as bottle neck. Totally opposite to attitude expressed by previous speaker. But Data Marts end up being very expensive and inefficient. eBay have built a virtual data mart; views onto a single pool of data. Far more efficient, in theory.

And then I had to slip away, and miss the final session before lunch…  More later…

There are some great sessions on the Programme, and I look forward to comparing the diverse ways in which Big Data concepts and methods are being put to work across a range of market segments. I also look forward to exploring answers to some of the questions I posed back in November.

As usual, the diary is filling up with meetings, briefings, and the odd conference session, but there are still some gaps to fill. If you’re at the event – or in the area – and want a chat, why not grab one of the slots over on Tungle?

Related articles

• Six months after “What is data science?” (radar.oreilly.com)
• O’Reilly Strata Conference: Only a few days left for early registration + reader discount (flowingdata.com)
• Big Data Examined At Inaugural O’Reilly Strata Conference (lockergnome.com)