County Arcade, Leeds

The global CloudCamp movement continues to grow, with events over the next few weeks in Denmark, Germany, Nigeria, Ghana, Kenya, and across the United States. And now, I’m very pleased to announce that the English city of Leeds is joining the party.

CloudCamp events have been taking place in the UK for years, and the London gatherings have picked up real momentum. Outside London, we’ve seen a few events in Warrington, Newcastle, and Edinburgh. We believe that the time is now right for something more regular; a place in which the cloud-building, cloud-using, cloud-interested and cloud-exploring can come together for talk, beer, pizza and more… without having to jump on a train to the deep south.

CloudCamps are interesting events, with a real emphasis on informality. I’ve attended several around the world, and am always impressed by the energy in the room, and by the welcome extended to newcomers. As the main CloudCamp site describes,

“CloudCamp is an unconference where early adopters of Cloud Computing technologies exchange ideas. With the rapid change occurring in the industry, we need a place we can meet to share our experiences, challenges and solutions. At CloudCamp, you are encouraged you to share your thoughts in several open discussions, as we strive for the advancement of Cloud Computing. End users, IT professionals and vendors are all encouraged to participate.”

Jeremy Jarvis at Brightbox and Karyn Fleeting and Joel Turner at Tinderbox Media have been driving this event forward, and they’ve invited me on board to help out. I also get to be MC on the night.

We’ve got speakers and sponsors committed, with more of both to come. If you think you should be one of those doing the speaking or the sponsoring, do let us know.

So, if you like talking Cloud, if your boss has ordered you to learn Cloud, or if you’re just keen to understand a little more about what this Cloud thing can do for you, stick the evening of 14 June in your diary, sign up (for free) on Eventbrite, and come along to the Hilton DoubleTree in Leeds for an evening of fun, learning, beer, and more.

See you there!

Image of the County Arcade in Leeds by Flickr user Francisco Perez

Image: NASA

Amazon’s globe-encircling cloud infrastructure is compelling to many. From Virginia to California, from Ireland to Singapore, and from Japan to Brazil; wherever you find yourself there’s a local instance of the same familiar set of services. And, in all likelihood, Australia will soon be added to the list. For those primarily interested in just serving both Europe and the US, the list of options grows to include Rackspace, GoGrid, CloudSigma and a few others. And yet, despite the buying power and increasing ubiquity of these larger players, there seems to be plenty of space left for smaller entrants. For prospective customers only concerned with a single country or region, for example, the choices are almost too many to count, and choosing between them becomes a complex and multi-faceted affair.

Take Brightbox, for example. As my family know all too well, those pesky timezones mean that many of my evenings are punctuated with calls to or from the States, where so much of the innovation in this sector continues to take root and grow. Either that, or I’m creeping out of a sleeping house to catch early trains for the 150 mile journey south to London. It was therefore refreshing to talk to someone in this industry whose offices are only 50 miles away in the UK city of Leeds.

Established back in 2005 as a Ruby shop capable of hosting apps on dedicated hardware, Brightbox has evolved to place increasing emphasis upon the provision of infrastructure. In 2010, the company began to seriously explore the possibility of offering a generic cloud infrastructure environment. This was in the days before OpenStack, but Eucalyptus existed and was attracting interest. But according to Brightbox co-founder Jeremy Jarvis, 2010′s Eucalyptus lacked some key resilience features (load balancing, multi-data centre capabilities, etc) that the team believed were critical… so they built their own system from the ground up. And, at the end of September last year, Brightbox Cloud entered general availability.

The Brightbox cloud operates out of two UK data centres, with planning underway for a third. Both data centres are now owned and operated by Telecity, which acquired the two independent data centre providers with whom Brightbox had launched. Brightbox owns the racks and (Dell) hardware, and also ensures provision of redundant network access into the data centres. Customers are predominantly drawn from across Europe, but Jarvis says he’s seeing some customers coming from as far away as Australia, New Zealand, and Brazil. The company sees its potential growth through eventual expansion to data centres on the European mainland, but Jarvis says he’s “much less interested in setting up yet another” North American operation. The company is profitable, employs ten staff, and is seeing steady growth in usage.

Brightbox does not (yet) offer a web management console, but Jarvis describes this as a conscious decision and also something of an asset. The company has instead focused their attention upon crafting a rich, capable and intuitive API (and associated command-line interface). According to Jarvis, the developers that the company tends to target have responded favourably to the API, describing it as “nice” and “more consistent” than the various APIs offered by Amazon’s growing suite of services.

A focus upon developers (and the growing Dev/Ops movement) was also a strategic decision, and Jarvis cites examples in which ‘mere developers’ have proved instrumental in securing significant contracts with Brightbox from their employers. Corporate purchasing processes may, finally, be evolving. Despite the success of Platform as a Service (PaaS) offerings such as Heroku and EngineYard, Jarvis also suggests that Brightbox is seeing growing evidence that developers are seeking more fine-grained control over infrastructure than PaaS typically offers. A platform abstracts the underlying complexity of infrastructure, making it easier for application builders to focus upon creating the specific services they wish to provide. But abstractions typically require compromises, with configuration decisions being made for everyone on the platform on the basis of ‘normal’ requirements. For developers with non-normal requirements (and they may actually be the majority of users), IaaS is more likely to offer the fine-grained control that they need.

But the cloud infrastructure world has come a long way since Brightbox began planning their product two years ago. OpenStack has arrived, and (despite a growing body of nay-sayers) is credible. Rackspace, HP, and others are on the cusp of delivering real clouds to real customers on the back of its codeline. Eucalyptus appears to have turned a corner, and pulled back from a brink that I (and others) saw them teetering on the edge of. Canonical’s marriage of Ubuntu to OpenStack is now just one of several ways to get the same cloud code, capabilities and apis onto machines running inside your own data centre. Amazon just keeps on doing what Amazon does, incrementally adding features, cutting prices, and becoming ever-harder to not choose.

In that world, surely a little cloud provider operating their own bespoke solution from the wrong end of the Leeds-London railway line, in a country on the wrong side of both the English Channel and the Atlantic Ocean has no hope? Surely they should just pack up shop, and either adopt OpenStack/Eucalyptus fast… or find a new line of work?

Jeremy Jarvis disagrees, vehemently. And he’s not alone. Look at Edinburgh’s Flexiant, Glasgow’s SymetriQ, and a whole host of other companies that have built their own solutions from nothing. Others, of course, have recognised the value in taking OpenStack, Eucalyptus, Microsoft’s Windows Azure, and similarly established technologies, and making them their own. Look at Eduserv’s Swindon data centre, or the hosted desktops from East Yorkshire’s own GoCloud.

So how can Brightbox (or Flexiant, or SymetriQ, or any of the other non-conformers) compete? How, indeed, can they survive? Jarvis suggests that “Buy British” continues to carry weight here. Even without raising the scary (and often, actually, wholly irrelevant) spectre of PATRIOT Act-powered snooping, a significant proportion of UK (or European) companies like the idea of buying services from UK (or European) suppliers. They like that the documentation is spelled correctly. They like that telephone support is (more or less) in their timezone. They like that the development team shows up at local events, and that it’s a person buying the drinks in the bar, rather than the disembodied marketing budget of some far-off corporation.

Purchasing decisions for something like cloud infrastructure are complicated. Often, they’re probably quite illogical. Price isn’t always the deciding factor (and even if it were, smaller providers like Brightbox aren’t ridiculously expensive in comparison to their larger competitors). Having been the friendly face at a local developer event might swing that big contract. Or having a “nice” API. Or implementing niche features in firewalls, or networking, or port forwarding might each grab the attention — and loyalty — of specific sectors of the long tail. Tesco might sell ‘everything’ to ‘everyone,’ but we still have room in our lives for the SPAR corner shop, and for the upscale deli with the nice cakes.

The same’s true in the cloud, although I can’t help feeling that we’re going to see quite a rapid decline in entirely new cloud infrastructures as the next generation of niche cloud boutiques take OpenStack or Eucalyptus and mould them to their requirements. They probably won’t be making those decisions because (like we Clouderati) they agonise endlessly about interoperability or portability or the de facto standard of the Amazon Web Services stack. For most of their SME customers, those things simply don’t matter. Instead, they’ll be adopting OpenStack or Eucalyptus because the grunt work (and the marketing) has been done. It simply costs less to take something off the shelf than to develop it yourself from scratch. But for companies like Brightbox, where that investment has already been made? Well, for them there may still be plenty of prospective customers out there.

Related articles

• Eucalyptus grabs $30M from IVP to push the open-source private cloud forward (venturebeat.com)
• Announcing Brightbox Cloud – the UK’s first true IaaS platform! (brightbox.co.uk)
• Platform as a service ready to rev up, says Gartner (zdnet.com)
• Hosting providers aim to challenge cloud leaders (zdnet.com)
• Joyent Brings Its Public Cloud To Europe (techweekeurope.co.uk)
• Rackspace launches OpenStack-powered next-gen public cloud (venturebeat.com)
• Ubuntu Cloud Portal: Brightbox 12.04 daily images now available, discounts for testers and Ubuntu members (cloud.ubuntu.com)
• Ubuntu Linux heads to the clouds (zdnet.com)

Image via Wikipedia

Most of us recognise that the Earth is warming and that — despite our planet’s temperatures having dramatically risen and fallen before — we humans must accept some measure of responsibility for the current changes.

Already consuming at least 1.1-1.5% of global power, and only forecast to grow ever-more rapacious, the data centres that power our information economy are surely one area in which we can, should, and must find ways to reduce consumption. And, although by no means perfect, data centre builders, operators and suppliers are paying attention. Individual servers are becoming more efficient. Buildings are being cooled by the wind and by toilets. Waste heat is warming homes. Newer facilities are being sited to save the planet (and money) by drawing power from hydro-electric schemes rather than coal, oil, gas, or the once-again-unpopular-and-scary Nuclear.

And then there’s solar. On the face of it, the ultimate renewable. The sun shines. It does so every day. It will continue to do so every day for millions of years to come. Every watt diverted to a data centre has absolutely no effect whatsoever upon the sunlight that the rest of us receive today, or that we shall receive tomorrow. There’s certainly a location trade-off to be made, as the sunshine rarely reaches the ground for much of the day in locations where the air is cool and damp enough for all that fresh air cooling to work, but there should certainly be a case to be made for both. If you’re in Iceland or Ireland or Oregon, sunlight is not something to rely upon for meeting your ravenous appetite for energy. But if you’re in sunnier climes, surely it’s a no-brainer?

Apple, for one, is constructing a 171 acre (69 hectare) solar farm next to its North Carolina data centre. That’s big. It’s apparently the biggest commercial deployment in the United States and — with some help from WolframAlpha — it’s also equivalent to 1.6 Vatican Cities or 13 Great Pyramids.

More tellingly, it’s also a lot of fields and trees.  If all those mirrors reduce the impact of Apple’s data centre on the environment, then maybe — just maybe — the loss of forest and farmland and open space is a cost worth paying. But Amazon’s James Hamilton does some maths, which he shares in a recent blog post, and he reckons that all the mirrors in that 69 hectares contribute around 3.2 MW to the data centre’s power requirements; requirements that Hamilton calculates to be around 78 MW. That’s 4%, which really is not that much at all. It’s worth noting that the figures for the solar array’s power output and the data centre’s consumption are only Hamilton’s estimates, but he tends to know what’s he’s talking about. In the absence of solid numbers from Apple, you can do a lot worse than believe the maths of someone like Hamilton.

So… why the enthusiasm for solar? Are data centre operators taking reductions where they can get them, and calculating that a 4% contribution to the fuel bill is worth 69 hectares? They might be right. Maybe. Or are they being more callous and calculating than that? Are the ‘savings’ on energy bills and greenhouse emissions essentially negligible, and are trees and fields being turned over to big mirrors simply so that companies can look good? They get a tick in the “green” box. They get a mention from Greenpeace. Customers, consumers and law makers think they care. Whilst actually, they don’t care at all.

Related articles

• The Switch SuperNAP data centre – one of the most impressive I’ve been in (greenmonk.net)
• Apple slaps mega-solar panel field on new ENORMO data centre (go.theregister.com)
• In cooler climates, outdoor air systems can save energy dollars for data centres, retailers, warehouses (cleanbreak.ca)
• Colt Telecom and Verne Global’s dual renewably sourced data centre in Iceland (greenmonk.net)
• Facebook Data Centre Uses As Much Power As The Entire County It’s In (gizmodo.com.au)
• Greenpeace, Facebook in truce over energy (news.smh.com.au)
• Solar power gets public vote (evoenergy.co.uk)
• The revenge of Thomas Edison (business.financialpost.com)
• California’s Military Bases Could Generate 30x More Electricity From Solar Power Than They Consume (treehugger.com)
• Apple’s iCloud Has Solar Lining (wired.com)

Piyush Lumba, Director of Product Management for Azure Data Services at Microsoft, talks about what the Marketplace can do today and shares some of his perspectives on ways that the nascent data market space could evolve.

Following up on a blog post that I wrote at the start of 2012, this is the eighth in an ongoing series of podcasts with key stakeholders in the emerging category of Data Markets.

Related articles

• Data Market Chat: the podcasts are a-coming…(cloudofdata.com)
• Data Market Chat: Tyler Bell discusses Factual (cloudofdata.com)
• Data Market Chat: Hjálmar Gíslason discusses DataMarket.com (cloudofdata.com)
• Data Market Chat: Chris Hathaway discusses AggData (cloudofdata.com)
• Data Market Chat: Flip Kromer discusses Infochimps (cloudofdata.com)
• Data Market Chat: Nick Edouard discusses BuzzData (cloudofdata.com)
• Data Market Chat: Stephen O’Grady of RedMonk examines the bigger picture (cloudofdata.com)
• Data Market Chat: Shion Deysarkar discusses Datafiniti (cloudofdata.com)
• Big data in the cloud (radar.oreilly.com)
• ‘Roswell’: Another key component of Microsoft’s cloud strategy (zdnet.com)

Image by Kevin Krejci via Flickr

The CloudCamp unconference returned to London for the 14th time this evening, regaling a capacity crowd in the Crypt below Clerkenwell’s St James Church with several hours of discussion and debate on the somewhat elusive topic of ‘Big Data’.

Rather rough notes of the proceedings follow, after the break.

LEF‘s Simon Wardley kicked proceedings off as usual, once again managing to pepper an on-topic canter through the topic with a seemingly never-ending stream of Flickr images of cats… and analogies to electricity. You possibly had to be there? His core message, though? There’s nothing new under the sun… and the cycles of change just keep on coming.

Next, Peter Matthews from CA Labs, on “is big data mutually compatible with the cloud?” Erm, yes. Data volumes with big data are so large that it’s difficult to move it around… which creates opportunities for lock-in that vendors may wish to seize. And then he was out of time.

Next, Fujitsu’s Mark Wilson on ‘Structuring Big Data.’ He’s actually talking about Linked Data, a topic I’ve dug into before here and over on semanticweb.com – Linked Data could be/ might be the effective realisation of the decade-old Semantic Web dream. Big Data means masses of unstructured or semi-structured content, presenting a management headache of previously unanticipated proportions. Linked Data, he argues, creates the mechanism to link all of this data together from across disparate sources. Yes, but it’s easier to say than to do… And in 5 minutes he really couldn’t explain enough to persuade the audience. Linked Data should be “the optimal reference source,” he said. It should be “a broker for all data sources,” and we should “think about integration, not duplication.” Yeeeeees… But.

Next, Canonical’s Nick Barcet, talking around scalability, Ubuntu, package management, configuration management, etc. Not wholly sure what the point was, I’m afraid.

Next, Chris Swan from UBS – big data and security. “If you’ve got security controls that aren’t properly monitored, then they don’t matter.”

Next, Tom Leyden of Amplidata – Big “Unstructured” Data in the Cloud. Data storage to increase 30x over the next decade, but staff will only increase 50% over the same period. Challenge in the 90s, as existing storage and analysis technologies struggled to cope with new data volumes. Seeing similar problems today with data streaming from sensor web, etc. Traditional file systems cannot cope. Object Storage the way forward ?

Next, Alex Farquhar – “Cloud v Big Data.” Not really versus… but intersection of the two. Too much discussion of his company, Forward. Just talking about how his company uses cloud to provision IT resources. Might work as a conference presentation or case study – not sure it fits as a 5 minute lightning chat. Around 60TB of data at Forward. Diverse and vital. Using Hadoop cluster – 24 nodes on-premise. Rationale (proximity to the cluster) seemed odd. That can be true, but not clear that it really needs to be the case here?

Next, Alaric Snell-Pym, on Scaling Hadoop. Trying to overcome Hadoop’s I/O bottleneck. Explaining basics of Hadoop and Map/Reduce – no one else has. Explains use of HDFS and ‘selective reading’ to manage lots of small tables and overcome the problems of I/O.

Next, Matt Wood from Amazon. Talking about genetics and the human genome. It’s an analogy. Human Genome Project took years and millions of dollars. Development of gene sequencing machines led to a step change – dramatic drop in cost of sequencing DNA. Like the cloud, anyone? But… the machines create an analysis challenge, because they generate so much data. Cloud offers “collection of productivity tools” to help scientists work with this data collaboratively and (relatively) affordably. A perfect example of a lightning presentation, unlike most of those who preceded him.

And finally, an impromptu slot from HP’s Joe Weinman. A quick overview of current thinking behind his latest book. This one could have gone for much longer… Good stuff.

And that’s the lightning talks finished. Now, the panel, and Simon Wardley’s search for “experts” and “volunteers.”

…and unfortunately, your scribe was ‘volunteered’ as an ‘expert’ by Mr Wardley… and here end the notes. It was great to have Amazon’s Werner Vogels sneak in, and lob comments into the panel, though…

Great event, though with the usual mix of people you wish could have talked for longer… and people you wish wouldn’t have spoken.

Related articles

• The brave new world of big data & Hadoop (venturebeat.com)
• Big VCs Invest In Big Data Startup Continuuity (techcrunch.com)

Image via Wikipedia

Last week, open standards body OASIS unveiled yet another shiny new standards effort. The OASIS Topology and Orchestration Specification for Cloud Applications (TOSCA) Technical Committee hopes to make it “easier to deploy cloud applications without vendor lock-in,” and to support moving from one cloud to another. The usual suspects — the likes of IBM, CA, and Cisco — are on board. The usual holdouts — Google and Amazon, of course — are not. So what is TOSCA trying to achieve? How does it fit alongside all the dead, dying, or ponderously deliberating cloud standardisation efforts that have gone before? And without the giants of the cloud, is there really any point bothering?

As I’ve probably mentioned before, involvement in various national and international standardisation efforts played a big part in my early career. I went to the working group meetings in odd (but often beautiful) locations. I participated in the conference calls. I engaged on the mailing lists. I drafted and edited and reviewed the documents. I completely buy into the idea that there is a place for foundational standards, developed through consensus-building and maintained for the long haul by organisations that stand apart from the vested interests and their competing agendas.

I also believe that there’s a time and a place for these standardisation efforts. Do it too soon, and we end up ossifying something that needs to be in a state of flux. When you don’t know what the best way to prepare a meal is, it’s too soon to print the recipe book. We need to try different approaches, and we need to be able to throw away the attempts that didn’t work out. More worryingly, standardisation efforts can be used for political ends. They can be little more than a rod with which to beat the (usually dominant) competition. At best a distraction, or a talking shop for those unwilling or unable to just get on and do something. At worst, one amongst a toolchest of dirty tricks in a broader war for hearts, minds, and — ultimately — wallets.

The cloud market is a fascinating place. There are leaders and there are followers. There is innovation, and there is competition. There is agreement, and there is debate. For all the rhetoric, and all the posturing, we really don’t yet know the right answer to many of the cloud’s questions.

Maybe TOSCA and the Open Data Center Alliance and IEEE and the rest are — still — too early, and should be content to let the market thrash out a few more of these issues before anyone tries to write anything down? And when it is time to write some stuff down, let’s make sure we focus on specific, finite, tangible, atomic tasks rather than “the cloud.” As Dave Roberts commented in regard to TOSCA’s scope;

“That goal is so large, that I think it’s probably unbounded. When problems get unbounded, the best you can ever hope to achieve is to solve a large enough subset of the problem that the solution is still interesting. If you can’t achieve that, people ignore the solution because it fundamentally doesn’t help them. There is always an ‘interesting’ part of the problem space that they have to solve a different way, and that undercuts the use of the partial ‘solution.’”

And as for Tosca? Things didn’t end well for her, did they? Might TOSCA’s fate, too, be sealed?

Related articles

• Tech giants back standard for cloud portability – InfoWorld (infoworld.com)
• Proposed spec aims to nix cloud lock-in (gigaom.com)
• On TOSCA and Cloud Standards. MyPOV (diversity.net.nz)

Image of Stephen Wolfram via Wikipedia

British-born computer scientist Stephen Wolfram sees ongoing efforts to extend the Internet’s top-level domains (TLDs) beyond the familiar .com, .org, .uk etc as an opportunity to raise the profile of machine-readable data. In a blog post published yesterday, he argues that a new .data domain would increase “exposure of data on the internet—and [provide] added impetus for organizations to expose data in a way that can efficiently be found and accessed.” Whilst wholly in favour of Wolfram’s stated aim, I can’t help feeling that his suggested solution is at best unnecessary and at worst a worrying segregration of data from the ‘proper’ web that everyone else will continue to exploit.

Back in June of last year, the body responsible for coordinating the global domain name system approved a plan to permit new top-level domains (the letters after the final dot in an internet address — the .com in cloudofdata.com, the .uk in bbc.co.uk, the .edu in harvard.edu). Until recently, these top-level domains have been tightly controlled, with a small set of generic domains (.edu, .gov, .mil, .org, etc), a larger set of country domains (.uk, .fi, .nz, etc) and one or two others such as .eu. From tomorrow, anyone with $185,000 will be able to submit a proposal to create and manage a new top level domain, and it’s possible that there could eventually be thousands of them. Wolfram is keen to ensure that data doesn’t miss out on the ‘opportunity.’

As Wolfram himself recognises, there is already an awful lot of machine-readable data on the web. Some of it sits embedded within the web pages that humans read, with specially formatted code waiting to be triggered by the calendars, the address books, or the browser plugins of site visitors. Some of it is packaged up in data files, offered for download. And some of it waits inside a database, ready to be delivered in response to an API call or a query typed into a web form.

There is a growing enthusiasm for exposing this data for reuse. Government transparency agendas have driven public sector data sites like data.gov.uk and data.gov. Similarly, efforts such as data.open.ac.uk and data.southampton.ac.uk see universities beginning to consciously collect data sets together and offer them up for reuse. Similar efforts in the commercial world are less easy to point to, but that reticence has nothing whatsoever to do with the lack of a ford.data, boeing.data, ge.data or astrazeneca.data domain!

In some ways, the convention for gathering significant chunks of data on a data.xxx.yyy site echoes Wolfram’s intention, but with a number of advantages. Data without context is far less valuable than data with context. Much of that context may be inferred from the domain in which the data lives, with data delivered from a .gov or .edu (or .gov.uk or .ac.uk) site perhaps interpreted differently to data hosted on .com, .biz, or .xxx. Southampton University, the Open University, and the US Federal Government are able to gather data up and make it available for download via their existing data. sites if they choose. This offers human visitors to their sites a degree of convenience, whilst retaining the power and brand attributes of their existing domain. Gov.data, gov.uk.data, open.ac.uk.data, southampton.ac.uk.data, though? All are messy, in ways that Wolfram’s own wolfram.data would admittedly not be, and all are simply additional registrations that the institutions would have to pay for in order to stop someone else grabbing the domain.

At the end of the day, the machines don’t actually care. The existing data.open.ac.uk-type sites are human conveniences, not machine enablers. The computers, and the software they run, are quite capable of crawling the public web and finding accessible data wherever it lies on a site. There are plenty of reasons to continue embedding little snippets of data inside human readable web pages, regardless of whether you have a data.wolfram.com or a wolfram.data site. Content negotiation is becoming increasingly capable, such that there really is no need for what Wolfram calls a ‘parallel construct to the ordinary web’ at all. A human being arriving at a web site sees human readable content, whilst various software tools would automatically be presented with very different data or functions, optimised to their capabilities and requirements.

By all means, let us show the curious some of the existing techniques that work in making data more easily accessible. By all means, let us identify the gaps, the issues, the problems (none of which a new TLD even begins to address). Yes, let us definitely and unambiguously set about “highlighting the exposure of data on the internet—and providing added impetus for organizations to expose data in a way that can efficiently be found and accessed.”

But please, let us not be distracted by the false hope that adding yet another TLD to the babel that ICANN is about to unleash can do anything more than consign data to some online ghetto, wallowing unwanted, unloved and unused as companies and their customers lavish love, attention, and clicks upon the .com domain over on the ‘proper’ web.

Thanks to Raphaël Troncy, whose tweet first drew the story to my attention.

Related articles

• Is It Time For Computers To Have Their Own .Data Domains? (techcrunch.com)
• ICANN Pushes Ahead With January 12 Launch For New Top-Level Domains (wired.com)
• The biggest change in DNS since Dot-Com (wired.com)

In many ways, the core concept of the data marketplace is nothing new. Companies like Bloomberg, Nielsen and Experian have built (extremely) profitable businesses by aggregating data, quality checking it, and selling it on. Often their customers could have gone directly to the source(s) and paid far less, but they don’t. The convenience and quality assurance of dealing with a single — reputable — source is perceived to have value. A brand like Bloomberg’s is associated with trustworthiness and authority, and the brand of the marketplace is far more prominent than the data sets upon which it is built.

Similar sites have also served the needs of those seeking data for free, with IBM’s ManyEyes project, Freebase (acquired by Google), Hans Rosling’s Gapminder or The Guardian‘s Data Store amongst those typically mentioned. Current government enthusiasm for ‘transparency’ has fed all of these sites with data, and led to creation of large government-specific data repositories such as data.gov.uk.

The commercial services like Bloomberg have tended to focus upon specific domains (finance, in Bloomberg’s case) or types of data. They have also tended to be eye-wateringly expensive; aimed squarely at the small market segment for whom the data are mission-critical and the fees are affordable. The free services like Gapminder also tend to focus (global development statistics in this case). Other, perhaps, than experiments like ManyEyes, both the free and the commercial sites tended to aim for a degree of comprehensiveness and authority. They wanted to become the place to turn for their type of data.

But for the new generation of data markets, the picture becomes far less clear. They tend to be catholic in their data acquisition policies, they typically don’t even attempt comprehensiveness, they mix free (almost all of them hold identical large swathes of government data from the US, the UK, and elsewhere) with commercial data, and they continue to feel their way toward business models that might prove sustainable for the long haul. Perhaps more seriously, they appear almost schizophrenic with respect to brand projection, attempting to push both their own brand and those of the data sets they host in ways that can confuse far more often than they enlighten.

In attempting to differentiate themselves, today’s data markets are seeking to add features and functionality in order to be seen as far more than simply places to buy third-party data. They want to become recognised for quality assurance, for data enrichment, or for tools and capabilities that make working with the data easier or more powerful. They want to become sticky, and they want to be seen as different from their competitors. The trick, though, is to explain those features and those differences in ways that make sense to potential customers. Those customers will ultimately pay for functionality and utility, not for gimmicks or under-the-hood technological distinctions that have no real impact upon getting on with the job in hand. Are today’s data markets describing their features in ways that help prospective customers to understand why they should be chosen over the alternatives? Not really. At least, not yet.

Also, as RedMonk’s Stephen O’Grady touched upon amongst a set of related issues, we’ve really not begun to see much evidence of price competition. There are too few suppliers, each with their fiercely loyal bands of tame users (‘customers’), and too few people prepared to shop around for the best deal.

The new data markets are still young. Understandably, they are still feeling their way in order to understand what the market wants, how much it is prepared to pay for what it wants, how large the market might be, and what their individual niche within that broader market might look like. Earlier models, based upon almost monopolistic domination of specific verticals and polarised pricing, offer some lessons but are ultimately unsatisfactory blueprints for this more competitive, open, and complex environment. Beyond specific domains like finance (which may be ripe for disruption), the data markets must struggle to convince prospective customers that they have something of value to offer. Those customers may already have their own processes for obtaining data. They may generate the data themselves, or expect — as so many do — to be able to access what they need for free. They are perhaps suspicious of data produced by third parties who are, in other contexts, their competitors, and they are almost certainly unwilling to allow ‘the competition’ to benefit from their own data. They invariably do not understand the costs associated with gathering and quality-assuring data, or the challenge of preparing different data sets in order that they may meaningfully be combined. And into this, the fledgling data markets must insert themselves, market themselves, and sell themselves. They must change behaviours, they must challenge presumptions, they must alter working practices, and they must persuade their new customers that all of this pain is worth paying for. A tall order, indeed, but necessary if any of them are to realise their potential.

The European Commission, at least, begins to comprehend the scale of the challenge. A set of projects are currently being finalised, and this year will see European SMEs given the funding to boot-strap a number of new data sources. With Commission funding, it is hoped, the chosen projects will be able to explore models by which data can be created, curated, shared and re-used in a manner that is cost-effective and ultimately sustainable. The funding should enable these projects to reach viable scale, and give participants the freedom to explore alternative commercial models. The projects will be announced shortly, but only time will tell if the funding and the incentives are sufficient to break through the barriers that prevented any of these markets from forming by themselves.

But outside the rather artificial bubble created by European public funding, there is a lot of work to do. Investors are intrigued by — but still wary of — the opportunity. Infochimps is spending its way through over $1.5 million of investment, Factual has almost $30 million, and companies like Talis and Microsoft are making not-insignificant investments in their own efforts. We’re all still experimenting, but with the real market for these services currently falling far short of the money at stake, it mustn’t be long before investors start asking harder questions. Back in 2010, Pete Soderling and Pete Forde described data as a $100 billion market. The data markets may be after a significant chunk of that but, today, they’re not even close.

The ways that data markets are attempting to differentiate themselves, and the work being done to understand the market opportunity here, will have to wait for subsequent posts.

Disclosures: I am a former employee of and current shareholder in Kasabi’s parent company, Talis. The European Commission is, from time to time, a client.

Related articles

• With Factual, 1 API now unlocks data for 55 million places (gigaom.com)
• ‘Roswell’: Another key component of Microsoft’s cloud strategy (zdnet.com)
• Infochimps Acquires Y Combinator Startup Data Marketplace, Expanding Brand Holdings and Online Presence (prweb.com)

The border between the USA and Canada, in Washington State

In my last post I looked at the USA PATRIOT Act, and at some of the ways in which it exemplifies differences in attitude and approach on either side of the Atlantic. In our increasingly connected world, these differences begin to pose quite serious challenges for those wishing to join up, to aggregate, and to operate at scale. In this post I’ll take a look at one particular case in which matters must soon come to a head; the current enthusiasm for cross-border data flows, and what GigaOM’s Derrick Harris refers to as “free trade for data.”

At first glance, the concept of free trade for data makes a lot of sense. Of course data should be able to move with reasonable freedom from country to country. As someone who once had the job title of ‘Interoperability Focus’ I’m bound to agree that international standards should normally be used to promote interoperability and transparency. Few, surely, would not welcome cross-border arrangements to encourage entrepreneurial reuse of data, or to ensure that people in different countries can access popular online services headquartered overseas?

Unfortunately, the bulk of the evangelism around reaching new agreements here is nationalistic, partisan, and closely tied to particular world views. From here in Europe, American posturing on the topic grates. From America, we Europeans no doubt appear protectionist and over-cautious. And elsewhere in the world, governments and companies with valid contributions to make cry out to be heard amidst the trans-Atlantic babel.

Derrick’s post from last November is a case in point, discussing submissions by US companies (Visa, Mastercard, Microsoft, Oracle, Salesforce, etc) to a US body (the National Foreign Trade Council), in an attempt to influence US government policy with respect to its peers around the world. The topic was Cross-Border data sharing, but similar companies from overseas were not involved. Nokia? Vodafone? Baidu? SAP? HSBC? Nope, nope, nope, nope, nope. Companies embracing a particular world view, rooted in a particular culture, come together to draw up recommendations that (probably in good faith) reflect that world view and those cultural norms. Whether implicitly or explicitly, those recommendations then seek to project that world view onto other countries, other cultures. We all do it. We all bring our baggage, our beliefs, our presumptions. Sometimes we know when we’re doing it, and we can either carry on regardless, or we can attempt to account for alternative approaches. But all too often the cultural norms are so ingrained that we forget they’re there. We assume that they’re normal. We assume that they’re shared. And then we’re surprised when Australians balk at receiving the ‘summer’ release of software in June, when Americans don’t consider that piece of personal data to be sensitive, or when Europeans think Government really should be involved in regulating a social networking site.

If we want to ensure the unimpeded flow of data across borders — and we should — then we need to begin by recognising that the places and people on either side of that border are very likely to be different. Their attitudes are different. Their needs are different. Their aspirations are different. Their laws are different. In the early days of the Internet and the web, legislation, policy and even expectation did not really exist. Almost by default, US attitudes and presumptions tended to apply unless a particular country cared enough to institute something different inside their own borders. As we become more and more interested in territoriality and jurisdiction with respect to data, that naive innocence no longer applies. There is no longer a blank canvas upon which the innovators can paint their hopes. We have policies, regulations, and laws. We have populations that have experienced today’s web, and we have a media quick to interject its perspective.

Today, far more than at the web’s birth, we have to engage in public dialogue about what we want to achieve, and why. We cannot achieve Derrick’s aspiration of free trade for data and leave every local law, policy and procedure untouched. But nor can we achieve it by projecting a single world view around the globe, sweeping all of those prior laws aside.

Rather than entrench behind a US ‘position,’ a European ‘position’ (including 24 variant positions, two abstensions and an opt-out), a Chinese ‘position,’ and so on, can’t we begin to understand what some of the real opportunities — and concerns — might be?

Big companies that operate internationally (like those developing the document (PDF) Derrick discussed) absolutely need to come together, to share their perspectives on the pain of moving data around. But for those companies only to be American is insane, and counter-productive. Nokia has perspectives to share here, as do HSBC or Vodafone. Let’s hear them in the same forum. Let’s also hear individual governments, speaking up for the concerns and desires of their citizens. Let’s hear the citizens themselves, when they care enough to express an opinion. But let’s hear all of it early, before it becomes entrenched in a set of contradictory official statements.

Then we might arrive at a sensible approach to ensuring free trade for data, rather than the projection of an American ideal upon the rest of us.

And that sounds like a good topic for one of those panels at Davos later this month…

Related articles

• Advancing the free flow of information (googlepublicpolicy.blogspot.com)
• PATRIOT Act clouds picture for tech – POLITICO.com Print View (mbcalyn.com)
• Border accord (economist.com)
• ANALYSIS: What the new border deal means for you (cbc.ca)

The PATRIOT Act’s acronymic name may evoke harmless images of bunting, parades, and national anthems, but the reality is rather different. A product of America’s post-9/11 entrenchment, the Uniting (and) Strengthening America (by) Providing Appropriate Tools Required (to) Intercept (and) Obstruct Terrorism Act of 2001 affords the Federal Government wide-ranging and far-reaching powers that show little — if any — respect for geographic boundaries or inconveniently contradictory local legislation. A US company (like Microsoft or Amazon) is subject to the Act’s powers all around the world. A US citizen’s data, stored in a US company’s data centre that is physically situated in the United States is subject to the Act, and everyone might be reasonably comfortable with that. But so is a German citizen’s data, stored in an Amazon data centre in Ireland; and German, Irish and European lawmakers appear almost powerless to intercede.

European countries tend to be stricter about use (and abuse) of personally identifiable information than the US. Although surveys identify some national differences, it also appears that Europeans broadly embrace the approach taken by their governments. And, anecdotally, conversations with European and American entrepreneurs and European and American individuals repeatedly point to rather different sets of basic presumptions operating on either side of the Atlantic. Europe’s Data Protection Directive, and its implementation in national legislation such as the UK’s Data Protection Act, are clear about the ways in which a citizen can expect data about themselves to be collected, stored, shared and used. The penalties for intentional abuse could probably be tougher, but the sentiment remains clear. The Safe Harbor Principles provide mechanisms by which US companies can self-certify that their normal operating procedures meet European standards (Microsoft, Google, Amazon and many others do this). The February 2010 ‘model clauses‘ that Microsoft embraced last month codify some of these protections in a manner that — theoretically — makes it easier for customers’ lawyers to understand what Microsoft will do with their data. It’s unlikely that the legalese (PDF) will actually make things any clearer for the average customer, though.

So, from the perspective of Europe’s governments and citizens, and for US companies that choose to trade here, things appear more or less ok. Personally identifiable data can be collected, stored, shared and used, but only within a set of constraints that Europeans broadly seem comfortable with. Unfortunately, all those Safe Harbor self certifications and model clause endorsements are summarily ignored whenever the PATRIOT Act is invoked. Data Protection Directive requirements not to transfer data to random third parties are trumped by PATRIOT Act powers enabling the US Federal Government to take what it wants. Data Protection Directive stipulations that citizens be informed when their data are taken are over-ruled by the PATRIOT Act’s cloak of secrecy. And on and on the list of contradictions continues. And the PATRIOT Act wins every time, because its powers, its penalties, and its backers are so much scarier than the officials in Brussels. Despite tougher language, it’s not clear that sweeping changes to Europe’s data protection directive will really resolve the contradictions. Indeed, once enshrined in law, the proposals will most likely result in more polarisation, not less.

In Europe too, of course, there are exemptions to the data protection legislation specifically intended to permit reasonable use of data by law enforcement agencies and others. This makes sense, and it could be argued that the PATRIOT Act is simply more of the same. But it’s not, because European law enforcement agencies must demonstrate a far clearer need before they’re allowed to — legally — start rooting through a citizen’s data.

It is unlikely that the PATRIOT Act is routinely invoked, or that US officials spend much time reading Europeans’ email. The cloud — even the parts run by US companies — remains broadly safe, secure, and reliable. Safe Harbor provisions, model clauses, and the ability to insist that data normally resides in one territory or another remain an effective means of ensuring that day-to-day cloud operations meet user needs whilst complying with relevant local, regional and international legislation. But, every now and again, the PATRIOT Act will be invoked, and data will be taken. Whilst it’s something to be aware of, it’s probably not something for most people to lose too much sleep over. You’re more likely to lose data yourself, or have it escape into the wild because of an error in your own systems or a malicious hack by a competitor. And you could and would be held accountable for those breaches, in a way that you almost certainly wouldn’t for a PATRIOT Act data seizure.

So the PATRIOT Act may not be as scary as it might now appear. But it remains a visible illustration of a rather more worrying issue; a belief that the laws of one country should be able to trample over the laws of other countries at will — even inside those countries. Further, it suggests a (growing?) disconnect between the attitudes and expectations on either side of the Atlantic. And one particular aspect of that is the subject for my next post.

Image by Flickr user John Morgan

Related articles

• Microsoft’s European ‘cloud pact’ still does not protect data against FISA, Patriot Act (zdnet.com)
• E.U. Regulations Become Microsoft Cloud Selling Point (blogs.wsj.com)
• Microsoft Pushes Back on EU Cloud Concerns as European Rivals Move In (wired.com)
• Defense giant ditches Microsoft’s cloud citing Patriot Act fears (zdnet.com)
• PATRIOT Act and privacy laws take a bite out of US cloud business (arstechnica.com)
• Microsoft boosts Office 365 regulatory compliance (infoworld.com)
• Updated European law will close Patriot Act data access loophole (zdnet.com)
• Exclusive: European data protection law proposals revealed (zdnet.com)