In many ways, the core concept of the data marketplace is nothing new. Companies like Bloomberg, Nielsen and Experian have built (extremely) profitable businesses by aggregating data, quality checking it, and selling it on. Often their customers could have gone directly to the source(s) and paid far less, but they don’t. The convenience and quality assurance of dealing with a single — reputable — source is perceived to have value. A brand like Bloomberg’s is associated with trustworthiness and authority, and the brand of the marketplace is far more prominent than the data sets upon which it is built.

Similar sites have also served the needs of those seeking data for free, with IBM’s ManyEyes project, Freebase (acquired by Google), Hans Rosling’s Gapminder or The Guardian‘s Data Store amongst those typically mentioned. Current government enthusiasm for ‘transparency’ has fed all of these sites with data, and led to creation of large government-specific data repositories such as data.gov.uk.

The commercial services like Bloomberg have tended to focus upon specific domains (finance, in Bloomberg’s case) or types of data. They have also tended to be eye-wateringly expensive; aimed squarely at the small market segment for whom the data are mission-critical and the fees are affordable. The free services like Gapminder also tend to focus (global development statistics in this case). Other, perhaps, than experiments like ManyEyes, both the free and the commercial sites tended to aim for a degree of comprehensiveness and authority. They wanted to become the place to turn for their type of data.

But for the new generation of data markets, the picture becomes far less clear. They tend to be catholic in their data acquisition policies, they typically don’t even attempt comprehensiveness, they mix free (almost all of them hold identical large swathes of government data from the US, the UK, and elsewhere) with commercial data, and they continue to feel their way toward business models that might prove sustainable for the long haul. Perhaps more seriously, they appear almost schizophrenic with respect to brand projection, attempting to push both their own brand and those of the data sets they host in ways that can confuse far more often than they enlighten.

In attempting to differentiate themselves, today’s data markets are seeking to add features and functionality in order to be seen as far more than simply places to buy third-party data. They want to become recognised for quality assurance, for data enrichment, or for tools and capabilities that make working with the data easier or more powerful. They want to become sticky, and they want to be seen as different from their competitors. The trick, though, is to explain those features and those differences in ways that make sense to potential customers. Those customers will ultimately pay for functionality and utility, not for gimmicks or under-the-hood technological distinctions that have no real impact upon getting on with the job in hand. Are today’s data markets describing their features in ways that help prospective customers to understand why they should be chosen over the alternatives? Not really. At least, not yet.

Also, as RedMonk’s Stephen O’Grady touched upon amongst a set of related issues, we’ve really not begun to see much evidence of price competition. There are too few suppliers, each with their fiercely loyal bands of tame users (‘customers’), and too few people prepared to shop around for the best deal.

The new data markets are still young. Understandably, they are still feeling their way in order to understand what the market wants, how much it is prepared to pay for what it wants, how large the market might be, and what their individual niche within that broader market might look like. Earlier models, based upon almost monopolistic domination of specific verticals and polarised pricing, offer some lessons but are ultimately unsatisfactory blueprints for this more competitive, open, and complex environment. Beyond specific domains like finance (which may be ripe for disruption), the data markets must struggle to convince prospective customers that they have something of value to offer. Those customers may already have their own processes for obtaining data. They may generate the data themselves, or expect — as so many do — to be able to access what they need for free. They are perhaps suspicious of data produced by third parties who are, in other contexts, their competitors, and they are almost certainly unwilling to allow ‘the competition’ to benefit from their own data. They invariably do not understand the costs associated with gathering and quality-assuring data, or the challenge of preparing different data sets in order that they may meaningfully be combined. And into this, the fledgling data markets must insert themselves, market themselves, and sell themselves. They must change behaviours, they must challenge presumptions, they must alter working practices, and they must persuade their new customers that all of this pain is worth paying for. A tall order, indeed, but necessary if any of them are to realise their potential.

The European Commission, at least, begins to comprehend the scale of the challenge. A set of projects are currently being finalised, and this year will see European SMEs given the funding to boot-strap a number of new data sources. With Commission funding, it is hoped, the chosen projects will be able to explore models by which data can be created, curated, shared and re-used in a manner that is cost-effective and ultimately sustainable. The funding should enable these projects to reach viable scale, and give participants the freedom to explore alternative commercial models. The projects will be announced shortly, but only time will tell if the funding and the incentives are sufficient to break through the barriers that prevented any of these markets from forming by themselves.

But outside the rather artificial bubble created by European public funding, there is a lot of work to do. Investors are intrigued by — but still wary of — the opportunity. Infochimps is spending its way through over $1.5 million of investment, Factual has almost $30 million, and companies like Talis and Microsoft are making not-insignificant investments in their own efforts. We’re all still experimenting, but with the real market for these services currently falling far short of the money at stake, it mustn’t be long before investors start asking harder questions. Back in 2010, Pete Soderling and Pete Forde described data as a $100 billion market. The data markets may be after a significant chunk of that but, today, they’re not even close.

The ways that data markets are attempting to differentiate themselves, and the work being done to understand the market opportunity here, will have to wait for subsequent posts.

Disclosures: I am a former employee of and current shareholder in Kasabi’s parent company, Talis. The European Commission is, from time to time, a client.

Related articles

• With Factual, 1 API now unlocks data for 55 million places (gigaom.com)
• ‘Roswell’: Another key component of Microsoft’s cloud strategy (zdnet.com)
• Infochimps Acquires Y Combinator Startup Data Marketplace, Expanding Brand Holdings and Online Presence (prweb.com)

The border between the USA and Canada, in Washington State

In my last post I looked at the USA PATRIOT Act, and at some of the ways in which it exemplifies differences in attitude and approach on either side of the Atlantic. In our increasingly connected world, these differences begin to pose quite serious challenges for those wishing to join up, to aggregate, and to operate at scale. In this post I’ll take a look at one particular case in which matters must soon come to a head; the current enthusiasm for cross-border data flows, and what GigaOM’s Derrick Harris refers to as “free trade for data.”

At first glance, the concept of free trade for data makes a lot of sense. Of course data should be able to move with reasonable freedom from country to country. As someone who once had the job title of ‘Interoperability Focus’ I’m bound to agree that international standards should normally be used to promote interoperability and transparency. Few, surely, would not welcome cross-border arrangements to encourage entrepreneurial reuse of data, or to ensure that people in different countries can access popular online services headquartered overseas?

Unfortunately, the bulk of the evangelism around reaching new agreements here is nationalistic, partisan, and closely tied to particular world views. From here in Europe, American posturing on the topic grates. From America, we Europeans no doubt appear protectionist and over-cautious. And elsewhere in the world, governments and companies with valid contributions to make cry out to be heard amidst the trans-Atlantic babel.

Derrick’s post from last November is a case in point, discussing submissions by US companies (Visa, Mastercard, Microsoft, Oracle, Salesforce, etc) to a US body (the National Foreign Trade Council), in an attempt to influence US government policy with respect to its peers around the world. The topic was Cross-Border data sharing, but similar companies from overseas were not involved. Nokia? Vodafone? Baidu? SAP? HSBC? Nope, nope, nope, nope, nope. Companies embracing a particular world view, rooted in a particular culture, come together to draw up recommendations that (probably in good faith) reflect that world view and those cultural norms. Whether implicitly or explicitly, those recommendations then seek to project that world view onto other countries, other cultures. We all do it. We all bring our baggage, our beliefs, our presumptions. Sometimes we know when we’re doing it, and we can either carry on regardless, or we can attempt to account for alternative approaches. But all too often the cultural norms are so ingrained that we forget they’re there. We assume that they’re normal. We assume that they’re shared. And then we’re surprised when Australians balk at receiving the ‘summer’ release of software in June, when Americans don’t consider that piece of personal data to be sensitive, or when Europeans think Government really should be involved in regulating a social networking site.

If we want to ensure the unimpeded flow of data across borders — and we should — then we need to begin by recognising that the places and people on either side of that border are very likely to be different. Their attitudes are different. Their needs are different. Their aspirations are different. Their laws are different. In the early days of the Internet and the web, legislation, policy and even expectation did not really exist. Almost by default, US attitudes and presumptions tended to apply unless a particular country cared enough to institute something different inside their own borders. As we become more and more interested in territoriality and jurisdiction with respect to data, that naive innocence no longer applies. There is no longer a blank canvas upon which the innovators can paint their hopes. We have policies, regulations, and laws. We have populations that have experienced today’s web, and we have a media quick to interject its perspective.

Today, far more than at the web’s birth, we have to engage in public dialogue about what we want to achieve, and why. We cannot achieve Derrick’s aspiration of free trade for data and leave every local law, policy and procedure untouched. But nor can we achieve it by projecting a single world view around the globe, sweeping all of those prior laws aside.

Rather than entrench behind a US ‘position,’ a European ‘position’ (including 24 variant positions, two abstensions and an opt-out), a Chinese ‘position,’ and so on, can’t we begin to understand what some of the real opportunities — and concerns — might be?

Big companies that operate internationally (like those developing the document (PDF) Derrick discussed) absolutely need to come together, to share their perspectives on the pain of moving data around. But for those companies only to be American is insane, and counter-productive. Nokia has perspectives to share here, as do HSBC or Vodafone. Let’s hear them in the same forum. Let’s also hear individual governments, speaking up for the concerns and desires of their citizens. Let’s hear the citizens themselves, when they care enough to express an opinion. But let’s hear all of it early, before it becomes entrenched in a set of contradictory official statements.

Then we might arrive at a sensible approach to ensuring free trade for data, rather than the projection of an American ideal upon the rest of us.

And that sounds like a good topic for one of those panels at Davos later this month…

Related articles

• Advancing the free flow of information (googlepublicpolicy.blogspot.com)
• PATRIOT Act clouds picture for tech – POLITICO.com Print View (mbcalyn.com)
• Border accord (economist.com)
• ANALYSIS: What the new border deal means for you (cbc.ca)

The PATRIOT Act’s acronymic name may evoke harmless images of bunting, parades, and national anthems, but the reality is rather different. A product of America’s post-9/11 entrenchment, the Uniting (and) Strengthening America (by) Providing Appropriate Tools Required (to) Intercept (and) Obstruct Terrorism Act of 2001 affords the Federal Government wide-ranging and far-reaching powers that show little — if any — respect for geographic boundaries or inconveniently contradictory local legislation. A US company (like Microsoft or Amazon) is subject to the Act’s powers all around the world. A US citizen’s data, stored in a US company’s data centre that is physically situated in the United States is subject to the Act, and everyone might be reasonably comfortable with that. But so is a German citizen’s data, stored in an Amazon data centre in Ireland; and German, Irish and European lawmakers appear almost powerless to intercede.

European countries tend to be stricter about use (and abuse) of personally identifiable information than the US. Although surveys identify some national differences, it also appears that Europeans broadly embrace the approach taken by their governments. And, anecdotally, conversations with European and American entrepreneurs and European and American individuals repeatedly point to rather different sets of basic presumptions operating on either side of the Atlantic. Europe’s Data Protection Directive, and its implementation in national legislation such as the UK’s Data Protection Act, are clear about the ways in which a citizen can expect data about themselves to be collected, stored, shared and used. The penalties for intentional abuse could probably be tougher, but the sentiment remains clear. The Safe Harbor Principles provide mechanisms by which US companies can self-certify that their normal operating procedures meet European standards (Microsoft, Google, Amazon and many others do this). The February 2010 ‘model clauses‘ that Microsoft embraced last month codify some of these protections in a manner that — theoretically — makes it easier for customers’ lawyers to understand what Microsoft will do with their data. It’s unlikely that the legalese (PDF) will actually make things any clearer for the average customer, though.

So, from the perspective of Europe’s governments and citizens, and for US companies that choose to trade here, things appear more or less ok. Personally identifiable data can be collected, stored, shared and used, but only within a set of constraints that Europeans broadly seem comfortable with. Unfortunately, all those Safe Harbor self certifications and model clause endorsements are summarily ignored whenever the PATRIOT Act is invoked. Data Protection Directive requirements not to transfer data to random third parties are trumped by PATRIOT Act powers enabling the US Federal Government to take what it wants. Data Protection Directive stipulations that citizens be informed when their data are taken are over-ruled by the PATRIOT Act’s cloak of secrecy. And on and on the list of contradictions continues. And the PATRIOT Act wins every time, because its powers, its penalties, and its backers are so much scarier than the officials in Brussels. Despite tougher language, it’s not clear that sweeping changes to Europe’s data protection directive will really resolve the contradictions. Indeed, once enshrined in law, the proposals will most likely result in more polarisation, not less.

In Europe too, of course, there are exemptions to the data protection legislation specifically intended to permit reasonable use of data by law enforcement agencies and others. This makes sense, and it could be argued that the PATRIOT Act is simply more of the same. But it’s not, because European law enforcement agencies must demonstrate a far clearer need before they’re allowed to — legally — start rooting through a citizen’s data.

It is unlikely that the PATRIOT Act is routinely invoked, or that US officials spend much time reading Europeans’ email. The cloud — even the parts run by US companies — remains broadly safe, secure, and reliable. Safe Harbor provisions, model clauses, and the ability to insist that data normally resides in one territory or another remain an effective means of ensuring that day-to-day cloud operations meet user needs whilst complying with relevant local, regional and international legislation. But, every now and again, the PATRIOT Act will be invoked, and data will be taken. Whilst it’s something to be aware of, it’s probably not something for most people to lose too much sleep over. You’re more likely to lose data yourself, or have it escape into the wild because of an error in your own systems or a malicious hack by a competitor. And you could and would be held accountable for those breaches, in a way that you almost certainly wouldn’t for a PATRIOT Act data seizure.

So the PATRIOT Act may not be as scary as it might now appear. But it remains a visible illustration of a rather more worrying issue; a belief that the laws of one country should be able to trample over the laws of other countries at will — even inside those countries. Further, it suggests a (growing?) disconnect between the attitudes and expectations on either side of the Atlantic. And one particular aspect of that is the subject for my next post.

Image by Flickr user John Morgan

Related articles

• Microsoft’s European ‘cloud pact’ still does not protect data against FISA, Patriot Act (zdnet.com)
• E.U. Regulations Become Microsoft Cloud Selling Point (blogs.wsj.com)
• Microsoft Pushes Back on EU Cloud Concerns as European Rivals Move In (wired.com)
• Defense giant ditches Microsoft’s cloud citing Patriot Act fears (zdnet.com)
• PATRIOT Act and privacy laws take a bite out of US cloud business (arstechnica.com)
• Microsoft boosts Office 365 regulatory compliance (infoworld.com)
• Updated European law will close Patriot Act data access loophole (zdnet.com)
• Exclusive: European data protection law proposals revealed (zdnet.com)

Image by fatboyke (Luc) via Flickr

I am in the Belgian city of Brussels at the moment, which means that my mobile phone is ‘roaming;’ off my UK network and being charged a scary amount of money to access data. Travelling to Europe is less scary than going elsewhere in the world, as I’m ‘only’ charged about £3 per Mb here. That’s half what I’d be charged to use data in the United States, but is still an obscene amount of money for my mobile phone company to be extracting from me.

There are plenty of excuses made about termination fees and cross-border this, and intra-jurisdictional that, but at the end of the day a very small number of large telcos are getting away with what is essentially a scam. Much of the time, I’m actually connected to the local network of my own mobile phone company, rendering many of their excuses even more pathetically irrelevant than they might otherwise be.

Still, this post isn’t meant to be a complaint about mobile phone companies. I have every faith in the European Commission, and its power to bring more pressure to bear in reducing those charges. I just wish they’d get on with it.

What this post was meant to be was a complimentary post about a new startup that’s attempting to do something about this. And there’s a cloud connection, too.

Enter Onavo, a startup that uses a small app (iPhone only, for now) to reduce the amount of data you consume when roaming; thereby lowering the size of the bill your mobile phone company sends you. The company claims reductions in data consumption of “up to 80%” in optimal circumstances. I’m not seeing that big a saving, but mine is still significant.

Basically, when you turn Onavo on for a trip and start roaming, the company’s servers step in and start compressing images etc before they are delivered to a mobile carrier and — via the mobile carrier — to your phone. Compressed content means smaller content, which means less bandwidth used, which means smaller bills for me. Should you happen to find affordable wifi, Onavo gets out of the way and the uncompressed content again becomes available to you.

If you’re going to be travelling, have an iPhone, and want to give it a try, the company has given me some invite codes to share. Simply visit www.onavo.com/codata using Safari on your iPhone, and the first 100 will get into the beta programme. It’s working well for me, and might for you too.

Onavo is not paying for this post; they let me into the beta programme, I found it useful, and now I’m telling you that I did.

And the Cloud connection? Onavo’s co-founder is Guy Rosen; he of the Jack of All Clouds blog, and those invaluable State of the Cloud graphs. Speaking of which – aren’t we due another one, Guy?

Related articles

• Maxroam Signs Airline To Disrupt EU Cell Roaming For 78 Million Passengers (techcrunch.com)
• Tighter regulation of mobile data roaming casts shadow over Barcelona from Brussels (robbratby.com)
• EU wants to bring down mobile phone roaming charges (newsinfo.inquirer.net)

A short report that I was commissioned to write for the European Public Sector Information Platform has just been published.

The rise of the App: a PSI opportunity? introduces (smartphone) apps and app stores to those in European governments responsible for meeting their obligations under the 2003 Public Sector Information (PSI) Directive.

Unfortunately somewhat tangential to the more recent (and cooler?) enthusiasm for Open Data, governments’ compliance with the PSI Directive has largely failed to engage the community of active and enthusiastic developers who might build compelling tools atop all that data.

Although elected officials might love showing their friends a council-branded iPhone app that knows where all the publicly-funded swimming pools within a single local government area are located, is that really a useful tool for anybody? Would it not be more useful to see the data made available in forms, formats and locations regularly frequented by communities of third party developers? Then you might see all the swimming pools, you might cross (mostly meaningless) local government boundaries, and you might pull in other leisure activities, so that a real user can ask ‘where can I swim?’ or ‘where can I go and have some fun?,’ instead of the rather unlikely ‘where can I swim in a council swimming pool?’ If you care that much about swimming in the pools of your local council, won’t you know where they are? And if you don’t, how likely are you to download an app to answer the question? Once you’ve answered it once (surely a Google query, rather than an app download) the app is useless.

As well as being formally released via council, region, agency and national web sites, should freely reusable public sector data not be actively contributed to Factual, Infochimps and the like? So long as a free copy is available somewhere, is it really a problem if someone else can take that data, add value to it, and make a little bit of money?

Public Sector Information is wide-ranging, comprehensive, and authoritative. It is truly insane for these rich resources not to underpin a wealth of applications originating in both the public and private sectors.

All we need to do is abolish some of the weirder licensing restrictions, disabuse some governments of the idea that PSI will make them rich, and make the data easy to find, easy to select, easy to get, easy to integrate, and easy to keep current. Easy, huh? Let’s do it.

The European Commission recognises that the PSI Directive is due a refresh, and is currently consulting on next steps in this area.

Related articles

• Europe begins revamp of rules on re-use of public info (go.theregister.com)
• Transport for London locks app users out of online travel data feed (guardian.co.uk)
• “Shiny app syndrome” and Gov 2.0 (radar.oreilly.com)

As European readers are doubtless aware, the EC has traditionally been a generous funder of research across Europe’s member states, with Digital Libraries, the Semantic Web and more owing much to the largesse of Europe’s massive ‘Framework Programme‘ funding cycles. We’re currently in the midst of the Seventh Framework Programme, and a few hundred of the academics and technologists hoping to secure some of the €Millions available for ‘Technologies for Information Management‘ have gathered in soggy Luxembourg to hear what they can bid for, to hear how to bid, and to engage in the funding world’s rather bizarre equivalent of Blind Date by pitching their wares to prospective partners.

I’ve been asked to talk about some of the trends and issues around ‘Big Data,’ to provide a context for the technological discussions to follow, and to illustrate some of the ways in which cutting edge implementations of the sort likely to be proposed might solve real problems.

My slides are on Slideshare and embedded, above, and begin by suggesting that the use of ‘the language of catastrophe’ in describing the ‘flood’ of information around us perhaps sets everything off on the wrong foot. Yes, there’s a lot of information out there… but there’s not too much, and maybe we shouldn’t be attempting to control all of it anyway…

European Commission-funded projects were amongst the first to make serious attempts to ‘control’ and ‘manage’ the early Web, with some suggestions that we could (and should) catalogue Web pages just like books in a library. Some of those laboriously curated, instantly obsolete, and hopelessly under-representative Web ghettos still exist today; but the mainstream Web has moved far beyond them, embracing more scalable and effective combinations of machine processing and lightweight community recommendation. Even in their heyday, those for whom these resources were created were all too often to be found applying their efforts to routing around these obstructions to the free flow of information across the Web.

As the volume of data available to us grows, it presents massive new opportunities as well as significant technological and social challenges. Twitter is just one example of the rise of the ‘real-time’ Web, and connected devices such as the iPhone and Google’s Android devices fundamentally shift the ways in which we consume and contribute to the ever-accelerating flows of data.

Use, re-use and control of data, too, are increasingly topical issues with which we should be concerned. The rise of licensing frameworks such as CC0 and the Open Data Commons are part of an attempt to reduce ambiguity in the ways that data may be repurposed, and expectations grow daily that data will be available; whether from Government, community groups or the private sector. Privacy, security, provenance and trust all come into play, with a path to be diplomatically steered between those too blasé to recognise the real issues at stake and those too cautious to countenance progress perceived to be at their expense.

Moving the quantities of data involved is becoming a serious challenge, too, and a generation of researchers accustomed to ‘simply’ throwing data into the Cloud for later analysis, sharing or retrieval must increasingly grapple with latency and bandwidth. Physical proximity of data to computation might actually matter again, and more than one of the Cloud’s biggest players have been heard to suggest that physical media might actually be the most cost-effective way to move these mountains of data around the globe.

I look forward to seeing what the nascent projects likely to coalesce over the next 24 hours contribute to understanding and progress with any or all of these… and hope that my presentation plays its part in shaping their thinking, their proposals and their outcomes.

Related articles by Zemanta

• One More Step Towards the European Digital Library (twidox.com)
• ‘Mixed results’ from EU R&D funds (news.bbc.co.uk)
• Semantic Web and Digital Libraries Conference (cont.) (ivan-herman.name)

On the face of it, there surely can’t be many places more ready for the Semantic Web than the world’s great museums. Vast quantities of richly structured data describing visually compelling objects that people already flock to see? A raison d’etre that is about sharing information and enhancing understanding? A tradition, stretching back several hundred years, of describing things clearly?

If my conversation with Koven Smith of New York City’s Metropolitan Museum of Art is any indication, though, the truth would appear rather different.

He talks, depressingly but all too familiarly, of vendor lock-in, competing institutional priorities, and a tradition of remarkable insularity within institutions that really should be more open.

Even in the face of this, though, Koven is able to describe work at the Met to leverage third party tools such as Open Calais and Semantic MediaWiki in enriching possible interactions with the museum’s institutional memory.

Have a listen, and see what you think.

I am aware of similar work at Amsterdam’s Rijksmuseum and behind the European Commission‘s struggling Europeana, and should try to secure similar conversations with them. Is anyone aware of other examples?

Related articles by Zemanta

• How will we interact with all this data?