In many ways, the core concept of the data marketplace is nothing new. Companies like Bloomberg, Nielsen and Experian have built (extremely) profitable businesses by aggregating data, quality checking it, and selling it on. Often their customers could have gone directly to the source(s) and paid far less, but they don’t. The convenience and quality assurance of dealing with a single — reputable — source is perceived to have value. A brand like Bloomberg’s is associated with trustworthiness and authority, and the brand of the marketplace is far more prominent than the data sets upon which it is built.

Similar sites have also served the needs of those seeking data for free, with IBM’s ManyEyes project, Freebase (acquired by Google), Hans Rosling’s Gapminder or The Guardian‘s Data Store amongst those typically mentioned. Current government enthusiasm for ‘transparency’ has fed all of these sites with data, and led to creation of large government-specific data repositories such as data.gov.uk.

The commercial services like Bloomberg have tended to focus upon specific domains (finance, in Bloomberg’s case) or types of data. They have also tended to be eye-wateringly expensive; aimed squarely at the small market segment for whom the data are mission-critical and the fees are affordable. The free services like Gapminder also tend to focus (global development statistics in this case). Other, perhaps, than experiments like ManyEyes, both the free and the commercial sites tended to aim for a degree of comprehensiveness and authority. They wanted to become the place to turn for their type of data.

But for the new generation of data markets, the picture becomes far less clear. They tend to be catholic in their data acquisition policies, they typically don’t even attempt comprehensiveness, they mix free (almost all of them hold identical large swathes of government data from the US, the UK, and elsewhere) with commercial data, and they continue to feel their way toward business models that might prove sustainable for the long haul. Perhaps more seriously, they appear almost schizophrenic with respect to brand projection, attempting to push both their own brand and those of the data sets they host in ways that can confuse far more often than they enlighten.

In attempting to differentiate themselves, today’s data markets are seeking to add features and functionality in order to be seen as far more than simply places to buy third-party data. They want to become recognised for quality assurance, for data enrichment, or for tools and capabilities that make working with the data easier or more powerful. They want to become sticky, and they want to be seen as different from their competitors. The trick, though, is to explain those features and those differences in ways that make sense to potential customers. Those customers will ultimately pay for functionality and utility, not for gimmicks or under-the-hood technological distinctions that have no real impact upon getting on with the job in hand. Are today’s data markets describing their features in ways that help prospective customers to understand why they should be chosen over the alternatives? Not really. At least, not yet.

Also, as RedMonk’s Stephen O’Grady touched upon amongst a set of related issues, we’ve really not begun to see much evidence of price competition. There are too few suppliers, each with their fiercely loyal bands of tame users (‘customers’), and too few people prepared to shop around for the best deal.

The new data markets are still young. Understandably, they are still feeling their way in order to understand what the market wants, how much it is prepared to pay for what it wants, how large the market might be, and what their individual niche within that broader market might look like. Earlier models, based upon almost monopolistic domination of specific verticals and polarised pricing, offer some lessons but are ultimately unsatisfactory blueprints for this more competitive, open, and complex environment. Beyond specific domains like finance (which may be ripe for disruption), the data markets must struggle to convince prospective customers that they have something of value to offer. Those customers may already have their own processes for obtaining data. They may generate the data themselves, or expect — as so many do — to be able to access what they need for free. They are perhaps suspicious of data produced by third parties who are, in other contexts, their competitors, and they are almost certainly unwilling to allow ‘the competition’ to benefit from their own data. They invariably do not understand the costs associated with gathering and quality-assuring data, or the challenge of preparing different data sets in order that they may meaningfully be combined. And into this, the fledgling data markets must insert themselves, market themselves, and sell themselves. They must change behaviours, they must challenge presumptions, they must alter working practices, and they must persuade their new customers that all of this pain is worth paying for. A tall order, indeed, but necessary if any of them are to realise their potential.

The European Commission, at least, begins to comprehend the scale of the challenge. A set of projects are currently being finalised, and this year will see European SMEs given the funding to boot-strap a number of new data sources. With Commission funding, it is hoped, the chosen projects will be able to explore models by which data can be created, curated, shared and re-used in a manner that is cost-effective and ultimately sustainable. The funding should enable these projects to reach viable scale, and give participants the freedom to explore alternative commercial models. The projects will be announced shortly, but only time will tell if the funding and the incentives are sufficient to break through the barriers that prevented any of these markets from forming by themselves.

But outside the rather artificial bubble created by European public funding, there is a lot of work to do. Investors are intrigued by — but still wary of — the opportunity. Infochimps is spending its way through over $1.5 million of investment, Factual has almost $30 million, and companies like Talis and Microsoft are making not-insignificant investments in their own efforts. We’re all still experimenting, but with the real market for these services currently falling far short of the money at stake, it mustn’t be long before investors start asking harder questions. Back in 2010, Pete Soderling and Pete Forde described data as a $100 billion market. The data markets may be after a significant chunk of that but, today, they’re not even close.

The ways that data markets are attempting to differentiate themselves, and the work being done to understand the market opportunity here, will have to wait for subsequent posts.

Disclosures: I am a former employee of and current shareholder in Kasabi’s parent company, Talis. The European Commission is, from time to time, a client.

Related articles

• With Factual, 1 API now unlocks data for 55 million places (gigaom.com)
• ‘Roswell’: Another key component of Microsoft’s cloud strategy (zdnet.com)
• Infochimps Acquires Y Combinator Startup Data Marketplace, Expanding Brand Holdings and Online Presence (prweb.com)

The PATRIOT Act’s acronymic name may evoke harmless images of bunting, parades, and national anthems, but the reality is rather different. A product of America’s post-9/11 entrenchment, the Uniting (and) Strengthening America (by) Providing Appropriate Tools Required (to) Intercept (and) Obstruct Terrorism Act of 2001 affords the Federal Government wide-ranging and far-reaching powers that show little — if any — respect for geographic boundaries or inconveniently contradictory local legislation. A US company (like Microsoft or Amazon) is subject to the Act’s powers all around the world. A US citizen’s data, stored in a US company’s data centre that is physically situated in the United States is subject to the Act, and everyone might be reasonably comfortable with that. But so is a German citizen’s data, stored in an Amazon data centre in Ireland; and German, Irish and European lawmakers appear almost powerless to intercede.

European countries tend to be stricter about use (and abuse) of personally identifiable information than the US. Although surveys identify some national differences, it also appears that Europeans broadly embrace the approach taken by their governments. And, anecdotally, conversations with European and American entrepreneurs and European and American individuals repeatedly point to rather different sets of basic presumptions operating on either side of the Atlantic. Europe’s Data Protection Directive, and its implementation in national legislation such as the UK’s Data Protection Act, are clear about the ways in which a citizen can expect data about themselves to be collected, stored, shared and used. The penalties for intentional abuse could probably be tougher, but the sentiment remains clear. The Safe Harbor Principles provide mechanisms by which US companies can self-certify that their normal operating procedures meet European standards (Microsoft, Google, Amazon and many others do this). The February 2010 ‘model clauses‘ that Microsoft embraced last month codify some of these protections in a manner that — theoretically — makes it easier for customers’ lawyers to understand what Microsoft will do with their data. It’s unlikely that the legalese (PDF) will actually make things any clearer for the average customer, though.

So, from the perspective of Europe’s governments and citizens, and for US companies that choose to trade here, things appear more or less ok. Personally identifiable data can be collected, stored, shared and used, but only within a set of constraints that Europeans broadly seem comfortable with. Unfortunately, all those Safe Harbor self certifications and model clause endorsements are summarily ignored whenever the PATRIOT Act is invoked. Data Protection Directive requirements not to transfer data to random third parties are trumped by PATRIOT Act powers enabling the US Federal Government to take what it wants. Data Protection Directive stipulations that citizens be informed when their data are taken are over-ruled by the PATRIOT Act’s cloak of secrecy. And on and on the list of contradictions continues. And the PATRIOT Act wins every time, because its powers, its penalties, and its backers are so much scarier than the officials in Brussels. Despite tougher language, it’s not clear that sweeping changes to Europe’s data protection directive will really resolve the contradictions. Indeed, once enshrined in law, the proposals will most likely result in more polarisation, not less.

In Europe too, of course, there are exemptions to the data protection legislation specifically intended to permit reasonable use of data by law enforcement agencies and others. This makes sense, and it could be argued that the PATRIOT Act is simply more of the same. But it’s not, because European law enforcement agencies must demonstrate a far clearer need before they’re allowed to — legally — start rooting through a citizen’s data.

It is unlikely that the PATRIOT Act is routinely invoked, or that US officials spend much time reading Europeans’ email. The cloud — even the parts run by US companies — remains broadly safe, secure, and reliable. Safe Harbor provisions, model clauses, and the ability to insist that data normally resides in one territory or another remain an effective means of ensuring that day-to-day cloud operations meet user needs whilst complying with relevant local, regional and international legislation. But, every now and again, the PATRIOT Act will be invoked, and data will be taken. Whilst it’s something to be aware of, it’s probably not something for most people to lose too much sleep over. You’re more likely to lose data yourself, or have it escape into the wild because of an error in your own systems or a malicious hack by a competitor. And you could and would be held accountable for those breaches, in a way that you almost certainly wouldn’t for a PATRIOT Act data seizure.

So the PATRIOT Act may not be as scary as it might now appear. But it remains a visible illustration of a rather more worrying issue; a belief that the laws of one country should be able to trample over the laws of other countries at will — even inside those countries. Further, it suggests a (growing?) disconnect between the attitudes and expectations on either side of the Atlantic. And one particular aspect of that is the subject for my next post.

Image by Flickr user John Morgan

Related articles

• Microsoft’s European ‘cloud pact’ still does not protect data against FISA, Patriot Act (zdnet.com)
• E.U. Regulations Become Microsoft Cloud Selling Point (blogs.wsj.com)
• Microsoft Pushes Back on EU Cloud Concerns as European Rivals Move In (wired.com)
• Defense giant ditches Microsoft’s cloud citing Patriot Act fears (zdnet.com)
• PATRIOT Act and privacy laws take a bite out of US cloud business (arstechnica.com)
• Microsoft boosts Office 365 regulatory compliance (infoworld.com)
• Updated European law will close Patriot Act data access loophole (zdnet.com)
• Exclusive: European data protection law proposals revealed (zdnet.com)

Formed as a partnership between Microsoft and Accenture, Avanade focuses upon delivering IT solutions based upon Microsoft’s suite of technologies and products, including Cloud offerings such as Azure and the company’s Business Productivity Online Standard Suite (BPOS).

I discussed Azure in a podcast with Microsoft’s Amitabh Srivastava last year, and in this latest conversation Scott and Bob share some of the experiences Avanade has gained in rolling out over 1.3 million BPOS seats to a wide range of enterprise clients.

This podcast was recorded on Wednesday 28 July, 2010.

During our conversation, we referred to the following resources;

• Accenture
• Amazon
• App Fabric
• Avanade
• Avanade Cloud Computing
• Azure
• BPOS
• Hotmail
• Java
• Lotus Notes
• Microsoft
• Microsoft Live@Edu
• PHP
• Werner Vogels

Related articles by Zemanta

• Microsoft shares (officially) its future BPOS plans (zdnet.com)
• Capgemini now championing Microsoft BPOS over Google Apps (zdnet.com)
• Microsoft launches BPOS Syndication Partner programme (channelweb.co.uk)
• Microsoft gives partners free cloud tools (infoworld.com)
• Is Sharepoint 2010 Cloud Ready? (readwriteweb.com)

“All too often, insider analyst research and commentary is costly, inaccessible or ineffective to the entrepreneurs, investors and innovators who need timely insights most. We’re changing this by making high-quality expert reports, data and commentary easy to access and highly interactive for our community in key emerging technology areas.”

I was therefore delighted when VP of Research Michael Wolf got in touch before Christmas to ask if I’d be interested in providing them with content.

My first piece, which is published today, explores some of the issues faced by Cloud service providers and users in negotiating the attitudes and laws of the different jurisdictions through which their data may pass. I would welcome any comments on the piece, and must thank Microsoft‘s Matt Deacon, Sun Microsystems‘ Michelle Dennedy, IBM‘s Harriet Pearson, Rackspace‘s Simon Abrahams, Computacenter‘s Matthew Yeager and F5 Networks‘ Lori MacVittie for taking the time to talk with me as I prepared it. Thanks, also, to Celeste LeCompte at GigaOM for editing and for Americaniszing my spellingz… Any misunderstandings, misrepresentations, typos etc, are of course my own.

I look forward to my next GigaOM assignment!

For those on this side of the Atlantic with a particular interest in the topic, London’s Powered By Cloud event (where I participated in a panel last year) is back for a second edition on 8 and 9 February. Of particular interest in this context is an addition to this year’s event; 10 February’s dedicated to a Cloud Law Summit.

Related articles by Zemanta

• GigaOM Pro Shakes up Market Research Industry with Innovative Program (prweb.com)
• GigaOM Pro Launches Analyst Relations Program – Get a Free Pro Subscription (jkontherun.com)
• Introducing GigaOM Pro’s Analyst Relations Program (gigaom.com)

It might seem that the mega-bucks reports from the likes of Gartner, Forrester et al are the preserve of CxOs with vast desks upon which they can array the multitudinous documents to which their employers’ subscription entitles them. The truth, though, is that these documents — which notionally sell for hundreds or thousands of dollars apiece — can often be downloaded in exchange for some contact details. The source of this apparent largesse is not the analysts’ own website, but rather the sites of companies looked upon with particular favour within a given report. The back-room exchanges of kudos, cash, or mutual back-slapping that sees these assets made freely available are well understood, and legitimate.

And so it was with a recent report from Forrester, made freely available to all via the web site of 3Tera; the co-founder of which I spoke with recently in one of my podcasts.

The report is ‘Deliver Cloud Benefits Inside Your Walls,’ dated 13 April 2009, and produced by Forrester’s James Staten with input from Simon Yates, John Rymer, Frank Gillett and Lauren Nelson. Billed as “the first document in the ‘Private Cloud’ series,” it would appear that Forrester, at least, has no hang-ups about the notion of a Private Cloud. Whilst some purists become incredibly agitated about this ‘dilution’ of their dream, realists, pragmatists and (it would appear) analysts are simply getting on with it.

According to the report’s Executive Summary,

“While the excitement about cloud computing centers on Amazon Elastic Compute Cloud (EC2) and other public infrastructure-as-a-service products, many enterprise infrastructure and operations professionals are taking this concept in-house and building their own internal clouds. These pools of virtual machines can be built upon either virtual server or high-performance computing (HPC) grid foundations and can be operated according to the specific security and process requirements of the business. But to deliver the fundamentally better economic value of cloud architectures within your walls, these clouds require a dynamic platform (or automated workload management) and developer self-service interfaces. There’s a growing list of vendors eager to help you deploy an internal cloud, but be sure you understand that these solutions are more building blocks than complete solutions and must be customized to your specific needs.”

It would be impossible to cover the space in much detail in a report comprising just nine pages of substantive content, but Staten and his co-authors do a reasonable job of outlining some high level benefits of internal deployment whilst also flagging some of the issues in need of addressing. Prominent amongst these is the suggestion that ‘bypassing IT Ops’ to hand developers their own internal Cloud is not necessarily something to be done lightly;

“Whether you buy the hype or not, enterprise application developers are finding the self-service, pay-as-you-go, instant deployment values of cloud computing platforms appealing. Developers can go to a Web page, sign up with a credit card, and instantly instantiate any number of virtual machines and applications without any IT ops involvement. Interviews conducted by Forrester show that many
enterprise developers are doing just that.

But IT ops processes and procedures — and enterprise architecture rules for that matter — exist to ensure that the overarching needs and policies of the business are fulfilled and followed. Although making time in the deployment process to accommodate these demands may hinder time-to-market, often there are very good reasons to do so”

This is true, but overlooks the reality that those processes and procedures arose to manage a very different environment. There is a middle ground to steer here, and it is one that should make a great deal of sense. Virtualisation, elasticity and more create significant opportunities for the cost-effective provision of Enterprise IT. Rather than simply accepting the status quo of those established processes and procedures, or routing around them with the departmental credit card and some cheap Dells or EC2 instances, there is a real opportunity for IT and business teams to engage in fresh dialogue; to understand the match between changing expectations, changing requirements, and changing possibilities.

A throwaway comment also points, tellingly, to the real need for organisational change if we are to optimise the benefits of these new opportunities;

“We can stand up a LAMP stack or Windows VM within 24 hours now — one hour to provision the VM and 23 hours to move the money.”

Elsewhere in the document, the Forrester team suggests that,

“With today’s virtual infrastructure solutions and a growing list of internal cloud platform technologies, it’s fairly easy for an enterprise to start up a cloud-like environment within its own domain.”

The document goes on to suggest that, in Q3 of 2008, around 4% of surveyed enterprises in Europe and North America had done so, with a significant number implementing or seeking budget. More than half of those surveyed, though, were either ‘Not Aware’ or ‘Not Interested’ in an Internal Cloud. For ‘small businesses’ of fewer than 100 employees that figure rose to a massive 80%. It would be interesting to see the same figures for ‘Public Cloud’ utilisation… although there is likely to be significantly more under-reporting of public Cloud use as so much of it will be by individuals and teams who are below the corporate radar.

With one of the significant cost reductions in the public Cloud being directly related to more efficient utilisation of virtual and physical machines, the corresponding saving within the enterprise — whilst still significant — is likely to be smaller, and there may still be a need for a hybrid arrangement to permit ‘cloud bursting’ at times of particularly high load.

The document draws out three ‘internal Cloud solutions’ — 3Tera, Elastra and Zimory — all of which are worth a look, and then goes on to very briefly touch on three alternative approaches. Here, more than elsewhere in the document, Forrester seem not to go into enough detail. They touch upon a DIY approach based upon EUCALYPTUS (now available in commercial form, of course, although the authors don’t mention this), contracting with a systems integrator such as IBM to secure a bespoke solution, and ‘waiting for the major virtualisation vendors to show up.’

There’s an awful lot of activity already underway from these companies, with whom enterprises probably already have a relationship, and it seems unhelpful to pass over them so quickly. What about Windows Azure, for example, which Microsoft is increasingly suggesting should be available for local utilisation?

So, on the whole this document is a useful overview. As might be expected in something of this length, it ends up raising more questions than it answers (which drives Forrester customers back to their Analysts, of course), and there are one or two areas in which it leaves odd gaps. Maybe the next document in the series will begin to fill some of those gaps?

Related articles by Zemanta

• SaaS And Cloud – A Strong Combination (teabreak.pk)
• Forrester Backs Private Clouds – Will Others Follow Suit? (gigaom.com)
• Cloud computing shapes up as big trend for 2009 (infoworld.com)
• Is Your Future in ‘The Cloud’? (usnews.com)
• Cloud Computing’s Three-Horse Race (gigaom.com)
• Cloud computing — value is an assumption, but cost matters (news.cnet.com)

Everyone who wants to do so should have had their chance to read the Open Cloud Manifesto by now, and to see the list of companies putting their names to it. As expected, Microsoft, Google and Amazon are not there.

Reuven Cohen of Enomaly is one of those involved in bringing the Manifesto to fruition, and I spoke to him on Thursday evening last week to hear more.

Related articles by Zemanta

• A Cloud Manifesto Controversy (blogs.wsj.com)
• Amazon, Microsoft reject ‘Open Cloud Manifesto’ (news.cnet.com)
• Cloud Computing Manifesto ? (As If We Need One) (lockergnome.com)

Long known for their dominance on the desktop, Microsoft is now making a concerted effort to carve out a space of its own in the Cloud with Windows Azure. Amitabh Srivastava, the company’s Corporate Vice President with responsibility for Azure, spoke with me this week and the result has just been released as one of my podcasts.

Production of this podcast was supported by Talis, and show notes are available on their Nodalities blog.

Amitabh talks about the company’s aspirations for Azure, and reports on the success of the current ‘Community Technology Preview;’ effectively a beta programme for Azure ahead of its formal launch towards the end of this year. We also discuss the nature of Azure, digging into the question of where Azure will be deployed, and the extent to which it will interact with desktops and data centres beyond Microsoft’s control.

In line with recent announcements from the likes of 3Tera, Amitabh stresses the importance that Microsoft sees in a comprehensive Service Level Agreement (SLA) for Azure.

Have a listen, and see what you think.

Related articles by Zemanta

• Microsoft cloud to get ‘full’ SQL Server soon? (theregister.co.uk)
• Microsoft updates Windows Azure tools (news.cnet.com)
• Microsoft: Azure not pie in the sky (news.cnet.com)
• Ballmer: Azure ready for release by end of year (infoworld.com)
• Microsoft to Lead the Cloud Computing Market (themindstorms.blogspot.com)
• Beware the dazzle of Microsoft’s Azure lure (theregister.co.uk)

Open, Interoperable services in the Cloud. Sounds like a no-brainer, eh? Well over the past 24 hours or so, a plethora of blogs, mailing lists and social networks have been getting extremely agitated over some nascent efforts intended to support exactly that. And, what’s more, the vast majority of those getting most agitated haven’t even seen the document in question.

I have seen the document. I have spoken with some of the people behind it, and I even have at least one podcast discussing the Manifesto and its aspirations. But those conversations were conducted under embargo, and I respect embargoes. I’ll be releasing my formal coverage of the Manifesto when that embargo lifts next week.

I can talk about the current storm without breaching those terms, and try to cut through some of the name-calling and innuendo to shine a little light on what’s actually going on.

This all began with a blog post by Steven Martin. Steven works at Microsoft, leading to commentary suggesting that;

“Earlier today Microsoft savaged an undisclosed effort to develop an ‘Open Cloud Manifesto’”

and

“Microsoft criticizes drafting of secret ‘Cloud Manifesto’”

Microsoft is a big organisation, and one that has long held the extremely enlightened view that its employees should be allowed to express their own opinions. Steven works for Microsoft. That doesn’t necessarily mean that when he writes on his blog Microsoft is speaking, despite his relatively senior position.

Sam Johnston then joined the fray (subsequently followed by most of the big tech sites on the blogosphere, the majority of which seemed to simply take the original post at face value, attribute it to the Redmond giant, and apply a healthy dose of pro/anti-Microsoft bias to the mix in order to increase the confusion.)

Both Steven and Sam make a number of extremely valid points, but these have been buried under the Microsoft-bashing directed at Steven by some and — especially — a readily apparent history of tension between Sam and Enomaly co-founder Reuven Cohen, one of those behind the Manifesto.

Sam goes on to establish a competing effort, the ‘Cloud Computing Manifesto.’ Whilst there may be a need for such a move if the community is not served by the Open Cloud Manifesto, to divide and confuse just days before we’ll all be able to read the first public draft of the Open Cloud Manifesto and decide for ourselves seems unhelpful.

By next week, everyone will be able to read the Open Cloud Manifesto. By next week, everyone will be able to see the companies that have put their weight behind it. By next week, everyone will be able to see the companies that chose not to support it, and the companies that — whether by conspiracy or otherwise — were not invited into the founding group.

Whether we like it or not, the politics of aligning powerful and competing interests often leads inevitably to the need for conversations in back rooms, and to a process in which some are earlier to the table than others. Having been involved in the evolution of quite a number of de facto and de jure standards and specifications, I’m well aware that there are a plethora of pros and cons associated with most processes for reaching consensus. The fact that this particular effort began life as a series of quiet conversations with movers and shakers does not mean that it’s inevitably a conspiracy or an attempt to bully the community into line.

Let’s stop second guessing the motives of its instigators and the wording of the document, take a weekend to do something more important, and then show those behind the Open Cloud Manifesto the courtesy of reading their document when it’s published. At that point, if there are things to criticise, then we can do so. But equally, if there are points to support and celebrate then we must do that to.

This endless speculation and innuendo helps none of us, and I expect that I can — easily — count the commentators and stirrers who have read the document on the fingers of one hand.

Related articles by Zemanta

• Thunder in the Cloud Over Openness (gigaom.com)
• The Pot Calling The Kettle Black (cloudave.com)
• Open Cloud Manifesto – The Fight Begins (cloudave.com)
• Death To The Embargo (techcrunch.com)
• Out of Order 2.0 (techcrunchit.com)

Sage is a titan of the UK software scene but has struggled recently, both with expansion beyond Europe and (like others) with shifting their business toward a hosted capability.

According to Duane Jackson of competitor KashFlow (also picked up by Dennis Howlett and Ben Kepes at CloudAve), Sage are only weeks away from rolling out SageLive;

“On a laptop tucked away in a quite corner I spied a program running in a web browser with the title “SageLive Beta”. I got chatting to the guy demonstrating it (without saying who I am). He confirmed it’s their newest product, in development for 18 months and due for release in January. Until now they’ve tried to keep it under wraps. I commented that it seemed to be a new approach for Sage to develop a product from scratch rather than acquire an existing company. They were obviously very proud of what they’d achieved and rightly so.”

This is interesting, but I’m intrigued to understand more about why Sage were showing this ‘in a quiet corner’ at a big trade show? Is the product still ‘under wraps’? If it is, why on earth was it there for the competition to oggle? And if it’s not, why aren’t they talking about it? Maybe Duncan Hawes, Development Manager for the product, would like to set the record straight?

Duane suggests that SageLive is ‘posting to a Sage 50 database in the backend,’ which (if I understand correctly) means they’re limiting the reach of the new offering to just the UK because of the data structures used in that particular product line. If true, this would seem something of a missed opportunity for alignment of the company’s confused product mish-mash.

It would also be interesting to have a lot more information about the target audience for this product; small businesses, medium businesses, big corporations? And is it an alternative for existing customers, part of a drive to acquire competitors’ customers, or an attempt to address pockets of non-consumption?

Come on Duncan; tell us what you’re building up there in Newcastle. I drove past your building on Sunday, too. If only I’d known this then, I could have popped in for a chat!

Update 3/12/2008 – The Financial Times reports on Sage’s results…

Related articles by Zemanta

• Sage Upping the SaaS Ante
• Online Accounting: State of the Market
• MYOB brand to be phased out
• Small Business Strategies For The New Year

Recent advances in the business of Cloud Computing form just such a picture, and reach out to encompass previously unrelated aspects of Web 2.0, the Semantic Web, Platform Computing, Software as a Service (SaaS) and the economics of Disruption. Not merely some game of buzzword bingo on an unprecedented scale, it is becoming increasingly easy to see the opportunities for a significant shift in the way that we access computational resources; and to recognise that the walls separating organisations from their peers, their partners, their competitors and their customers will become ever-more permeable to the flow of data upon which those distant machines will compute.

There is much to understand that is already known in related fields, and much to discover that only becomes possible in this space. One early challenge is in carving a discrete niche for the place toward which we are moving with such rapidity. Far more than ‘just’ the Cloud; an evolution on from the playful flippancy that diminishes so many of Web 2.0′s poster children; and difficult to relate to the mainstream misconceptions of the Semantic Web’s complexity. Yet this new place is the sum of these parts, and far greater than they can ever be alone. So do we extend the already ephemeral notion of Cloud Computing? Do we appropriate the ‘next big thing’ label of Web 3.0? Or do we need a healthily fresh attitude to business computing’s apparently insatiable desire to apply labels?

First, though, let us consider the shape of this thing that is taking on more substance with each passing day.

Reporting on last week’s Web 2.0 Summit in San Francisco, CNET’s Dan Farber notes that

“The cloud was omnipresent,”

before going on to close his report with;

“cloud computing won’t be very compelling without what is variously called Web 3.0 or the Semantic Web.”

Indeed.

For too long, the emphasis in Cloud Computing circles has been almost exclusively upon provision of rapidly scalable and ad hoc remote computing on top of cost-effective commodity hardware. The Cloud play from Salesforce, Amazon’s EC2 and the rest has been dominated by the implicit assumption that these Cloud-based resources are an extension of the corporate data centre; a way to simply reduce the costs of enterprise computing.

There is value in this business, but there are bigger opportunities.

Nick Carr is amongst those to fear that a small number of players may come to dominate the provision of Cloud resources. He outlines many of these arguments in his latest book, The Big Switch, and more recently has been involved in an interesting discussion with Tim O’Reilly on the topic. Justin Leavesley shares some of Talis‘ views on the economics behind all this over on Nodalities, broadly agreeing with Tim O’Reilly;

“It’s pretty clear that utility cloud computing is highly capital intensive so it should come as no surprise that there are powerful economies of scale to be had. But the bottom line is that you are talking about plant and power. These are rival goods, scarce resources that are created and consumed. This is not different from many utility industries with one exception: the distribution network has global reach, already exists and is very cheap compared to existing utility distribution networks. It is a lot cheaper to access a computing resource on the other side of the planet than it is to send electricity or gas across the globe… [So] what is to stop economies of scale turning this into a global natural monopoly?

Actually, unless there are some large network effects, quite a lot stops single companies ruling entire industries. For a start, without network effects, economies of scale tend to run out: the curve is usually U-shaped. Telecoms, Gas, rail companies have strong network effects from their infrastructure-it makes little sense to have duplicate rail networks or gas networks in a country. Utility computing does not have this advantage because the distribution network is not owned by them.”

Continuing the conversation, Carr captures the usual widely held perception of Cloud Computing nicely;

“The history of computing has been a history of falling prices (and consequently expanding uses). But the arrival of cloud computing – which transforms computer processing, data storage, and software applications into utilities served up by central plants – marks a fundamental change in the economics of computing. It pushes down the price and expands the availability of computing in a way that effectively removes, or at least radically diminishes, capacity constraints on users. A PC suddenly becomes a terminal through which you can access and manipulate a mammoth computer that literally expands to meet your needs. What used to be hard or even impossible suddenly becomes easy.”

This is quite true, but continues and further entrenches the misapprehension that the Cloud is little more than an adjunct to the corporate data centre; a misapprehension that we shall get down to challenging in a moment.

First, though, there is a growing recognition that today’s market leaders will inevitably need to become more interoperable if this business segment – and they – are to grow. The proprietary nature of their offerings today may allow them to innovate ahead of the standards process (that will be shaped in large part by the lessons they learn), and the relatively high cost of switching to a competitor today may give each the critical mass upon which to invest and grow, but the characteristics of the current market are clearly the characteristics of a nascent market; computing’s new Wild West. As so often before, standardisation, true competition, mainstream adoption and commoditisation will all follow as we move toward phases 2 and 3 of Gartner analyst Thomas Bittman‘s intriguing ‘evolution of the Cloud Computing market.’ Similarly, Erica Naone offers a useful overview of Cloud Computing’s open source component in Technology Review this month. None of the projects she covers are a significant challenge to Amazon’s EC2, Microsoft’s Azure, Salesforce’s force.com or Google’s App Engine… yet. But together they help to keep these commercial entrants honest, and remind all of us that switching costs can be brought very low indeed if the pain of the status quo becomes too great.

Writing ‘Welcome to the Data Cloud?‘ for ZDNet last month, I began to explore the important role that data could and should play in the Cloud;

“Just as ‘we’ used to duplicate and under-utilise computational resources, so we do something very similar with our data. We expensively enter and re-enter the same facts, over and over again. We over-engineer data capture forms and schemas, making collection exorbitantly expensive, whilst often appearing to do all we can to limit opportunities for re-use. Under the all-too-easy banners of ’security’ and ‘privacy’ we secure individual data stores and fail to exploit connections with other sources, whether inside or outside the enterprise.

In a small way, the efforts of the Linked Data Project’s enthusiasts have demonstrated how different things should be. The cloud of contributing data sets grows from month to month, and the number of double-headed arrows denoting a two-way linkage is on the rise. Even the one-way relationships that currently dominate the diagram are a marked improvement on ‘business as usual’ elsewhere on the data Web; even in these cases, data from a third party is being re-used (by means of a link across the web) rather than replicated or re-invented. Costs fall. Opportunities open up. Both resources, potentially, improve. The strands of the web grow stronger.“

It is here, in the use and reuse of data, that the potential of the Cloud will be realised. Back in the previously cited conversation between Nick Carr and Tim O’Reilly, O’Reilly himself came very close to saying so;

“In short, Google is the ultimate network effects machine. ‘Harnessing collective intelligence’ isn’t a different idea from network effects, as Nick argues. It is in fact the science of network effects - understanding and applying the implications of networks.

I want to emphasize one more point: the heart of my argument about Web 2.0 is that the network effects that matter today are network effects in data. My thought process (outlined in The Open Source Paradigm Shift and then What is Web 2.0?, went something like this:

1.  The consequence of IBM’s design of a personal computer made out of commodity, off- the-shelf parts was to drive attractive margins out of hardware and into software, via Clayton Christensen’s ‘law of conservation of attractive profits.’ Hardware became a low margin business; software became a very high margin business. 
2.  Open source software and the standardized protocols of the Internet are doing the same thing to software. Margins will go down in software, but per the law of conservation of attractive profits, this means that they will go up somewhere else. Where? 
3.  The next layer of attractive profits will accrue to companies that build data-backed applications in which the data gets better the more people use the system. This is what I’ve called Web 2.0.

It’s network effects (perhaps more simply described as virtuous circles) in data that ultimately matter, not network effects per se.”
(my emphasis) 

Talis CTO Ian Davis would appear to agree, commenting;

“People need to be  investing in their data as the long term carrier of value, not the applications around them… the data is more likely to persist than the software so it’s important to get the data right and take care of it.”

Salesforce CEO Marc Benioff, too, used his Dreamforce User Conference this month to move a company long associated with the ‘data centre extending’ Cloud firmly in the direction of embracing data and the network. As Krishnan Subramanian noted on Cloud Ave before the keynote,

“Till now, the Force.com platform served business users to develop apps that can be used internally within an organization. They have to tap into Force.com APIs from outside platforms to offer customer facing web apps. With the new initiative, it becomes easy for customers to allow the internet users to “interact” with their data.”

Over on VentureBeat, Anthony Ha had more;

“Salesforce.com wants to become an even big player in the cloud computing market with a new service called Force.com Sites, which allows companies to host public-facing web applications in the Force.com platform. That means Salesforce — nominally a maker of customer relationship management (CRM) software, but also an increasingly important platform for business-related applications — is moving closer to direct competition with cloud giants like Amazon Web Services and the Google App Engine.”

Locked away within an organisation, and only accessed by that organisation’s applications, data cannot be put to full use. Much of the value in each individual datum lies in comparing it to other measurements, in delving into detail and in pulling right back to observe the bigger picture.

Organisations believing that either the big picture or the detail reside within their own systems alone are woefully misguided. Even the most specialised, the most proprietary, the most confidential of data only reveal their true value when placed in context, and that context is all the richer when informed by numerous perspectives.

Cloud Computing, and the various *aaS movements, have finally brought us to a place where the fiercely guarded and tightly delineated boundaries between the organisation and those outside it may become permeable in ways that should benefit the organisation rather than threaten it. Data is just a resource. In the terminology of Geoffrey Moore most data is often mere context, and there are savings to be made both in reusing the data of others or in re-selling necessary context to those prepared to pay. Some data, of course, is core to the business, and this may continue to receive the same reverence and protection that we misguidedly apply to the entire database today. Even here, though, the opportunities afforded by (controlled?) sharing may outweigh any desire to maintain data protectionism.

The language of Groundswell offers opportunities to go further, to embrace and to exploit the behaviours and the motivations of customers and the wider Web.

There is clearly far more to write in clarifying this view of both the components and the whole, but as it passes 2,000 words this particular blog post has perhaps gone on long enough.

For now, then, I should conclude by asking what role the Semantic Web has to play in any of this.

The Semantic Web, with its unadulterated recognition of the primacy of the web’s hyperlink? The Semantic Web, designed from the outset to convey context and relationships derived from data spread across the Web? The Semantic Web, supported by technologies that operate openly and at Web scale?

Isn’t it obvious yet?

Returning to the Web 2.0 Summit with which this post began, another presentation was from Kevin Kelly, founding editor of Wired Magazine. As I wrote this post, I referred to Steve Gillmor and Nicole Ferraro, from whose reports I inferred that Kelly had built upon an earlier presentation (that I greatly enjoyed), in which he argued;

“You have to be open to having your data shared… which is a much bigger step than just sharing your web pages or your computer.”

Fact-checking before hitting publish, I notice that last week’s video is now up, here, and Kevin’s championing of the primacy of data in the cloud resonates with every word I’ve just written.

Yep. Here we go, on a journey toward Kevin Kelly’s “World Wide Database.”

In subsequent posts I’ll explore some more of the detail, and I hope you’ll stick around for the journey.

Storm Clouds image © ‘shidairyproduct‘ 2008. Shared on Flickr, and licensed with a Creative Commons Attribution License. Converted to a jigsaw by Big Huge Labs.