The conjecture may be correct. The NSA, the FBI, TLA and ETLA might be plugged right into the data centres of the internet’s giants, slurping down your messages, searches and calls. If they are, that’s potentially serious. But we don’t actually know that they are, yet. Until then, reporters, bloggers, analysts and pundits are speculating and considering implications. That’s a good and useful thing to do. But they really need to stop suggesting that they’re reporting facts.

Whether PRISM turns out to be as wide-ranging as suggested or not, a lot of confusion is being caused by misinformed, malicious or badly phrased speculation. There is rarely smoke without fire, but real damage is being done here. As David Meyer notes in a piece for GigaOM this morning,

“All of this is likely to prove very problematic indeed for U.S. cloud firms trying to push further into the European market.

Imagine you’re a European government wanting to move your IT systems into the cloud. For some, nationalism and protectionism already come into play at this point – witness the French (of course) and the two national clouds that they have under development.

Now imagine you’re a U.S. firm trying to drum up business in that context. You can say you have an EU data center and you’re even willing to set up a mini-cloud in the country, just to put everyone’s mind at rest. You can say it and you can mean it, but can you really be surprised when you get laughed at because everyone now sees U.S. internet companies as being in league with the NSA? Even if you’re Amazon, which isn’t part of PRISM, you have a problem.” (my emphasis)

Most countries around the world have a legal means to access data stored on servers operating on their soil. The degree of judicial oversight – or evidence – required varies widely from one country to the next, but it is widely accepted that law enforcement agencies should be able to gain access to data under certain circumstances. It was also widely believed that this doesn’t actually happen terribly often.

Alleged information on PRISM obtained by The Guardian would suggest that this programme is able to go much further;

“Companies are legally obliged to comply with requests for users’ communications under US law, but the PRISM program allows the intelligence services direct access to the companies’ servers.”

The piece continues,

“The PRISM program allows the NSA, the world’s largest surveillance organisation, to obtain targeted communications without having to request them from the service providers and without having to obtain individual court orders.

With this program, the NSA is able to reach directly into the servers of the participating companies and obtain both stored communications as well as perform real-time collection on targeted users.”

Over in the States, the Washington Post makes even more extreme claims;

“The National Security Agency and the FBI are tapping directly into the central servers of nine leading U.S. Internet companies, extracting audio and video chats, photographs, e-mails, documents, and connection logs that enable analysts to track foreign targets, according to a top-secret document obtained by The Washington Post.” (my emphasis)

Companies such as Google and Apple, originally accused of active cooperation, were quick to issue carefully worded denials, explicitly challenging the ”direct access to… servers” claim. The New York Times, at least, is being careful;

“The New York Times has not confirmed the authenticity of the documents, and several of the Internet companies issued statements strongly denying knowledge of or participation in the program.” (my emphasis)

Elsewhere, headlines, ‘news’ and editorials leap gleefully into the melee, proclaiming that government agencies are ‘lying,’ warning that data is being read by the NSA and FBI, accusing tech firms of collusion, and worse.

I don’t know the extent to which the powers attributed to PRISM are real. I also don’t know how often — if ever — they’ve actually been used. Nor do most of the others commenting so knowledgeably on this story. Just bear that in mind, as you read what they write.

Prism image contributed to Wikimedia Commons by ‘Ffffnm‘.

Related articles

• PRISM – US Gov. mining data from Google, y, msn, skype, youtube, and FB (washingtonpost.com)
• Google, Facebook And Apple Deny Participation In NSA PRISM Surveillance Program (techcrunch.com)
• PRISM program lets FBI, NSA secretly mine data from 9 U.S. tech companies (report) (venturebeat.com)
• How the NSA lied about not tracking Americans with PRISM (venturebeat.com)
• NSA spying scandal fallout: Expect big impact in Europe and elsewhere (Updated) (gigaom.com)
• How likely is the NSA PRISM program to catch a terrorist? (bayesianbiologist.com)
• How Congress unknowingly legalized PRISM in 2007 (washingtonpost.com)
• Secret program gives NSA, FBI backdoor access to Apple, Google, Facebook, Microsoft data (theverge.com)
• NSA has direct access to tech giants’ systems for user data, secret files reveal (guardian.co.uk)
• By the numbers: The NSA’s super-secret spy program, PRISM (foreignpolicy.com)

Image © BBC 2013

I recently set up a new Tumblr site, in addition to my main blog here at cloudofdata.com. It’s been running for a few days now, and I’m deliberately posting just one entry each day to explore a topical news item, issue, or trend.

We are all inundated by a constant stream of press releases, tweets, briefings, presentations, scheduled conversations and chance encounters. Sometimes, I find, it can be difficult to take a step back from the flow in order to identify — and reflect upon — the important pieces of the whole.

With One Short Thought Each Day, and inspired by GigaOM Pro‘s excellent Today In… features, I’m setting out to distil a single post each day from the flurry of signals that target me and my inbox. Each day’s post might be sparked by just one press release, or by some strange combination of signals that either contradict or reinforce one another. It won’t necessarily be the most important story of the day, or the one that everyone else covers. But hopefully it will be interesting, and relevant to others in the Cloud Computing/ Big Data/ Open Data-type areas in which I work.

I intend to keep the posts short, to try and publish every working day, and to never publish more than once per day.

All of the posts are available on Tumblr, here. They are also available as an RSS feed, and over on the sidebar of my main blog. A single tweet to my Twitter account announces the publication of a new piece, and you can also sign up with MailChimp to have the day’s post delivered to your inbox — I promise not to use your email address for anything else.

My main blog and website continue as before, and I will set aside some time soon to improve the rather bare-bones look of the Tumblr site and the email template!

I hope you find this useful, but do let me know if there are ways in which you’d like to see length, topic, tone, or format tweaked to increase the value you get. Frequency, unless there’s a veritable howl of protest, is non-negotiable… at one per day.

Technological advances and shifting business requirements affect Government just as much as they do the enterprise. Fragmentation and siloisation can be used to explain sprawling data centre estates, which officials now work hard to consolidate. Shifting requirements, over-engineering, cronyism and politics can all take the blame for massive cost-overruns on boil the ocean projects that rarely realise their promise. Lighter, nimbler, more responsive, more accountable and more cost-effective approaches to Government IT are undeniably required, and it’s true that cloud can — and should — play a role here.

But simply proclaiming that the cloud is Government’s first choice moving forward misses the point. The system’s current weaknesses (and strengths) are deep-seated. They are ingrained, and institutionalised. Imposing cloud by decree from on high simply gives the mass of public sector IT buyers a new religion to blindly follow… and provides the thoughtful and the recalcitrant with a new bogeyman to loathe.

As the Cabinet Office press release this month states,

In future, when procuring new or existing services, public sector organisations should consider and fully evaluate potential cloud solutions first – before they consider any other option.

Huddle (which, deservedly, has done well out of the UK Government’s cloud push) co-founder Alastair Mitchell blogged about the UK’s Cloud First announcement at the time, and I was uncomfortable.

A tweet last night from the Open Data Center Alliance reignited that discomfort. The tweet wasn’t actually addressed to me at all, but to my friend Ben Kepes.

@opendatacenter I’m still not sure #CloudFirst is a good idea. Cloud by merit, not by dictat! /cc @benkepes

— Paul Miller (@PaulMiller) May 20, 2013

My grammatically suspect response gets right to the heart of my concern, despite the 140 character limitation of Twitter.

Cloud computing has a powerful role to play in Government and in the commercial sector. But it should build market share on merit — by being the best or the cheapest or the quickest or the most interoperable or the most flexible. By being — demonstrably — the best tool for the job. Dictat, decree, mandate; all of these are market skewing pieces of positive discrimination and, at the end of the day, positive discrimination is still discrimination. And unhelpful.

Some Government users will always find and choose cloud all by themselves. Huddle’s been racking up Government successes for years, without a Government enforcer to kick down the doors for them. But others (probably the majority) will prove harder to engage and persuade. Build it and they will come is almost certainly a strategy that will lead to burning cash far faster than you sign up customers. So there is certainly a need for evangelism, and demonstration, and the liberal use of carrots to entice the unsure, the unwilling and the unconvinced. Let’s show Government buyers the ways in which cloud-based solutions help the Government to do its multitude of jobs. Let’s implement programmes that ease the pain of moving from current practice to cloud-enabled practice. Let’s help SMEs (like Huddle) get heard by buyers and decision makers used to being wined and dined by Accenture, IBM, HP and Microsoft. Let’s smooth the path to the cloud, not force everyone along it.

Maybe Francis Maude’s next cloud pronouncement could be “Cloud, please,” rather than the current “Cloud, or else.”

‘Police State‘ image by Flickr user ‘katesheets‘

Related articles

• Press release: Government adopts ‘Cloud First’ policy for public sector IT (gov.uk)
• Cloud must now be your first choice for big IT projects, says UK government (zdnet.com)
• U.K. Government Mandates Cloud First IT Procurement (informationweek.com)
• UK Deputy CIO Attacks EU Plan To Certify Cloud Providers (misco.co.uk)

Image © Mission Bay Conference Center

Last month, RightScale’s State of the Cloud report got me thinking about the rise of multi-cloud solutions. Next month, I’ll be moderating a Mapping Session at GigaOM’s Structure event to work out how, where, when, why and if this trend is going to prove significant.

Hybrid clouds, in which one public cloud and one private cloud are used together, are becoming increasingly common solutions to a range of business challenges. RightScale’s figures suggest growing interest in something more complex and, potentially, more interesting; multi-cloud. In a multi-cloud arrangement, customers build solutions combining one or more public clouds with one or more private clouds. This has the potential to significantly increase complexity, without necessarily delivering a comparable increase in value.

In my post last month, I suggested that many of these multi-cloud deployments were essentially accidental. A quick email exchange with RightScale shows that their survey respondents would appear to disagree; multi-cloud, for them, is a conscious business decision.

I’m intrigued, and so were GigaOM. So we’re putting on a Mapping Session at next month’s Structure conference to explore the issue further. I’ll be joined at the front of the room by fellow GigaOM Pro Analysts Ben Kepes and David Linthicum, as well as GigaOM Research Director Jo Maitland.

A GigaOM Mapping Session is more like a workshop than a regular panel. Although there are analysts at the front of the room, they’re really there to stimulate and guide a conversation with every single person in the room. We don’t have all the answers. We’re there to explore the topic, and to work out what — if anything — it might mean. The perspectives of customers, practitioners, suppliers and investors are an integral part of the process. When it works well, everyone comes away with a broader perspective than when they entered the room. Ideas are born, perceptions are sharpened, the germs of deals are done, and directions for future GigaOM research are painted. Mapping Sessions are free for registered Structure attendees, but separate registration is required. Numbers are capped, to ensure plenty of opportunity for discussion. If you’ve got perspectives to share, please do register to join us on 20 June in San Francisco.

Multi-cloud clearly throws up a host of intriguing issues for cloud vendors, their customers, and the ecosystem of cloud management-type providers (like survey author RightScale). What, if anything, do cloud vendors need to do in order to encourage, support, or hinder multi-cloud adoption? What’s the value proposition behind multi-cloud for customers, and what do the providers of cloud management services need to do in order to capitalise upon an emerging trend? Is this a long-term trend, or a short-lived opportunity?

Many of those stakeholders will be in the room, speaking, listening, and interacting. Do join us, and them.

From time to time, I write about Open Data. This has nothing to do with that. The Open Data Center Alliance is interested in data centres, not data. The Alliance was established back in 2010 with Intel driving things forward, and now claims over 300 member organisations, including the likes of BMW, Lockheed Martin, Microsoft, Deutsche Bank and Marriott Hotels.

According to the ODCA website,

we came together to deliver a unified voice for emerging data center and cloud computing requirements. Our mission is to speed the migration to cloud computing by enabling the solution and service ecosystem to address IT requirements with the highest level of interoperability and standards.

Much of the Alliance’s work involves identifying customer requirements and capturing these in a series of usage models. In theory, prospective customers can modify these usage models in defining their own requirements, and suppliers can tap new business by aligning their offerings to the models. I’ve not seen much evidence that this is happening at scale yet, but the Alliance site does state that

we anticipate quick industry response to the requirements, initial POCs of solutions beginning in 2012. This could accelerate over $50 billion in cloud service investments and is expected to save $25 billion through IT operational efficiency due to cloud adoption.

One of those usage models is concerned with Virtual Machine Interoperability in a Hybrid Cloud Environment (pdf), and last month it was augmented by the release of a Proof of Concept document (pdf) which

outlines testing criteria and procedures for documenting how hypervisor and VM solutions from both ODCA members and non-members interoperated in real-world enterprise cloud scenarios.

Quoted in an April press release to mark publication of the PoC document, ODCA executive director Marvin Wheeler commented that

true interoperability of hypervisor and virtual machine solutions between clouds is critical to advancing enterprise ready cloud implementations. We encourage global IT managers and cloud solution and service providers to join us at the Forecast 2013 VM interoperability panel on June 17 to take part in the collaborative discussions that will help drive the next phase of VM interoperability in the enterprise cloud.

The basic premise is a simple one; a virtual machine started on one hypervisor or class of physical server should run equally well when moved to run on a different hypervisor or physical server. Further, there is a presumption that there is a credible business requirement for this capability. As enterprise users of cloud increasingly find themselves adopting a hybrid approach spanning their own data centre, co-location facilities, hosting sites and diverse public clouds, the likelihood of them needing to run their standard Windows or Linux virtual machines on top of more than one hypervisor certainly increases. It’s less clear, though, that an inability to reliably move a Linux VM from the KVM hypervisor in your own data centre to the Xen hypervisor at a cloud provider is causing significant business pain today; it’s often reasonably straightforward, for example, to simply select a different cloud provider able to support your chosen KVM hypervisor.

But even if a lack of VM interoperability isn’t presenting an insurmountable barrier to business right now, it’s still one more thing to think about when pulling a set of disparate services together. If we can cost-effectively and pragmatically remove or diminish its ability to complicate, then that’s presumably a good thing.

ODCA’s PoC work took the usage requirements the organisation had already defined, and applied them to a specific set of documented tests. As IDG’s Joab Jackson notes in his InfoWorld piece, the results were not great.

Running through all the different possible combinations of hypervisors and OSes, the researchers found that 13 test cases resulted in warnings, and 19 test cases failed entirely. Only in two cases did the VM work flawlessly across two different hypervisors. In both of these cases, a VM created with Xen worked without troubles on a Microsoft hyper-V environment — in one case running Ubuntu and in the other case running Windows Server.

The researchers do note that they set the bar for success pretty high, and that several of the ‘warning’ states would still result in a functional VM. It does appear clear, though, that customers with a real need for multi-VM interoperability face significant challenges in efficiently managing workloads across different virtualisation infrastructures.

And that brings us to the panel at Forecast, which can hopefully help quantify the true scale of the problem… and indicate some of the ways in which vendors are working to fix the broken bits. In terms of the panel discussion itself, I look forward to delving into at least the following:

• how big a problem is the current state of VM interoperability?
• do the ODCA VM Interoperability use cases prioritise the right things?
• what do customers really need, and are they getting it?
• how are vendors really responding to requirements such as those defined by ODCA?

If you have opinions to share, perspectives to offer, facts with which to illuminate, or questions to ask, I do hope you’ll join us.

ODCA has given me a couple of free passes to Forecast. If you want to come along and we know one another, then please do get in touch.

Note: ODCA is covering my flight to San Francisco, and accommodation during the event.

Image of event venue, San Francisco’s Westin St. Francis, by Flickr user ‘prayitno‘

Related articles

• Open Data Center Alliance Tackles Cloud Lock-In (datacenterknowledge.com)
• Virtual machines inching toward portability (infoworld.com)
• 5 Key Essentials of Cloud Workloads Migration (cloudave.com)

But, as we all become increasingly dependent upon data sourced from third parties, aspects of this rather esoteric pastime are beginning to matter to a far broader audience. Products, recommendations, decisions and entire businesses are being constructed on top of data sourced from trusted partners, from new data brokers, from crowdsourced communities, or simply plucked from across the open web. Without an understanding of where that data came from, and how it was collected, interpreted or maintained, all of those products, recommendations, decisions and businesses stand upon very shaky foundations indeed.

Data attribution is increasingly important, but it will be essential to make sure that the rules, tools and norms which emerge are both lightweight and pragmatic. Now is not the time to get heavy-handed and pedantic about where the comma goes.

Former colleague Leigh Dodds recently offered a useful discussion of the rationale behind data attribution. Early on, he describes the related (and, often, sloppily interchangeable) notions of attribution and citation;

It might also be useful to distinguish between:

• Attribution — highlighting the creator/publisher of some data to acknowledge their efforts, conferring reputation
• Citation — providing a link or reference to the data itself, in order to communicate provenance or drive discovery

This distinction is important in some circumstances, but it can also be useful to consider a simpler, more selfish, but ultimately more scalable justification. Attribution (and citation) of data quite simply provides an audit trail, enabling you, your bosses, your investors, or your customers, to know more about the data upon which actions are based.

Creators want credit, consumers want to trust

Much of the serious consideration of attribution comes from a relatively small cadre of data owners or creators, who (understandably, perhaps) want credit for their hard work. Perhaps they want to prove use in order to secure future funding or advancement, or perhaps they simply want to track where their data ends up. Through a series of licenses, contracts and Terms & Conditions statements, these creators have done much in codifying the ways that data should be referred to. Leigh discusses some of the licensing terms in his post, but it’s probably fair to say that none of them have really caught on outside a few rather narrowly scoped groups of co-dependent developers and data providers.

Data owners’ requirements for credit run the gamut, from loosely phrased requests for a link back to their website all the way past this rather excessive example quoted by Leigh to end up as lengthy tomes of draconian legalese;

The attribution must be no smaller than 70% of the size of the largest bit of information used, or 7px, whichever is larger. If you are making the information available via your own API you need to make sure your users comply with all these conditions.

All too often, that sort of self-defeating prescription is enough to send prospective users back to Google, in search of a less demanding alternative.

For consumers of data, or for those wondering where the data behind a product or decision came from, things are rather simpler. On the whole, those on this side of the divide are simply looking for a pointer which enables them to learn more. Is my company’s multi-million dollar change of direction based upon detailed data from stable Governments, credible banks and respected analysts, or did the person responsible use some numbers the found on their friend’s blog?

Carefully crafted rules regarding attribution’s wording, placement, colour, size and typeface are an irrelevance, probably deserving to be ignored or ridiculed.

Far better, and far more likely of success, to simply encourage users and re-users of data to sensibly point back (however they like) to their principal sources.

Remembering all the ancestors is a bit daft

Data set A is modified and added to in order to create data set A1. Data set A1 is modified and added to in order to create data set A2. Data set B is modified and added to in order to create data set B1. Data set B1 is modified and added to in order to create data set B2. Data Set C modifies and extends data sets A2 and B2. It seems reasonable to acknowledge the contribution made to C by A2 and B2, but some would argue (loudly) that A, B, A1 and B1 also need to be acknowledged in C. This is one aspect of ‘attribution stacking’, and attribution stacking is, quite simply, stupid.

If I am the creator of data set C, I am selecting A2 and B2 because they are the right data sets for my purpose. That selection will be based upon a range of criteria, including the scope and coverage of the data. The selection will also be based upon my impression of the brands responsible for A2 and B2, and that impression (implicitly or explicitly) will include some awareness of the processes they use to select, validate and manage the data they use. It’s for them to carefully select, validate and provide attribution for A1 and B1, not for me. And it’s A1 and B1′s job to do the same for A and B, not me.

Things get even worse in some open source data projects, where all the individual contributors expect to be acknowledged. Inside the project (and on its website, etc), that’s fine and sensible. Outside, though? It’s ridiculous. So if data set A were created by individuals Aa, Ab, Ac, Ad and all their friends right up to Az, under some licenses there would be an expectation that every single one of those individuals be acknowledged by name in any mention of data sets A, A1, A2 or C. A massive administrative burden for any downstream users of the data set, and of no real benefit to anyone whatsoever. This desire for glory really does need to be challenged, if it is not to stifle free and fair downstream use and reuse of the data. Within the project building A, it may be vital to know that user Aa is a bit sloppy, or that user Ad has a nasty habit of making the data say what she thinks it should rather than what it actually does. But it is the responsibility of the project behind A to put processes and procedures in place to address these issues, and to ensure that all of its participants receive appropriate credit within the project for their contribution. By the time we reach A1 or A2, though, those internal details no longer matter. A1 chose to use A because those processes exist. After an initial evaluation of those processes and their implementation, A1 can — and should — simply trust them, rather than endlessly second-guessing them.

Tracking and Trust are different

Ultimately, the motivations of data creators and data re-users are very different. The processes and procedures put in place by creators and owners in search of kudos or statistics may actively obstruct the use and reuse that they profess to want. Complex forms of attribution, aligned to heavy-handed enforcement of infringements, do nothing to encourage a far broader community of use to emerge.

By attempting to count — and manage — the small number of uses today, data creators are stifling growth that otherwise is ready to explode. A perfect example of the saying (which may not translate beyond Britain’s shores!) of ‘biting off your nose to spite your face.’ Think about it…

Leigh ends his post with

Attribution should be a social norm that we encourage, strongly, in order to acknowledge the sources of our Open Data.

Other than broadening it from ‘Open Data’ to just ‘data,’ I couldn’t agree more. But let’s keep it lightweight, simple, and pragmatic.

Note: or perhaps this post should have been called “When you stand on a giant’s shoulders, it’s a good idea to say thank you.”

Image of Eduardo Paolozzi‘s sculpture of Sir Isaac Newton by Flickr user ‘monkeywing‘

Related articles

• How Do We Attribute Data? (ldodds.com)

Albert Einstein, you may have heard, was a clever man. He scribbled equations on blackboards, thought big thoughts, and all of that. But, allegedly, he also said

Everything should be made as simple as possible, but not simpler.

These words have resonated with me recently, as I’ve heard pitches from one company after another, all of which are trying to cut through the complexity of data to make it accessible. Their goals appear laudable, but all too often I find myself wondering how simple this stuff can be? If we make it too simple, do we run the risk of unleashing a flood of half-baked ‘analysis,’ undertaken by people who really shouldn’t be allowed near a calculator, let alone a Hadoop cluster? On the other hand there’s a persuasive argument to be made for democratising access to data and tools, freeing organisations from over-reliance upon their new High Priests of Data.

Every data question should not require a data scientist, but maybe we really shouldn’t be making it too easy for people to tackle the hard questions without support from someone who knows what they’re doing.

How simple, then, is too simple? And can we use data in a different way; in order to offer enough simplicity, smartly?

One company which has tried to do that in an interesting way is Datahero, and I spoke with co-founder Chris Neumann this week to learn some more.

There’s been some good coverage of the company over the past few weeks, and if you’ve not come across them before then it’s worth taking a skim through the Related articles at the end of this piece. I won’t bother repeating those pieces here.

The thing that interested me — other than Chris’ obvious passion and enthusiasm for his subject — was the way in which Datahero plans to use a mix of data analysis, user experience design and machine learning in order to guide the user toward analyses and visualisations that are likely to be of use to them. Chris is quick to stress that the company isn’t looking too closely at the data values its customers upload. Instead, the system studies the structure of the data (this column contains dates, this column contains place names, etc) and any associated metadata in order to make recommendations for logical ways to visualise the dataset.

As the number of users grows, the roadmap also includes Amazon-style recommendations. If a lot of other people uploading their quarterly sales forecast graph it in a certain way, then it makes sense to recommend that type of graph to a new user who uploads data with a similar structure. The recommendation won’t always be right, but it should go a long way toward minimising the fear of staring at columns and columns of data without a good idea about where to turn first.

Smart recommendations, clever algorithms, and an engaging UI will not — on their own — turn everyone into a data scientist. But nor, really, should they. What they can do, though, is what Neumann described as “enabling the 99%;” those people who have a dump of data from Mailchimp or Google Analytics or Salesforce or Excel, and who don’t really know how to begin sensibly visualising the multitude of columns and rows of numbers.

It’s an intriguing idea, and it will be interesting to see how successfully the system can deliver value to a potentially huge pool of individuals who find themselves data-rich, but skills-poor.

Even if technically successful, of course, the challenge will be persuading those same users to continue paying for the service. If they think they’ll use it often enough to pay for, won’t they end up acquiring enough of the skills to work with their own data anyway? Unless, of course, Datahero manages to grow and add complexity along with its users…

Image of Albert Einstein, ©1947. Image sourced from the Library of Congress’ Prints and Photographs division with identifier cph.3b46036. Shared on Wikipedia, and deemed to be in the Public Domain.

Related articles

• Datahero Turns Data Into Rich Visuals Without The Need For A Data Analyst (techcrunch.com)
• ‘Data vis for the 99 percent’: DataHero launches its free service (venturebeat.com)
• Datahero launches to the public, making genius-level data science simple, automated and free (thenextweb.com)
• Visualization startup Datahero opens its doors and delivers data analysis for the masses [GigaOM] (gigaom.com)

Enigma pulls data from tens of thousands of public data sets, and then offers up an interface that makes it pretty straightforward to trawl through the whole lot in search of the data points that you actually need. As the company’s Marc DaCosta introduced it, a “search and discovery platform for public data.”

At present, everything in Enigma is publicly available data. It’s mostly from the USA right now, and is acquired by a combination of screen scraping/ crawling of .gov sites… and calling government agencies to request that they ship CDs of offline material. DaCosta stresses that the data is all — theoretically — available to any US citizen, but that it’s not really that easy for them to access. Even with the advent of central sites like data.gov, primary data remains spread across a multitude of portals and web pages, stored in a dizzying array of machine readable formats… and (all too often) as PDFs full of tabular data. DaCosta finishes his preamble by stressing the need for

infrastructure to acquire, index and search public data.

The team’s Disrupt pitch and the subsequent back and forth with the judges provides a pretty good overview. It is worth 12 minutes of your time to watch over on TechCrunch.

During the Q&A, Freestyle Capital’s Dave Samuel raised the same concern I did during my meeting with the Enigma team. The set of possibilities with public data is extremely large, but still finite. Far more so than Google, there are real limitations to the sorts of questions that it makes sense to ask. Presented with a search box, users struggle to understand what’s possible and what’s sensible. Simple tricks would go a long way toward helping here, such as highlighting today’s most popular queries, or providing sets of sample queries as Wolfram Alpha does. Users need to learn how to work with aggregations of data such as this, and the onus is on sites like Enigma to help grow their potential market through lightweight and accessible education.

In many ways, though, I think the biggest opportunity for Enigma doesn’t lie in their website at all. The real opportunity lies in their api, and in licensing it to third parties who will construct entirely new Enigma-powered vertical applications or integrate Enigma data into existing software such as an investor’s due diligence systems.

In and of itself, the data Enigma has gathered is interesting, but neither indispensable nor unique. The real value comes from offering easy, reliable, comprehensive and cost-effective means to get this data integrated into existing workflows, and that’s where the api needs to shine. There’s also an opportunity to encourage/enable companies to upload their own data into the platform, letting them combine it with the public data already there. This is, apparently, on the road map. I would imagine that this corporate data will initially only be available to the company that owns it, but there are a whole set of other opportunities around sharing data with supply chain partners, and even making it available to anyone.

Nice (and fast!) as the website is once you understand how it works, I tend to see it far more as a shop window than as a revenue-generating service in its own right. The Enigma team disagrees, seeing subscription-based access to the website as one of their two main products. We shall see who proves more right in due course!

Finally, for now, the company faces a real challenge in scaling. It needs to pull in more data from inside the US, and it needs to broaden its coverage outside the US. Its data acquisition processes, although highly automated, remain pretty labour intensive. And its data teams need to understand the data they’re working with. Some government data is rigorously documented, whereas other data sets are almost unintelligible. Growth, as the Enigma team recognises all too well, requires far more than simply pointing their crawlers at some new web domains.

March and April were busy with travel. May’s pretty quiet on the travel front, with plenty of opportunity to get some proper work done before heading to Brussels at the end of the month. The next trip States-side looks like being to San Francisco in June. I wonder who I’ll meet there… and what they’ll go on to win?

Picture of the Enigma cryptographic machine by Flickr user Chris Brown.

Related articles

• And The Winner Of TechCrunch Disrupt NY 2013 Is… Enigma! (techcrunch.com)
• This TechCrunch Disrupt Winner Could Be The Future Of Search (businessinsider.com)
• Enigma brings the deep, dark world of public data to light (venturebeat.com)
• Enigma Makes Unearthing And Sifting Through Public Data A Breeze (techcrunch.com)
• Beating out tough competition, ‘Google for public data’ Enigma wins Disrupt NY (venturebeat.com)
• Enigma Reinvents Public Data And Wins TechCrunch Disrupt Battlefield (nibletz.com)

Xeround was founded back in 2005, initially delivering scalable data management solutions into the data centres of telcos such as T-Mobile. The company’s work on scaling MySQL and offering it in the cloud brought it to wider attention, and they seemed to maintain momentum by adding additional cloud partners. Last year, the company rolled out a freemium offering to entice a far wider set of potential customers. It would appear, though, that any influx of paying customers was insufficient to meet the cost of this rapid expansion across disparate cloud providers.

In last night’s blog post, Xeround’s Avigail Ofer writes;

Xeround’s leadership forum has recently decided to re-focus the company’s effort. This means we will no longer be able to support our service over public clouds, across all of our currently active data centers.

This presumably means a return towards the company’s roots, offering scalable data management solutions inside enterprise data centres. As a strategic decision it may make sense, although that market is far more crowded than it was in 2005. For now, though, the company website continues to talk almost exclusively about their Database as a Service (DBaaS) offering, hosted in public clouds. Anyone looking to see what else Xeround does today, or where it will be active after 15 May, will get few clues from the site. Current and prospective customers must, inevitably, feel anxious. What else might be consigned to the pyre?

Strategic refocusing, cutting underperforming products, and even pivoting are facts of business life. Well-known consumer services like Flickr and Twitter were the result of pivots that must have seemed painful at the time. In those cases, and others, pundits and commentators bemoaned the death of whatever went before… but few would dispute the success of the new ventures. Customers worry about what they have. I still miss Tungle, and have no idea how I will cope when Google cruelly sacrifices the invaluable Reader. But boards and managers need to worry about where they’re going. Sometimes, maintaining the status quo simply does not make sense. You’re breaking even or — worse — burning cash. You can see a new competitor that’s faster, better, or cheaper than you. Your revenues or usage figures are headed the wrong way. All are signs that something needs to change. And if you’ve got a better idea about how to focus resources, why not go for it?

Successful companies aren’t necessarily the ones with the most brilliant people, or the strongest teams, or the best idea, or the biggest pile of cash. Successful companies are those with people who are good enough, who can work together well enough, on an idea that’s intriguing enough, attracting sufficient cash to get noticed. If all of those pieces fall into place, successful companies then need a good dose of luck, and a great big dollop of smart timing. The market needs to be ready, customers need to be in the mood to buy, and the competitive landscape needs to be tilted in your favour. Sometimes, those pieces don’t all align, and the company needs to shift focus a little and try again.

Trying again is fine. There’s no shame in it. But when you try again people remember things about how you behaved the last time. They won’t hold failure or pivoting or downsizing or refocusing against you. But if they are considering becoming an investor, partner or customer, they’ll remember how you treated investors, partners, and customers last time around.

And that’s why Xeround’s timing for the shutdown bothers me. A week (or two, if you’ve given them cash) is not a lot of time to move off a database that might be baked deep into your business. Customers must be waking up this morning, and scrambling to adapt. Their roadmap is out the window. Their vacation is on hold. Their work to attract the next client abandoned, as they scrabble to keep the database running for the clients they already have. Xeround did that. And it didn’t need to.

Pivot if you want, Xeround, but you’ll have a much better hope of maintaining current and future custom if you treat your users with respect. And a week or two’s notice for the removal of something that may very well be mission critical is not respectful.

I did try to reach Xeround’s PR firm for comment, without success. I shall update this post if I receive anything relevant from them or the company.

Hat tip to Barb Darrow at GigaOM, who was quick to cover the story… and to update her post as more facts emerged.

Image of a burning Viking ship from istockphoto.

Related articles

• Xeround Provides Auto-Scaling & High-Availability for your MySQL Database in the Cloud – The Official Rackspace Blog (rackspace.com)
• Biggest SQL Disasters (i-programmer.info)

Surveys. Along with the loathsome infographic, they have become the default tool of the lazy PR person. I get loads of them, pretty much every day. And, almost without fail, they are a shocking waste of everyone’s time. Three out of the four people we gave some money to said our product was best. I don’t care. And I’m definitely not going to write about it. Every now and then, though, one that’s worth a little attention comes along. RightScale’s latest State of the Cloud, released last week, is one of those.

The sample is relatively small (just 625) and, as Network World‘s Brandon Butler emphasises, they

came from a community to [sic] people that had somehow, some way interacted with RightSacle [sic] in the past — they had registered on the company’s website, dropped a business card in a bowl at a trade show, or are a customer (although RightScale says only 30% of respondents were customers). The firm’s VP of Marketing Kim Weins says respondents to RightScale’s survey admittedly have some ‘affinity’ to cloud computing.

RightScale’s analysis of respondent demographics bears this out, with over half (51%) describing themselves as being from ‘Tech Software, Services, Hardware’ businesses. The broader community of cloud-consumers in manufacturing, financial services, media and the rest is far less well represented.

Survey Demographics. Image © RightScale 2013

The survey digs into a number of areas (see the Related articles, below), but it was the discussion of ‘multi-cloud’ that particularly caught my attention.

Image © RightScale 2013

According to the survey, 77% of respondents from enterprises with 1,000+ employees (see Disclaimer for an important proviso) had some sort of multi-cloud policy. The majority of those (47%) are pursuing some form of on- and off-premise hybrid strategy, whilst 15% are employing multiple private clouds and 15% are employing multiple public clouds. Amongst enterprises with 1,000+ employees that are already using a public cloud solution, the numbers are even higher; 88% are pursuing a multi-cloud policy, with 53% opting for a hybrid approach, 22% for multiple public clouds, and 13% for multiple private clouds. In both cases, it is unclear whether respondents could describe themselves as using multiple public and multiple private clouds.

A hybrid approach, in which some computing is done on-premise (or in a long-term and highly managed co-location or hosting arrangement) and some is done in the public cloud makes an awful lot of sense. Some data may be deemed too expensive or too sensitive to routinely move in and out of the cloud. Some workloads are steady, routine, and highly optimised to run most cost-effectively in a supported, SLA-wrapped and tightly managed dedicated facility. Other workloads demonstrate dramatic peaks and troughs in usage, or require irregular access to resources which are too expensive to maintain — frequently idle — in-house. Even in the most regulated, paranoid or old-fashioned organisations, there are plenty of workloads for which a hybrid solution is ideal.

More interesting than the simple use of a hybrid solution is the suggestion that companies are using more than one public cloud, more than one private cloud, and perhaps more than one of both. There is, perhaps, a clear opportunity here for cloud management solutions (such as RightScale’s, of course), but to really understand the nature of the opportunity we need to understand far more about why the trend we’re seeing is occurring. There’s insufficient data in RightScale’s published results to be sure, but possible explanations include, one, some, or all of the following.

Eggs and Baskets

It’s a conscious decision. Public clouds suffer outages. Private cloud providers get acquired, go bust, or change direction. To continue delivering core services to key customers, the modern enterprise is forced to spread its services across different solutions providers. The gradually increasing credibility of enterprise cloud solutions from HP, IBM, Microsoft and others may diminish this perceived requirement.

The Search for Features

It may be deliberate, but is more likely to be an accidental byproduct of something else. Different public and private cloud providers excel in different areas. They provide support for different virtual machine instances, their apis are better at some actions than others, they offer services in different jurisdictions, and with varying levels of support. The modern enterprise, in meeting a wide range of business requirements, will inevitably find itself selecting different clouds for different reasons. This is likely to continue for some time to come, despite the growing feature completeness of competing offerings.

The Departmental Power Grab

It may be deliberate, but is more likely to be an accidental byproduct of something else. The organisation has no centralised policy regarding the cloud. Different departments and divisions are pursuing their own formal strategies. Their differing requirements lead, almost inevitably, to the selection of different solutions. As the CIO attempts to reassert control, there may be scope for solutions providing a sensible view across these competing services. Alternatively, there may be an explicit move to consolidate, creating opportunities for those able to assist in migrating services from one provider to another.

Shadow IT

It may be deliberate, but is more likely to be an accidental byproduct of something else. The different solutions are being selected by individuals and small teams. They may be selecting those they’ve previously used elsewhere, or they may be employing a wide range of criteria from the frivolous to the robust. Differing requirements and perspectives lead to a wide range of overlapping services being selected. If the CIO attempts to reassert control, there may be scope for solutions providing a sensible view across these competing services. Alternatively, there may be an explicit move to consolidate, creating opportunities for those able to assist in migrating services from one provider to another.

 Play Time

It may be deliberate, but is more likely to be an accidental byproduct of something else. Developers are casting the net wide, experimenting and exploring. The organisation may be littered with accounts for different cloud services that someone tried, explored for a while, and then moved away from. There may well be a large number of intranet-type applications that were built during the exploration phase, deployed to a cohort of internal users, and then simply left running.

Oops

It’s unlikely to have been a conscious business decision. No one knew someone else had signed up for that other cloud. The person who did it left last year, and we’re really not sure if there are some nasty dependencies in our upcoming product…

What do you think? Why are we seeing people using more than one public cloud, or more than one private cloud? Is it a trend that’s likely to continue? Is it an opportunity, or a cause for concern?

I suspect that the vast majority of ‘multi-public’ and ‘multi-private’ responses are not the result of a conscious and explicit decision to use more than one cloud. Instead, they’re simply a byproduct of the industry’s current state of evolution. One cloud, today, is better for some things, and its competitor is better for others. Mainstream enterprise adopters need both those capabilities, and therefore end up having to use two clouds.

Potential adopters of multi-cloud, though, would benefit from a more considered exploration of the issues. There are good reasons to learn and use more than one cloud, but there is significant scope for increased complexity in following such a path. Cloud management providers like Enstratius and RightScale may diminish some of that complexity, but it doesn’t go away altogether.

Multi-cloud should be a conscious decision.

Disclaimer: It is important to note that the RightScale survey does not identify the number of responses upon which analysis is being performed. The total number of survey respondents was 625, but subsequent discussion in their report appears to be based upon subsets of the whole. Figures on enterprise cloud usage, for example, are drawn from “organisations with a hybrid cloud strategy” (p.6). There is no information on the proportion of respondents classed as ‘enterprise,’ nor the proportion of those that may have a cloud strategy. The information that 29% currently run apps in a hybrid cloud may point to 29% of 625, or 29% of 3. Similarly, discussion (p.7) of multi-cloud implementation is based upon enterprise respondents with more than 1,000 employees! Unhelpfully, the descriptive text uses “all respondents,” “all enterprise respondents,” “respondents” and “enterprises” interchangeably. It would be useful if RightScale could provide explicit sample sizes next to each of their graphs.

Image of feet in a bed by Flickr user Nany Mata.

Related articles

• RightScale survey points toward preference for ‘multi-cloud’ deployments (zdnet.com)
• RightScale sees uptick in cloud adoption and multi-cloud use [GigaOM] (gigaom.com)
• Cloud adoption growing, but how big depends on who you ask (networkworld.com)
• RightScale State of the Cloud 2013: A New Industry Survey (rightscale.com)
• With Experience Comes Wisdom in Cloud Computing, Survey Shows (forbes.com)
• Survey Says: Multi-Cloud Usage Growing, DevOps on the Rise (datacenterknowledge.com)