But as Kasabi CTO Leigh Dodds is keen to stress, you don’t need to be a Semantic Web developer or enthusiast to derive value from what Kasabi has to offer.

Like many others in the nascent data market space, Kasabi’s commercial models are still emerging, and Leigh shares some of the thinking that lies behind their current proposition.

There is some background noise on this recording, which will hopefully not affect your enjoyment of the conversation.

Kasabi is an initiative of Talis. I am a former employee of and current shareholder in Talis.

Following up on a blog post that I wrote at the start of 2012, this is the ninth in an ongoing series of podcasts with key stakeholders in the emerging category of Data Markets.

Related articles

• Data Market Chat: the podcasts are a-coming…(cloudofdata.com)
• Data Market Chat: Tyler Bell discusses Factual (cloudofdata.com)
• Data Market Chat: Hjálmar Gíslason discusses DataMarket.com (cloudofdata.com)
• Data Market Chat: Chris Hathaway discusses AggData (cloudofdata.com)
• Data Market Chat: Flip Kromer discusses Infochimps (cloudofdata.com)
• Data Market Chat: Nick Edouard discusses BuzzData (cloudofdata.com)
• Data Market Chat: Stephen O’Grady of RedMonk examines the bigger picture (cloudofdata.com)
• Data Market Chat: Shion Deysarkar discusses Datafiniti (cloudofdata.com)
• Data Market Chat: Piyush Lumba discusses Microsoft’s Windows Azure Marketplace (cloudofdata.com)

Richard Wallis

Our conversation is also an excuse for an experiment. I have been producing audio-only podcasts here and elsewhere for a number of years, but have always tended to avoid producing video. It’s more effort, it requires more bandwidth at both ends of the conversation, and I’ve never really been convinced that it adds very much to a conversation between two people. Anecdotal evidence would also suggest that my current podcasts are consumed in environments where video would not work; washing dishes, walking dogs, and sitting on buses.

However, rather than just continue to presume that my biases are correct, I’ve decided to give video a try. Richard kindly agreed to participate, and the result is available on YouTube and embedded here.

An audio-only version is also available for download if you prefer. The introductory remarks in this version are slightly different to those on the video, as they come straight from the original conversation.

It’s perhaps unfair to draw too many conclusions from this first attempt, but a few things are immediately apparent. The whole process takes an awful lot longer. The files are larger, so processing and uploading times increase 2-3 fold. Uploading a separate audio file also takes a bit of time. Simply dumping the Skype recording into iMovie worked just fine… but I’ve (so far) not managed to find any way to balance the audio levels. Garageband lets me do this with my audio-only podcasts, but iMovie doesn’t seem to, so Richard’s side of the conversation comes across as quite a bit louder than mine.

Having done one, I’m still not convinced that the video adds anything to the conversation. But what do you think? If you’ve listened to any of my podcasts, please take a moment to complete the short survey over at SurveyMonkey. Your responses will help me to decide where to go next.

Many thanks.

Image by Kevin Krejci via Flickr

The CloudCamp unconference returned to London for the 14th time this evening, regaling a capacity crowd in the Crypt below Clerkenwell’s St James Church with several hours of discussion and debate on the somewhat elusive topic of ‘Big Data’.

Rather rough notes of the proceedings follow, after the break.

LEF‘s Simon Wardley kicked proceedings off as usual, once again managing to pepper an on-topic canter through the topic with a seemingly never-ending stream of Flickr images of cats… and analogies to electricity. You possibly had to be there? His core message, though? There’s nothing new under the sun… and the cycles of change just keep on coming.

Next, Peter Matthews from CA Labs, on “is big data mutually compatible with the cloud?” Erm, yes. Data volumes with big data are so large that it’s difficult to move it around… which creates opportunities for lock-in that vendors may wish to seize. And then he was out of time.

Next, Fujitsu’s Mark Wilson on ‘Structuring Big Data.’ He’s actually talking about Linked Data, a topic I’ve dug into before here and over on semanticweb.com – Linked Data could be/ might be the effective realisation of the decade-old Semantic Web dream. Big Data means masses of unstructured or semi-structured content, presenting a management headache of previously unanticipated proportions. Linked Data, he argues, creates the mechanism to link all of this data together from across disparate sources. Yes, but it’s easier to say than to do… And in 5 minutes he really couldn’t explain enough to persuade the audience. Linked Data should be “the optimal reference source,” he said. It should be “a broker for all data sources,” and we should “think about integration, not duplication.” Yeeeeees… But.

Next, Canonical’s Nick Barcet, talking around scalability, Ubuntu, package management, configuration management, etc. Not wholly sure what the point was, I’m afraid.

Next, Chris Swan from UBS – big data and security. “If you’ve got security controls that aren’t properly monitored, then they don’t matter.”

Next, Tom Leyden of Amplidata – Big “Unstructured” Data in the Cloud. Data storage to increase 30x over the next decade, but staff will only increase 50% over the same period. Challenge in the 90s, as existing storage and analysis technologies struggled to cope with new data volumes. Seeing similar problems today with data streaming from sensor web, etc. Traditional file systems cannot cope. Object Storage the way forward ?

Next, Alex Farquhar – “Cloud v Big Data.” Not really versus… but intersection of the two. Too much discussion of his company, Forward. Just talking about how his company uses cloud to provision IT resources. Might work as a conference presentation or case study – not sure it fits as a 5 minute lightning chat. Around 60TB of data at Forward. Diverse and vital. Using Hadoop cluster – 24 nodes on-premise. Rationale (proximity to the cluster) seemed odd. That can be true, but not clear that it really needs to be the case here?

Next, Alaric Snell-Pym, on Scaling Hadoop. Trying to overcome Hadoop’s I/O bottleneck. Explaining basics of Hadoop and Map/Reduce – no one else has. Explains use of HDFS and ‘selective reading’ to manage lots of small tables and overcome the problems of I/O.

Next, Matt Wood from Amazon. Talking about genetics and the human genome. It’s an analogy. Human Genome Project took years and millions of dollars. Development of gene sequencing machines led to a step change – dramatic drop in cost of sequencing DNA. Like the cloud, anyone? But… the machines create an analysis challenge, because they generate so much data. Cloud offers “collection of productivity tools” to help scientists work with this data collaboratively and (relatively) affordably. A perfect example of a lightning presentation, unlike most of those who preceded him.

And finally, an impromptu slot from HP’s Joe Weinman. A quick overview of current thinking behind his latest book. This one could have gone for much longer… Good stuff.

And that’s the lightning talks finished. Now, the panel, and Simon Wardley’s search for “experts” and “volunteers.”

…and unfortunately, your scribe was ‘volunteered’ as an ‘expert’ by Mr Wardley… and here end the notes. It was great to have Amazon’s Werner Vogels sneak in, and lob comments into the panel, though…

Great event, though with the usual mix of people you wish could have talked for longer… and people you wish wouldn’t have spoken.

Related articles

• The brave new world of big data & Hadoop (venturebeat.com)
• Big VCs Invest In Big Data Startup Continuuity (techcrunch.com)

Image of Stephen Wolfram via Wikipedia

British-born computer scientist Stephen Wolfram sees ongoing efforts to extend the Internet’s top-level domains (TLDs) beyond the familiar .com, .org, .uk etc as an opportunity to raise the profile of machine-readable data. In a blog post published yesterday, he argues that a new .data domain would increase “exposure of data on the internet—and [provide] added impetus for organizations to expose data in a way that can efficiently be found and accessed.” Whilst wholly in favour of Wolfram’s stated aim, I can’t help feeling that his suggested solution is at best unnecessary and at worst a worrying segregration of data from the ‘proper’ web that everyone else will continue to exploit.

Back in June of last year, the body responsible for coordinating the global domain name system approved a plan to permit new top-level domains (the letters after the final dot in an internet address — the .com in cloudofdata.com, the .uk in bbc.co.uk, the .edu in harvard.edu). Until recently, these top-level domains have been tightly controlled, with a small set of generic domains (.edu, .gov, .mil, .org, etc), a larger set of country domains (.uk, .fi, .nz, etc) and one or two others such as .eu. From tomorrow, anyone with $185,000 will be able to submit a proposal to create and manage a new top level domain, and it’s possible that there could eventually be thousands of them. Wolfram is keen to ensure that data doesn’t miss out on the ‘opportunity.’

As Wolfram himself recognises, there is already an awful lot of machine-readable data on the web. Some of it sits embedded within the web pages that humans read, with specially formatted code waiting to be triggered by the calendars, the address books, or the browser plugins of site visitors. Some of it is packaged up in data files, offered for download. And some of it waits inside a database, ready to be delivered in response to an API call or a query typed into a web form.

There is a growing enthusiasm for exposing this data for reuse. Government transparency agendas have driven public sector data sites like data.gov.uk and data.gov. Similarly, efforts such as data.open.ac.uk and data.southampton.ac.uk see universities beginning to consciously collect data sets together and offer them up for reuse. Similar efforts in the commercial world are less easy to point to, but that reticence has nothing whatsoever to do with the lack of a ford.data, boeing.data, ge.data or astrazeneca.data domain!

In some ways, the convention for gathering significant chunks of data on a data.xxx.yyy site echoes Wolfram’s intention, but with a number of advantages. Data without context is far less valuable than data with context. Much of that context may be inferred from the domain in which the data lives, with data delivered from a .gov or .edu (or .gov.uk or .ac.uk) site perhaps interpreted differently to data hosted on .com, .biz, or .xxx. Southampton University, the Open University, and the US Federal Government are able to gather data up and make it available for download via their existing data. sites if they choose. This offers human visitors to their sites a degree of convenience, whilst retaining the power and brand attributes of their existing domain. Gov.data, gov.uk.data, open.ac.uk.data, southampton.ac.uk.data, though? All are messy, in ways that Wolfram’s own wolfram.data would admittedly not be, and all are simply additional registrations that the institutions would have to pay for in order to stop someone else grabbing the domain.

At the end of the day, the machines don’t actually care. The existing data.open.ac.uk-type sites are human conveniences, not machine enablers. The computers, and the software they run, are quite capable of crawling the public web and finding accessible data wherever it lies on a site. There are plenty of reasons to continue embedding little snippets of data inside human readable web pages, regardless of whether you have a data.wolfram.com or a wolfram.data site. Content negotiation is becoming increasingly capable, such that there really is no need for what Wolfram calls a ‘parallel construct to the ordinary web’ at all. A human being arriving at a web site sees human readable content, whilst various software tools would automatically be presented with very different data or functions, optimised to their capabilities and requirements.

By all means, let us show the curious some of the existing techniques that work in making data more easily accessible. By all means, let us identify the gaps, the issues, the problems (none of which a new TLD even begins to address). Yes, let us definitely and unambiguously set about “highlighting the exposure of data on the internet—and providing added impetus for organizations to expose data in a way that can efficiently be found and accessed.”

But please, let us not be distracted by the false hope that adding yet another TLD to the babel that ICANN is about to unleash can do anything more than consign data to some online ghetto, wallowing unwanted, unloved and unused as companies and their customers lavish love, attention, and clicks upon the .com domain over on the ‘proper’ web.

Thanks to Raphaël Troncy, whose tweet first drew the story to my attention.

Related articles

• Is It Time For Computers To Have Their Own .Data Domains? (techcrunch.com)
• ICANN Pushes Ahead With January 12 Launch For New Top-Level Domains (wired.com)
• The biggest change in DNS since Dot-Com (wired.com)

Image via Wikipedia

For real-world applications of Linked Data and the Semantic Web, the long-running Semantic Technology Conference is hard to beat. For getting a real handle on the Cloud Computing landscape, GigaOM‘s Structure Conference is also a leading light.

Working across both areas as I do, these events tend to figure prominently in my calendar for the year*. Last year, both took place in San Francisco during the same week. I tried to attend both, and therefore succeeded in spending most of my week in cabs, shuttling between meetings at the two venues. I saw very few sessions that I wasn’t personally involved in, and the experience wasn’t a huge success.

This year the conference organisers have taken pity on me, and moved their events to opposite ends of June. The Semantic Technology Conference is up first, back at the Hilton Union Square from 5-9 June. Structure follows, returning to the Mission Bay Conference Centre on 22 and 23 June.

I’ll be attending both, and probably doing various official things during each event. At the moment, the only thing we’ve definitely nailed down is a special live appearance by The Semantic Link crew on the evening of 5 June; we’ll be taking a look at the highlights expected for the conference, offering some tips for those new to the event and its multitude of parallel sessions, and generally bringing our usual podcast chatter to the stage.

So if you’re in town around the time of either event, do get in touch; it promises to be an interesting month.

* Disclosure: WebMediaBrands pay me to host the monthly Semantic Link podcast, and to write a monthly column on SemanticWeb.com. GigaOM pay me to curate the Infrastructure/ Cloud Computing channel on their Pro site. I attended and participated in both of these events before that was the case, and still would today without the contractual relationship.

Related articles

• GigaOM Structure 2011 (datacenterknowledge.com)
• Semantic Technology Conference (fakeiitian.com)

• My latest monthly column on SemanticWeb.com; Big Data Presents a Big Opportunity?
• My latest weekly wrap-up on GigaOMPro; Rosslyn Analytics, Microsoft Finding Value in Data Aggregation
• The teaser piece on GigaOM’s public Cloud site; In Exploiting the Data Market, Trust Is Key

I spot a theme building…

I’ve been a fan of Om Malik‘s boutique analyst site, GigaOM Pro, pretty much from the outset, and happily renew my subscription each year. The site covers a wide range of industry topics, and those Quarterly Wrap-ups are worth the fee all by themselves.

I’ve written a few reports for them in the past, but was delighted when Mike Wolf got in touch to see if I fancied trying my hand at curation on their Infrastructure/Cloud channel.

So next week (from 31 January) I’m going to be gathering and commenting upon links from around the web, writing a daily ‘Today in Infrastructure,’ and finishing off with a Weekly Update. If you’re not (yet!) a subscriber, why not sign up for a free seven day trial and join me for the start of my little adventure?

If there’s something you think I should be covering, do let me know.

Related articles

• Free GigaOM Pro Webinar: The Scalable Cloud (gigaom.com)
• GigaOm raises $2.5M, claims 10,000 Pro subscribers (venturebeat.com)
• The Cloud has a place, even inside heavily regulated industries (cloudofdata.com)
• Talking Scalable Clouds with GigaOM Pro and Limelight Networks (cloudofdata.com)
• GigaOM Structure 2011 (datacenterknowledge.com)
• Join GigaOM at Big Data on March 23 in New York City (gigaom.com)

We recorded the first episode earlier this week and it’s now online, along with an introduction to the series from myself.

Please do have a listen, and let me know what you think whilst I work out how to automate adding these to my sidebar on this site…

I look forward to lining up some great conversations – and guests – in the year ahead. We will record and publish regularly each month. Where we can agree a topic well ahead of the recording date, I’ll let you know via the @TheSemanticLink twitter account, and invite questions for me to throw at both regulars and guests.

If there are particular topics you’d like to see covered, particular guests you’d like us to secure, or other comments you’d like to make, please do let me know…

Related articles

• Top 10 Semantic Web Products of 2010 (readwriteweb.com)
• Looking forward to The Semantic Link podcast on SemanticWeb.com (cloudofdata.com)

‘Big Data‘ is currently capturing the imagination, attracting hype, investment and ambitious startups in almost equal measure. Kim and Eric Norlin’s excellent Defrag and Glue events have gained big-name company, with O’Reilly‘s Strata and GigaOM‘s Structure both set to arrive in the first quarter of 2011. Venture firms like IA Ventures have emerged, specifically targeted at finding, funding, and profiting from the big Big Data idea. Giants of the web from Yahoo! and Amazon to Twitter and Facebook solve their own Big Data problems in very different ways, contributing valuable code and experience to the community whilst simultaneously diluting focus and adding to the cacophony.

Flippantly reckoned by many to be ‘anything that requires more than a single machine to run,’ the Big Data reality remains somewhat harder to pin down. To those seeking routine business insight, that mammoth Excel spreadsheet they laboriously query overnight at the end of each month might quite justifiably be thought of as ‘Big.’ At the other end of the scale, data wizards scorn anything that doesn’t require a room full of servers, a mountain of empty pizza boxes, and the careful construction of a bespoke data ingest, management and querying system atop the most bare-bones version of the Linux kernel they can find. Somewhere between the two, a growing mass of cheaply gathered data holds out the promise of invaluable insight. Remote sensors, web clickstreams, social graph interactions, purchaser (and non-purchaser) behaviours. All these, and more, have much to tell planners, builders, makers, sellers, and buyers. If only we could formulate the right questions. If only we could devise the right sampling strategies. If only we had big enough machines to ask lots of questions using lots of sampling strategies. If only we had big enough machines to not bother sampling at all.

On the hardware side of things, even humble domestic laptops typically ship with at least two cores these days; two separate little computers ready to do the data processor’s bidding. Four, eight, sixteen and more cores are not far behind, but mainstream software products typically fail to exploit anything more than a single core. Push Excel as hard as you like, and it won’t do more than take one of your computer’s multiple cores to the max. On that 12-core Mac Pro you persuaded the boss to buy, only one core will be hard at work on your data. Twitter, Mail, YouTube, and ripping DVDs  will each be giving other cores a little light exercise whilst others sit idly by, waiting for the arrival of operating systems and applications capable of exploiting multi-core power. The same is true as jobs grow and move to run across multiple machines, whether under your desk, in your data centre, or out in the Cloud. Those big datasets need to be carved up and shared amongst the available computers before any analysis takes place. You’re typically not accessing a ‘big computer in the Cloud’ at all… but lots of relatively small (commodity) computers, and it takes careful planning and smart software to manage the division and recombination of those jobs in a cost-effective manner. Projects such as Joseph Hellerstein‘s Berkeley Orders of Magnitude (BOOM) begin to demonstrate some of the potential for working natively with multiple processors, but there’s a long way to go before those advances reach the mainstream.

Hadoop, Cassandra, MapReduce, Dynamo, Voldemort. These, and more, are solutions developed by the likes of Yahoo!, Facebook, Google, Amazon and LinkedIn to tackle the influx of data that each faced – and for which each had failed to find an existing solution. Hadoop, with the addition of Cloudera‘s commercial polish, is rapidly emerging as the front runner for an off the shelf Big Data solution, but all of these tools remain rather narrow in their abilities. Find the type of data or the nature of query for which each of these was built and its performance will be unbeatable, but we are a very long way from Big Data’s equivalent of the jack-of-all-trades SQL-powered relational database of old.

And there, for many enterprises, lies the problem. Useful Google searches require the crawler, index and UI to do a relatively small number of essentially similar tasks, very quickly, very cost-effectively, and at massive scale. Focus on that finite set of problems, and you build a solution that delivers the experience we’ve all come to know. Each type of data manipulation or analysis requires a different tool, differently optimised, with the inevitable result that a typically diverse organisation may require a plethora of Big Data tools to get their work done. Or they might just continue to muddle along with Oracle or mySQL, churning inefficiently through their data analysis jobs for interminably long periods of time. These relational database tools are understood, they are mature, and they get the job done. Except in the most data-intensive industries, they have a market presence that will be difficult to disrupt.

The Big Data space is seeing remarkable innovation, but there is a long way to go in order to lift it out of the domain of the technically proficient specialist and place it on desktops across the organisation. As IA Ventures’ Brad Gillespie notes, “Excel is where the world’s data lives… [and] Big Data has to get to that place… so that a CMO can leverage it directly.”

And in all of this fervent of innovation, to return to the title of the post, it strikes me that Big Data is becoming disconnected from the fabric of the web itself. Oh, much of the data certainly comes from the Web, and a lot of it might even be queried on the Web after processing. But, somewhere along the line, the linkedness of the Web has either been forgotten or ignored. That rich set of connections, interconnections and associations has been reduced to a table, an index, or a (large) set of key-value pairs. And in the process, something fundamental has gone away.

This is enough for now, though. Looking more closely at different Big Data approaches, and exploring the potential for re-introducing the Web must wait for future posts.

Related articles

• Cloudera raises $25M to help deal with the enterprise data deluge (venturebeat.com)
• Strata Week: Building data startups (radar.oreilly.com)
• Big Data and a Critique of Geek Culture (readwriteweb.com)
• Big Data and NoSQL March to the Enterprise (nytimes.com)
• Does ‘big data’ equal big opportunity for storage vendors? (news.cnet.com)
• New York Times Event Shows the Promise of Big Data (programmableweb.com)
• Overwhelmed Executives Still Crave Big Data, Says Survey (readwriteweb.com)

Almost exactly two years ago, I announced my intention to leave Talis and strike out on my own. A year ago, the final piece of that transition concluded when we published the last episode of the Semantic Web Gang; a round table podcast that I began inside Talis and subsequently produced with their sponsorship.

Fast forward to the end of 2010, and it’s time to revisit the value that a monthly round table conversation offered to the community. Talking with friends, customers and contacts, many of them pointed to the useful role that something like the round table discussion format plays in cutting through marketing hype, community groupthink and competitor obfuscation. The old Semantic Web Gang did that, and it did it because it comprised a great group of contributors who knew their stuff.

Working with Eric Franzon at semanticweb.com we are about to recreate some of that value by launching The Semantic Link, and I am really pleased with the panel of regulars that Eric and I have signed up; a completely new group of contributors who also know their stuff.

Our first episode will be recorded in early December, and should be pushed out via semanticweb.com (and iTunes, no doubt) before Christmas. We’ll then settle down to a regular cycle, recording and publishing each month, dissecting whichever trends, stories, initiatives and companies happen to be topical. I’m also keen to get guests on the show from time to time, especially where they’re involved in something truly interesting and newsworthy.

If you have ideas for topics you would like to hear us dissect, please do let me know.