Image by Kevin Krejci via Flickr

The CloudCamp unconference returned to London for the 14th time this evening, regaling a capacity crowd in the Crypt below Clerkenwell’s St James Church with several hours of discussion and debate on the somewhat elusive topic of ‘Big Data’.

Rather rough notes of the proceedings follow, after the break.

LEF‘s Simon Wardley kicked proceedings off as usual, once again managing to pepper an on-topic canter through the topic with a seemingly never-ending stream of Flickr images of cats… and analogies to electricity. You possibly had to be there? His core message, though? There’s nothing new under the sun… and the cycles of change just keep on coming.

Next, Peter Matthews from CA Labs, on “is big data mutually compatible with the cloud?” Erm, yes. Data volumes with big data are so large that it’s difficult to move it around… which creates opportunities for lock-in that vendors may wish to seize. And then he was out of time.

Next, Fujitsu’s Mark Wilson on ‘Structuring Big Data.’ He’s actually talking about Linked Data, a topic I’ve dug into before here and over on semanticweb.com – Linked Data could be/ might be the effective realisation of the decade-old Semantic Web dream. Big Data means masses of unstructured or semi-structured content, presenting a management headache of previously unanticipated proportions. Linked Data, he argues, creates the mechanism to link all of this data together from across disparate sources. Yes, but it’s easier to say than to do… And in 5 minutes he really couldn’t explain enough to persuade the audience. Linked Data should be “the optimal reference source,” he said. It should be “a broker for all data sources,” and we should “think about integration, not duplication.” Yeeeeees… But.

Next, Canonical’s Nick Barcet, talking around scalability, Ubuntu, package management, configuration management, etc. Not wholly sure what the point was, I’m afraid.

Next, Chris Swan from UBS – big data and security. “If you’ve got security controls that aren’t properly monitored, then they don’t matter.”

Next, Tom Leyden of Amplidata – Big “Unstructured” Data in the Cloud. Data storage to increase 30x over the next decade, but staff will only increase 50% over the same period. Challenge in the 90s, as existing storage and analysis technologies struggled to cope with new data volumes. Seeing similar problems today with data streaming from sensor web, etc. Traditional file systems cannot cope. Object Storage the way forward ?

Next, Alex Farquhar – “Cloud v Big Data.” Not really versus… but intersection of the two. Too much discussion of his company, Forward. Just talking about how his company uses cloud to provision IT resources. Might work as a conference presentation or case study – not sure it fits as a 5 minute lightning chat. Around 60TB of data at Forward. Diverse and vital. Using Hadoop cluster – 24 nodes on-premise. Rationale (proximity to the cluster) seemed odd. That can be true, but not clear that it really needs to be the case here?

Next, Alaric Snell-Pym, on Scaling Hadoop. Trying to overcome Hadoop’s I/O bottleneck. Explaining basics of Hadoop and Map/Reduce – no one else has. Explains use of HDFS and ‘selective reading’ to manage lots of small tables and overcome the problems of I/O.

Next, Matt Wood from Amazon. Talking about genetics and the human genome. It’s an analogy. Human Genome Project took years and millions of dollars. Development of gene sequencing machines led to a step change – dramatic drop in cost of sequencing DNA. Like the cloud, anyone? But… the machines create an analysis challenge, because they generate so much data. Cloud offers “collection of productivity tools” to help scientists work with this data collaboratively and (relatively) affordably. A perfect example of a lightning presentation, unlike most of those who preceded him.

And finally, an impromptu slot from HP’s Joe Weinman. A quick overview of current thinking behind his latest book. This one could have gone for much longer… Good stuff.

And that’s the lightning talks finished. Now, the panel, and Simon Wardley’s search for “experts” and “volunteers.”

…and unfortunately, your scribe was ‘volunteered’ as an ‘expert’ by Mr Wardley… and here end the notes. It was great to have Amazon’s Werner Vogels sneak in, and lob comments into the panel, though…

Great event, though with the usual mix of people you wish could have talked for longer… and people you wish wouldn’t have spoken.

Related articles

• The brave new world of big data & Hadoop (venturebeat.com)
• Big VCs Invest In Big Data Startup Continuuity (techcrunch.com)

Image via CrunchBase

Chris Hathaway sees basic location information scattered across the websites of hundreds — or thousands — of coffee shop chains, hotel groups, and fast food joints, but argues that it’s almost impossible to do anything more sophisticated with the data than find your closest Starbucks. His company, AggData, is attempting to fill what he sees as a gap in the market; scraping addresses and other facts off company websites to create simple files of store locations that can then be enriched with coordinate data and sold.

Customers for this data include competitors, market researchers, consultants, and even the companies themselves; as is so often the case, it can be easier to buy data on store locations from a third party than to find the authoritative sources within your own organisation. AggData is strongest in the US today, but also offers a growing body of data for other countries. Although the data files are structurally simple, Chris sees plenty of opportunity to continue collecting and selling data to a growing community of customers.

Unlike Factual, which was the focus of last week’s podcast, AggData is not currently interested in combining data from different sources. Customers download separate files on the locations of Starbucks, Peets and Tim Hortons, and not a single aggregated set of coffee shop locations. The AggData model is also predicated upon using their own scripts to extract data from third party sites; asked if he would accept a file of WalMart store locations supplied by WalMart, Hathaway explained why he would — and does — decline.

Have a listen to learn more about AggData, and to hear Chris’ perspectives on the potential role of semantic technologies in making his job easier. And check back on Thursday for the next podcast in the series; Hjálmar Gíslason of DataMarket.com.

Following up on a blog post that I wrote at the start of 2012, this is the second in a series of podcasts with key stakeholders in the emerging category of Data Markets. Other conversations, all of which will be published here, have been scheduled with BuzzData, DataMarket.com, Factual, Infochimps, Kasabi, and Microsoft. I am still adding conversations to the series, and intend to talk with more companies and with analysts and investors with insight to share. 

Related articles

• Nurturing the market for Data Markets (cloudofdata.com)
• Data Market Chat: the podcasts are a-coming… (cloudofdata.com)
• Data Market Chat: Tyler Bell discusses Factual (cloudofdata.com)

Image via Wikipedia

For real-world applications of Linked Data and the Semantic Web, the long-running Semantic Technology Conference is hard to beat. For getting a real handle on the Cloud Computing landscape, GigaOM‘s Structure Conference is also a leading light.

Working across both areas as I do, these events tend to figure prominently in my calendar for the year*. Last year, both took place in San Francisco during the same week. I tried to attend both, and therefore succeeded in spending most of my week in cabs, shuttling between meetings at the two venues. I saw very few sessions that I wasn’t personally involved in, and the experience wasn’t a huge success.

This year the conference organisers have taken pity on me, and moved their events to opposite ends of June. The Semantic Technology Conference is up first, back at the Hilton Union Square from 5-9 June. Structure follows, returning to the Mission Bay Conference Centre on 22 and 23 June.

I’ll be attending both, and probably doing various official things during each event. At the moment, the only thing we’ve definitely nailed down is a special live appearance by The Semantic Link crew on the evening of 5 June; we’ll be taking a look at the highlights expected for the conference, offering some tips for those new to the event and its multitude of parallel sessions, and generally bringing our usual podcast chatter to the stage.

So if you’re in town around the time of either event, do get in touch; it promises to be an interesting month.

* Disclosure: WebMediaBrands pay me to host the monthly Semantic Link podcast, and to write a monthly column on SemanticWeb.com. GigaOM pay me to curate the Infrastructure/ Cloud Computing channel on their Pro site. I attended and participated in both of these events before that was the case, and still would today without the contractual relationship.

Related articles

• GigaOM Structure 2011 (datacenterknowledge.com)
• Semantic Technology Conference (fakeiitian.com)

• My latest monthly column on SemanticWeb.com; Big Data Presents a Big Opportunity?
• My latest weekly wrap-up on GigaOMPro; Rosslyn Analytics, Microsoft Finding Value in Data Aggregation
• The teaser piece on GigaOM’s public Cloud site; In Exploiting the Data Market, Trust Is Key

I spot a theme building…

‘Big Data‘ is currently capturing the imagination, attracting hype, investment and ambitious startups in almost equal measure. Kim and Eric Norlin’s excellent Defrag and Glue events have gained big-name company, with O’Reilly‘s Strata and GigaOM‘s Structure both set to arrive in the first quarter of 2011. Venture firms like IA Ventures have emerged, specifically targeted at finding, funding, and profiting from the big Big Data idea. Giants of the web from Yahoo! and Amazon to Twitter and Facebook solve their own Big Data problems in very different ways, contributing valuable code and experience to the community whilst simultaneously diluting focus and adding to the cacophony.

Flippantly reckoned by many to be ‘anything that requires more than a single machine to run,’ the Big Data reality remains somewhat harder to pin down. To those seeking routine business insight, that mammoth Excel spreadsheet they laboriously query overnight at the end of each month might quite justifiably be thought of as ‘Big.’ At the other end of the scale, data wizards scorn anything that doesn’t require a room full of servers, a mountain of empty pizza boxes, and the careful construction of a bespoke data ingest, management and querying system atop the most bare-bones version of the Linux kernel they can find. Somewhere between the two, a growing mass of cheaply gathered data holds out the promise of invaluable insight. Remote sensors, web clickstreams, social graph interactions, purchaser (and non-purchaser) behaviours. All these, and more, have much to tell planners, builders, makers, sellers, and buyers. If only we could formulate the right questions. If only we could devise the right sampling strategies. If only we had big enough machines to ask lots of questions using lots of sampling strategies. If only we had big enough machines to not bother sampling at all.

On the hardware side of things, even humble domestic laptops typically ship with at least two cores these days; two separate little computers ready to do the data processor’s bidding. Four, eight, sixteen and more cores are not far behind, but mainstream software products typically fail to exploit anything more than a single core. Push Excel as hard as you like, and it won’t do more than take one of your computer’s multiple cores to the max. On that 12-core Mac Pro you persuaded the boss to buy, only one core will be hard at work on your data. Twitter, Mail, YouTube, and ripping DVDs  will each be giving other cores a little light exercise whilst others sit idly by, waiting for the arrival of operating systems and applications capable of exploiting multi-core power. The same is true as jobs grow and move to run across multiple machines, whether under your desk, in your data centre, or out in the Cloud. Those big datasets need to be carved up and shared amongst the available computers before any analysis takes place. You’re typically not accessing a ‘big computer in the Cloud’ at all… but lots of relatively small (commodity) computers, and it takes careful planning and smart software to manage the division and recombination of those jobs in a cost-effective manner. Projects such as Joseph Hellerstein‘s Berkeley Orders of Magnitude (BOOM) begin to demonstrate some of the potential for working natively with multiple processors, but there’s a long way to go before those advances reach the mainstream.

Hadoop, Cassandra, MapReduce, Dynamo, Voldemort. These, and more, are solutions developed by the likes of Yahoo!, Facebook, Google, Amazon and LinkedIn to tackle the influx of data that each faced – and for which each had failed to find an existing solution. Hadoop, with the addition of Cloudera‘s commercial polish, is rapidly emerging as the front runner for an off the shelf Big Data solution, but all of these tools remain rather narrow in their abilities. Find the type of data or the nature of query for which each of these was built and its performance will be unbeatable, but we are a very long way from Big Data’s equivalent of the jack-of-all-trades SQL-powered relational database of old.

And there, for many enterprises, lies the problem. Useful Google searches require the crawler, index and UI to do a relatively small number of essentially similar tasks, very quickly, very cost-effectively, and at massive scale. Focus on that finite set of problems, and you build a solution that delivers the experience we’ve all come to know. Each type of data manipulation or analysis requires a different tool, differently optimised, with the inevitable result that a typically diverse organisation may require a plethora of Big Data tools to get their work done. Or they might just continue to muddle along with Oracle or mySQL, churning inefficiently through their data analysis jobs for interminably long periods of time. These relational database tools are understood, they are mature, and they get the job done. Except in the most data-intensive industries, they have a market presence that will be difficult to disrupt.

The Big Data space is seeing remarkable innovation, but there is a long way to go in order to lift it out of the domain of the technically proficient specialist and place it on desktops across the organisation. As IA Ventures’ Brad Gillespie notes, “Excel is where the world’s data lives… [and] Big Data has to get to that place… so that a CMO can leverage it directly.”

And in all of this fervent of innovation, to return to the title of the post, it strikes me that Big Data is becoming disconnected from the fabric of the web itself. Oh, much of the data certainly comes from the Web, and a lot of it might even be queried on the Web after processing. But, somewhere along the line, the linkedness of the Web has either been forgotten or ignored. That rich set of connections, interconnections and associations has been reduced to a table, an index, or a (large) set of key-value pairs. And in the process, something fundamental has gone away.

This is enough for now, though. Looking more closely at different Big Data approaches, and exploring the potential for re-introducing the Web must wait for future posts.

Related articles

• Cloudera raises $25M to help deal with the enterprise data deluge (venturebeat.com)
• Strata Week: Building data startups (radar.oreilly.com)
• Big Data and a Critique of Geek Culture (readwriteweb.com)
• Big Data and NoSQL March to the Enterprise (nytimes.com)
• Does ‘big data’ equal big opportunity for storage vendors? (news.cnet.com)
• New York Times Event Shows the Promise of Big Data (programmableweb.com)
• Overwhelmed Executives Still Crave Big Data, Says Survey (readwriteweb.com)

Almost exactly two years ago, I announced my intention to leave Talis and strike out on my own. A year ago, the final piece of that transition concluded when we published the last episode of the Semantic Web Gang; a round table podcast that I began inside Talis and subsequently produced with their sponsorship.

Fast forward to the end of 2010, and it’s time to revisit the value that a monthly round table conversation offered to the community. Talking with friends, customers and contacts, many of them pointed to the useful role that something like the round table discussion format plays in cutting through marketing hype, community groupthink and competitor obfuscation. The old Semantic Web Gang did that, and it did it because it comprised a great group of contributors who knew their stuff.

Working with Eric Franzon at semanticweb.com we are about to recreate some of that value by launching The Semantic Link, and I am really pleased with the panel of regulars that Eric and I have signed up; a completely new group of contributors who also know their stuff.

Our first episode will be recorded in early December, and should be pushed out via semanticweb.com (and iTunes, no doubt) before Christmas. We’ll then settle down to a regular cycle, recording and publishing each month, dissecting whichever trends, stories, initiatives and companies happen to be topical. I’m also keen to get guests on the show from time to time, especially where they’re involved in something truly interesting and newsworthy.

If you have ideas for topics you would like to hear us dissect, please do let me know.

Bill’s slides are here, and mine are embedded below.

Thanks to Paul for the invite, and to the audience for braving an intermittently (very) wet Manchester evening to spend a few hours discussing Linked Data, the Semantic Web, and related topics; it was fun.

Related articles by Zemanta

• Google Acquires Metaweb To Make Search Smarter (techcrunch.com)
• Tim Berners-Lee Doesnt Seem to Think Linked Data Requires RDF (ironick.typepad.com)
• CubicWeb – The Semantic Web is a Construction Game!: OSCON 2010, The O’Reilly Open Source Convention – O’Reilly Conferences, July 19 – 23, 2010, Portland, OR (oscon.com)
• 15 sessions on marketing and the semantic web (chiefmartec.com)

A somewhat hectic June concluded with my second trip of the month to California. Whilst the first excursion led to Santa Clara and an interesting insight into attempts by Hitachi Data Systems to reinvent its relationship with the wider community, the second journey took me to San Francisco and a pair of events that lie at the heart of the advances that underpin much of my current work.

The Semantic Technology Conference celebrated its sixth year in 2010, moving from San Jose to the Hilton Union Square in Downtown San Francisco. This was my fourth (I think!) year, and alongside the “30% increase in attendance” to around 1,200, I noticed a refreshing realism amongst presenters, exhibitors and attendees; the hyperbole and inflated expectations of the “Google killers” thankfully seemed mostly to have been replaced by good ideas, sound business models, and steadily growing customer bases. Richard MacManus has been doing a good job of distilling some of the news from the event, and pushing it out on ReadWriteWeb.

Across town at UCSF’s Mission Bay Campus, GigaOM‘s smaller and younger Structure Conference broke out of its previous 1-day format to pack two days with various insights into the shifting Cloud Computing landscape; insights that were reported as they happened on GigaOM and elsewhere, and preserved on video for you to watch at your leisure.

Both events were smoothly run (at least one even brings in the same professional A/V team every year, don’t they Curtiss?), buzzing, and well worth attending. Unfortunately, for me at least, ‘attendance’ meant a near-unending stream of meetings and briefings in and around the venues, rather than much time spent listening to either carefully orchestrated programme. Lining up briefings, meetings and the rest seemed like such a good idea as this trip was planned. Participating in two events simultaneously, I persuaded myself, would be fine. And it mostly was, even though there were definitely times that I couldn’t quite remember where I was. Sitting down with John Hagel, looking blank, then saying “Hmm… so tell me about the Cloud,” was probably not the best way to get started with someone I’d been looking forward to meeting for a very long time. Delayed jetlag from the week before? Jetlag from the outbound flight earlier that same week? MUNI-lag from crossing back and forth between venues? Too much coffee? Not having quite got around to eating that day? Too many appointments in rapid succession? Or just impending senility? Whatever the reason, Hagel coped admirably with his pathetic interviewer’s opening gambit, and we went on to have a great conversation. More on that, his latest book, and his team’s Cloud Computing: Storms on the Horizon paper in due course, I suspect.

Despite starting from very different places (a sizeable chunk of the SemTech audience would not be out of place at an academic conference, whilst most of Structure’s attendees probably stalked the halls of Tim O’Reilly’s earliest Web 2.0 events), the perceived gulf between the real worlds of Cloud Computing and the Semantic Web grows narrower all the time. Companies like Facebook attend one to talk about the Open Graph protocol, and address issues of scale at the other. With Cloud infrastructures’ (yes, that apostrophe is correctly placed) capabilities ever-more commoditised by the day, value and differentiation move toward putting those assets to work in tackling real business challenges; challenges that require data, challenges that depend upon business buy-in, and challenges that benefit from lessons the Semantic Web community has learned in grappling with its transition toward a range of (hopefully) sustainable data businesses.

I have long believed in the fusion of the two (‘Cloud of Data,’ after all), but it was both refreshing and reassuring to observe the growing convergence in hallway conversation at each event. At SemTech, many were reaching toward ‘the Cloud’ (mostly, they meant ‘using Amazon’) in its simplest, purest, form. At Structure, many of those with whom I spoke were reaching for Semantic Technologies and some of Linked Data’s promise, without initially recognising that for which they sought. The Semantic Web, to them, was all too often just a discredited pipe dream from the Web’s distant past. The Semantic Web community of today is a very different place, and one with which Structure’s attendees might find much in common. The danger for the Semantic Web community, of course, is that they once more become absorbed in internal debate or the purity of the model. They may have powerful ideas and technologies to offer, but they need to reach out and describe those in ways that resonate with an industry that is ripe for hearing these messages. Delay too long, or descend too far into re-imagining technical underpinnings, and this fleeting opportunity will be lost; the bright young things of Silicon Valley will just develop something else that does the job almost as well, then iterate rapidly as they mine the flow to observe what works, and what doesn’t.

Personally, I am still absorbing the cacophony of ideas, contacts and opportunities arising from my week in San Francisco, but I am sure that there are plenty of projects, stories, podcasts and other follow-ups to appear here and elsewhere over the next few months.

And finally, an appeal to the organisers; can you locate these events a little closer together next year, please?!

Disclosures: Costs for my trip to Santa Clara were met in full by HDS. Both the Semantic Technology Conference and GigaOM’s Structure Conference supplied Analyst Passes for admission to their events.

Related articles by Zemanta

• Get an Insider’s Knowledge of the Cloud at Structure 2010 (gigaom.com)
• The Value of the Cloud: It’s Not Just About Cost (gigaom.com)
• Semantic Web Gang podcast looks back at the Semantic Technology Conference (zdnet.com)
• Attendance Up 30% at SemTech 2010, the World’s Largest Semantic Technology Conference (prweb.com)
• Facebook & The Semantic Web (downes.ca)
• Velocity, Structure Conference Coverage (datacenterknowledge.com)

The final version of that report is available today, both in commentable form via JISC’s JISCPress tool and for download as a PDF.

JISC’s associated call to fund Linked Data projects will be available imminently, and £750,000 is available to share between successful bidders. Watch the JISC funding pages, and get ready to bid. If you’re not inside a UK university yourself, then find a friendly partner who is… as projects have to be led by a University. And good luck!

Related articles by Zemanta

• DRAFT report explores Linked Data potential in UK Universities (cloudofdata.com)

“GoodRelations is a language that can be used to describe very precisely whatyour business is offering. Some people call GoodRelations a ‘data dictionary’, others prefer ‘schema’ or ‘ontology’. But the name of the thing is not important. Important is that you can use GoodRelations to create a small data package that describes your productsand their features and prices, yourstores and opening hours, payment options and the like.

You simply paste this data package into your Web page using W3C’s RDFa format.”

Martin and Jamie talk about the vocabulary itself, before moving on to the some of the ways in which it transforms commerce on the Web for early adopters such as Best Buy and O’Reilly.

The conversation – and its implications – are also covered in this post on ZDNet today. Links to some of the sites and resources mentioned during our chat are included here, after the fold.

• Best Buy
• Bing
• DBpedia
• DERI
• Drupal Ubercart
• GoodRelations
• GoodRelations Cookbook
• GoodRelations Developer Wiki
• Google
• Google Product Feed Converter
• HTML 5
• Freebase
• Linked Data
• Magento
• Metaweb
• O’Reilly Media
• osCommerce
• Ping The Semantic Web
• RDF
• RDFa
• RDFS
• Sindice
• SPARQL
• STI
• UPC Code
• Universität der Bundeswehr München
• VirtueMart
• Yahoo!
• Yahoo! BOSS

Related articles by Zemanta

• Example Best Buy Product RDFa Markup Released (beta) (jay.beweep.com)
• Best Buy jump starts data web marketing (chiefmartec.com)
• Talking with Martin Hepp about solving the paradox of choice (jonudell.net)
• Thoughts on RDFa (jay.beweep.com)