Here’s a stream of consciousness from the morning’s keynotes at this sold-out event.

Conference chair Edd Dumbill is introducing things, talking about William Smith‘s nineteenth century map of geological strata in the British Isles, the rise of industrialisation, and the move to towns. Edd suggests that a similar set of inflections are happening today in the world of data; ‘the start of something big.’

“In the same way that the industrial revolution changed what it meant to be human, the data revolution is changing what it means to be alive.”

The first of this morning’s keynotes; Hilary Mason from link shortener bit.ly.

Data and the people who work with data; “The state of the data union is strong.” Data scientists have an identity – a place to rally around – with Strata.

We have accomplished much, begging, borrowing and stealing from lots of domains. We have the tools. We have the capacity to spin up infrastructure in the Cloud. We have the algorithms to explore data, and to learn from it.

The most important thing we have now that we didn’t have before… is momentum. People are paying attention.

There are still challenges though. Timeliness of data is an issue, especially in real-time. We need to develop systems that can do robust analysis against a moving stream of data. We need to be able to store data in ways that let us operate on it in real-time. Hadoop… amazing ‘because I can run a query and get the result back before I forget why I submitted the query in the first place.’ We need training. We need imagination, not more ad optimisation networks. We have a real opportunity to do something better.

Opportunities (expressed in context of bit.ly); Bit.ly gets lots of data from people shrinking web links. They learn a lot about people; what they like, what they want, what they’re doing. bit.ly also gets rich segmentation data; location, context, etc. bit.ly sees global data, for example clicks on bit.ly links from Egyptian domains.

Now that we have all this data, it offers a window on to the world. What can we do with it? Make the world a better place? What would you do with all of this data?

Next up, James Powell from Thomson Reuters to talk about privacy and behavioural data in B2B contexts. Thomson Reuters gathers large amounts of global data, and filters it for customers. Time and context key; 700,000 updates a second through financial systems, 5,000,000 documents per day served through Open Calais, etc. Thomson Reuters interested in ways to filter information better.

Need to think about B2B implications of behavioural data, especially as we sell/exchange increasing volumes of data with partners. Consumers reasonably comfortable with giving up some personal data in return for a ‘better’ product (Amazon recommendations, etc), that probably doesn’t scale to the enterprise. For example, Open Calais customers submitting large numbers of dummy queries to obfuscate what they’re really looking for…

Key problem that needs to be addressed is ambiguity; many systems in this space still rely upon implicit assumptions, whilst the enterprise is used to explicit contracts. Tension – or recipe for disaster?

Keys to success – need to treat behavioural data differently/better, and avoid the mistake of simply continuing consumer trends.

Next, Mark Madsen from Third Nature, talking about ‘the Mythology of Big Data.’

Lots of assumptions underlying conversations about Big Data. ‘Every technology carries within itself the seeds of its own destruction.’ Code is a commodity; things that a lot of people have built profitable careers around have started to move down-market. Libraries, packages, etc make it easier for third parties to stitch things together rather than start from scratch.

The central myth underlying Big Data that’s erupted over the past 18-24 months; the myth of the gold rush. Everyone wants to be a data scientist. But just like the gold rush, success takes capital. It takes corporate engagement, and infrastructure. The ‘myth tells us you can go it alone… and you can’t.’

1950s-60s – data as product. 1970s-80s – data as byproduct. 1990s-2000s – data as assset. 2010- data as substrate (data as the basis for competition). ‘The real data revolution is in business structure and processes and how the use information.’

Using Big Data; the point isn’t necessarily about ‘Big.’ Much valuable data inside an enterprise is only GB or TB in size. We get tied up in ‘big’ way too much. It’s not really about data either; it’s about applying data. Without an application, it’s trivia.

Next, Amazon CTO Werner Vogels. An overview of how Amazon Web Services look at the data processing being done on their infrastructure by customers… Government, Finance, COmmerce, Pharma… all making use of tools. Plugging The Fourth Paradigm book from Microsoft Research (which is very good).

Vogels – big data is big data when your data sets become so large that you have to innovate to manage them. Customers view big data as collection and curation of data for competitive advantage… with the presumption that bigger is better. For recommendations etc, that is probably true.

There are a number of categories of data, where quality is far more important than quantity.

In the past, data tended to be collected to answer questions. Now, trend to collecting as much as possible before developing the questions you want answered, and the algorithms you will need to use for the analysis.

To do this, you should not be worried by data storage, data processing, etc – which is why you should embrace the scalable Cloud.

Data analysis pipeline; collect – store – organise – analyse – share.

AWS Import/Export – “you shouldn’t underestimate the bandwidth of a FedEx box.” Indeed.

“This is Day 1 for Cloud infrastructure.’

Next up, Microsoft’s Zane Adam talking about data marketplaces. Windows Azure DataMarket; Data as a Service, free or at cost. One stop shop for data (one of many one stop shops, unfortunately!) DataMarket is interesting… but this is far too much of a product pitch for the keynote track.

90 days since launch – 5,000+ subscriptions, 3 Million transactions to date. Given Microsoft’s presence and reach, aren’t those figures a bit low?

“There’s a lot of data out there… but it’s not all good.” A Data Marketplace gives customers access to good data. Does it? Do Microsoft vet every fact in a submitted data set? What would a single bad data set do to the marketplace’s brand recognition?

Regardless of where you stand on some of the questions of detail with respect to the Linked Data meme, it’s clear that significant enthusiasm is being marshalled behind both the concept and the opportunities that it promises.

Dion Hinchcliffe looks at some of the means by which enterprise data can be more visible on (and useful to) the Web in a ZDNet post this week. The ‘Semantic Web & Linked Data’ are included, and Dion writes;

“By far the most sophisticated and complex of the three approaches to open data presented here, [Linked Data is] highly suitable for certain applications that have rich data sets that need powerful means of processing and consumption. In particular, scientific, technical, medical, mapping, and certain government domains are highly suitable for this approach. It remains unclear if Linked Data will finally trigger the boom in the Semantic Web so use with care. However, definite consideration should be applied, given the potential of the approach to create data sets with extraordinarily high function. Businesses already managing their data with Semantic Web technologies will be the most likely candidates for adoption.”

We’re certainly seeing plenty of talk — and some interesting beginnings — in the Government domain, and organisations such as Thomson Reuters and the BBC are also taking compelling steps around the periphery of their core businesses.

Education offers another interesting set of opportunities, and Jason Ohler’s piece in Educause Quarterly (and a related podcast I recorded with him whilst I was still at Talis) illustrates one view of that opportunity.

Here in the UK, the Joint Information Systems Committee (JISC) is beginning to take note. They first funded a review of ‘Semantic Web Technologies‘ back in 2005, then revisited the topic with ‘Semantic Technologies in Learning and Teaching‘ (and a related workshop in the south of France later this year). I’ll be recording a podcast with the manager of that project, Thanassis Tiropanis, later this month.

JISC have also asked me to conduct a short piece of work to look specifically at the opportunity presented to the Higher Education community by Linked Data, and this work will run over the next few months. I’m certainly keen to learn about concrete examples, and to hear reasoned arguments for and against in order to submit comprehensive findings and recommendations. So if you have something to say, please do get in touch.

Related articles by Zemanta

• Web3 and Enterprise Linked Data. The Middleware Revolution (aldobucchi.com)
• Open government and Linked Data; now it’s time to draft… (johnbreslin.com)
• ReadWriteWeb Interview With Tim Berners-Lee, Part 1: Linked Data (readwriteweb.com)
• Linked Data is Blooming: Why You Should Care (readwriteweb.com)

Thomson Reuters‘ Open Calais team have clearly been busy, with several announcements at the Semantic Technology Conference here in San Jose.

On 15 June the company rolled out version 4.1 of Open Calais, embracing Spanish language content and the notion of ‘social tags;’

“OpenCalais is a great semantic data extraction engine. If you write an article about the relative merits ofPorsche and BMW at the test track in Leipzig, we’ll diligently identify Porsche and BMW as companies and Leipzig as a geography. We’ll create Linked Data URIs to represent these things and open up access to theLinked Data ecosystem so you can enhance your article with other content assets.

But… sometimes you just want a great description. The kind of tags a human would put on the article. Like “Car racing” or “Automobiles”. The kind of tag that would, for example, be very searchable and therefore …. SEO’able (that is definitely is not a word).

In 4.1 we’re introducing OpenCalais Social Tags. Social Tags is our attempt to emulate how a human might tag the document. Social Tags does some fairly sophisticated analysis of your entire document and maps it to a knowledgebase based on Wikipedia and other assets. From that process we generate Social Tags.“

This morning, they followed through with a further pair of announcements. Firstly, CNET has been joined by The Huffington Post, DailyMe and the UK’s Mail Online in integrating Open Calais into their workflow.

One of the more interesting aspects of the earlier CNET announcement was the contribution of data back into the pool;

“CNET joins Thomson Reuters as one of the first commercial media companies to publish core data assets for public, programmatic use on the open semantic Web. CNET will leverage OpenCalais’ connection to the rapidly expanding ‘Linked Data cloud’ to allow its original content — such as tech product reviews on laptops, TVs, smart phones, and digital cameras; news articles and blog posts from its CNET News editorial staff; and parts of its core technology product catalog – to be available for public use.”

It will be interesting to see whether these latest media properties are able and willing to do something similar.

The second element announced today in many ways mirrors Amazon’s recent Import/Export service. Thomson Reuters, too, have recognised that it remains impractical to move large quantities of data over the network, and today announced their ‘Archive Express’ which will process up to 20 million documents off a physical storage device within 24 hours, free of charge.

Related articles by Zemanta

• CNET Partners with Thomson Reuters on Linked Data Initiative (readwriteweb.com)
• Linked Data and the Enterprise: a viable two-way street (cloudofdata.com)
• Semantic technology gains publishing foothold (news.cnet.com)

Harvard University’s Berkman Center for Internet & Society unveiled their Media Cloud research tool today, bringing Semantic Web goodness from Thomson Reuters’ Calais and affordably scalable Cloud oomph from Amazon Web Services together in powering exploration of a fascinating topic.

As the press release notes,

“Media Cloud was conceived by Berkman Fellow Ethan Zuckerman and Berkman Faculty Co-Director Yochai Benkler.  It was inspired by their debate over whether the blogosphere largely echoed traditional media or was instead a source for original news and democratic agenda-setting.

‘While daily newspapers struggle for survival, political, niche and special interest blogs continue to capture consumer interest,’ said Yochai Benkler, Faculty Co-Director of the Berkman Center. ‘In the midst of this upheaval, it is difficult to know where stories begin, who sets the agenda, and how these dramatic changes impact news coverage on the whole.  We created Media Cloud to help researchers and the public get quantitative answers to these challenging questions.’”

The site itself provides more detail, notably,

“Print newspapers are declaring bankruptcy nationwide. High-profile blogs are proliferating. Media companies are exploring new production techniques and business models in a landscape that is increasingly dominated by the Internet. In the midst of this upheaval, it is difficult to know what is actually happening to the shape of our news. Beyond one-off anecdotes or painstaking manual content analysis, there are few ways to examine the emerging news ecosystem.”

By systematically harvesting full-text content from both new and traditional news sources and passing it through Calais for entity extraction, the Berkman team is able to build a coherent and normalised pool of news that they – and others – can begin to interrogate in order to answer pressing questions as to the shifting news landscape. Today’s site is just the beginning, and the team has ambitious plans to increase the number of sources they harvest, to provide ever-richer visualisations on their site, and to expose public APIs to the underlying data in order that third parties can consume it for their own purposes.

Ahead of the launch I spoke with Barak Pridor, CEO of Thomson Reuters‘ ClearForest subsidiary, and Berkman Fellow Stephen Schultze. Both stressed the beta nature of the site, and the fact that the team are still working to optimise a number of areas. That aside, I was impressed by some of the early results.

Barak pointed to the role Calais is playing in enabling the Berkman team to conduct comparisons within and between different news sources, extracting key entities (personal names, companies, places, etc) from the stream of text and normalising the various ways in which we refer to them (IBM, International Business Machines, etc).

Stephen discussed some of the background to the project, and was keen to emphasise the platform nature of their activity; although focussed on a destination web site today, the intention is to expose much of the data via a series of APIs that third parties will be able to consume. In the context of recent comparable moves from the New York Times and Britain’s Guardian, this is clearly to be welcomed and will accelerate the innovation around this incomparable pool of content.

Whilst the site does not currently expose the full richness of the data being harvested, it is already possible to see a number of interesting visualisations.

The maps above, for example, compare the coverage devoted to all the world’s countries by three very different news organisations; the BBC, the Wall Street Journal and FOX News. The apparent non-coverage of Iran is surprising, and suggests an algorithm still in need of tweaking. That apart, it’s actually surprising to note the very similar spread of coverage. European snobbishness about the insularity of US news – and the farcical nature of FOX ‘news’ – may not survive exposure to this data, although we can, of course, reassure ourselves that ‘coverage’ doesn’t always equate to ‘insight,’ ‘analysis’ or ‘truth.’ As more data become available, even those preconceived notions may face an overdue battering.

The system is already capable of addressing more complex questions, and exploring temporal aspects of the ways in which stories break, spread and relate to one another across different media. Stephen shared a number of evolving visualisations that made it possible for me to explore several trends at once, and I look forward to these tools arriving on the site.

The project is directly funded by the Berkman Center at present, and a desire to enrich and stabilise the platform means that significant addition of new news resources may be some months off. I, for one, look forward to seeing more non-US resources such as the UK’s Financial Times and Guardian, or their international equivalents.

The project’s APIs are also being finalised at this point and will be released in due course, along with the source code upon which Media Cloud runs. It will be interesting to see the balance between API access and software download at that point.

Media Cloud represents an excellent example of the uses to which the technologies I cover can be put. It’s not a Semantic Web project. It’s not a Cloud Computing project. It’s an intriguing exploration, that puts those technologies (invisibly) to work in helping to get the job done. That’s the way it should be.

And, as Stephen commented when asked what he was most interested to see,

“the most amazing stuff will be what other folks use it for in asking and answering their own questions.”

Indeed.

Thomson Reuters is already supporting this project by contributing Calais and some technical expertise. I wonder whether some of our more enlightened news organisations might like to help the Berkman get more data in there, faster? As more newspapers close each week, and as we (supposedly) become ever-more insular in the choices we make about the news we consume, there’s a real need to understand the ways in which the media are changing. Many eyes on this data set is one sure way to ensure that the commentary moving forward is informed and informative.

Related articles by Zemanta

• Lawsuit to Determine Fair Use for Blog Links, Headlines (crooksandliars.com)
• Newspapers Suddenly Adapt To Socal Media; Nearly 60 Percent Offer User-Gen Content (paidcontent.org)
• Yes, Twitter is a source of journalism (mathewingram.com)
• Boston Journalists Launch GlobalPost.com, Alternative to Traditional Media’s Shrinking International Coverage (xconomy.com)
• What’s the Best Business Model for Newspapers? (blogs.harvardbusiness.org)

I am travelling to San Diego next week to speak about the importance of evolving Enterprise attitudes to data. Borrowing some nice turns of phrase from Sir Tim Berners-Lee‘s recent TED talk and JP Rangaswami‘s keynote to Powered by Cloud, amongst other things I’ll be suggesting that they ‘stop hugging their data’ and move ‘from data centre to data centric.’

The Linked Data initiative, which began in March of 2007 as a community project supported by W3C‘s Semantic Web Education & Outreach (SWEO) Interest Group (of which I was a member), has been a huge success. Described by Berners-Lee as ‘the Web done right,’ the notion of Linked Data rests upon the acceptance of four simple principles, yet opens the door to previously unanticipated re-use of data scattered across the Web.

The most rapid adoption has, unsurprisingly, been seen in terms of liberally licensed data already visible on the Web in some form. DBpedia, for example, is a community effort to extract structured information from Wikipedia and expose the individual facts for use across the Web. There have also been examples — as always justified by hacker mentality, ‘academic freedom,’ the imprimatur of ‘research,’ or the expectation that the perpetrators are ‘too small’ to be noticed — in which data have been appropriated to the cause without due care and attention to the rights of the data owner, but these isolated cases should certainly not detract from the value of the broader effort.

Public Interest data from organisations such as the BBC has also begun to appear in the ‘Linked Data Cloud‘ (click on individual data sets for more),  and the frequency and strength of reciprocal links between participating resources grows rapidly.

Enterprise data is effectively invisible to this Cloud, which brings me back to Andreas’ first post. In it, he asks;

“Since the [Linked Data] cloud is kind of the basic infrastructure which drives the whole process – this layer should remain a freely accessible one. But how could new business models be built on top of it (and constantly spend money on maintaining and extending the underlying infrastructure)?

Where could enterprises start using Linked Data? Only by retrieving data from the ‘outside’ and mash it up with the ‘inside’ – only one way?”

I can certainly see cases in which cautious corporates will be willing to consume without contributing in return, and there’s clearly work to do in demonstrating the value that they could gain from more balanced participation; participation that should never mean unwillingly ‘giving away’ competitive advantage or sensitive data.

We have an annoying tendency to view data in our databases as an indivisible mass, vigorously and unthinkingly applying the same (expensive) protections to an uninteresting and low-value factoid of underlying context as we do to the core attributes of our next big lead.

Andreas concludes this post by suggesting something very similar to JP Rangaswami’s notion of ‘data centric’;

“Information has no ‘place’ anymore, energy can’t be shipped around the world. We should rethink the meaning of a ‘data store’ and information will flow without flooding us. Linked Data might become the essence.”

Andreas’ second post followed after he’d listened to the most recent episode of the Semantic Web Gang, which I Chair. During the show, recorded last month, we discussed the latest release from Thomson Reuters‘ Open Calais activity, which sees it embrace Linked Data’s principles whilst continuing to run and grow a viable global business.

Andreas extrapolates from the conversation to suggest that a viable business model for the data-curating Enterprise might be to expose timely and accurate enrichments to the Linked Data ecosystem; enrichments that customers might pay a premium to access more quickly or in more convenient forms than are available for free. He also sees a market for application builders that optimise the flow of information, and both of these are certainly possible.

The Linked Data — the Data Web — opportunity is far greater, though, and too little attention is being devoted to it by Linked Data’s advocates as they concentrate their efforts on big public datasets of the sort Berners-Lee discussed in Long Beach last week. Big public data sets are important, and Berners-Lee is right to suggest that more Open and Linked access to the outputs of scholarship will help in our efforts to tackle many of the world’s ills. There’s as much value locked up inside our commercial enterprises too, though, and yet the rationale that will ultimately lead to us unlocking this is quite different.

It is that rationale which we need to get right, almost certainly without mentioning ‘RDF’, ‘Semantic Web,’ or even ‘Open.’

And if you’re in Southern California next week too, why not come and say ‘Hi’…?

‘Sunrise over San Diego‘ image © Alon Banks, 2007

Related articles by Zemanta

• TED2009: Tim Berners-Lee (boingboing.net)
• Linked data will change health care (ourownsystem.com)
• The Semantic Web in Action (sciam.com)
• Tim Berners-Lee: Linked data (thenumerati.net)
• Building coherence at bbc.co.uk (blogs.talis.com)
• Calais 4.0 Released: Linked Data Meets the Commercial Web (readwriteweb.com)
• Calais Release 4 and the Linking Data cloud… (ivanherman.wordpress.com)

The first show of the year was recorded yesterday, and just went up on the Semantic Web Gang site. Have a listen, to hear the Gang and I talk about Thomson Reuters’ Calais web service, Linked Data, and the possibility that Google might be getting all semantic on us.

For those interested in the Thomson Reuters activity, last year I also recorded an interview with the CEO of Thomson Reuters’ Clearforest subsidiary, Barak Pridor.

Related articles by Zemanta

• Calais 4.0 Released: Linked Data Meets the Commercial Web
• Enabling the Linked Data Ecosystem
• OpenCalais will link to Linked Open Data identifiers
• Thomson Reuters’ Calais goes commercial, offering a starting point for semantic web startups
• What one information junkie wants
• Open Calais carves enterprise channel

As well as linking to ‘usual suspects’ in the Linked Data space such as the CIA Factbook, GeoNames, DBpedia, Musicbrainz and the like, Thomson Reuters is taking the additional step of providing access to parts of their information empire. Already recognised as one of the premiere sources for financial and company information on the global stage, the company is choosing to embed Thomson Reuters services such as stock tickers, information on corporate Boards, etc.

In a world in which everyone can participate, and in which citizen journalism, crowd sourcing and the rest are leading to powerful and wide-ranging disruptions to traditional media, Thomson Reuters is betting that authoritative, timely and branded data has value, and that opportunities exist to build that value by giving some previously expensive information away for free in order to jump-start a range of new and lucrative services.

Freebase employees once talked about a desire to become the canonical source of certain facts on the web; the go-to place for a class of information. Might Thomson Reuters have similar ambitions in the business information market, and might the undeniable (and free) value of the Calais web services be the hook that encourages a whole swathe of developers building applications for this market to perpetuate – and strengthen – the company’s reach?

This isn’t cynical, this isn’t devious, this isn’t ‘wrong,’ and there is absolutely no suggestion that the currently free services will ever incur charges for use. Instead, it’s an interesting example of a titan of Old Media thinking creatively about ways in which it can continue to have value in a changing world.

Casa Battló © Larry Miller 2003, and shared on Flickr under Creative Commons License.

Related articles by Zemanta

• Updated Profile: Calais
• Calais Release 4 and the Linking Data cloud…
• Cross-Pollinating DBpedia and Freebase
• Freebase Officially Linked Data with Release of RDF Service