Here’s a stream of consciousness from the morning’s keynotes at this sold-out event.

Conference chair Edd Dumbill is introducing things, talking about William Smith‘s nineteenth century map of geological strata in the British Isles, the rise of industrialisation, and the move to towns. Edd suggests that a similar set of inflections are happening today in the world of data; ‘the start of something big.’

“In the same way that the industrial revolution changed what it meant to be human, the data revolution is changing what it means to be alive.”

The first of this morning’s keynotes; Hilary Mason from link shortener bit.ly.

Data and the people who work with data; “The state of the data union is strong.” Data scientists have an identity – a place to rally around – with Strata.

We have accomplished much, begging, borrowing and stealing from lots of domains. We have the tools. We have the capacity to spin up infrastructure in the Cloud. We have the algorithms to explore data, and to learn from it.

The most important thing we have now that we didn’t have before… is momentum. People are paying attention.

There are still challenges though. Timeliness of data is an issue, especially in real-time. We need to develop systems that can do robust analysis against a moving stream of data. We need to be able to store data in ways that let us operate on it in real-time. Hadoop… amazing ‘because I can run a query and get the result back before I forget why I submitted the query in the first place.’ We need training. We need imagination, not more ad optimisation networks. We have a real opportunity to do something better.

Opportunities (expressed in context of bit.ly); Bit.ly gets lots of data from people shrinking web links. They learn a lot about people; what they like, what they want, what they’re doing. bit.ly also gets rich segmentation data; location, context, etc. bit.ly sees global data, for example clicks on bit.ly links from Egyptian domains.

Now that we have all this data, it offers a window on to the world. What can we do with it? Make the world a better place? What would you do with all of this data?

Next up, James Powell from Thomson Reuters to talk about privacy and behavioural data in B2B contexts. Thomson Reuters gathers large amounts of global data, and filters it for customers. Time and context key; 700,000 updates a second through financial systems, 5,000,000 documents per day served through Open Calais, etc. Thomson Reuters interested in ways to filter information better.

Need to think about B2B implications of behavioural data, especially as we sell/exchange increasing volumes of data with partners. Consumers reasonably comfortable with giving up some personal data in return for a ‘better’ product (Amazon recommendations, etc), that probably doesn’t scale to the enterprise. For example, Open Calais customers submitting large numbers of dummy queries to obfuscate what they’re really looking for…

Key problem that needs to be addressed is ambiguity; many systems in this space still rely upon implicit assumptions, whilst the enterprise is used to explicit contracts. Tension – or recipe for disaster?

Keys to success – need to treat behavioural data differently/better, and avoid the mistake of simply continuing consumer trends.

Next, Mark Madsen from Third Nature, talking about ‘the Mythology of Big Data.’

Lots of assumptions underlying conversations about Big Data. ‘Every technology carries within itself the seeds of its own destruction.’ Code is a commodity; things that a lot of people have built profitable careers around have started to move down-market. Libraries, packages, etc make it easier for third parties to stitch things together rather than start from scratch.

The central myth underlying Big Data that’s erupted over the past 18-24 months; the myth of the gold rush. Everyone wants to be a data scientist. But just like the gold rush, success takes capital. It takes corporate engagement, and infrastructure. The ‘myth tells us you can go it alone… and you can’t.’

1950s-60s – data as product. 1970s-80s – data as byproduct. 1990s-2000s – data as assset. 2010- data as substrate (data as the basis for competition). ‘The real data revolution is in business structure and processes and how the use information.’

Using Big Data; the point isn’t necessarily about ‘Big.’ Much valuable data inside an enterprise is only GB or TB in size. We get tied up in ‘big’ way too much. It’s not really about data either; it’s about applying data. Without an application, it’s trivia.

Next, Amazon CTO Werner Vogels. An overview of how Amazon Web Services look at the data processing being done on their infrastructure by customers… Government, Finance, COmmerce, Pharma… all making use of tools. Plugging The Fourth Paradigm book from Microsoft Research (which is very good).

Vogels – big data is big data when your data sets become so large that you have to innovate to manage them. Customers view big data as collection and curation of data for competitive advantage… with the presumption that bigger is better. For recommendations etc, that is probably true.

There are a number of categories of data, where quality is far more important than quantity.

In the past, data tended to be collected to answer questions. Now, trend to collecting as much as possible before developing the questions you want answered, and the algorithms you will need to use for the analysis.

To do this, you should not be worried by data storage, data processing, etc – which is why you should embrace the scalable Cloud.

Data analysis pipeline; collect – store – organise – analyse – share.

AWS Import/Export – “you shouldn’t underestimate the bandwidth of a FedEx box.” Indeed.

“This is Day 1 for Cloud infrastructure.’

Next up, Microsoft’s Zane Adam talking about data marketplaces. Windows Azure DataMarket; Data as a Service, free or at cost. One stop shop for data (one of many one stop shops, unfortunately!) DataMarket is interesting… but this is far too much of a product pitch for the keynote track.

90 days since launch – 5,000+ subscriptions, 3 Million transactions to date. Given Microsoft’s presence and reach, aren’t those figures a bit low?

“There’s a lot of data out there… but it’s not all good.” A Data Marketplace gives customers access to good data. Does it? Do Microsoft vet every fact in a submitted data set? What would a single bad data set do to the marketplace’s brand recognition?

I don’t think I’ve ever really been convinced by the argument that everything will end up in the data centres of Amazon.

The straightforward provision of commodity Cloud Computing is an important – and growing – area, and one that will continue to expand as interfaces become simpler, FUD is challenged, and prices maintain their relentless march towards the bottom. Everyone has something they could usefully, sensibly, and cost-effectively run in a commodity Cloud such as those offered by Amazon, Rackspace, Flexiant, and others. In this space, basic stability, security and reliability combine with a compelling – and diminishing – pricing proposition to create commodity services targeted squarely to lowest common denominator functionality. Here, market forces may (inevitably?) lead to an eventual reduction in the number of providers. Cost, although not the only consideration, is both important and compelling. Although markets like competition, there may even be a single winner here, one day.

Layered all around the basic, routine, grunt-work computation that these commodity public clouds handle so well, many organisations find themselves having to cope with a wide range of other use cases and data sets. Some require specialist hardware (like the GPUs that Amazon has recently begun selling access to). Some demand particular regulatory and legislative hoops to be jumped through. Some have quirky requirements around latency in data transfer or speed of in-CPU processing. Some have lots of data, and issues with regard to getting the stuff from one location to another with a sensible balance between transfer cost and time.

All of these are certainly capable of being addressed in the Cloud, but the economics and the business rationale begin to shift. For the data owner, cost may no longer be quite so significant a factor. Reliability may matter more, or speed, or the audit trail. For the Cloud provider, these requirements no longer look like the lowest common denominator. It’s not cost-effective to provide these capabilities to everyone and still keep the price low. It becomes more sensible to segment, to divide, and to create bespoke offerings of various kinds. Some of these services require such specific things in terms of network topology, physical building layout, and staff expertise that it may even become counter-productive to have these services in the same building as the commodity Cloud. Here, there’s plenty of room for new entrants, plenty of scope for competition, and plenty of opportunity to differentiate in terms of price, location, support, and a host of other factors. This segment of the Cloud is only just getting started.

In these contexts, we see compelling arguments made for on-premise private clouds, off-premise private clouds, hybrid clouds, community clouds and the rest. Some of the arguments made in favour of private and hybrid certainly are part of the FUD we see in this space, but beneath the noise, the security scares, and the vested interests of SysAdmins and sellers of data centre components, there lies a grain of truth. Not everything is most sensibly run on a cheap VM, rented from Amazon (or Rackspace, or whoever) with your credit card, and physically located half way round the planet.

Unfortunately, it can be difficult to make sensible decisions about which type of cloud works best in each situation, and large swathes of the market are doing everything in their power to add to the confusion.

Having accepted that the basic offering from a public cloud provider is not the solution for my particular requirements, where do I turn next?

Do I listen to the (convincing) pitch from a vendor of ‘community cloud’ solutions for my domain? If I’m in Healthcare, they come with HIPAA and European Data Protection Directive, and all sorts of other accreditations. For dealing with sensitive patient data, this may be just what I need… but does the wily salesman also persuade me to run staff email and the hospital volleyball club website on this over-specified (and expensive) infrastructure?

Do I listen to the (convincing) pitch from a vendor of virtualisation software? If I’ve got a reasonably sized data centre with some life left in it, I may see the value of virtualising all of that expensive hardware, and running current applications in house more efficiently. But instead of gradually reducing my in-house costs, do I continue to add more machines as current ones reach end of life, or as new requirements come along?

Do I listen to the (convincing) pitch from my co-location facility, which happily sells me a ‘private cloud’ that may fail to deliver some of the economies of scale so central to the main Cloud proposition?

Do I listen to the horror stories, stick my head in the sand, and simply keep ordering servers until every single one of my competitors undercuts my costs and I go out of business?

These, and more, are certainly possible. But let’s return to that UK project I mentioned right at the start.

Flexible Services for the Support of Research (FleSSR) is

“a new cloud pilot project looking at utilising hybrid private-public IaaS cloud infrastructure to provide computational and data services to the academic research community. The project is a collaboration between the Oxford e-Research Center, IT Service @ University or Reading, e-Science Centre @ STFC, Eduserv, EoverI, Eucalyptus INC and Canonical Ltd.”

The ten month project is funded by the Joint Information Systems Committee (JISC), an organisation that supports the innovative use of IT across UK universities.

Now, to a degree, the project’s mindset must be influenced by its partners. IT staff at Reading and STFC are incumbents with turf to protect (or new vistas to discover, map, and claim). Eduserv has a new data centre that they’d like to fill with willing clients. It would be easy to be cynical, but knowing some of the people involved, I see no real reason to be. It is perfectly reasonable to suggest that a ‘community’ the size of UK Higher Education would realise value in replicating less (not nothing) at every university campus across the country, and bringing much of that together in some sort of Cloud. That Cloud might use public infrastructure, or it might be served up from an organisation such as Eduserv, which is known to the community, aware of the community’s requirements, quirks and foibles, and (importantly) not-for profit (and therefore cheaper?).

Personally, I’d always rather presumed that an organisation like Eduserv (or JISC itself) would act on behalf of the community to procure a competitive price on access to the resources of Amazon, Rackspace, or one of the others. I’m not convinced that most UK research computation needs any sort of special treatment that couldn’t be met from Amazon’s Dublin data centre… unless the community itself can somehow beat – and continue to beat – Amazon on price. Somewhat surprisingly, that’s exactly what some calculations in two posts by Eduserv’s Andy Powell suggest could happen. By including network costs and other charges over and above the basic storage cost, Andy finds Amazon, Rackspace and Dropbox to be more expensive than anticipated, and posits that Eduserv (connected to every UK university free of charge via JISC’s high speed JANET service, and constrained in the ways it can generate profit from services sold to universities by its charitable status) might actually work out cheaper.

There’s a lot of work to do in terms of fleshing out the assumptions behind some of Andy’s figures, but the whole industry certainly benefits when people conduct exercises like these out in the open, for all to see. If Andy has made mistakes, the vendors should be quick to jump in and correct them. If his assumptions miss the mark, public debate can redress the balance.

The Cloud is not all about price. But more transparency around the true cost of computing in the Cloud – and in your data centre – means that we can all make more informed decisions.

Thanks for sharing, Andy – and hopefully readers will be willing and able to look over your calculations and share their own views.

Note: this post was conceived and written in the United Kingdom. By reading this post you agree to comply with UK usage, and will henceforth pronounce the word ‘niche’ from the title as ‘neesh,’ not ‘nitch.’ Or maybe not.

Related articles

• Rackspace launches managed cloud services (zdnet.com)
• Are hybrid clouds the path to cloud-computing nirvana? (venturebeat.com)
• We’ll Take Care of Your Cloud, While You Manage Your Business (rackspacecloud.com)
• Trust Is Key For Cloud Success And What Can We Do About It? (cloudave.com)

Furthermore, ‘Open Access‘ and ‘Repository’ typically go hand in hand. If you support Open Access you need a repository, and if you question the role of repositories you’re in the pocket of evil publishers who want to lock up everything ever written and lease reading rights back to the employers of those who wrote the stuff in the first place.

Nonsense.

Open Access is an important component of today’s scholarly ecosystem. It’s not the only answer, and it’s not perfect, but it does have a significant part to play. Institutions have a role in preserving, disseminating and exploiting the work of their employees, but these are very different tasks that may benefit from different solutions. In too many cases, the repository is by default seen as a preservation mechanism and a dissemination vehicle, and as such it may fail to cost-effectively achieve either aim.

There are some large, well known, and research-intensive institutions where it might be possible to make a compelling argument for projecting a strong institutional image around a single ‘home’ for all of that research output. Never mind, for a moment, that so much research today is the result of inter-institutional collaboration, or that the eminent researcher might wish to take ‘their’ research publications with them as they move from Oxford to Harvard to York during their glittering career.

Alongside those institutions sit a plethora of others where research of equal quality is also being conducted; there just, maybe, isn’t quite as much of it. Bombarded by ‘advice’ and funding, and desperate to keep up with the Russell Group, ever-more institutions blindly join the repository cult and wonder why their new toys do not fill to overflowing with the jewels of scholarly erudition.

As research becomes increasingly data-rich, the whole cycle looks set to repeat. The recently released Panton Principles for Open Data in Science are to be welcomed, but I’ll bet the institutional response will all too often be the commissioning of a ‘data repository’ to sit alongside the ‘publication repository’ they already don’t use.

All of which is a rather long-winded way of introducing the fact that Eduserv’s Andy Powell has asked me to facilitate a breakout afternoon on ‘Policy Issues’ at the Repositories in the Cloud event Eduserv and JISC are holding in London on Tuesday.

“This free event, organised jointly by Eduserv and the JISC, will bring together software developers, repository managers, service providers, funding and advisory bodies to discuss the potential policy and technical issues associated with cloud computing and the delivery of repository services in UK HEIs.”

In a post on 11 February, Andy invited participants to share some of their views ahead of the meeting, and on 19 February he wrote about some of his own thoughts.

Like Andy, I struggled somewhat to nail down a coherent set of thoughts about the issue of pushing today’s repositories into the Cloud. On one level, I wonder whether the vast majority of institutions with small (and relatively low traffic) repositories would see much of a tangible efficiency gain or cost saving by moving off an in-house computer to rent an equivalent Virtual Machine from Amazon, Rackspace, or any of their competitors. If we’re talking about IT systems within a typical university, there are others (email, calendaring, pools of compute resource for research jobs, etc) that appear more immediately compelling for the shift Cloud-ward. Which is not to say that there isn’t a clear opportunity for someone trusted to step into this space and offer a SaaS repository to which institutions might affordably subscribe. Eduserv? Mimas? Edina? The British Library? The National Archives? Duraspace? Any could, and if we’re not ready for something more then at least one probably should.

However, a bolder reconsideration of what repositories are and what they’re for might very well lead to something interesting, sustainable, and perfectly suited for benefitting from Cloud Computing’s strengths.

Why does a paper have to be ‘deposited’ in a repository? Why does a single paper with three authors from three institutions have to be deposited in three separate institutional repositories? Why does that same paper have to be deposited – separately – in the subject repository favoured by scholars in the relevant discipline? Why does the institution’s very reasonable desire to protect, preserve, promote and disseminate its excellence mean that it has to run systems in perpetuity that preserve and permit access? Why do we address the fundamentally different (perhaps even contradictory) problems of access and preservation in the same system? Why can’t the individual researcher easily assemble a view across their publication history, regardless of the institution within which they happened to reside as they wrote each paper? Why don’t the assemblages of papers reflect personal, professional and disciplinary relationships, alongside (or instead of) the contractual accident of employee-employer relationships? Why isn’t the wealth of metadata implicit to any publication (authors, subjects, dates, citations, and more) available and actionable, both inside the repository and far beyond it across the Web? Why isn’t there a tight and active association between the paper and the data from which its findings were derived (something for which Internet Archaeology was demonstrating utility a very long time ago)?

Scholarly papers principally comprise text, augmented by the occasional static image. They’re not big, and they don’t tend to change very fast. In many ways, they represent a fairly easy problem set with which to work. As more and more data becomes key to research in a growing number of subject areas, the problems are set to become far larger and far more difficult. For individual universities to even consider replicating the process by which they all ended up with their repositories of text surely seems madness in this data-rich environment. Even with levels of uptake as low as those seen in too many text repositories, the issues of data management, curation, access and dissemination are too great to be sensibly solved in the institutional machine room. Services like InfoChimps and Amazon’s own Public Data Sets offering show some of the ways that we might begin to work with data at scale. Might we, for example, come to recognise as Amazon has that it’s actually cheaper and quicker to entrust large data sets to FedEx rather than transmit them over the Internet?

‘The answer’ might be some central service for the community, funded by JISC like the Arts & Humanities Data Service (AHDS) of old. Or it might be something different, something nimbler, more responsive, more flexible to individual, institutional, and disciplinary requirements, and something more scalable to new disciplines; institutional support for and use of existing Cloud infrastructures extending far beyond UK Higher Education, aligned with a clear understanding of the separation between preservation and access.

I certainly don’t have all the answers, but I do believe that simply asking whether or not we should move existing repositories to the Cloud is to miss the point. Rather, we should ask what role the Cloud might play in addressing the business requirements to which the institutional repository was our initial – faltering – response. The answer might very well be ‘None,’ but I doubt it.

I look forward to Tuesday’s discussion. I’m not going there to push my personal view that individual institutions frequently shouldn’t be building, running or populating their own repositories at all. I’m going there to facilitate the discussion those in the room want to have, and to learn from their experiences and their perspectives.

Related articles by Zemanta

• Does a Citation Advantage Exist for Mandated Open Access Articles? (scholarlykitchen.sspnet.org)
• Scholarly content and the cliff edge: the place of subject ‘repositories’ (hangingtogether.org)
• Scholarly Communications must be Scalable (downes.ca)
• Beyond Open Access: Open Publishing (opendotdotdot.blogspot.com)
• 57 college presidents declare support for public access to publicly funded research in the US (scienceblog.com)
• Mandelson says academics are ‘set in aspic’ (guardian.co.uk)

Amid mounting international concern that the guidance lasers aboard Jeff Bezos‘ new Floating Amazon Cloud Environment would interfere with Rudolph‘s sense of direction, sources close to the Amazon Web Services team tell me that they’ve been forced to alter priorities and switch attention to an early release of the next product on their roadmap.

Today sees the release of Amazon‘s latest web service; the Hadoop-powered Elastic MapReduce;

“Using Amazon Elastic MapReduce, you can instantly provision as much or as little capacity as you like to perform data-intensive tasks for applications such as web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research. Amazon Elastic MapReduce lets you focus on crunching or analyzing your data without having to worry about time-consuming set-up, management or tuning of Hadoop clusters or the compute capacity upon which they sit.”

The company’s press release quotes VP for Product Management & Developer Relations, Adam Selipsky, who notes;

“Some researchers and developers already run Hadoop on Amazon EC2, and many of them have asked for even simpler tools for large-scale data analysis. Amazon Elastic MapReduce makes crunching in the cloud much easier as it dramatically reduces the time, effort, complexity and cost of performing data-intensive tasks.”

MapReduce was brought to prominence by Google, and is one of the principal techniques at that company’s disposal in enabling them to break massive data sets into manageable chunks suitable for cost-effective processing on the commodity hardware for which they are known. The abstract for a Google research paper on the topic outlines the value proposition reasonably succinctly;

“MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google’s clusters every day.“

Hadoop is a Yahoo!-nurtured Open Source equivalent to Google’s MapReduce, managed as a project of the Apache Software Foundation, and reputedly scalable to handle many petabytes of data distributed across thousands of CPUs.

As Adam noted in the press release, customers (such as the New York Times and Netflix) are already using Hadoop on Amazon’s Web Services. Today’s announcement makes it easier to cost-effectively and transparently commission (and decommission) the required compute resources. This is the ‘elasticity’ referred to in the new service’s name, and is an increasingly important aspect of the current generation of Cloud-based compute services; much of the economic value proposition lies in only using (and therefore paying for) the resources you actually need to complete a task. If demand increases, the number of (virtual) machines available should rapidly increase to cope, and they should shut back down just as rapidly when the demand passes;

“Amazon Elastic MapReduce enables you to use as many or as few compute instances running Hadoop as you want. You can commission one, hundreds, or even thousands of instances to process gigabytes, terabytes, or even petabytes of data. And, you can run as many job flows concurrently as you wish. You can instantly spin up large Hadoop job flows which will start processing within minutes, not hours or days. When your job flow completes, unless you specify otherwise, the service automatically tears down your instances.“

Elastic MapReduce is currently available only for data centres in Amazon’s US region (so non-US customers can use the service; they just have to be able/willing to transfer the data beyond their borders), and is priced in addition to existing EC2 instances with Elastic MapReduce on a $US0.10 per hour ‘small’ instance costing a further $US0.015 per hour (yes, 1 and a half cents per hour) and on a $US0.80 per hour ‘extra large’ instance costing a further $US0.12 per hour.

Elastic MapReduce is another nice example of slow, incremental improvement to Amazon’s core Web Services offer.

It remains to be seen, as developers get down to using it for real, whether it’s pitched as a low-end disruptor that simply rounds out another piece of the emerging AWS whole, or if it’s a viable competitor in its own right to the recently announced Cloudera which sees taking Hadoop to mainstream enterprise customers as its raison d’etre;

“Cloudera can help you install, configure and run Hadoop for large-scale data processing and analysis. Get Cloudera’s Distribution for Hadoop and start working with Big Data today.“

Update: Amazon’s Jeff Barr provides a lot more detail in a post to the AWS Blog.

Related articles by Zemanta

• Learn to develop software for Hadoop (makezine.com)
• Amazon Blows Away Fourth Quarter Earnings Expectations (businessweek.com)
• Grand Challenges in Database Self-Management (perspectives.mvdirona.com)
• Cloud platforms of the future: Hadoop and Eucalyptus (news.cnet.com)
• Cloudera aims to take Hadoop commercial (vnunet.com)
• The Meta Cloud – Flying data centers enter fourth dimension (theregister.co.uk)
• Cloudera floats commercial Hadoop distro (theregister.co.uk)
• Free Cool Tools for Educators (johnmwillis.com)
• Hadoop-Focussed Startup, Cloudera Raises $5 Million (gigaom.com)
• Hadoop, a Free Software Program, Finds Uses Beyond Search (nytimes.com)
• Cloudera launched to offer commercialized Hadoop (insidehpc.com)
• Cloudera (battellemedia.com)
• LinkLog: Streaming Data, Distributed Execution Engines (dorai.wordpress.com)

Twitter is aflutter once again this morning, this time over a Wall Street Journal suggestion that ‘IBM in talks to buy Sun.’ I am not able to comment on the veracity of the rumour itself, but it’s clear that Sun needs to do something in order to strengthen its position in a competitive market. Selling to IBM is certainly one route, but an easier one might be the provision of a more complete Sun-badged proposition.

Elsewhere on WSJ.com this morning, in news that seems extremely unlikely to be unconnected, Don Clark reports on Sun’s

“plans to offer its own cloud-style services. Sun also plans to offer software, as well as hardware, to other companies that want to build clouds.”

Alongside competitive enterprise server hardware and Sun’s widely used stable of open source software (Solaris, Java, MySQL, OpenOffice, etc), this latest announcement of ‘Sun Cloud Storage’ (equivalent to Amazon’s Simple Storage Service, S3) and ‘Sun Cloud Compute’ (equivalent to Amazon’s Elastic Compute Cloud, EC2) should make Sun a serious player in the Cloud Computing space in a way that their abortive network.com never really did.

So why is anyone discussing either a desire on Sun’s part to sell, or a desire on IBM’s part to consider buying?

I’ve greatly enjoyed the insights of Sun CEO Jonathan Schwartz, especially as enunciated most recently on his blog in two videos discussing community adoption of Sun’s open source software and the commercial models Sun deploys to monetise that community. Despite Jonathan’s arguments, though, it seems to me that Sun lacks a fundamental piece of the whole; an effective and highly visible professional services arm. IBM has this. HP, with the purchase of EDS, has this. Accenture and gang are this, but nothing makes them choose to use or recommend Sun over its competitors today.

As Jonathan discusses in the first of the videos I pointed to (YouTube version embedded below, in two parts), Sun has been successful in encouraging use and innovation around a suite of open source operating systems, tools and applications.

Indeed, it was little more than a year ago that the company announced plans to spend some $800 million in acquiring European open source web database company MySQL. The problem is that these solutions are all freely downloadable from the Web, and the inevitable professional services and consultancy work associated with enterprise delivery — which could generate so much revenue — goes to far more companies than just Sun.

Alongside the software, Sun has a competitive range of hardware offerings in the enterprise space, and sells these in competition with IBM, HP, Dell and the rest.

By omitting a compelling and enveloping professional services proposition, Sun damages its own ability to capitalise upon its software and hardware efforts. Potential customers download Sun software, and then run it on anything; Sun gets a very small slice of the hardware sales. Sun isn’t doing badly at selling hardware, but maybe a more rounded services proposition would enable them to do better, despite Jonathan’s points in the commercial innovation video.

With more emphasis on offering a comprehensive package of solutions — whilst not removing choice and the vibrant open source community of which Jonathan speaks — might Sun not be a more obvious choice for customers in need of services and support?

An acquisition might, as Larry Dignan writes, make sense. But there’s plenty of life left in a standalone Sun, too… if it can monetise more of those downloading free software or steer more of those who ‘just need a server’ towards one with a Sun badge on the front. Professional Services are the road to both.

Related articles by Zemanta

• Sun’s missing mojo: MIA until when? (news.cnet.com)
• IBM, Amazon foreshadow bevy of connecting clouds (news.cnet.com)
• Cisco’s Data Center Moves: Who Wins, Who Loses? (gigaom.com)
• Cisco’s Data Center Play Reinvents The Server (gigaom.com)
• Sun takes another swing at cloud computing (infoworld.com)
• Sun CEO: Open source = free advertising (news.cnet.com)
• Sun Microsystem Business Revenues (Latest Discovery) (it-sideways.com)
• Schwartz: Not worried about Sun’s future (infoworld.com)

Harvard University’s Berkman Center for Internet & Society unveiled their Media Cloud research tool today, bringing Semantic Web goodness from Thomson Reuters’ Calais and affordably scalable Cloud oomph from Amazon Web Services together in powering exploration of a fascinating topic.

As the press release notes,

“Media Cloud was conceived by Berkman Fellow Ethan Zuckerman and Berkman Faculty Co-Director Yochai Benkler.  It was inspired by their debate over whether the blogosphere largely echoed traditional media or was instead a source for original news and democratic agenda-setting.

‘While daily newspapers struggle for survival, political, niche and special interest blogs continue to capture consumer interest,’ said Yochai Benkler, Faculty Co-Director of the Berkman Center. ‘In the midst of this upheaval, it is difficult to know where stories begin, who sets the agenda, and how these dramatic changes impact news coverage on the whole.  We created Media Cloud to help researchers and the public get quantitative answers to these challenging questions.’”

The site itself provides more detail, notably,

“Print newspapers are declaring bankruptcy nationwide. High-profile blogs are proliferating. Media companies are exploring new production techniques and business models in a landscape that is increasingly dominated by the Internet. In the midst of this upheaval, it is difficult to know what is actually happening to the shape of our news. Beyond one-off anecdotes or painstaking manual content analysis, there are few ways to examine the emerging news ecosystem.”

By systematically harvesting full-text content from both new and traditional news sources and passing it through Calais for entity extraction, the Berkman team is able to build a coherent and normalised pool of news that they – and others – can begin to interrogate in order to answer pressing questions as to the shifting news landscape. Today’s site is just the beginning, and the team has ambitious plans to increase the number of sources they harvest, to provide ever-richer visualisations on their site, and to expose public APIs to the underlying data in order that third parties can consume it for their own purposes.

Ahead of the launch I spoke with Barak Pridor, CEO of Thomson Reuters‘ ClearForest subsidiary, and Berkman Fellow Stephen Schultze. Both stressed the beta nature of the site, and the fact that the team are still working to optimise a number of areas. That aside, I was impressed by some of the early results.

Barak pointed to the role Calais is playing in enabling the Berkman team to conduct comparisons within and between different news sources, extracting key entities (personal names, companies, places, etc) from the stream of text and normalising the various ways in which we refer to them (IBM, International Business Machines, etc).

Stephen discussed some of the background to the project, and was keen to emphasise the platform nature of their activity; although focussed on a destination web site today, the intention is to expose much of the data via a series of APIs that third parties will be able to consume. In the context of recent comparable moves from the New York Times and Britain’s Guardian, this is clearly to be welcomed and will accelerate the innovation around this incomparable pool of content.

Whilst the site does not currently expose the full richness of the data being harvested, it is already possible to see a number of interesting visualisations.

The maps above, for example, compare the coverage devoted to all the world’s countries by three very different news organisations; the BBC, the Wall Street Journal and FOX News. The apparent non-coverage of Iran is surprising, and suggests an algorithm still in need of tweaking. That apart, it’s actually surprising to note the very similar spread of coverage. European snobbishness about the insularity of US news – and the farcical nature of FOX ‘news’ – may not survive exposure to this data, although we can, of course, reassure ourselves that ‘coverage’ doesn’t always equate to ‘insight,’ ‘analysis’ or ‘truth.’ As more data become available, even those preconceived notions may face an overdue battering.

The system is already capable of addressing more complex questions, and exploring temporal aspects of the ways in which stories break, spread and relate to one another across different media. Stephen shared a number of evolving visualisations that made it possible for me to explore several trends at once, and I look forward to these tools arriving on the site.

The project is directly funded by the Berkman Center at present, and a desire to enrich and stabilise the platform means that significant addition of new news resources may be some months off. I, for one, look forward to seeing more non-US resources such as the UK’s Financial Times and Guardian, or their international equivalents.

The project’s APIs are also being finalised at this point and will be released in due course, along with the source code upon which Media Cloud runs. It will be interesting to see the balance between API access and software download at that point.

Media Cloud represents an excellent example of the uses to which the technologies I cover can be put. It’s not a Semantic Web project. It’s not a Cloud Computing project. It’s an intriguing exploration, that puts those technologies (invisibly) to work in helping to get the job done. That’s the way it should be.

And, as Stephen commented when asked what he was most interested to see,

“the most amazing stuff will be what other folks use it for in asking and answering their own questions.”

Indeed.

Thomson Reuters is already supporting this project by contributing Calais and some technical expertise. I wonder whether some of our more enlightened news organisations might like to help the Berkman get more data in there, faster? As more newspapers close each week, and as we (supposedly) become ever-more insular in the choices we make about the news we consume, there’s a real need to understand the ways in which the media are changing. Many eyes on this data set is one sure way to ensure that the commentary moving forward is informed and informative.

Related articles by Zemanta

• Lawsuit to Determine Fair Use for Blog Links, Headlines (crooksandliars.com)
• Newspapers Suddenly Adapt To Socal Media; Nearly 60 Percent Offer User-Gen Content (paidcontent.org)
• Yes, Twitter is a source of journalism (mathewingram.com)
• Boston Journalists Launch GlobalPost.com, Alternative to Traditional Media’s Shrinking International Coverage (xconomy.com)
• What’s the Best Business Model for Newspapers? (blogs.harvardbusiness.org)

The paperback edition of Nick’s book has just been published, and includes an additional chapter that introduces ‘The Cloud 20′; representative examples of the different ways in which companies big and small, new and old are engaging with the Cloud Computing phenomenon.

I was delighted when Nick agreed to speak with me over the weekend, and the resulting podcast has just been released so you can listen in on our conversation.

As well as talking about the book, which most readers of this blog have doubtless already read, we looked more broadly at some of the trends shaping consumer and corporate attitudes to the Cloud. How does the current economic climate, for example, affect the rate of adoption of utility computing or of Cloud-based applications such as those from Google?

Have a listen, and let us know what you think…

Related articles by Zemanta

• What kinda apps are best suited for ‘Cloud deployment’ : 4 Solutions (techpluto.com)
• Alternate Cloud 20 List (johnmwillis.com)
• A National Higher Ed Computing Cloud (q-ontech.blogspot.com)
• Review: The Big Switch (fiberevolution.com)
• Eli Lilly using cloud computing (insidehpc.com)
• The Cloud 20 (roughtype.com)
• Cloud Cafe #30 What is a Cloud from the Beginning (johnmwillis.com)
• Overcast: Conversations on Cloud Computing (elasticvapor.com)
• eweek Podcast: The Evolution of Cloud Computing (elasticvapor.com)
• A Unified Ontology for Cloud Computing (elasticvapor.com)
• Year in review: The ‘cloud’ soars (news.cnet.com)

Event organisers are feeling the squeeze as advertising, travel and ‘training’ budgets present easy targets to Finance Directors seeking to balance their books in the current economic climate.

Amidst announcement after announcement of cancelled and radically down-sized trade shows and conferences, one bright spot in the event management space appears to be anything related to ‘The Cloud.’ There is an understandable perception that Cloud Computing will save money, so that ticks boxes back at HQ. There is also a perception that a sound understanding of the Cloud (and yes, it’s more than simply outsourcing your Data Centre to save some money) will position companies to come out of this economic downturn extremely well placed to exploit new opportunities and grow.

One of those events to cross my radar just before Christmas was Powered By Cloud, which is being held in London – just around the corner from the UK Parliament – on 2 and 3 February. According to the site, attendees will learn;

“What does [the Cloud] mean for your business model?

How fast will this happen?

How can I make money from Cloud Computing?

What technologies will be used?

What are the implications for consumers, privacy and security?

What is the future of Cloud Computing?”

Speakers on the programme look like a nice mix of solutions providers, customers and analysts, and include Amazon‘s Simone Brunozzi, Dave Armstrong from Google, Woodson Martin from Salesforce, Rightscale CEO Michael Crandell, Elastichosts CEO Richard Davies, Simon Wardley and Sam Sethi.

If you’ve got a couple of days to spare, can convince the Finance Director to agree (tell ‘em Cloud Computing saves money…), and can get to London then this looks like a pretty good investment for that diminished travel budget. It might even be worth enduring Heathrow to reach.

And, thanks to Philip Low at event organisers BroadGroup, here’s something that might even make the Finance Director smile… If you use discount code ‘SPKR‘ when you register, you can get in cheaper and save even more money. Enjoy!

Related articles by Zemanta

• Clouds Merge, Search Challenge Emerges
• Salesforce.com rolls out Service Cloud
• Sun takes another swing at cloud computing
• Nebulous cloud computing
• HP dismisses cloud ‘hype’
• Editorial: The trend for storing information on remote servers known as ‘clouds’ is a good thing
• Eli Lilly using cloud computing
• Is the Pentagon ready to stick its head in the cloud?
• Top Issues for CIO’s
• Someone say Recession? Not in the Cloud.
• 6 Ways to Help Your Business Weather the Economic Storm

It began, as so many things do these days, with an idle tweet.

On 21 November, Amazon Web Services‘ Deepak Singh pointed to a new page describing the company’s ‘Public Data Sets on Amazon Web Services.’

Lidija Davis covered the news for ReadWriteWeb two days later and on 4 December Amazon issued its formal press release, prompting a flurry of coverage from Mike Arrington at TechCrunch, Larry Dignan at ZDNet, Krishnan Subramanian at CloudAve, and many others.

Alongside broader discussion of this move, members of the W3C-backed Linking Open Data project delved into the synergies via their public mailing list and Linking Open Data enthusiast Kingsley Idehen‘s company issued a Press Release suggesting ways in which their products might fit within this shifting data landscape.

So what have Amazon done, what does it mean, and how does it ‘bring the Cloud of Data closer’ as the title of this post suggests?

Amazon’s web page describes their offer quite succinctly;

“Public Data Sets on AWS provides a centralized repository of public data sets that can be seamlessly integrated into AWS cloud-based applications. AWS is hosting the public data sets at no charge for the community, and like all AWS services, users pay only for the compute and storage they use for their own applications.”

As Krishnan noted in his post,

“By doing this, Amazon is helping research community save money on storage and bandwidth costs associated with assessing these public data from any EC2 instances they use in their research. When the data in question is in hundreds of terabytes or petabytes, we are talking about huge cost savings here.”

In addition, OpenLink’s press release gives an indication of the efficient manner in which services and data already hosted by Amazon can be plugged together as needed;

“As a vital contribution to the momentum behind the burgeoning Web of Linked Data, [OpenLink's product] Virtuoso provides a simple deployment mechanism for highly integrated knowledge bases emerging from the Linking Open Data community. For example, it is now possible to deploy personal or service-specific renditions of DBpedia within 1.5 hours, compared to an 8 – 22 hour effort when performed from scratch.”
(my links)

By offering free hosting for public data, then, Amazon are doing the wider community a huge service. Much of the data there today is reasonably readily available from other sources, so the biggest immediate benefits are those of speed and cost outlined above by Krishnan and OpenLink. For existing or potential users of Amazon’s Web Services to power their applications, this is yet another reason to consider Amazon.

Harvard Medical School‘s Dr. Peter Tonellato was quoted in Amazon’s release, and he is unlikely to be alone;

“Public Data Sets on AWS will enable me and many of my colleagues to collaborate with each other by sharing our commonly used data sets, research environments and tools. We can set up a controlled environment in minutes, run our computational analysis for a couple of hours, and shut down the environment. Our results are completely repeatable. I only pay for the compute time I use, and more importantly I can spend more time focusing on research, not downloading and setting up computational infrastructure.“

The bigger long-term contribution of this Amazon initiative may actually lie with data that are difficult or impossible to find online today. In a previous existence at the Archaeology Data Service (ADS), for example, my colleagues and I were always being contacted by individuals and organisations with data that they wanted to see online; individuals and organisations that lacked the skills, resources or mandate to mount and maintain the data themselves. How many of those organisations will beat a path to Amazon’s door now… and what sort of resource might we see emerge as a result?

However…

Krishnan concludes his post with a reality-check, commenting;

“this data stored on AWS servers are useful only if the researchers use Amazon EC2 for their computing needs… even if they could tap into it from external platforms, it doesn’t mean much if these public datasets are accessible using some kind of API from their original source itself.”

In other words, much (most? all?) of the advantage Amazon is offering evaporates if developers then have to pull the hosted data off Amazon’s servers and into their own applications running locally or via a competing Cloud provider such as Google.

Although the way in which it is recognised and monetised is finally shifting, data is still valuable, and Amazon (and others) clearly recognise the benefits of enticing users to entrust data to their offering, whilst (almost) imperceptibly making it that little bit more painful to use the data somewhere else.

Kingsley Idehen is quoted as saying,

“The Web’s potential as a globally distributed information space that plugs into disparate databases has never been in question. What has remained unclear is how a federated Web of linked databases would be delivered in a manner consistent with the Web’s core architecture, without compromising its simplicity.”

It is in moving us toward this open vision that Amazon’s offering (although undoubtedly an important step along the way) is ultimately lacking. For that, we may well require the open and linked approach of Semantic Web offerings from companies such as Talis and Kingsley’s OpenLink. These recognise the futility of expecting all data to migrate to a single service provider, whilst still ensuring that those on the ‘inside’ may gain the benefits of proximity on the network, pre-computation of certain indices, etc. Amazon and its services clearly have a place within that emerging ecosystem, but it is a place that they will need to share with others.

The worthwhile philanthropic aspects of Amazon’s announcement apart, the company is certainly doing its part to evangelise the benefits of moving data to the Cloud, and this is to be wholeheartedly welcomed.

CIOs are recognising the benefits of Cloud-based computation, and their resistance to the loss of control implied by individual cost centres’ embracing of SaaS solutions such as Salesforce is diminishing. The proposition of accessing data in the Cloud, at will, is even more profound, and the benefits to be gained require careful and compelling explanation in the face of inevitable fears regarding issues such as data integrity.

Showing everyone the benefits to be gained in sharing disparate public data sets is one more step along the way to widespread acceptance of the value in easing restrictions over access to more sensitive resources.

Related articles by Zemanta

• The evolution of an all encompassing world of clouds
• Amazon Web Services: Bigger Than Amazon
• Public Data Goes on Amazon’s Cloud
• The Amazon Cloud: no longer a mid-Altantic kludge
• Amazon Tries to Lure Scientific Community into the Clouds