Image via Wikipedia

Openness is undeniably cool right now, at least if you move in the slightly odd circles that I do. Openly available scientific papers are disrupting the world of scholarly publishing (which may not be all good, but that’s a post for another day). Openly available university courses are finally beginning to work out how to offer meaningful accreditation to students. Openly accessible data from government agencies around the world bulks out almost every data marketplace, and anchors many an analysis. Openly available code for cloud infrastructure or networking is challenging the hold of the tech world’s giants. Everywhere you look, ‘incumbents’ are apparently being ‘challenged’ and ‘disrupted’ by the power of open.

The truth, of course, is a little more complex and a lot more nuanced, as business models shift and evolve just like they always have. In sustainable systems, some people still need to be rewarded (often through being paid) for their effort. And in sustainable systems, paying someone can often be a pretty straightforward means of ensuring that you have a throat to choke if something breaks; big companies adopting open source often seek a proper financial relationship with someone who installs and maintains the ‘free’ software or hardware they’re depending upon.

One area of openness that I’ve been involved with for about ten years is that of open licensing for both creative works and data. And it’s come a very long way.

Here in Europe, for example, the (badly flawed) 2003 Public Sector Information Directive is under review, and there’s every likelihood that the replacement will make a number of sensible moves toward greater openness, transparency, and reusability for publicly funded data. As the EPSI Platform site notes today, Andrés Nin proposes going a step further than the European Commission is currently contemplating, by instituting a common open license across Europe;

“The creation of a single public information re-use space in Europe requires much more, it requires a common European OpenData license applicable to all data generated by European public administrations.”

I would certainly welcome a model license that European member states might be enabled to use. I’d also welcome — and support — vigorous efforts to dissuade individual member states or ministries from their usual practice of tweaking and otherwise modifying perfectly good documents in order to demonstrate how ‘special’ or ‘different’ their circumstances apparently are. When will they all realise that they are neither as special nor as different as they like to think?

But — and it’s a big but — it seems unwise, premature, and unhelpful to even begin to suggest that such a license might be mandated across Europe. It isn’t required, and attempts to develop a single document that everyone could accept would be an unhelpful distraction that would result in something so bureaucratic, so ringed in opt-outs and prevarications, as to be utterly worthless. It would also, in all likelihood, be one of those exercises in which the process very quickly subsumed the point. A prime candidate for, in the words of an old boss, being too busy to be effective.

Related articles

• Branding academic publishers ‘enemies of science’ is offensive and wrong (guardian.co.uk)
• Open Data: Europe Starts to Get It (opendotdotdot.blogspot.com)
• Open Data in Europe gets a huge boost from new EU rules (thenextweb.com)

The OpenStack activity was unveiled by Rackspace, NASA, and their partners back in July, and is on track to deliver functional initial releases in the next few weeks. We discuss the relationship between OpenStack’s deliverables and earlier developments from Rackspace and NASA’s Nebula project, and begin to explore the implications of an Open Source Cloud Computing stack for the wider industry.

This podcast was recorded on Friday 10 September, 2010.

During our conversation we referred to the following resources;

• Amazon
• Amazon EC2
• Amazon S3
• Apache license
• Citrix
• Cloud.com
• Dell
• GoGrid
• HP (and my podcast with Russ Daniels)
• IBM (and my podcasts with Kristof Kloeckner and Ric Telford)
• KVM
• Linux
• Microsoft (and my podcast with Amitabh Srivastava)
• mySQL
• NASA
• NASA Nebula (and my article for GigaOM Pro)
• OpenStack
• OpenStack code launchpad
• OpenStack design conference, November 2010
• Oracle
• Rackspace (and my podcast with Lew Moorman)
• Rackspace CloudFiles
• VirtualBox
• Werner Vogels, Amazon CTO, discussing ‘false Clouds’ at GigaOM’s Structure conference
• XenServer

Related articles by Zemanta

• OpenStack sets some Texas-sized deadlines (zdnet.com)
• OpenStack: Rackspace and NASA Nebula Join Forces for Open Cloud Ecosystem (readwriteweb.com)
• NASA gives OpenStack instant credibility (zdnet.com)
• Impact of OpenStack Project Goes Beyond the Cloud Industry Leaders (readwriteweb.com)
• Space IT, the final frontier (radar.oreilly.com)
• What’s next for OpenStack’s cloud efforts (news.cnet.com)
• Rackspace Announces Open Source Cloud Platform (informationweek.com)

Tim Berners-Lee has been amongst those rallying to the cause, and working with Governments here and overseas to realise the opportunities in — first — simply getting data out and — second — ensuring the structure and linkages required if Government data is to form a useful foundation upon which others really can build.

Tim O’Reilly, too, has been pushing his notion of ‘Government as a Platform’ for some time, driving toward next week’s gov2.0 summit in Washington, DC. His arguments reached a broader audience with yesterday’s guest spot on TechCrunch.

O’Reilly points to some of the uses of Government data in his TechCrunch post, and I’m continuing my own efforts to secure podcast interviews with some of the more interesting examples that I come across.

One of the things, I think, that is most interesting about this ‘platform’ that O’Reilly describes is the myriad ways in which it potentially benefits so many different constituencies.

Government itself should certainly become more efficient, making better use of its own information ‘simply’ because it’s so much easier to see what’s there. John Sheridan of the UK Government’s Office of Public Sector Information (OPSI) touched on some of these issues in our recent conversation, and the potential for discovering synergies across the Departmental and Agency divide is surely just beginning to be realised.

‘Activists’ of various kinds will have easier access to information in support of their various causes. Some of this information will, undoubtedly, be used to embarrass the authorities, and much of it will be skewed to present the truth in rather odd ways. Rather than simply leading to more informed activism by a vocal minority, however, David Eaves in Vancouver and Sunlight Labs’ David James both argue that better availability of data will make it easier for everybody to hold Government to account.

Researchers, such as Jim Hendler’s team at Rensselaer, are turning to resources like Data.gov in search of interesting technological problems and large pools of data upon which to test new techniques and ideas. The resulting data exhaust (in Hendler’s case, RDF versions of Data.gov resources) is then available for others to use in further innovation.

And (perhaps) most interestingly of all, the well understood trinity of open source software, commodity hardware and near-ubiquitous connectivity is coming together with increasingly available data to make the public sector information space — for far too long the expensive preserve of the big Consultancy firms and their ilk — interesting to start-ups and innovators.

Sunlight Labs‘ Apps for America2 contest is into its final stage (don’t forget to vote!), and the range of applications received is a clear indication that innovation in and around Government information is both possible and long overdue. Amongst the three finalists, This We Know appeals to my Semantic Web interests because of the particular technological approach they’ve adopted, but there’s plenty to admire in all three.

The test will come down the line, when working with Government data is no longer incentivised by competition glory and prize money, when it’s no longer the hot new source of Big Data for academic exploration, and when the activist arms race levels out at a new plateau of comparable informedness. When that day arrives, will enthusiastic entrepreneurs still be competing to extract additional capability from the Data.gov and Guardian apis, or will all of the hits to these sites emanate from the cubicles of Accenture, IBM and EDS?

Whilst I certainly hope that these mainstream exploiters of public sector information embrace the possibilities, it would be a sad day if independent grass-roots innovation atop the Platform of Government were so short-lived.

As such, I’ll be watching next week’s proceedings in Washington with interest.

Image of the Scottish Parliament by Stéphane Goldstein, shared under Creative Commons license on Flickr.

Back in May, ReadWriteWeb reported on a Motion put before legislators in the Canadian city of Vancouver. Duly passed, the Motion commits the city to three closely related ‘open’ agendas;

• the City of Vancouver will move as quickly as possible to adopt prevailing open standards for data, documents, maps, and other formats of media;
• the City of Vancouver, when replacing existing software or considering new applications, will place open source software on an equal footing with commercial systems during procurement cycles;
• the City of Vancouver will freely share with citizens, businesses and other jurisdictions the greatest amount of data possible while respecting privacy and security concerns.

Last week I spoke with David Eaves, a co-author of the Motion, both to understand the city’s rationale, and to explore intentions for the third area — Open Data — in a little more depth. The result has just been released as a podcast, which is available below.

Production of this podcast was supported by Talis, and show notes are available on their Nodalities blog.

As more and more data become available as a matter of course, the examples set by organisations such as MySociety become increasingly attainable for us all. Other than ensuring that it is ‘open,’ do we need to be asking for more from those making data available? And once it’s there, will its use and scrutiny move beyond the enthusiasts and activists to encompass the population at large?

David shares his views on these and other questions during our conversation.

Related articles by Zemanta

• What is an Open City? (mt-soft.com.ar)
• City of Vancouver embraces open data, standards and source (cbc.ca)
• Doors are finally opening to public data (bookmarkdevil.com)
• City of Vancouver embraces open data, standards and source (slumpedoverkeyboarddead.com)
• Vancouver, BC Wants to be an Open City (readwriteweb.com)
• Vancouver mulls making itself an ‘open city’ (cbc.ca)
• Vancouver’s Open Data, Open Standards, Open Source and the Vancouver Public Library (madanalogy.com)

As the company’s press materials note,

“Garlik, the online identity expert, was founded by Mike Harris, founding CEO of Egg plc, former Egg CIO Tom Ilube and former British Computer Society president Professor Nigel Shadbolt. As the first company to develop a web-scale commercial application of semantic technology, Garlik enables consumers to protect themselves against identity theft and financial fraud.”

According to Wikipedia, ‘Egg… is now the world’s largest internet bank,’ so effective management of identity information is clearly nothing new to Ilube and his team.

Founded in 2005, Garlik has secured some £4.5million from 3i, Doughty Hanson and Noble Venture Finance to offer products such as their DataPatrol solution for tracking sensitive personal information online, and the less ‘serious’ measure of online status, QDOS.

Behind the scenes, data is aggregated from across the open Web and various proprietary databases, and stored in Garlik’s own RDF triple store.

Now the company is releasing their triple store — 4store — under a GNU GPL license and making it available for download. Capable of scaling to handle as many as 60billion triples (perhaps at least three times more than their closest competitors), 4store has the potential to address many concerns about the scalability of triple store technology.

I took the opportunity to talk with Garlik’s Tom Ilube and 4store’s designer, Steve Harris, before the launch and the result has just been released as a podcast.

Related articles by Zemanta

• Nigel Shadbolt talks about Web Science, the Semantic Web, Linked Data, and Garlik (blogs.talis.com)
• Online fraud at highest level ever (telegraph.co.uk)
• Happy 20th Birthday, World Wide Web (sciam.com)
• ReadWriteWeb Interview With Tim Berners-Lee, Part 2: Search Engines, User Interfaces for Data, Wolfram Alpha, And More… (readwriteweb.com)

Sun Microsystems used the CommunityOne East event in New York City this past March to unveil their Cloud Computing offering. I spoke with the company’s Juan Carlos Soto recently, to learn more.

Today, David Douglas (Senior VP, Cloud Computing) opened CommunityOne West in San Francisco discussing ‘Communities, Open Source Platforms, and Clouds.’ I joined the live webcast to see what he had to say.

Dave Douglas kicks off, talking to the importance of ‘community’. He stresses the underlying value of open – source code, protocols, formats, ideas.

“‘Open’ lowers barriers to adoption and innovation.”

A lot of the ideas he’s highlighting are similar to Tim O’Reilly’s call to ‘do stuff that matters;’ but oddly Dave doesn’t mention this.

Lew Tucker, Sun’s Cloud CTO, gets up on stage to talk about Sun’s Cloud Computing with Dave. Their opening gambit is around the on-demand nature of the Cloud, with its ability to pull up (and shut down) Cloud resources on demand, with a credit card. Lew argues that the Cloud doesn’t create lock-in, as it’s based upon open software such as Apache, Solaris and Linux.

Sun’s Storage Service, announced in March, is still on track to be available this summer… so no surprise unveiling from the stage today.

Lew shows some demonstrations of the Sun Compute and Storage Services, building upon those we saw in March to manage resources in the data centre via GUI.

Dave mentioned that ‘several thousand’ Sun staff currently use the Sun Cloud internally, every day, “in Open Office” and elsewhere. Is this ‘just’ Cloud-based file storage, or something more?

On an intriguing mix of laptops, other examples from Sun Partners include Vertica and webappVM. The examples definitely leaned towards the sysadmin and developer crowd, and I look forward to seeing some user-facing apps down the line. Dave cites ‘dozens and dozens’ of partners, as their logos flash up on screen behind him.

Lew suggests that the Cloud introduces a change from ‘Download -> Install -> Config’ to ‘Deploy,’ with the implication that this will always be easier.

Turning to Security, Lew points to a new ‘secure hardened VM for OpenSolaris,’ available on Amazon S3 today. The Center for Internet Security has assessed this new VM and verified it as secure.

Eric Baldeschwieler from Yahoo! gets up on stage, to talk about the ways in which Apache Hadoop is being used at Yahoo! – and their use of the Sun Cloud.

I look forward to hearing more, face to face, during June’s Semantic Technology and Cloud Computing tour around Silicon Valley; Menlo Park is already on my itinerary, along with sojourns to San Jose, San Francisco and Sunnyvale. Anyone else got things they want to show me, June 14-21?

Related articles by Zemanta

• Sun Microsystems Forecast: We Are Partly Cloudy (sitepoint.com)
• Sun Announces Open Cloud Platform & API (java.sys-con.com)
• Sun Microsystems Plans To Compete With Amazon Web Services (cloudave.com)
• Sun Open Cloud – Detailed Review (taragana.com)
• Sun’s new mantra: Call us the ‘cloud company’ (news.cnet.com)
• Sun breaks through the clouds (theregister.co.uk)

Chander Kant

I spoke with Chander recently to learn more about the company, and to understand the particular issues associated with moving backup data to the Cloud.

Production of this podcast was supported by Talis, and show notes are available on their Nodalities blog.

In addition to leveraging Cloud storage as part of a rounded backup strategy for on-premise data, we also discussed Zmanda’s approach to providing useful backups of data already stored in the Cloud.

Related articles by Zemanta

• Is Your Future in ‘The Cloud’? (usnews.com)
• Amazon’s Cloud Grows in Europe (businessweek.com)

“EUCALYPTUS — Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems — is an open-source software infrastructure for implementing Elastic/Utility/Cloud computing using computing clusters and/or workstation farms. The current interface to EUCALYPTUS is interface-compatible with Amazon.com’s EC2 (arguably the most commercially successful Cloud computing service), but the infrastructure is designed to be modified and extended so that multiple client-side interfaces can be supported. In addition, EUCALYPTUS is implemented using commonly-available Linux tools and basic web service technology making it easy to install and maintain.”

Since then, we’ve seen EUCALYPTUS embraced by Sun Microsystems in their Cloud announcements and potentially downloaded to millions of machines around the world as part of the latest update to Linux’s popular Ubuntu distribution.

Today the UCSB research project takes the next step, announcing a successful Series A investment round led by Benchmark Capital that moves the team out of the University and onto a professional footing with $5.5 Million to spend. Project Director Wolski becomes CTO, with Woody Rollins as CEO and Matt Reid as VP Sales & Marketing rounding out Eucalyptus Systems‘ fledgling management team. Wolski reports that the entire UCSB development team is moving across to the newly capitalised company, which is licensing IP from UCSB in return for an undisclosed equity stake. Benchmark’s Kevin Harvey takes a seat on the Board, which is Chaired by former AOL Europe CEO Andreas Von Blottnitz.

Wolski is quick to stress that Eucalyptus Systems is an open source company; there is no intention to start charging for software that is freely available for download today, and this will be actively maintained and developed moving forward. A point release is expected ‘in about a week’ to resolve minor issues with respect to Ubuntu 9.04, and version 1.6 of Eucalyptus will follow ‘in 6-8 weeks.’

Speaking of the company’s proposition to new customers, Wolski suggests that

“Eucalyptus Systems will enable businesses of any size to leverage their own IT resources to get the benefits of cloud computing without the concerns of lock-in, security ambiguity, and unexpected storage costs that can be associated with public clouds.”

Interestingly, Wolski describes the revenue model traditionally employed by companies seeking to make money from open source as no more than a way of boot-strapping Eucalyptus Systems toward their real goal. Rather than simply concentrating on the provision of for-profit support and consultancy, Wolski has his sights set on the sale of new Eucalyptus-powered software solutions directly addressed to Enterprise customers. It’s not yet clear whether these solutions will be SaaS offerings or for on-premise installation, but Wolski is confident that the company’s early customers will begin to receive their new software during the third quarter of 2009.

Although best known for providing an on-premise equivalent to Amazon’s Elastic Compute Cloud (EC2), Wolski stresses that the Eucalyptus architecture is API agnostic and could be extended to emulate other Cloud solutions relatively easily. This, allied with Eucalyptus’ industry-leading support for both the KVM and Xen hypervisors, raises the prospect of enterprise customers integrating their own (Eucalyptus powered) internal Cloud with different public Clouds, seamlessly and at will.

Related articles by Zemanta

• Karmic Koalas Love Eucalyptus (radar.oreilly.com)
• Rich Wolski discusses Grids, Clouds, and the past, present and future of EUCALYPTUS (cloudave.com)
• Eucalyptus to be Included in Next Ubuntu Release (berkeleyclouds.blogspot.com)
• Without APIs, there is no cloud computing (news.cnet.com)
• Sun to take to the cloud (infoworld.com)
• Ubuntu now has “Cloud Computing Inside” (news.cnet.com)
• Ubuntu: Karmic Koala to make cloud ‘dance’ (news.zdnet.com)

Most readers of this blog are probably well aware that a new version of the Ubuntu Linux distribution is coming this week, and that it will be putting code from the Open Source EUCALYPTUS Project to work in simplifying the creation of private Clouds that look remarkably like Amazon’s EC2. You’ve probably also read RightScale‘s announcements with respect to Ubuntu, and heard that Sun Microsystems were also making supportive noises about EUCALYPTUS and the EC2 API before their recent change in circumstances.

Earlier today I spoke with Simon Wardley of Canonical (the commercial organisation that sells support and consultancy for Ubuntu) to hear a little more about what those downloading Ubuntu will get… and what it might mean for the rapidly shifting Cloud landscape.

Production of this podcast was supported by Talis, and show notes are available on their Nodalities blog.

Related articles by Zemanta

• Ubuntu gets pre-Koala cloud love (theregister.co.uk)
• Karmic Koalas Love Eucalyptus (radar.oreilly.com)
• Mark Shuttleworth Now Sees Oracle As a Big Open Source Kahuna (ostatic.com)
• Ubuntu promises DIY Amazon cloud (theregister.co.uk)
• Ubuntu now has “Cloud Computing Inside” (news.cnet.com)

Amid mounting international concern that the guidance lasers aboard Jeff Bezos‘ new Floating Amazon Cloud Environment would interfere with Rudolph‘s sense of direction, sources close to the Amazon Web Services team tell me that they’ve been forced to alter priorities and switch attention to an early release of the next product on their roadmap.

Today sees the release of Amazon‘s latest web service; the Hadoop-powered Elastic MapReduce;

“Using Amazon Elastic MapReduce, you can instantly provision as much or as little capacity as you like to perform data-intensive tasks for applications such as web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research. Amazon Elastic MapReduce lets you focus on crunching or analyzing your data without having to worry about time-consuming set-up, management or tuning of Hadoop clusters or the compute capacity upon which they sit.”

The company’s press release quotes VP for Product Management & Developer Relations, Adam Selipsky, who notes;

“Some researchers and developers already run Hadoop on Amazon EC2, and many of them have asked for even simpler tools for large-scale data analysis. Amazon Elastic MapReduce makes crunching in the cloud much easier as it dramatically reduces the time, effort, complexity and cost of performing data-intensive tasks.”

MapReduce was brought to prominence by Google, and is one of the principal techniques at that company’s disposal in enabling them to break massive data sets into manageable chunks suitable for cost-effective processing on the commodity hardware for which they are known. The abstract for a Google research paper on the topic outlines the value proposition reasonably succinctly;

“MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google’s clusters every day.“

Hadoop is a Yahoo!-nurtured Open Source equivalent to Google’s MapReduce, managed as a project of the Apache Software Foundation, and reputedly scalable to handle many petabytes of data distributed across thousands of CPUs.

As Adam noted in the press release, customers (such as the New York Times and Netflix) are already using Hadoop on Amazon’s Web Services. Today’s announcement makes it easier to cost-effectively and transparently commission (and decommission) the required compute resources. This is the ‘elasticity’ referred to in the new service’s name, and is an increasingly important aspect of the current generation of Cloud-based compute services; much of the economic value proposition lies in only using (and therefore paying for) the resources you actually need to complete a task. If demand increases, the number of (virtual) machines available should rapidly increase to cope, and they should shut back down just as rapidly when the demand passes;

“Amazon Elastic MapReduce enables you to use as many or as few compute instances running Hadoop as you want. You can commission one, hundreds, or even thousands of instances to process gigabytes, terabytes, or even petabytes of data. And, you can run as many job flows concurrently as you wish. You can instantly spin up large Hadoop job flows which will start processing within minutes, not hours or days. When your job flow completes, unless you specify otherwise, the service automatically tears down your instances.“

Elastic MapReduce is currently available only for data centres in Amazon’s US region (so non-US customers can use the service; they just have to be able/willing to transfer the data beyond their borders), and is priced in addition to existing EC2 instances with Elastic MapReduce on a $US0.10 per hour ‘small’ instance costing a further $US0.015 per hour (yes, 1 and a half cents per hour) and on a $US0.80 per hour ‘extra large’ instance costing a further $US0.12 per hour.

Elastic MapReduce is another nice example of slow, incremental improvement to Amazon’s core Web Services offer.

It remains to be seen, as developers get down to using it for real, whether it’s pitched as a low-end disruptor that simply rounds out another piece of the emerging AWS whole, or if it’s a viable competitor in its own right to the recently announced Cloudera which sees taking Hadoop to mainstream enterprise customers as its raison d’etre;

“Cloudera can help you install, configure and run Hadoop for large-scale data processing and analysis. Get Cloudera’s Distribution for Hadoop and start working with Big Data today.“

Update: Amazon’s Jeff Barr provides a lot more detail in a post to the AWS Blog.

Related articles by Zemanta

• Learn to develop software for Hadoop (makezine.com)
• Amazon Blows Away Fourth Quarter Earnings Expectations (businessweek.com)
• Grand Challenges in Database Self-Management (perspectives.mvdirona.com)
• Cloud platforms of the future: Hadoop and Eucalyptus (news.cnet.com)
• Cloudera aims to take Hadoop commercial (vnunet.com)
• The Meta Cloud – Flying data centers enter fourth dimension (theregister.co.uk)
• Cloudera floats commercial Hadoop distro (theregister.co.uk)
• Free Cool Tools for Educators (johnmwillis.com)
• Hadoop-Focussed Startup, Cloudera Raises $5 Million (gigaom.com)
• Hadoop, a Free Software Program, Finds Uses Beyond Search (nytimes.com)
• Cloudera launched to offer commercialized Hadoop (insidehpc.com)
• Cloudera (battellemedia.com)
• LinkLog: Streaming Data, Distributed Execution Engines (dorai.wordpress.com)