It might seem that the mega-bucks reports from the likes of Gartner, Forrester et al are the preserve of CxOs with vast desks upon which they can array the multitudinous documents to which their employers’ subscription entitles them. The truth, though, is that these documents — which notionally sell for hundreds or thousands of dollars apiece — can often be downloaded in exchange for some contact details. The source of this apparent largesse is not the analysts’ own website, but rather the sites of companies looked upon with particular favour within a given report. The back-room exchanges of kudos, cash, or mutual back-slapping that sees these assets made freely available are well understood, and legitimate.

And so it was with a recent report from Forrester, made freely available to all via the web site of 3Tera; the co-founder of which I spoke with recently in one of my podcasts.

The report is ‘Deliver Cloud Benefits Inside Your Walls,’ dated 13 April 2009, and produced by Forrester’s James Staten with input from Simon Yates, John Rymer, Frank Gillett and Lauren Nelson. Billed as “the first document in the ‘Private Cloud’ series,” it would appear that Forrester, at least, has no hang-ups about the notion of a Private Cloud. Whilst some purists become incredibly agitated about this ‘dilution’ of their dream, realists, pragmatists and (it would appear) analysts are simply getting on with it.

According to the report’s Executive Summary,

“While the excitement about cloud computing centers on Amazon Elastic Compute Cloud (EC2) and other public infrastructure-as-a-service products, many enterprise infrastructure and operations professionals are taking this concept in-house and building their own internal clouds. These pools of virtual machines can be built upon either virtual server or high-performance computing (HPC) grid foundations and can be operated according to the specific security and process requirements of the business. But to deliver the fundamentally better economic value of cloud architectures within your walls, these clouds require a dynamic platform (or automated workload management) and developer self-service interfaces. There’s a growing list of vendors eager to help you deploy an internal cloud, but be sure you understand that these solutions are more building blocks than complete solutions and must be customized to your specific needs.”

It would be impossible to cover the space in much detail in a report comprising just nine pages of substantive content, but Staten and his co-authors do a reasonable job of outlining some high level benefits of internal deployment whilst also flagging some of the issues in need of addressing. Prominent amongst these is the suggestion that ‘bypassing IT Ops’ to hand developers their own internal Cloud is not necessarily something to be done lightly;

“Whether you buy the hype or not, enterprise application developers are finding the self-service, pay-as-you-go, instant deployment values of cloud computing platforms appealing. Developers can go to a Web page, sign up with a credit card, and instantly instantiate any number of virtual machines and applications without any IT ops involvement. Interviews conducted by Forrester show that many
enterprise developers are doing just that.

But IT ops processes and procedures — and enterprise architecture rules for that matter — exist to ensure that the overarching needs and policies of the business are fulfilled and followed. Although making time in the deployment process to accommodate these demands may hinder time-to-market, often there are very good reasons to do so”

This is true, but overlooks the reality that those processes and procedures arose to manage a very different environment. There is a middle ground to steer here, and it is one that should make a great deal of sense. Virtualisation, elasticity and more create significant opportunities for the cost-effective provision of Enterprise IT. Rather than simply accepting the status quo of those established processes and procedures, or routing around them with the departmental credit card and some cheap Dells or EC2 instances, there is a real opportunity for IT and business teams to engage in fresh dialogue; to understand the match between changing expectations, changing requirements, and changing possibilities.

A throwaway comment also points, tellingly, to the real need for organisational change if we are to optimise the benefits of these new opportunities;

“We can stand up a LAMP stack or Windows VM within 24 hours now — one hour to provision the VM and 23 hours to move the money.”

Elsewhere in the document, the Forrester team suggests that,

“With today’s virtual infrastructure solutions and a growing list of internal cloud platform technologies, it’s fairly easy for an enterprise to start up a cloud-like environment within its own domain.”

The document goes on to suggest that, in Q3 of 2008, around 4% of surveyed enterprises in Europe and North America had done so, with a significant number implementing or seeking budget. More than half of those surveyed, though, were either ‘Not Aware’ or ‘Not Interested’ in an Internal Cloud. For ‘small businesses’ of fewer than 100 employees that figure rose to a massive 80%. It would be interesting to see the same figures for ‘Public Cloud’ utilisation… although there is likely to be significantly more under-reporting of public Cloud use as so much of it will be by individuals and teams who are below the corporate radar.

With one of the significant cost reductions in the public Cloud being directly related to more efficient utilisation of virtual and physical machines, the corresponding saving within the enterprise — whilst still significant — is likely to be smaller, and there may still be a need for a hybrid arrangement to permit ‘cloud bursting’ at times of particularly high load.

The document draws out three ‘internal Cloud solutions’ — 3Tera, Elastra and Zimory — all of which are worth a look, and then goes on to very briefly touch on three alternative approaches. Here, more than elsewhere in the document, Forrester seem not to go into enough detail. They touch upon a DIY approach based upon EUCALYPTUS (now available in commercial form, of course, although the authors don’t mention this), contracting with a systems integrator such as IBM to secure a bespoke solution, and ‘waiting for the major virtualisation vendors to show up.’

There’s an awful lot of activity already underway from these companies, with whom enterprises probably already have a relationship, and it seems unhelpful to pass over them so quickly. What about Windows Azure, for example, which Microsoft is increasingly suggesting should be available for local utilisation?

So, on the whole this document is a useful overview. As might be expected in something of this length, it ends up raising more questions than it answers (which drives Forrester customers back to their Analysts, of course), and there are one or two areas in which it leaves odd gaps. Maybe the next document in the series will begin to fill some of those gaps?

Related articles by Zemanta

• SaaS And Cloud – A Strong Combination (teabreak.pk)
• Forrester Backs Private Clouds – Will Others Follow Suit? (gigaom.com)
• Cloud computing shapes up as big trend for 2009 (infoworld.com)
• Is Your Future in ‘The Cloud’? (usnews.com)
• Cloud Computing’s Three-Horse Race (gigaom.com)
• Cloud computing — value is an assumption, but cost matters (news.cnet.com)

“EUCALYPTUS — Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems — is an open-source software infrastructure for implementing Elastic/Utility/Cloud computing using computing clusters and/or workstation farms. The current interface to EUCALYPTUS is interface-compatible with Amazon.com’s EC2 (arguably the most commercially successful Cloud computing service), but the infrastructure is designed to be modified and extended so that multiple client-side interfaces can be supported. In addition, EUCALYPTUS is implemented using commonly-available Linux tools and basic web service technology making it easy to install and maintain.”

Since then, we’ve seen EUCALYPTUS embraced by Sun Microsystems in their Cloud announcements and potentially downloaded to millions of machines around the world as part of the latest update to Linux’s popular Ubuntu distribution.

Today the UCSB research project takes the next step, announcing a successful Series A investment round led by Benchmark Capital that moves the team out of the University and onto a professional footing with $5.5 Million to spend. Project Director Wolski becomes CTO, with Woody Rollins as CEO and Matt Reid as VP Sales & Marketing rounding out Eucalyptus Systems‘ fledgling management team. Wolski reports that the entire UCSB development team is moving across to the newly capitalised company, which is licensing IP from UCSB in return for an undisclosed equity stake. Benchmark’s Kevin Harvey takes a seat on the Board, which is Chaired by former AOL Europe CEO Andreas Von Blottnitz.

Wolski is quick to stress that Eucalyptus Systems is an open source company; there is no intention to start charging for software that is freely available for download today, and this will be actively maintained and developed moving forward. A point release is expected ‘in about a week’ to resolve minor issues with respect to Ubuntu 9.04, and version 1.6 of Eucalyptus will follow ‘in 6-8 weeks.’

Speaking of the company’s proposition to new customers, Wolski suggests that

“Eucalyptus Systems will enable businesses of any size to leverage their own IT resources to get the benefits of cloud computing without the concerns of lock-in, security ambiguity, and unexpected storage costs that can be associated with public clouds.”

Interestingly, Wolski describes the revenue model traditionally employed by companies seeking to make money from open source as no more than a way of boot-strapping Eucalyptus Systems toward their real goal. Rather than simply concentrating on the provision of for-profit support and consultancy, Wolski has his sights set on the sale of new Eucalyptus-powered software solutions directly addressed to Enterprise customers. It’s not yet clear whether these solutions will be SaaS offerings or for on-premise installation, but Wolski is confident that the company’s early customers will begin to receive their new software during the third quarter of 2009.

Although best known for providing an on-premise equivalent to Amazon’s Elastic Compute Cloud (EC2), Wolski stresses that the Eucalyptus architecture is API agnostic and could be extended to emulate other Cloud solutions relatively easily. This, allied with Eucalyptus’ industry-leading support for both the KVM and Xen hypervisors, raises the prospect of enterprise customers integrating their own (Eucalyptus powered) internal Cloud with different public Clouds, seamlessly and at will.

Related articles by Zemanta

• Karmic Koalas Love Eucalyptus (radar.oreilly.com)
• Rich Wolski discusses Grids, Clouds, and the past, present and future of EUCALYPTUS (cloudave.com)
• Eucalyptus to be Included in Next Ubuntu Release (berkeleyclouds.blogspot.com)
• Without APIs, there is no cloud computing (news.cnet.com)
• Sun to take to the cloud (infoworld.com)
• Ubuntu now has “Cloud Computing Inside” (news.cnet.com)
• Ubuntu: Karmic Koala to make cloud ‘dance’ (news.zdnet.com)

Most readers of this blog are probably well aware that a new version of the Ubuntu Linux distribution is coming this week, and that it will be putting code from the Open Source EUCALYPTUS Project to work in simplifying the creation of private Clouds that look remarkably like Amazon’s EC2. You’ve probably also read RightScale‘s announcements with respect to Ubuntu, and heard that Sun Microsystems were also making supportive noises about EUCALYPTUS and the EC2 API before their recent change in circumstances.

Earlier today I spoke with Simon Wardley of Canonical (the commercial organisation that sells support and consultancy for Ubuntu) to hear a little more about what those downloading Ubuntu will get… and what it might mean for the rapidly shifting Cloud landscape.

Production of this podcast was supported by Talis, and show notes are available on their Nodalities blog.

Related articles by Zemanta

• Ubuntu gets pre-Koala cloud love (theregister.co.uk)
• Karmic Koalas Love Eucalyptus (radar.oreilly.com)
• Mark Shuttleworth Now Sees Oracle As a Big Open Source Kahuna (ostatic.com)
• Ubuntu promises DIY Amazon cloud (theregister.co.uk)
• Ubuntu now has “Cloud Computing Inside” (news.cnet.com)

Amid mounting international concern that the guidance lasers aboard Jeff Bezos‘ new Floating Amazon Cloud Environment would interfere with Rudolph‘s sense of direction, sources close to the Amazon Web Services team tell me that they’ve been forced to alter priorities and switch attention to an early release of the next product on their roadmap.

Today sees the release of Amazon‘s latest web service; the Hadoop-powered Elastic MapReduce;

“Using Amazon Elastic MapReduce, you can instantly provision as much or as little capacity as you like to perform data-intensive tasks for applications such as web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research. Amazon Elastic MapReduce lets you focus on crunching or analyzing your data without having to worry about time-consuming set-up, management or tuning of Hadoop clusters or the compute capacity upon which they sit.”

The company’s press release quotes VP for Product Management & Developer Relations, Adam Selipsky, who notes;

“Some researchers and developers already run Hadoop on Amazon EC2, and many of them have asked for even simpler tools for large-scale data analysis. Amazon Elastic MapReduce makes crunching in the cloud much easier as it dramatically reduces the time, effort, complexity and cost of performing data-intensive tasks.”

MapReduce was brought to prominence by Google, and is one of the principal techniques at that company’s disposal in enabling them to break massive data sets into manageable chunks suitable for cost-effective processing on the commodity hardware for which they are known. The abstract for a Google research paper on the topic outlines the value proposition reasonably succinctly;

“MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google’s clusters every day.“

Hadoop is a Yahoo!-nurtured Open Source equivalent to Google’s MapReduce, managed as a project of the Apache Software Foundation, and reputedly scalable to handle many petabytes of data distributed across thousands of CPUs.

As Adam noted in the press release, customers (such as the New York Times and Netflix) are already using Hadoop on Amazon’s Web Services. Today’s announcement makes it easier to cost-effectively and transparently commission (and decommission) the required compute resources. This is the ‘elasticity’ referred to in the new service’s name, and is an increasingly important aspect of the current generation of Cloud-based compute services; much of the economic value proposition lies in only using (and therefore paying for) the resources you actually need to complete a task. If demand increases, the number of (virtual) machines available should rapidly increase to cope, and they should shut back down just as rapidly when the demand passes;

“Amazon Elastic MapReduce enables you to use as many or as few compute instances running Hadoop as you want. You can commission one, hundreds, or even thousands of instances to process gigabytes, terabytes, or even petabytes of data. And, you can run as many job flows concurrently as you wish. You can instantly spin up large Hadoop job flows which will start processing within minutes, not hours or days. When your job flow completes, unless you specify otherwise, the service automatically tears down your instances.“

Elastic MapReduce is currently available only for data centres in Amazon’s US region (so non-US customers can use the service; they just have to be able/willing to transfer the data beyond their borders), and is priced in addition to existing EC2 instances with Elastic MapReduce on a $US0.10 per hour ‘small’ instance costing a further $US0.015 per hour (yes, 1 and a half cents per hour) and on a $US0.80 per hour ‘extra large’ instance costing a further $US0.12 per hour.

Elastic MapReduce is another nice example of slow, incremental improvement to Amazon’s core Web Services offer.

It remains to be seen, as developers get down to using it for real, whether it’s pitched as a low-end disruptor that simply rounds out another piece of the emerging AWS whole, or if it’s a viable competitor in its own right to the recently announced Cloudera which sees taking Hadoop to mainstream enterprise customers as its raison d’etre;

“Cloudera can help you install, configure and run Hadoop for large-scale data processing and analysis. Get Cloudera’s Distribution for Hadoop and start working with Big Data today.“

Update: Amazon’s Jeff Barr provides a lot more detail in a post to the AWS Blog.

Related articles by Zemanta

• Learn to develop software for Hadoop (makezine.com)
• Amazon Blows Away Fourth Quarter Earnings Expectations (businessweek.com)
• Grand Challenges in Database Self-Management (perspectives.mvdirona.com)
• Cloud platforms of the future: Hadoop and Eucalyptus (news.cnet.com)
• Cloudera aims to take Hadoop commercial (vnunet.com)
• The Meta Cloud – Flying data centers enter fourth dimension (theregister.co.uk)
• Cloudera floats commercial Hadoop distro (theregister.co.uk)
• Free Cool Tools for Educators (johnmwillis.com)
• Hadoop-Focussed Startup, Cloudera Raises $5 Million (gigaom.com)
• Hadoop, a Free Software Program, Finds Uses Beyond Search (nytimes.com)
• Cloudera launched to offer commercialized Hadoop (insidehpc.com)
• Cloudera (battellemedia.com)
• LinkLog: Streaming Data, Distributed Execution Engines (dorai.wordpress.com)

Deepak pointed to an Amazon page, which briefly reported that;

“Developers and businesses can now run their Amazon EC2 instances in the EU to help achieve lower latency, operate closer to other resources like Amazon S3 in the EU, and meet EU data storage requirements when required. The new European Region for Amazon EC2 contains two Availability Zones enabling you to easily and cost effectively run fault-tolerant applications with the same scalability, reliability and cost efficiency achieved with Amazon EC2 in the US.”

Amazon CTO Werner Vogels and Evangelist Jeff Barr were close behind, using Twitter to share the same news, and pointing to longer blog posts on Werner’s blog and the Amazon Web Services Blog. A nice example of Twitter at work.

As for the news they were reporting; it’s good to see local availability for EC2 resources join the existing European sites for S3 storage.

Werner’s post outlines a three-part rationale for the move, broadly characterised as;

• lower latency (faster) access to the servers for customers in Europe,
• cheaper use of data stored in the European parts of S3,
• compliance with European regulatory requirements regarding storage of personal (and other) data outside Europe.

All are useful, and different customers will certainly emphasise each differently. I wonder which was the most commonly cited? ‘Compliance’ was a major stumbling block to using EC2 with certain data in the past, but I wonder if customers for whom that was an issue have traditionally been that important in the grand scheme of things at Amazon?

So; speed, price, or privacy?