County Arcade, Leeds

The global CloudCamp movement continues to grow, with events over the next few weeks in Denmark, Germany, Nigeria, Ghana, Kenya, and across the United States. And now, I’m very pleased to announce that the English city of Leeds is joining the party.

CloudCamp events have been taking place in the UK for years, and the London gatherings have picked up real momentum. Outside London, we’ve seen a few events in Warrington, Newcastle, and Edinburgh. We believe that the time is now right for something more regular; a place in which the cloud-building, cloud-using, cloud-interested and cloud-exploring can come together for talk, beer, pizza and more… without having to jump on a train to the deep south.

CloudCamps are interesting events, with a real emphasis on informality. I’ve attended several around the world, and am always impressed by the energy in the room, and by the welcome extended to newcomers. As the main CloudCamp site describes,

“CloudCamp is an unconference where early adopters of Cloud Computing technologies exchange ideas. With the rapid change occurring in the industry, we need a place we can meet to share our experiences, challenges and solutions. At CloudCamp, you are encouraged you to share your thoughts in several open discussions, as we strive for the advancement of Cloud Computing. End users, IT professionals and vendors are all encouraged to participate.”

Jeremy Jarvis at Brightbox and Karyn Fleeting and Joel Turner at Tinderbox Media have been driving this event forward, and they’ve invited me on board to help out. I also get to be MC on the night.

We’ve got speakers and sponsors committed, with more of both to come. If you think you should be one of those doing the speaking or the sponsoring, do let us know.

So, if you like talking Cloud, if your boss has ordered you to learn Cloud, or if you’re just keen to understand a little more about what this Cloud thing can do for you, stick the evening of 14 June in your diary, sign up (for free) on Eventbrite, and come along to the Hilton DoubleTree in Leeds for an evening of fun, learning, beer, and more.

See you there!

Image of the County Arcade in Leeds by Flickr user Francisco Perez

Image via Wikipedia

The Open Knowledge Foundation (OKFN) promotes the creation, dissemination and use of ‘open knowledge.’ As part of this activity OKFN developed a data repository called CKAN, and has seen this become increasingly important to a range of data dissemination activities such as data.gov.uk and publicdata.eu.

In this podcast I talk with OKFN Director Rufus Pollock and CKAN Product Owner Irina Bolychevsky, to learn more about CKAN, its use in the context of open data, and the wider implications for dissemination of any data (whether open or closed).

Following up on a blog post that I wrote at the start of 2012, this is the tenth in an ongoing series of podcasts with key stakeholders in the emerging category of Data Markets.

Related articles

• Open Knowledge Releases Open Data Handbook 1.0 (readwriteweb.com)
• Data Market Chat: Leigh Dodds discusses Kasabi (cloudofdata.com)
• Data for the public good (radar.oreilly.com)
• Data Market Chat: Stephen O’Grady of RedMonk examines the bigger picture (cloudofdata.com)
• Data Market Chat: Nick Edouard discusses BuzzData (cloudofdata.com)

Image via CrunchBase

With Iceland’s DataMarket.com, Founder Hjálmar Gíslason is on his fourth startup, and ready to expand overseas. Focused upon becoming “Google for datanumbers,” DataMarket concerns itself with collecting and providing access to quantitative data; numbers from governments, international agencies, and commercial providers around the world.

Alongside the business of collecting data and making it available for download, DataMarket has invested in providing tools with which users can visualize data (typically in the form of a graph) and even compare results from diverse sources. Hjálmar sees these tools as part of a strategy to ensure that it is

“more desirable to use data on DataMarket than at the source.”

Hjálmar also discusses his view that four characteristics of data make it profitably sellable; proprietariness, timeliness, analysis, and curation.

Have a listen to learn more about DataMarket, and to hear Hjálmar’s thoughts on an industry segment that his company has done much to shape. And check back on Tuesday for the next podcast in the series; Ian Davis of Kasabi.

Following up on a blog post that I wrote at the start of 2012, this is the third in a series of podcasts with key stakeholders in the emerging category of Data Markets. Other conversations, all of which will be published here, have been scheduled with AggData, BuzzData, Factual, Infochimps, Kasabi, and Microsoft. I am still adding conversations to the series, and intend to talk with more companies and with analysts and investors with insight to share. 

Related articles

• Nurturing the market for Data Markets (cloudofdata.com)
• Data Market Chat: the podcasts are a-coming… (cloudofdata.com)
• Data Market Chat: Tyler Bell discusses Factual (cloudofdata.com)
• Data Market Chat: Chris Hathaway discusses AggData (cloudofdata.com)

Image via Wikipedia

Last week, open standards body OASIS unveiled yet another shiny new standards effort. The OASIS Topology and Orchestration Specification for Cloud Applications (TOSCA) Technical Committee hopes to make it “easier to deploy cloud applications without vendor lock-in,” and to support moving from one cloud to another. The usual suspects — the likes of IBM, CA, and Cisco — are on board. The usual holdouts — Google and Amazon, of course — are not. So what is TOSCA trying to achieve? How does it fit alongside all the dead, dying, or ponderously deliberating cloud standardisation efforts that have gone before? And without the giants of the cloud, is there really any point bothering?

As I’ve probably mentioned before, involvement in various national and international standardisation efforts played a big part in my early career. I went to the working group meetings in odd (but often beautiful) locations. I participated in the conference calls. I engaged on the mailing lists. I drafted and edited and reviewed the documents. I completely buy into the idea that there is a place for foundational standards, developed through consensus-building and maintained for the long haul by organisations that stand apart from the vested interests and their competing agendas.

I also believe that there’s a time and a place for these standardisation efforts. Do it too soon, and we end up ossifying something that needs to be in a state of flux. When you don’t know what the best way to prepare a meal is, it’s too soon to print the recipe book. We need to try different approaches, and we need to be able to throw away the attempts that didn’t work out. More worryingly, standardisation efforts can be used for political ends. They can be little more than a rod with which to beat the (usually dominant) competition. At best a distraction, or a talking shop for those unwilling or unable to just get on and do something. At worst, one amongst a toolchest of dirty tricks in a broader war for hearts, minds, and — ultimately — wallets.

The cloud market is a fascinating place. There are leaders and there are followers. There is innovation, and there is competition. There is agreement, and there is debate. For all the rhetoric, and all the posturing, we really don’t yet know the right answer to many of the cloud’s questions.

Maybe TOSCA and the Open Data Center Alliance and IEEE and the rest are — still — too early, and should be content to let the market thrash out a few more of these issues before anyone tries to write anything down? And when it is time to write some stuff down, let’s make sure we focus on specific, finite, tangible, atomic tasks rather than “the cloud.” As Dave Roberts commented in regard to TOSCA’s scope;

“That goal is so large, that I think it’s probably unbounded. When problems get unbounded, the best you can ever hope to achieve is to solve a large enough subset of the problem that the solution is still interesting. If you can’t achieve that, people ignore the solution because it fundamentally doesn’t help them. There is always an ‘interesting’ part of the problem space that they have to solve a different way, and that undercuts the use of the partial ‘solution.’”

And as for Tosca? Things didn’t end well for her, did they? Might TOSCA’s fate, too, be sealed?

Related articles

• Tech giants back standard for cloud portability – InfoWorld (infoworld.com)
• Proposed spec aims to nix cloud lock-in (gigaom.com)
• On TOSCA and Cloud Standards. MyPOV (diversity.net.nz)

I’ve been a fan of Om Malik‘s boutique analyst site, GigaOM Pro, pretty much from the outset, and happily renew my subscription each year. The site covers a wide range of industry topics, and those Quarterly Wrap-ups are worth the fee all by themselves.

I’ve written a few reports for them in the past, but was delighted when Mike Wolf got in touch to see if I fancied trying my hand at curation on their Infrastructure/Cloud channel.

So next week (from 31 January) I’m going to be gathering and commenting upon links from around the web, writing a daily ‘Today in Infrastructure,’ and finishing off with a Weekly Update. If you’re not (yet!) a subscriber, why not sign up for a free seven day trial and join me for the start of my little adventure?

If there’s something you think I should be covering, do let me know.

Related articles

• Free GigaOM Pro Webinar: The Scalable Cloud (gigaom.com)
• GigaOm raises $2.5M, claims 10,000 Pro subscribers (venturebeat.com)
• The Cloud has a place, even inside heavily regulated industries (cloudofdata.com)
• Talking Scalable Clouds with GigaOM Pro and Limelight Networks (cloudofdata.com)
• GigaOM Structure 2011 (datacenterknowledge.com)
• Join GigaOM at Big Data on March 23 in New York City (gigaom.com)

KnowledgeTree, a provider of cloud-based document management solutions based in Raleigh, North Carolina, today announced that customers on their enterprise payment plan will be able to download copies of documents, workflows and associated metadata for local backup.

As Krish noted over on CloudAve when the company recently announced their single sign-on capability, these are hardly groundbreaking advances. They are, however, important incremental enhancements and an indication of growing product maturity. To move beyond early adopters and niche use cases, ‘basic’ features such as integration with enterprise identity management processes and effective – tangible, almost – backup procedures are simply essential.

According to the release,

“With this feature, customer content is available for immediate download via secure HTTP, FTP and rsync connections, ensuring a quick and efficient backup. Downloads are encrypted and password-protected. Documents, while still available anytime and anywhere via the cloud, can now have backup versions housed on an organization’s own servers.”

This new backup capability goes further than the product’s existing ability to offer ZIP downloads of customer documents, as it preserves workflow, audit trails, and associated metadata.

I spoke with CEO Daniel Chalef ahead of the announcement, to learn a little more about the company’s direction. He was, of course, keen to stress that provision of this download capability should in no way be seen as a reflection on the robustness of either the Cloud or KnowledgeTree’s own (Amazon S3-powered) infrastructure. Rather, Chalef pointed to the continuing (mis-?) perception that auditory and compliance requirements may more easily be met by companies that are able to lay their hands upon local copies of key data.

With a focus upon customers in legal, financial and HR-type roles, KnowledgeTree has transitioned from providing on-premise solutions toward a mixed economy in which the product is available as either Software as a Service (SaaS), or as an on-premise installation typically addressed toward larger customers. The SaaS offering is updated on a 3-4 week release cycle, with the traditional on-premise version updated more slowly. A new version is due early in 2011, which will bring feature parity with the current SaaS release. Chalef argued that the company continues to grow both branches of the product, but I can’t help wondering how long it will truly remain cost-effective to offer – and maintain – both?

Chalef suggests that the company’s focus upon providing turn-key solutions designed to address the needs of its defined customer groups differentiates it from the competition. These, he argues, are primarily SugarCRM and Microsoft Sharepoint, both of which tend to present rather more of a blank canvas upon which customers (or VARs) are expected to build. A turn-key solution, especially deployed from the Cloud, should enable customers to realise value from their investment relatively quickly. The more significant investment in on-premise installations, bespoke development and VAR engagement may result in eventual cost savings or better alignment with business requirements, but it’s a long-term – and risky – bet. Press Releases over the past few months suggest that KnowledgeTree continues to innovate, fleshing out the feature set on a product that shows no sign of slowing.

Related articles

• KnowledgeTree Steps Up Its Enterprise Game (cloudave.com)

Formed as a partnership between Microsoft and Accenture, Avanade focuses upon delivering IT solutions based upon Microsoft’s suite of technologies and products, including Cloud offerings such as Azure and the company’s Business Productivity Online Standard Suite (BPOS).

I discussed Azure in a podcast with Microsoft’s Amitabh Srivastava last year, and in this latest conversation Scott and Bob share some of the experiences Avanade has gained in rolling out over 1.3 million BPOS seats to a wide range of enterprise clients.

This podcast was recorded on Wednesday 28 July, 2010.

During our conversation, we referred to the following resources;

• Accenture
• Amazon
• App Fabric
• Avanade
• Avanade Cloud Computing
• Azure
• BPOS
• Hotmail
• Java
• Lotus Notes
• Microsoft
• Microsoft Live@Edu
• PHP
• Werner Vogels

Related articles by Zemanta

• Microsoft shares (officially) its future BPOS plans (zdnet.com)
• Capgemini now championing Microsoft BPOS over Google Apps (zdnet.com)
• Microsoft launches BPOS Syndication Partner programme (channelweb.co.uk)
• Microsoft gives partners free cloud tools (infoworld.com)
• Is Sharepoint 2010 Cloud Ready? (readwriteweb.com)

I shared some of my views on this site soon after the event, but think you’ll like the range of opinions and insights from regular Gang members, participants in our audience, and conference organiser Tony Shaw. Have a listen, and see what you think.

As well as generating our regular audio podcast, the event organisers were kind enough to also video the session, and publish the video on their site. I’ve also embedded it, below.

In fact, there’s a nice set of resources from the conference taking shape over on the Semantic Universe site; take a look and see video from the Linked Data Panel (another of the sessions I moderated), the keynote sessions and more.

Related articles by Zemanta

• Tom Gruber talks about Siri, the Virtual Personal Assistant (cloudofdata.com)
• Don’t Miss: Semantic Web Gang, Web 3.0 Conference, Semantic Technology Conference (semanticsincorporated.com)
• Bing is not alone; similar techniques alive and well in existing vertical search (blogs.zdnet.com)

It might seem that the mega-bucks reports from the likes of Gartner, Forrester et al are the preserve of CxOs with vast desks upon which they can array the multitudinous documents to which their employers’ subscription entitles them. The truth, though, is that these documents — which notionally sell for hundreds or thousands of dollars apiece — can often be downloaded in exchange for some contact details. The source of this apparent largesse is not the analysts’ own website, but rather the sites of companies looked upon with particular favour within a given report. The back-room exchanges of kudos, cash, or mutual back-slapping that sees these assets made freely available are well understood, and legitimate.

And so it was with a recent report from Forrester, made freely available to all via the web site of 3Tera; the co-founder of which I spoke with recently in one of my podcasts.

The report is ‘Deliver Cloud Benefits Inside Your Walls,’ dated 13 April 2009, and produced by Forrester’s James Staten with input from Simon Yates, John Rymer, Frank Gillett and Lauren Nelson. Billed as “the first document in the ‘Private Cloud’ series,” it would appear that Forrester, at least, has no hang-ups about the notion of a Private Cloud. Whilst some purists become incredibly agitated about this ‘dilution’ of their dream, realists, pragmatists and (it would appear) analysts are simply getting on with it.

According to the report’s Executive Summary,

“While the excitement about cloud computing centers on Amazon Elastic Compute Cloud (EC2) and other public infrastructure-as-a-service products, many enterprise infrastructure and operations professionals are taking this concept in-house and building their own internal clouds. These pools of virtual machines can be built upon either virtual server or high-performance computing (HPC) grid foundations and can be operated according to the specific security and process requirements of the business. But to deliver the fundamentally better economic value of cloud architectures within your walls, these clouds require a dynamic platform (or automated workload management) and developer self-service interfaces. There’s a growing list of vendors eager to help you deploy an internal cloud, but be sure you understand that these solutions are more building blocks than complete solutions and must be customized to your specific needs.”

It would be impossible to cover the space in much detail in a report comprising just nine pages of substantive content, but Staten and his co-authors do a reasonable job of outlining some high level benefits of internal deployment whilst also flagging some of the issues in need of addressing. Prominent amongst these is the suggestion that ‘bypassing IT Ops’ to hand developers their own internal Cloud is not necessarily something to be done lightly;

“Whether you buy the hype or not, enterprise application developers are finding the self-service, pay-as-you-go, instant deployment values of cloud computing platforms appealing. Developers can go to a Web page, sign up with a credit card, and instantly instantiate any number of virtual machines and applications without any IT ops involvement. Interviews conducted by Forrester show that many
enterprise developers are doing just that.

But IT ops processes and procedures — and enterprise architecture rules for that matter — exist to ensure that the overarching needs and policies of the business are fulfilled and followed. Although making time in the deployment process to accommodate these demands may hinder time-to-market, often there are very good reasons to do so”

This is true, but overlooks the reality that those processes and procedures arose to manage a very different environment. There is a middle ground to steer here, and it is one that should make a great deal of sense. Virtualisation, elasticity and more create significant opportunities for the cost-effective provision of Enterprise IT. Rather than simply accepting the status quo of those established processes and procedures, or routing around them with the departmental credit card and some cheap Dells or EC2 instances, there is a real opportunity for IT and business teams to engage in fresh dialogue; to understand the match between changing expectations, changing requirements, and changing possibilities.

A throwaway comment also points, tellingly, to the real need for organisational change if we are to optimise the benefits of these new opportunities;

“We can stand up a LAMP stack or Windows VM within 24 hours now — one hour to provision the VM and 23 hours to move the money.”

Elsewhere in the document, the Forrester team suggests that,

“With today’s virtual infrastructure solutions and a growing list of internal cloud platform technologies, it’s fairly easy for an enterprise to start up a cloud-like environment within its own domain.”

The document goes on to suggest that, in Q3 of 2008, around 4% of surveyed enterprises in Europe and North America had done so, with a significant number implementing or seeking budget. More than half of those surveyed, though, were either ‘Not Aware’ or ‘Not Interested’ in an Internal Cloud. For ‘small businesses’ of fewer than 100 employees that figure rose to a massive 80%. It would be interesting to see the same figures for ‘Public Cloud’ utilisation… although there is likely to be significantly more under-reporting of public Cloud use as so much of it will be by individuals and teams who are below the corporate radar.

With one of the significant cost reductions in the public Cloud being directly related to more efficient utilisation of virtual and physical machines, the corresponding saving within the enterprise — whilst still significant — is likely to be smaller, and there may still be a need for a hybrid arrangement to permit ‘cloud bursting’ at times of particularly high load.

The document draws out three ‘internal Cloud solutions’ — 3Tera, Elastra and Zimory — all of which are worth a look, and then goes on to very briefly touch on three alternative approaches. Here, more than elsewhere in the document, Forrester seem not to go into enough detail. They touch upon a DIY approach based upon EUCALYPTUS (now available in commercial form, of course, although the authors don’t mention this), contracting with a systems integrator such as IBM to secure a bespoke solution, and ‘waiting for the major virtualisation vendors to show up.’

There’s an awful lot of activity already underway from these companies, with whom enterprises probably already have a relationship, and it seems unhelpful to pass over them so quickly. What about Windows Azure, for example, which Microsoft is increasingly suggesting should be available for local utilisation?

So, on the whole this document is a useful overview. As might be expected in something of this length, it ends up raising more questions than it answers (which drives Forrester customers back to their Analysts, of course), and there are one or two areas in which it leaves odd gaps. Maybe the next document in the series will begin to fill some of those gaps?

Related articles by Zemanta

• SaaS And Cloud – A Strong Combination (teabreak.pk)
• Forrester Backs Private Clouds – Will Others Follow Suit? (gigaom.com)
• Cloud computing shapes up as big trend for 2009 (infoworld.com)
• Is Your Future in ‘The Cloud’? (usnews.com)
• Cloud Computing’s Three-Horse Race (gigaom.com)
• Cloud computing — value is an assumption, but cost matters (news.cnet.com)

Amid mounting international concern that the guidance lasers aboard Jeff Bezos‘ new Floating Amazon Cloud Environment would interfere with Rudolph‘s sense of direction, sources close to the Amazon Web Services team tell me that they’ve been forced to alter priorities and switch attention to an early release of the next product on their roadmap.

Today sees the release of Amazon‘s latest web service; the Hadoop-powered Elastic MapReduce;

“Using Amazon Elastic MapReduce, you can instantly provision as much or as little capacity as you like to perform data-intensive tasks for applications such as web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research. Amazon Elastic MapReduce lets you focus on crunching or analyzing your data without having to worry about time-consuming set-up, management or tuning of Hadoop clusters or the compute capacity upon which they sit.”

The company’s press release quotes VP for Product Management & Developer Relations, Adam Selipsky, who notes;

“Some researchers and developers already run Hadoop on Amazon EC2, and many of them have asked for even simpler tools for large-scale data analysis. Amazon Elastic MapReduce makes crunching in the cloud much easier as it dramatically reduces the time, effort, complexity and cost of performing data-intensive tasks.”

MapReduce was brought to prominence by Google, and is one of the principal techniques at that company’s disposal in enabling them to break massive data sets into manageable chunks suitable for cost-effective processing on the commodity hardware for which they are known. The abstract for a Google research paper on the topic outlines the value proposition reasonably succinctly;

“MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google’s clusters every day.“

Hadoop is a Yahoo!-nurtured Open Source equivalent to Google’s MapReduce, managed as a project of the Apache Software Foundation, and reputedly scalable to handle many petabytes of data distributed across thousands of CPUs.

As Adam noted in the press release, customers (such as the New York Times and Netflix) are already using Hadoop on Amazon’s Web Services. Today’s announcement makes it easier to cost-effectively and transparently commission (and decommission) the required compute resources. This is the ‘elasticity’ referred to in the new service’s name, and is an increasingly important aspect of the current generation of Cloud-based compute services; much of the economic value proposition lies in only using (and therefore paying for) the resources you actually need to complete a task. If demand increases, the number of (virtual) machines available should rapidly increase to cope, and they should shut back down just as rapidly when the demand passes;

“Amazon Elastic MapReduce enables you to use as many or as few compute instances running Hadoop as you want. You can commission one, hundreds, or even thousands of instances to process gigabytes, terabytes, or even petabytes of data. And, you can run as many job flows concurrently as you wish. You can instantly spin up large Hadoop job flows which will start processing within minutes, not hours or days. When your job flow completes, unless you specify otherwise, the service automatically tears down your instances.“

Elastic MapReduce is currently available only for data centres in Amazon’s US region (so non-US customers can use the service; they just have to be able/willing to transfer the data beyond their borders), and is priced in addition to existing EC2 instances with Elastic MapReduce on a $US0.10 per hour ‘small’ instance costing a further $US0.015 per hour (yes, 1 and a half cents per hour) and on a $US0.80 per hour ‘extra large’ instance costing a further $US0.12 per hour.

Elastic MapReduce is another nice example of slow, incremental improvement to Amazon’s core Web Services offer.

It remains to be seen, as developers get down to using it for real, whether it’s pitched as a low-end disruptor that simply rounds out another piece of the emerging AWS whole, or if it’s a viable competitor in its own right to the recently announced Cloudera which sees taking Hadoop to mainstream enterprise customers as its raison d’etre;

“Cloudera can help you install, configure and run Hadoop for large-scale data processing and analysis. Get Cloudera’s Distribution for Hadoop and start working with Big Data today.“

Update: Amazon’s Jeff Barr provides a lot more detail in a post to the AWS Blog.

Related articles by Zemanta

• Learn to develop software for Hadoop (makezine.com)
• Amazon Blows Away Fourth Quarter Earnings Expectations (businessweek.com)
• Grand Challenges in Database Self-Management (perspectives.mvdirona.com)
• Cloud platforms of the future: Hadoop and Eucalyptus (news.cnet.com)
• Cloudera aims to take Hadoop commercial (vnunet.com)
• The Meta Cloud – Flying data centers enter fourth dimension (theregister.co.uk)
• Cloudera floats commercial Hadoop distro (theregister.co.uk)
• Free Cool Tools for Educators (johnmwillis.com)
• Hadoop-Focussed Startup, Cloudera Raises $5 Million (gigaom.com)
• Hadoop, a Free Software Program, Finds Uses Beyond Search (nytimes.com)
• Cloudera launched to offer commercialized Hadoop (insidehpc.com)
• Cloudera (battellemedia.com)
• LinkLog: Streaming Data, Distributed Execution Engines (dorai.wordpress.com)