Coming To A Place Near You: A Private Cloud Spiked With Big Data

Netflix similarity map

Yesterday, I moderated a couple of panels at the Big Data Cloud event. I have been a keynote speaker, panelist, moderator, and participant for many conferences in the last few years. It has always been a pleasure to see the cloud and big data becoming more and more mainstream. Here are my quick observations and insights from the event:

Private cloud getting momentum: As a public cloud proponent I thought I would never have to write this. But lately I have seen more and more interest in private cloud; new start-ups, established cloud vendors, and large legacy vendors are designing private or hybrid cloud solutions. Vendors have recognized that prospects and customers have started to take cloud very seriously but they still have the same concerns what they had few years back: security, moving data to public cloud, and giving up control. I am not interested in the private/public debate (though I do love to mess with fellow clouderati on Twitter on this topic). My take on this trend is that the vendors should do whatever it takes to move the organizations to the cloud, private or public. Once companies dip their toes, they themselves will realize what's good for them.

Big Data as a serious category: A few days back, I blogged about big data going mainstream. Coming out from this event, it felt like, today Big Data is where cloud was a couple of years back. When I asked people a few years back "What's Hadoop?" They would reply "Huh?" Now, everyone wants to know more about Hadoop, Hive, HBase, S4, Oozie, Pig, Cassandra, and other big data frameworks. They're interested in analyzing and comparing available solutions. They're asking all the right questions. The VC investment in this category has been record high. Hadoop World was a sold out event this year with 1500 participants. Milind Bhandarkar, Chief Architect with Greenplum Labs, mentioned that in 2008, during the first Hadoop summit, they had to coax people to come to the summit. The people who willingly came to the summit either worked for Yahoo or Facebook. We have come a long way and there's a long way to go but this is a rock solid category. As the first set of big data infrastructure companies settle in we will see people building killer applications and PaaS solutions specifically designed to leverage big data. It is encouraging to see more and more companies and venture capitalists recognizing that the data is worth a lot more if they have the right tools and right people — the data scientists — to do something interesting with it. For example, Greylock partners have hired DJ Patil as a "data scientist in residence" to help them with evaluating their opportunities and advising their portfolio companies on big data strategies.

Rise in popularity of open source frameworks: If you follow the history of open source you'll realize that when a proprietary way of doing things become popular, commercial vendors pose a lock-in threat, and things don't work as expected, developers get frustrated and start to work on filling that gap by building open source technology. Linux started that way and so were many other open source projects. This is why I am excited to see OpenStack gaining rapid momentum. It's slowly becoming a de-facto standard to build a commercial cloud solutions. I also like Cloud Foundry since many companies that I know of, ISVs and large IT shops, won't use a public PaaS. They would prefer to launch their own PaaS solution in the cloud. Without an open source solution, it does become a big challenge.