Hadoop: The new boardroom buzzword

A few years ago, the open source data platform Hadoop was the domain of first mover technology companies and entrepreneurial analysts frustrated with the limitations of traditional data storage systems’ capabilities experimenting on their own desktops to create new analytical models.

Today, the quirky moniker is part of boardroom banter and forms the technology base for one of SAS’ latest strategic alliances. We sat down for a conversation with Simon Gregory, EMEA business development director at Hortonworks, a software company specialized in developing applications for and ensuring support for the Hadoop platform. Along with Adrian Jones from SAS' Global Centre of Excellence for Information Management and Analytics, we set out to discover what the buzz is about.

Why is Hadoop different?

Adrian Jones: Hadoop shot to the fore after a difficult financial period, and people were looking to challenge the market place. Early adopters were looking to gain flexibility at an attractive price point, as looking for new things were a challenge with the constraints of the traditional data storage systems. So whether it is working with new data types, or greater volumes instead of just samples, Hadoop is just flexible and can cope with that. You can onboard more data quicker.

Simon Gregory: The types of data our customers try to drive into Hadoop are not the types of data that fits nicely into traditional systems or rows and columns. Hadoop lets you supplement existing systems with a new style of data, a complimentary platform that actually supports this diversity of data. And this works well with the emergence of mobile and social media, where the focus is on unstructured data.

Why were legacy data storage systems not able to adapt to these demands?

Adrian Jones: People, analysts especially, have been doing things on their desktops and own servers that management did not know about, running models on Hadoop to complement their work. I have met analysts who spent 60 percent of their time just doing data management, not analytics – the traditional systems were focused on SLAs and other structural limitations and it made it hard to join data together. IT departments were blissfully unaware for a long time, so were data warehouse vendors who missed what the business was trying to accomplish because they were too focused on what IT was telling them.

Simon Gregory: There was a sort of purchasing mechanism in the past in which line-of-business managers went to IT and said “I would like you to build me a program based on this business problem” and what happened was that an application would be built that was perfectly capable of handling the type and volume of data, but they would be created in silos. So what we hear our customers saying is that they ended up having a lot of data silos across their estate and what they really want is a single point of entry and the current application points are not giving them this.

Adrian Jones: It is not like IT intended not to be able to serve their business lines’ needs or wanted to created silos – it just happened over time. It is all a cultural thing – we need to educate and address the company culture, because in most companies we work the way we do due to legacies and processes which were determined by IT systems. We need to educate IT departments to be more of a flexible service model with a degree of structure around it while we educate business managers that this flexibility is actually good, they can innovate more. So you need to address the fears.

But what I see is that when you let analysts play with this, they get it – and BANG – it is an overnight turning point. But first, there is a period of convincing managers, gaining sponsors and advocates. So it is about opening people’s minds up to changing the way they have worked for sometimes 30 years.

How is Hadoop changing the way businesses collaborate around Big Data?

Simon Gregory: What is really interesting now is seeing the collaboration between the CTO and the IT infrastructure office - and more and more, between the CMO and the Chief Data Officer. This is a really interesting shift. Three years ago, Hadoop was the domain of mostly technology companies but now you are seeing traditional, even conservative companies starting to use it.

The business is starting to see data as an asset and the minute they do this, they can see how Hadoop can help them in moving forward. It is about accepting data that shifts your customers’ perception of your products, exposes information you did not have access to before. The cultural part is critical – when the business sees this as important, it becomes important. Also, analytics is an asset – not back office. Both of these worlds need to go hand in hand. Data is worthless without data analytics and data access. Hadoop alone is not the answer – but in the broader ecosystem, including analytics, that is where it fits together.

Adrian Jones: You are seeing companies revisiting their data strategies and seeing that some companies are unloading everything but the primary data into Hadoop to use their resources most appropriately. This way they can still maintain the integration to legacy systems without the cost profile. Customers are talking about how Hadoop fits into their data strategies – from all angles. Hadoop changes the way an analyst thinks and works: It sets the analyst free again and that is going to be very powerful. Creative analysts will do very well in this new paradigm!

About Hortonworks

Hortonworks was founded in 2011 by Yahoo! and Benchmark Capital. 24 lead engineers from Yahoo made up the initial workforce, naming the company from the famous Dr. Seuss book “Horton hears a Who!”. Hortonworks develops, distributes and supports the open source Apache Hadoop data platform. The Hortonworks Data Platform provides an open platform that integrates with existing IT platforms and upon which enterprises can build and deploy Hadoop-based applications. In the Nordics, Hortonworks have just hired their first local employees and initiated dialogue with strategic partners, including SAS Institute.

Simon Gregory on Hortonworks: We are a very passionate and community-driven company - but our aim is to put some rigor around this technology. We add all the functions that a business line would want: Security, Governance, Operations and Administration – all the functions you expect from an enterprise product, we are delivering around an open source product, to make clients secure in this environment. All of it is delivered into Apache, to let partners like SAS Institute plug directly into the platform with analytic tools.