Sensing a disturbance in the data

By Jim Harris, Blogger-in-Chief at Obsessive-Compulsive Data Quality (OCDQ)

Data analytics has historically focused on analyzing data after it stopped moving and was stored, often in a data warehouse. But recent years have seen the rise of event stream processing (ESP) where data is continuously analyzed while it’s in motion, within what’s referred to as event streams.

ESP captures the real-time value of data before it’s lost in the time lag between creation and storage – and before it’s lost in the time lag between analysis and action. ESP also detects meaningful patterns within event streams and, most important, deviations (disturbances) from those patterns to determine if an individual event within the stream should trigger an immediate action.

The need for ESP has increased in the era of big data – because not only do fast-moving, large volumes of varied data need to be processed quickly, but insights must be derived in real time. While ESP can be applied to many sources of big data, one source commanding attention these days is the data streaming from the Internet of Things (IoT) – the growing network of things connected to the Internet, ranging from industrial machines to a wide variety of consumer goods.

How big is the IoT?

By 2008, there were more Internet-connected things on Earth than humans. Estimates show that there will be more than 29 billion IoT devices by 2030. These connections are enabled by sensors and communication networks, often wireless. Across those connections flows real-time data that can be put to many uses.

The early adopters of IoT focused on inventory control and supply chain management systems for uniquely identifying, inventorying and tracking products in real time. But according to The Economist, 75% of global business leaders are now exploring the economic opportunities of IoT to create new markets or improve existing services. GE estimates that IoT has the potential to add $10-$15 trillion to global GDP over the next 20 years, essentially doubling the US economy.

While the potential of IoT is hard to ignore, it’s also important to realize it’s not as simple as slapping Internet-connected sensors on everything. I could put an Internet-connected sensor, for example, on my 1999 Mercury Mystique, but it wouldn’t allow me to do much more than use a GPS tracking smart phone app to help me find where I parked. Although that’s somewhat useful, and already available, much more useful Internet-connected things will require new products to be manufactured, with embedded – and more sophisticated – sensors, as well as computing systems specifically designed to make use of the IoT data.

Combining forces: IoT and ESP

As IoT technology advances, more “smart things” will continue to enter the marketplace. Internet-connected cars are an excellent example of the potential of IoT. Currently, most cars are serviced by periodically scheduling maintenance whether it’s needed or not (e.g., getting your oil changed every 3 months or 3,000 miles). Instead of scheduling periodic maintenance to find out if something is wrong, or else waiting for something to break down, the data streaming from the sensors in Internet-connected cars can be used to alert their owners when something specific is detected (e.g., high engine temperature, low oil level, slow brake response, decreasing tire pressure) so appropriate action can be taken.

The transportation industry has been an early investor in this technology since its business operations are disrupted when delivery vehicles break down. Internet-connected delivery vehicles can trigger maintenance to prevent breakdowns – or at least greatly minimize them – and keep business operations running smoothly. Even though only 10% of the cars on US roads were connected to the Internet in 2012, estimates show that connected cars will comprise 95% of all vehicles sold globally by 2030.

Governments are also exploring the benefits of IoT. Intel predicts city governments will spend $41 trillion in the next 20 years on infrastructure upgrades needed for IoT. For example, after a 2011 earthquake killed 185 people and severely damaged Christchurch, New Zealand (the country’s second-largest city), recovery efforts have included integrating a network of Internet-connected sensors into the physical infrastructure of the city as it is rebuilt. This is transforming Christchurch into a sensing city, where the data streaming from Internet-connected sensors are constantly monitored for air pollution, water use, traffic conditions and other quality-of-life aspects.

It will take some time for IoT to reach its full potential. But once it advances on a wider and deeper scale, the event streams emanating from IoT will transform data into something far more dynamic than ever before, giving us extrasensory perception to detect what is beyond our current analytic capabilities. Already, ESP and IoT have combined forces to help us automatically sense disturbances in data (in other words, detect anomalous patterns in the data streaming from Internet-connected sensors), so that we can take immediate action before these data disturbances disrupt business operations or negatively affect customer experiences.

Jim Harris is a recognized data quality thought leader with 20 years of enterprise data management industry experience. Jim is an independent consultant, speaker, and freelance writer. Jim is the Blogger-in-Chief at Obsessive-Compulsive Data Quality, an independent blog offering a vendor-neutral perspective on data quality. Jim is the host of the popular podcast OCDQ Radio, and is very active on Twitter, where you can follow him @ocdqblog.