We are creating some awesome events for you. Kindly bear with us.

Extracting timely actionable insights from data captured on remote devices

Extracting timely actionable insights from data captured on remote devices

Big data continues to grow bigger every second. The internet-of-things (IOT) or the rapidly expanding network of devices and sensors, from smart devices worn by people to industrial sensors, is a major contributor to that growth. 

OpenGov spoke to Mr. John Kreisa, Vice President for international marketing at Hortonworks about capturing and transmitting data securely and analysing it to generate actionable insights. He shared several big data use cases from the energy sector and government.

Founded in 2011, Hortonworks is a leading provider of big data platforms for capturing, storing, processing and analysing data. Mr. Kreisa said, “Our core DNA is all around big data technologies and bringing big data technologies to the market.” Hortonworks has two platforms. One is based on an open source project called Apache Hadoop, which is used for storing and analysing data at massive scale.

The second platform, Hortonworks DataFlow (HDF) is for capturing data in motion, data from remote locations, from sensors and devices. To bring that data safely and securely back to the core platform for processing and analysing. Hortonworks develops those technologies in open source and provides services, support and training around those.

Challenges in transmission of data from IOT devices

Picking up on the second platform mentioned by Mr. Kreisa, we asked about the challenges faced in transmission of transmission of data from IOT devices.

He replied that the ability to handle large volumes of data and to process the data as it’s in motion, to extract timely actionable insights as it’s coming back is critical.

Mr. Kreisa explained with an example, “If you are talking about energy, it might be some kind of an indicator of a potential problem in an energy transmission network. You have got a sensor out on a tower somewhere and it detects an anomaly. That might come over analysing thousands of signals and looking for a pattern, or it might be a spike. Either way, you need to be able to react to that in an actionable way.”

The second challenge is securing the data, bringing the data back in a manner that nobody else can have access to that data and there is full understanding of provenance and governance of that data.

“So that when it comes back to you, you can say that it was not accessed, it was not changed and if it had to be analysed, you know who did it and what were the circumstances around that,” said Mr. Kreisa.  

The third challenge is the requirement of backward communication from the devices at the edge. With limited connectivity to the Internet, or interrupted network coverage, the ability to communicate back might be a challenge.

So, the remote device that’s capturing data needs to have the ability to understand the connectivity. It needs to have the ability to collect the insights and transmit the data back when the connectivity is available. If bandwidth is low, it should only transmit the most important information back.

The software on the device has to be intelligent enough to do all this. In addition, these devices often have to run at very low power. So, the software itself has to run on a very small footprint.

This open source technology that Hotonworks runs is based on an Apache project called Apache NiFi. It handles the transmisison. Mr. Kreisa said, “Apache NiFi runs on a small footprint, it has the smarts. But we have created an even smaller footprint called MiNiFi. It can run on devices as small as a handheld.”

Smart meters and big data in the energy sector

We moved on to discuss a very important use case in the form of smart meters. Smart meters are a critical component of energy saving strategies in many countries around the world today.

A typical regular meter would take one reading a month, saying you have consumed this many kilowatt hours (kWh) of energy this month. Smart meters could be reading data every few minutes.

“And you are not just measuring the power consumption. You are also looking at things like the quality of the power, what were the high and low pieces of consumption within that 10-minute period. You are transmitting not just one, but 6 or 7 pieces of data at much higher frequency,” Mr. Kreisa said trying to provide an idea of the volumes and velocity of data involved.  

Government, or private industry need massive scale-up of ability to analyse and store that historical data. But they can use it to get better insights into power consumption. They can do a better job of designing power generation plants. Government which regulates power generation plants and sometimes runs them, can do a better job of projecting and predicting power consumption.

Mr. Kreisa talked about an interesting London-based company called Open Energi. They have what they call a virtual power station. They work both with the UK government and the National Power Grid. They are connected to major industrial and commercial power consumers. They put smart devices on those consumption mechanisms and they monitor power consumption. As consumption starts to get close to power generation capacity they can go and remotely turn those motors and pumps off for a little bit of time, which has no impact on the business. Suppose you have a refrigeration unit. They can turn that motor off for 5 minutes without affecting the temperature of the refrigerator at all.

Mr. Kreisa added, “As consumption comes up, there’s no need to create more power. They can actually sort of slow the consumption curve on the major industrial consumers of energy. The energy providers don’t have to fire up a coal plant to try to meet a spike in demand.”  

The dynamic demand technology automatically and invisibly adjusts power consumption to help manage fluctuations in electricity supply and demand. And it is driven by big data.

Mr. Kreisa shared another use case of customer-oriented application of big data in the energy sector. Centrica, the largest supplier of gas to domestic customers in the UK, uses platforms from Hortonworks in two different ways. One is to drive operational efficiencies across the company.

Then they have also created a customer application, so that when their technicians go to a customer, they have a complete view of all of the interactions with the customers, including payment and complaints. It helps the technician to do a better job when they go on a service call.

In addition, they have created a consumer dashboard where the consumer can see at a fine level of detail how much power they are consuming. They found was that by giving the consumer more visibility into the power consumption, they naturally reacted to reduce the consumption. “They see the spikes and lows, they know that they might be using power at a more expensive time. It might be better if I run the washing machine late at night,” Mr. Kreisa said.

And all this is enabled because they can capture it in a big data platform. Centrica is able to define correlations between data sets across the different business units that were previously isolated and identify new data patterns to provide a better customer experience.

Use cases abound across public sector, from defence/ intelligence to transportation/ healthcare

Mr. Kreisa said that nearly every branch of government has the opportunity to exploit big data. He talked about working with Metro Transit of St. Louis, which operates the public transportation system for the St. Louis metropolitan region in the USA. There the challenge was to provide more safe, reliable and accessible transport, while reducing costs. The two objectives were somewhat at odds with each other.

A lack of ongoing, reliable machine data from its buses forced a trade-off between the two. Mr. Kreisa elaborated, “Based on say every 10,000 miles, you would replace the brakes, regardless of how much they had been used. If one part broke in one bus, they might replace it in all the buses. Because their one initiative was safer, more efficient network.” But that would lead to rising costs.

With the platform, they were able to capture all the data from the buses and do predictive and proactive maintenance. They were replacing the parts only on those buses which fit a predictable pattern where those particular parts might fail. Thus, the government was able to reduce the operational costs and making the network more efficient.

Mr. Kreisa shared another use case from Barcelona, Spain. The government has a bike sharing program, where they were facing challenges in placing the bikes and incurring heavy costs in moving them. The government used Hadoop technology to capture bike usage patterns and monitor the entire network of bike share to improve efficiency of the sharing network and increase ridership.

Mr. Kreisa said that the initial use cases for government, and for industry in general are around renovating the existing architecture and driving efficiencies. Usage could include keeping more data online for longer as active archive or combining existing data from multiple sources.

Then they use the cost savings to go into some of the more innovative things, such as predictive applications.  They could also provide data back to citizens (open data) so that they might use it and build on it. 

Ultimately, it is about the government using the data smartly to improve the lives of citizens. 


Qlik’s vision is a data-literate world, where everyone can use data and analytics to improve decision-making and solve their most challenging problems. A private company, Qlik offers real-time data integration and analytics solutions, powered by Qlik Cloud, to close the gaps between data, insights and action. By transforming data into Active Intelligence, businesses can drive better decisions, improve revenue and profitability, and optimize customer relationships. Qlik serves more than 38,000 active customers in over 100 countries.


CTC Global Singapore, a premier end-to-end IT solutions provider, is a fully owned subsidiary of ITOCHU Techno-Solutions Corporation (CTC) and ITOCHU Corporation.

Since 1972, CTC has established itself as one of the country’s top IT solutions providers. With 50 years of experience, headed by an experienced management team and staffed by over 200 qualified IT professionals, we support organizations with integrated IT solutions expertise in Autonomous IT, Cyber Security, Digital Transformation, Enterprise Cloud Infrastructure, Workplace Modernization and Professional Services.

Well-known for our strengths in system integration and consultation, CTC Global proves to be the preferred IT outsourcing destination for organizations all over Singapore today.


Planview has one mission: to build the future of connected work. Our solutions enable organizations to connect the business from ideas to impact, empowering companies to accelerate the achievement of what matters most. Planview’s full spectrum of Portfolio Management and Work Management solutions creates an organizational focus on the strategic outcomes that matter and empowers teams to deliver their best work, no matter how they work. The comprehensive Planview platform and enterprise success model enables customers to deliver innovative, competitive products, services, and customer experiences. Headquartered in Austin, Texas, with locations around the world, Planview has more than 1,300 employees supporting 4,500 customers and 2.6 million users worldwide. For more information, visit www.planview.com.


SIRIM is a premier industrial research and technology organisation in Malaysia, wholly-owned by the Minister​ of Finance Incorporated. With over forty years of experience and expertise, SIRIM is mandated as the machinery for research and technology development, and the national champion of quality. SIRIM has always played a major role in the development of the country’s private sector. By tapping into our expertise and knowledge base, we focus on developing new technologies and improvements in the manufacturing, technology and services sectors. We nurture Small Medium Enterprises (SME) growth with solutions for technology penetration and upgrading, making it an ideal technology partner for SMEs.


HashiCorp provides infrastructure automation software for multi-cloud environments, enabling enterprises to unlock a common cloud operating model to provision, secure, connect, and run any application on any infrastructure. HashiCorp tools allow organizations to deliver applications faster by helping enterprises transition from manual processes and ITIL practices to self-service automation and DevOps practices. 


IBM is a leading global hybrid cloud and AI, and business services provider. We help clients in more than 175 countries capitalize on insights from their data, streamline business processes, reduce costs and gain the competitive edge in their industries. Nearly 3,000 government and corporate entities in critical infrastructure areas such as financial services, telecommunications and healthcare rely on IBM’s hybrid cloud platform and Red Hat OpenShift to affect their digital transformations quickly, efficiently and securely. IBM’s breakthrough innovations in AI, quantum computing, industry-specific cloud solutions and business services deliver open and flexible options to our clients. All of this is backed by IBM’s legendary commitment to trust, transparency, responsibility, inclusivity and service.

Send this to a friend