Government and enterprises, in an unprecedented period in history, have been compelled to accelerate and bring forward their digital transformation strategies. The pandemic has vaulted the governments and businesses headfirst into the next stage of digital transformation and online services.

Personalisation, efficiency and effective services are only possible with a comprehensive, 360^o view of citizens and customers. This understanding is built on and powered by data. Indeed, there is a good case to be made that digital transformation is likely to fall short unless it is based on a solid foundation of “data transformation.”

To become a truly data-driven organisation that operates in real-time, agencies must deploy multiple modernisation initiatives, including application modernisation, artificial intelligence, machine learning, cloud, edge computing and analytics.

However, modern applications no longer live in isolation. They are built on microservices and rely on other services to move data between applications. Nowadays, modern distributed applications comprise hundreds of thousands of remote services operating in multiple tiers, all of which are loosely coupled and need to exchange data with each other. In such distributed architectures, reliable data and message distribution are key.

Government missions are no longer static, but continuously in motion, handling the “now.” To be effective, agencies need to continually innovate to provide seamless and easy-to-use services and improve the resilience and security of digital platforms for citizens and other stakeholders. Agencies need a complete streaming data platform that treats data in motion, whether it is driving action or producing real-time analysis. Such a platform must be capable of supporting workloads anywhere, anytime across the globe – whether it is in the cloud, on-prem or at the edge – and connect them all.

Pertinently, stored data is not useful when it is not being queried or used to drive decision-making. Agencies must start thinking about creating a central connective tissue that conducts their data across their organisations and handles their data in motion as it is created and flows to those who need it.

This question then is:  Are data silos hindering innovation in organisations?

The Singapore government has taken bold measures to re-engineer government digital infrastructure to support modern application development. As organisations continue to migrate to the cloud and adopt microservice architectures, the data needs to be constantly streaming.

Agencies need a platform that draws together disparate applications, systems and teams with data being the backbone and making it easier to gain actionable insights. This platform should be able to unlock and repurpose the existing data for countless modern applications and use cases securely and efficiently.

The OpenGovLive! Virtual Breakfast Insight on 19 November 2021 was aimed at imparting knowledge on how public sector agencies can power mission outcomes, better serve citizens, ensure security and compliance, enhance IT efficiency and maximise productivity with a platform built for data in motion.

Harnessing data for responsive citizen engagement

Mohit Sagar, Group Managing Director and Editor-in-Chief, OpenGov Asia, kicked off the session with his opening address.

The world has fundamentally changed and the pandemic has forced paradigm shifts in culture and perspective. Today, citizens are looking for a rich, smooth experience from their government. Against that backdrop, agencies must anticipate what citizens want and deliver it in a timely and seamless manner. For Mohit, interactions must be engaging, intuitive, anticipatory and personalised.

The key to delivering a more engaging citizen experience is knowing how to utilise data to produce insights. “Data can be a cost. But if you use it correctly and effectively, it yields rich returns on investment – actionable insights,” contends Mohit.

However, the challenge is, for the most part in the p[ublic sector, that data sits in silos. This compartmentalised arrangement prevents organisations from having a full view. Data silos, Mohit firmly believes, limit the growth of agencies and hinder the provision of services. Isolated systems not only magnify cyber, risk and compliance challenges, but they also hamper innovation. They produce inaccurate analytics and ultimately, are the root cause of poor citizen service.

To harness data fully, having a full view of data across the organisation is critical. “If you only see part of the problem, you may solve the wrong thing,” he claims.

Government agencies must keep up with the pressures of citizen demands and to do so, data needs to be simplified, available and accessible, allowing agencies to retrieve it whenever and wherever necessary.

This begs the question: How can data be made more visible and harnessed? How much of the data needs to be real-time and how much of it does not have to be so?

Event streaming can allow governments to deliver faster, more personalised and responsive engagements with citizens and other stakeholders. “Data is in the transition from being static to event-driven, from batch to real-time, from siloed to integrated,” Mohit observes.

In closing, Mohit emphasises that the data decree of our time is: visible, accessible, understandable, linked, trustworthy, interoperable and secure. He implores delegates to partner with organisations that can help them strategise ways to leverage data in motion. Experts can assist them to deliver responsive citizen engagements and can make their digital transformation journey smoother, cost-effective and impactful.

Formulating data strategies in the public sector

During the fireside chat between Mohit and Simon Herbert, Chief Data Officer, New South Wales Department of Customer Service, Simon shared his experience of how the New South Wales government has utilised data.

NSW Department of Customer Service is the first of its kind in Australia to have developed a comprehensive and robust strategy to get a wider picture of citizens and organisations.

For citizens, data was gathered through a sentiment survey conducted about COVID-19 and other critical events in Australia.  About 700 people were interviewed every week across the different demographics to engage and understand what citizens felt about the pandemic as well as the myriad of disasters that beset the nation. The results of the survey were immensely helpful, “directly driving policy decisions”.

To aid businesses, the NSW government set up a business insights hub to comprehend how SMEs were hit by the pandemic. They acquired a wide range of data – accounting, credit, hiring – to analyse. Based on analysis of that data, travel, hospitality and the arts were identified as the worst-hit industries.

Agreeing with Mohit that “data is considered a strategic asset”, Simon shares his thoughts on how data could be used to deliver faster, more personalised and responsive engagements with citizens and other agency stakeholders.

The key to effective engagements, Simon feels,  is to have a holistic citizen-centric view. In the current scheme of things, the view of citizens is segmented by the various agencies providing different services that often overlap. As a result, data is neither complete nor it is shared regularly nor promptly.

Simon shared that the NSW government did face challenges around privacy and identity. While a privacy act restricts the sharing of data, in real terms, there were no unique identifiers to distinguish the individual in Australia. To overcome this, two key programmes were introduced: Life Journeys and Government Made Easy.

Life Journeys tracks and identifies the key elements of the life of a citizen – graduation, marriage etc – while Government Made Easy seeks to reduce the number of times a citizen needs to re-enter their personal details when interfacing with different government services.

A sound data strategy, architecture and SOPs are critical to growth. It is an accepted fact, Simon contends, that to increase resilience, productivity and economic growth, organisations need to use data much more effectively than they are now.

“Everyone knows that cloud is enabling,”. Having a single data lake and warehouse is one of the keys to building a strong data foundation. That requires testing, proof of concepts and being able to accept failure as part of the process.

Simon added that self-service is also vital. Having too much control can stifle innovation. “Metadata must be automated” so that as data gets loaded, it is analysed via machines and made usable.

To sum up, Simon opines that agencies need to understand the use cases of real-time or near real-time data. Having clarity about the data that needs to be in real-time is critical. The key is to utilise technology appropriately in accordance with the needs of organisations.

Utilising Data in Motion to transform organisations

Damien Wong, Vice President, APAC, Confluent, spoke next about the use of data in motion to power digital leadership.

“The world is changing,” Damien opines. “The world has changed for the current generation because technology is shaping how businesses need to respond to these changing expectations. The younger generation have never walked into a bank branch, and likely will never understand why anyone would ever need to do so since everything can be done online today,” he notes.

Business and public service perspectives have been dramatically altered. In the past, technology was
merely a support function. In today’s reality, technology is the business. While innovation was required only for growth in the past, business innovation is a matter of survival today. Running on yesterday’s data might have been sufficient earlier but is considered a failure by today’s standards. For Damien, coping with the digital transformation requires modern real-time data infrastructure.

Most organisations today, are “becoming software.” Ride-hailing, he said, was an excellent example.  Not too long ago, people needed a taxi, they would call a taxi dispatch service, wait for the ride to be confirmed and look out for the vehicle to arrive – there was no information on how long the taxi would take to arrive or the ETA to destinations. Today, all that information is given almost instantaneously on apps.

This transformation is happening everywhere, and it is drastically causing people to rethink their approaches and systems:

• Cloud: Rethinking Data Centres

The cloud has changed how organisations think about data centres and running technical infrastructure. Today, every company is moving to the cloud.

• Machine Learning: Rethinking Decision Making

Machine learning has changed how decisions are being made, and this happens increasingly in an automated manner, driven by software that communicates to other software.

• Mobile: Rethinking User Experience

Mobile devices and internet connectivity have dramatically changed the user experience of how customers interact with organisations and have raised the bar for expectations.

• Data in Motion: Rethinking Data

Event streaming has changed how people think about and how people work with the data that underlies all the other trends.

The traditional use of data at rest is to consolidate data into a warehouse and apply analytics. Data in motion is, on the other hand, understanding the predefined actions that will be taken when encountering a specific event or data stream.

The rise of event streaming can be traced back to 2010 when Apache Kafka was created by the future Confluent founders in Silicon Valley. From there, Kafka began spreading throughout Silicon Valley and across the US West Coast. In 2014, Confluent was created to turn Kafka into an enterprise-ready software stack and cloud offering, after which the adoption of Kafka started to accelerate. Today, tens of thousands of companies across all kinds of industries the world over are using Kafka for event streaming.

“Data in Motion is the central nervous system for today’s enterprises,” Damien contends. “And Apache Kafka is the event streaming technology powering Data in Motion.”

If Kafka is the engine (the core technology), then Confluent is the ready-to-use product around that.

Confluent is a natural candidate for real-time operations like command and control, cyber security and other anomaly detection solutions. It can enable event-driven architecture that helps modernise IT applications and hasten the addition of new citizen services or capabilities. Apart from that, data infrastructure for data in motion, Confluent will help organizations move towards multi- and hybrid- cloud and DR operations.

Damien affirmed the need for organisations to emphasise digital leadership if they want to stay relevant. To achieve digital leadership, Data in Motion is the ultimate key, and Confluent is the trusted partner for setting Data in Motion for organisations.

Interactive Discussion

After the informative presentations, delegates participated in interactive discussions facilitated by polling questions. This activity is designed to provide live-audience interaction, promote engagement, hear real-life experiences, and facilitate discussions that impart professional learning and development for participants.

The first poll asked delegates to rate their organisation’s use of data and data analytic tools for decision-making. Many of them felt that their organisations were doing good, with some tools in place (38%). The remaining votes were split between fair use of data in the decision-making process, although analysis is primarily a manual process (33%) and needing improvements in the tools to analyse (29%).

A delegate pointed out that his agency is working towards the democratisation of data. Currently, only a niche pool of data scientists works on the data. The idea is to have every staff within the organisation fully competent in utilising data.

In that regard, Mohit is convinced that data that is democratised and utilised by more people can result in more effective decision making.

Damien observes that there are areas for improvement. Agencies are thinking about ways to break down silos, getting the “right people get the right data at the right time,” and ensuring that the collaboration is facilitated.

On what their biggest data challenge is within their organisation, most delegates said it was that efforts to manage data are expensive (35%). The next largest segment (30%) were hampered by manually aggregating data to produce BI/reports for executive leadership. The remaining votes were split between being unsure of data quality and integrity (15%), the fact that current tools do not capture and make available the right data (10%) and how adding new data sources is difficult and time-consuming (10%).

Delegates most;y concurred that a significant challenge lies in getting buy-in. While data is being utilised in public services, a delegate remarked, not everyone believes in the need to leverage it. “People do not see the value of data holistically,” he opined.

Damien agreed that there is a need to change the perception of data as a bunch of statistics and to help people see how there can be real-time use cases in daily life and not just in corporate environments.

“Data is not seen as an asset,” Simon acknowledges. “There remains plenty of work in educating and motivating people to see the value of data and recognising data as an asset and an investment. Making data discoverable to raise the awareness to get it seen as an asset.”

Inquiring what delegates would consider as the greatest barrier to integrating more data and analytics into their day-to-day decision-making, most delegates indicated that they consider it limited access to data (29%). Compliance with data security and privacy requirements got 24% of the votes as id the inaccuracy of available data (24%). The remaining respondents indicated that they do not have the right tools to gather appropriate data (14%) or that there were cultural barriers (9%).

Even though Singapore is leading the charge in the field of digital transformation, there are mounting challenges. Mohit posits cultural barriers as an underlying problem. Agreeing with this point of view, a delegate said that communications play a vital role in shifting mindsets and culture – stakeholders need to understand what data can do for the organisation.

On the top driver of modernising applications in delegates’ organisations, responses were split between improving the speed and accuracy of business decisions (48%) and achieving better citizen service/experience (43%).

When asked about the biggest barrier to progress in the data journey of delegates’ organisations, most delegates indicated the disconnect between IT & business (30%) as the greatest barrier. The remaining votes were split between poor quality and availability of data (25%) and non-data-literate workforce (25%) and lack of leadership and commitment (20%).

The sixth question asked delegates to indicate what they felt were their organisations’ greatest strengths in terms of real-time data analytics and backend processing. Most of the delegates felt that deriving meaningful insights through real-time data analytics (41%) is their organisations’ greatest strength. That is followed by the ability to get voluminous data onto big data platform (35%) and the synchronisation of disparate data sources (24%).

In the final poll, an overwhelming majority of the delegates (65%) ranked the maturity of their organisation in processing real-time data as emergent (some processes and knowledge; non-standardised), followed by 25% for limited (ad-hoc, unstructured, uncontrolled or reactive) and 10% for structured (standardised, governance, scale, proactive).

Conclusion

Wrapping up the session, Damien thanked everyone for their enthusiastic participation and the robust discussions.

He appreciate that the session had offered insights on the key priorities of the public sector in Singapore when it came to data in motion and that every organisation is at a different stage and faced varied challenges. Damien reiterated the urgency of providing personalised and engaging citizen services in a rapidly changing world. With the right tools and the right partnership, better, faster and smoother citizen experiences can be created. It is clear that agencies will have to navigate through the lack of understanding towards data in motion with internal and external stakeholders.

Before bringing the session to an end, he encouraged the delegates to connect with him and the team if they would like to explore ways Confluent can help and support agencies in transforming their organisation.