We are creating some awesome events for you. Kindly bear with us.

EXCLUSIVE – Building a scalable data analytics platform for government

OpenGov Asia had a chance to speak with Dr Ole Nielsen again. This time, Dr Nielsen dons a new role – Director of Data and Analytics for the Chief Minister, Treasury and Economic Development Directorate in the ACT Government. Previously, he held the title of Director of Scientific Computing and Systems Engineering, Geoscience Australia.

Speaking to Dr Nielsen, his knowledge and wealth of experiences are evident. More importantly, Dr Nielsen is passionate about data. To this end, he is willing to explore new frontiers.

Agility in Government

This year, his team established the Australian Capital Territory Data Analytics Centre. The Centre aims to build data literacy across the ACT Government. With better data management and analytics capabilities, the government can ultimately deliver better policies and services for citizens. A team of eleven data scientists, analysts and engineers spearhead the Centre’s projects.

In the Centre, agility is key. Dr Nielsen’s team works in bursts of twelve weeks for any given initiative – one of the first of its kind in the ACT Government. The Centre takes on problem statements from other directorates in the ACT Government. If too little has been achieved by the end of the cycle for whatever reason, the governing body can choose to stop and ask the Centre to work on something else.

“We may not finish, but we will have something of value within that twelve weeks – or we may have uncovered issues e.g. with data access that prevents and initiative from progressing” explains Dr Nielsen.

Periods of intensive focus like these hold immense value. Realtime monitoring of water quality in rivers and lakes, payroll tax compliance, mapping businesses by type and location to inform potential investors, and even applications to promote the safety of victims to domestic violence, are just a few of the initiatives being tackled by the agile mindset.

Data Analytics Project for Government

Of these initiatives, the one he’s most excited about, is the enterprise grade data analytics platform his team developed in an impressive sixteen weeks based on learnings from a minimal viable product which took much longer. The cloud-based platform serves to promote better use of data by providing access and sharing of data between directorates in the ACT Government. At present, government data in Australia is neither stored centrally nor in a uniform manner. Every organisation houses their own data and it is stored either on paper or digitally in a range of systems. Additionally, under the traditional arrangement, security concerns arise when access to a system is granted to a third party. What could be an innocent act of extracting data could result in breaching the system.

“Traditionally, what we’ve seen is silo mindsets. Everyone works really well, but they’re not working together… and that does not always lead to the [desired] outcome,” Dr Nielsen explains. “This platform allows each agency [in the ACT Government] to have their own tenancy, their own area in this data platform, where they can ingest data from their own systems and make it available for sharing with other parts of the government, in a controlled, privacy-centric way. That way data can be shared without allowing third parties access to individual systems. This is a subtle distinction – access to the source system is very different from access to the data”

A DevOps team and agility were key to delivering a successful outcome. The extended team worked together towards a common goal, they automated pretty much everything and every two weeks, the team evaluated the project’s progress and assessed room for improvement. Consequently, this seriously slashed what could have been a sixteen-month project into sixteen weeks.

It’s a People Problem, Not Data

Despite the success of such models, there was some scepticism and not everyone were keen to jump on the bandwagon. After all, this model was novel and unfamiliar. People want detailed plans on how and when the outcome will be delivered – something the team’s modus operandi did not provide. To help getting buy-in for the vision and build trust, bouts of negotiation and relationship management were required.

“It’s difficult to change the status quo,” he said. “People are not afraid of change… people are only afraid of losing something or being blamed for failure.”

For example, taking a gamble on winning a million dollars from the lottery. While this could actually be very dangerous people are not afraid of such a change. On a more serious note, this is Dr Nielsen’s point: People have to take risks from time to time to achieve outcomes in a rapidly changing, ambiguous and complex environment. However, traditionally there is widespread fear of being seen to fail so deviations are often frowned upon. One wrong move could mar their reputation. No one wants a poor performance review.

Nevertheless, during the implementation process, the team remained very respectful of the existing ICT processes which helped build the trust that is required. They made it a point to have face-to-face interactions, rather than communicating over an ebb and flow of emails. For Dr Nielsen, this was a game changer. Both teams could understand each other’s point of view with immediacy and roll out a mutually agreed solution. In his words, the collaboration fostered on the ground was the biggest change that was achieved.

Dr Nielsen opines, “It takes time, it takes convincing, and it takes a great deal of compassion.”

What Goes Down Must Come Up

More can be expected from the enterprise grade data analytics platform. The platform will consistently be upgraded to meet everchanging demands. One such demand is a growing volume of data which is why the platform is built to be scalable; Another is the relentless need for new versions to be implemented.

As a real test of the platform’s scalability, the team would tear the platform down entirely and build it back up every day through automation. For the team, its cyclical rebirth, and indeed its ability to do so repeatedly without failure, is a litmus test of confidence in the product. This was inspired by the way Netflix and Australia Post are working, for example.

At this point, Dr Nielsen’s understanding of scalable shocked me. But he assures, “But that is how we need to work in the future.”

Once a change has been made, the automation script keeps a record. A new version can be rebuilt with the change in mind instead of revamping the existing system. Affectionately calling it a software product rather than a project, this process allows for a complete automated recreation of the platform whilst incorporating changes. If ever the system were to go down, an identical platform can be built from scratch in a matter of hours, Because of what automation can afford in terms of velocity, reproducibility and transparency., the successful regeneration process is a great source of motivation for the team. Especially in times of disaster, they are confident in mitigating risks.

How Data Continues to Challenge Government

The data analytics platform is a good instance of how governments can use technology for better services and policies. However, there continues to be challenges in realising that. Dr Nielsen shares what he thinks are some fundamental issues he’s seen.

First are systems which are incompatible with the demands of the times. Data is sitting in a hodgepodge of different systems, with different modes of governance. Accessing the data is difficult and the quality of data is also varied.

Furthermore, legislation which governs data in the past might conflict with what is needed today, evidenced by Australia’s new proposed Data Sharing and Release Bill which aims to set out more legislative clarity to promote better use of data

Third, there is a reluctance to share data within government. Dr Nielsen attributes this to culture. Officials are uncertain about another agency’s data quality, and hence might be misinterpreted. Questions about the ownership of the data linger. Not only do people fear a loss of control, they are also afraid about doing the wrong thing since there is a lack of clarity about what is within reasonable and legitimate bounds.

“There’s a whole range of issues. But they all boil down to uncertainty and fear. We need to have a lot of clarity, psychological safety and transparency around this,” said Dr Nielsen.

The condition is not limited within the realms of the government. Around Australia, Dr Nielsen notes a lack of public trust in government. There might be more trust in government if people saw more transparency and engagement from government organisations.

He says, “We as government need to earn the trust. You can’t tell people to trust you. You earn the trust by being transparent and delivering digital services of value to people.”

Dr Nielsen concludes, “It is a very exciting time to live right now. We have digital technology opening up, where everyone is interconnected, and you can get real-time services which are personalised. Although it is going to be difficult to navigate, it is also very exciting.”

Send this to a friend