Dr Jim Webber had just returned home from dropping his child at school when he found that a rear window had been smashed and that a startled burglar had fled back through it. Fortunately, a neighbour was able to jot down the thief’s number plate as he was speeding off and it was reported to the UK police.
The authorities are able to make use of a POLE (people-object-location-event) database in responding to such crimes. Dr Webber happens to be the chief scientist for the Graph Data Platform management company Neo4j used widely in government and law enforcement. “I believe the police response was partly guided by my software to track the criminals on the A3 road heading into London. They were able to apprehend the gang and return the stuff they had stolen from me and others.”
This is one of Dr Webber’s favourite stories to tell when asked about the functionality of graph data. It’s both personal and social.
The graph platform is essentially a collection of software systems that help people to understand data that is represented in a graph. “And by graph, we mean the mathematical thing, your edges and vertices if you remember that in college,” Jim, the Chief Scientist at Neo4j and visiting professor at Newcastle University, explains. “In short, it’s a system that stores and retrieves connected data fast. Very fast.”
Unlike other types of databases, Neo4j connects data as it’s stored, enabling queries never before imagined, at speeds never previously possible. Users are able to build data models that are circles, connected by arrows, and the arrows are assigned names, and each can produce very rich data models.
Elaborating Jim says, “We know in a relational model we store data in relations or rows. And if we’re being sophisticated, we can join rows across tables in some very clever ways.” But a Graph Data Platform is actually entity relationships that mirror how software people think of the world. In a graph, entities can be established that are related to each other in sometimes very rich and complex ways and sometimes very simple ways. Jim explained, “So we’re able to build these data models that are circles, connected by arrows, and the arrows have names, and we can produce really rich data models. So we can do all of these rich things because the data is smart. Data in traditional data structures are really ‘un-smart.’”
He said, for example, in a column-store, there is an implicit association of columns that is constraining. And a document-store contains all the data that kind of belongs together by virtue of being in the same document; but apart from that there is not much else.
The database is at the centre of the Neo4j platform. For example, it has systems for visualising and exploring graphs in human terms. It can run sophisticated algorithms to query who is the most popular person in the graph? While in the aggregate their database platform is probably no different than all the mature databases, the difference is that they process graphs rather than other data models.
THE CASE TO START THE GRAPH JOURNEY
Graph tech is still considered to be in the early stage of its evolution as many people have not used it or maybe have not even heard of it yet. As the category, founder Neo4j is early in the cycle. And although it feels late for Jim, as he has been plugging away at it for more than a decade, most people are just coming into this.
And Jim also has the perfect book for a graph neophyte. It is a book he authored titled “Graph Data Platforms for Dummies.” He said it is a very short book that, in one evening of humane reading, can explain everything one needs to know to understand the basics of graphs. The book is available for free on the Neo4j website.
Jim is quick to point out that the book should be seen as a snapshot. If there is a need to quickly understand this graph arena that seems to be a macro-trend in the industry, a CIO can quickly and easily browse through the book and understand the thinking and the use cases.
For anyone who may be sitting on the fence, Jim has a challenge, “What I ask of you is give me an afternoon of your time. Spend one afternoon working with Neo4j with a good part of your problem, with something meaty and useful, and if you can’t make progress in one afternoon, Neo4j and graphs are not for you. But if you can make progress in one afternoon, it will illuminate it for you. You’ll find out it’s not hard or mysterious. We designed this system to be explicitly humane, not arcane.”
There is a plethora of learning material available and a lot of information on government use cases, including Neo4j’s white papers. Yet, interestingly, graphs have frequently made their way into businesses and governments serendipitously. Keen engineers or big System Integrators (that the agencies often use), who are experienced with graphs, may have brought it to their assigned work and that’s how it made its way into the organisation.
The adoption of graphs will likely take more time, Jim conceded, especially among government agencies with well-established systems of record who are disinclined to immediate change. This reticence is understandable, as the public sector has systems of record that have been working relatively-well for them. So, it would make perfect sense for these agencies to be able to understand the associativity before commencing such migration or adoption.
Typical citizen information across diverse bands, such as tax, health and criminal records, are a great example. These cannot be clubbed together ad hoc into one big database – it just would not be workable. But what can be done is to put an “index” on top of all the information. Called a knowledge graph, this harvests all of the diverse information into a single organised graph. With it, a very sophisticated understanding of the needs of citizens can be created.
Another advantage of having a knowledge graph layered on top is that existing systems do not get disturbed. All sophisticated queries can be done at the knowledge graph level. If the physical row or the physical document from the underlying store is needed, it can be pulled up without disturbing or changing the original data.
In sum, the knowledge graph can collate information across sectors to address complicated queries. A government agency can then more easily conduct an evaluation: Do tax returns match the benefits being claimed?” Do a person’s criminal records allow them to be in a specific kind of job? Such graphs can also be deployed at a population level: What is the level of fraud in a tax system versus the benefits of the system?” Large scale, macro depictions are vital to intelligent, data-driven governance. This sort of information can be collected to help inform agency or government policy.
WHAT THEN IS HOLDING GRAPH BACK?
The biggest barrier to graph dominance is that there is already well-known technology out there. There is a lot of data technology, particularly in the relational world, that is very mature, that people know how to run and have been operating for many years. When agencies and CIOs already have a working solution, why would they take a chance on the graph?
The challenge for graphs in converting CIO’s is to make the impetus to adopt very compellingly. If the benefits are not tenfold or more, a CIO will rightly not even spend a few minutes on it. Further, other systems, while they may be incredibly expensive, essentially provide most government departments, another one or ten licenses basically for no charge because of existing agreements.
Intellectually a CIO could understand that graphs might be better here, but it involves learning new tech, bringing onboard a new supplier and negotiating another licensing agreement for substantial money. While the existing vendor may not be optimal, it can be sort of “shoehorned” in, and it’s free.
That’s not to say there aren’t opportunities for graphs to gain a foothold. When one takes equivalent queries from the relational world and ports them to graph, there is a paradigm-shifting minutes-to-millisecond experience. Something that might take several minutes in the relational world may only take several milliseconds in the graph world. For some people that is so compelling that they will make the jump.
For sectors like policing and health services, minutes are too long. If a fugitive is coming through an airport, waiting 30 minutes for a report to run can thwart a chance to detain them before they disappear into the general population. Similarly, in the medical world, information may be required to help make an immediate diagnosis. It can’t wait several minutes while the patient is in pain or is in a life-threatening condition.
Even things such as product recommendations are much better with graphs. For the digital-native client in today’s world looking to buy something on a web app or a phone app, they may wait for half a second and then they will move on.
Government agencies are challenged with complex, ever-evolving problems every day. While the answers exist somewhere in a vast amount of data, they are only identifiable with the right technology to make sense of the interconnectedness within the data. For proponents like Dr Jim Webber, that technology is a Graph Data Platform.
