OpenGov had the opportunity to interview Dan Rothman, Chief Technology Officer at the City of Boston. He talked about providing and maintaining all the enterprise-wide IT infrastructure for the City and discussed the consolidation of data centres and sharing infrastructure with communities surrounding Boston. He explained how the city produced near limitless bandwidth through the Boston Optical Fibre Network (BoNET), with very limited capital expenditure.
Can you tell us about your role at the City of Boston?
I am in the Department of Innovation and Technology (DOIT), which is the enterprise IT organisation for the City of Boston. Anything that is enterprise-wide is done within our group. The overall infrastructure that supports daily communications and computing needs is supported within DOIT. This includes IT security and network, data centre operations, mainframe operations, service management, telecommunications, radio networks, video networks.
All applications, infrastructure, that are not agency-specific, are supported through the DOIT department. For instance, the enterprise financial system, the enterprise system for hiring and payroll, the CRM systems which are built into the multiple agencies, those things are done through us. Those are permanent systems with broad reach. There are other silos of IT that are specific to the agencies.
Are there instances where something that should be enterprise-wide is not?
Sure, there are instances. For example, I really think that video surveillance should be managed on an enterprise-wide basis. There should be a common set of cameras that provide multiple agencies the video infrastructure they need.
But different agencies own their own infrastructure. Our transportation division, the police department, the school system, they all have their own infrastructure. We make them interoperable and we work on sharing them. But the reality is that this stuff should be a common resource that should be deployed for multiple uses.
So, there is room for improvement in that. We have been trying for multiple users to try to get their budgets transferred into a common video infrastructure budget. But that hasn’t happened yet.
Initially there were 7 different video management systems (VMS) were being used within the city. We achieved some consolidation and we got it down to 3 systems. Then we put a system on top of that, that allowed sort of a one-stop shopping access to all the systems. It didn’t really work out very well.
We ended up trying to make it better by making a separate network architecture available for video. Maybe we could have some common architecture to make it simpler. We also built up capability within the city for other agencies to adopt an existing VMS and to scale it up. We leveraged off inexpensive storage. If they adopted this, it would be cheaper than building their own infrastructure and storage.
We were also able to set some standards. We made sure we had the capacity to get everyone into compliance. We still have 3. But we got pretty much all the outliers to combine with the one system. That same system is shared by the state government, the state transportation authority, and by a bunch of state agencies.
There are other examples. There are often outliers who have not adopted a city-wide deployed system.
The complexity of city government is such that there are lots of different kinds of agencies. Some of them are purely under city, some are hybrids, where they are quasi-independent agencies. Maybe they are not reporting directly to the mayor but to some other type of government body. So, we don’t necessarily always have a mandate to force change a lot of times. We have to cajole, we have to use a carrot and stick approach.
So sometimes we will make a facility available. We will get participation in the process from a group. We will try to get other silos of IT to participate. It’s never 100% successful. But even if we get 90% of city agencies to adopt something, it’s better than nothing.
Can you tell us about the origins and use of Boston Fiber Optic Network (BoNET)[1]?
We were dependent upon telecoms for data services. Agencies had T1s or T3s, lines that offered sufficient bandwidth but very expensive and not terribly scalable.
Around 7 years ago, the City of Boston was able to get Comcast to give them dark fibre[2] in lieu of some legal mandate for shadow conduit[3] in a construction. Legal permitting process mandated that a certain shadow conduit for our city had to be put in place, any time trenches were dug within the city.
In lieu of them doing that conduit, we said give us x number of strands of fibre in these locations. That allowed us to get this fibre network for free. Initially its use was focused on public safety and the main hub for city operations, providing connectivity to few key large buildings and the public safety infrastructure.
Basically, with that we could push as much data as we wanted. We initially threw out 1 GB, with a 2 GB backhaul. We expanded over the years. Now we are up to a 100 GB backbone, at some locations we are pushing 10 GB at the edge.
With very limited capital expenditure, we were able to produce near limitless bandwidth for the people within the city. It let us do a lot of things that wouldn’t be practical otherwise. In most cases, if you have a 1000 video cameras, you don’t want to have those going across significant lengths, because video is a monster. But having this infrastructure, we can consolidate that video.
It also lets us sort of become the Internet provider for all the city employees, for libraries, for schools, because we are able to throw as much bandwidth as we need across that network, to the end-point.
You were talking about consolidation of data centres (in a pre-interview chat). What is being done in that area?
Initially we had 8 data centres within the City of Boston footprint. These were not purpose-built data centres. They tend not to meet the highest tiers for redundancy and resilience, because they were just ordinary city buildings converted to that use.
So, we have been doing a couple of things. One, we have been trying to migrate city infrastructure into purpose built data centres, managed professionally and meeting higher standards. So, initially we moved our production environments into premier data centre space in the city, which were way better than our environment.
Our main data centre for city hall is in the floodplain, below sea level. There were concerns around that. We would probably need a multi-million Dollar investment to upgrade the environmental controls to meet our future standards, energy and many other things.
So, the next step was to do a Request for Proposal (RFP) for data centre space outside of the city’s limits, to give ourselves some geographical diversity and be able to avert concerns of regional disasters. We ended up building space within another commercial data centre, around 240 miles (386 km). That was far enough to provide geographical diversity, in terms of weather issues like hurricanes. It was also far enough to be on separate power grids. That helps improve resilience.
Once we had built that, all the other city’s data centres wanted to have presence outside of physical footprint for resilience. So, we made that available. We have been in the process of building it out, the second data centre and making space available to other agencies to collapse those 8 into 2. But it’s not a mandate. We make it available, they can adopt it or not.
How are you preparing for the future?
I don’t think we are not going to be limited by bandwidth in the
foreseeable future, at least for a decade or two.
There might be a paradigm shift in technology. But right now, technologies such as small cell are all dependent on the fibre of our backhaul. They still need to get to that fibre connectivity at some point. At the moment, a lot of the wireless technology is very vulnerable, it’s fragile. So, having that fibre pathway, which we have in place, is going to be critical for the future because there is no resilient, non-fibre based solution at this point.
In dealing with IT infrastructure, what are the primary cybersecurity concerns you face?
Today we have systems which are not traditional IT systems and the people installing them are not IT companies. The guy installing security cameras may know enough to put an IP address in there and configure it. But he doesn’t know enough to properly turn off the ports and protocols that are unnecessary. He may for convenience, leave the default passwords, so that any of his customers can get into the camera easily and tweak it. He is not an IT guy and he doesn’t know that it is bad practice.
There are similar issues with say an intelligent air conditioning system, where you are collecting power consumption data and using that to fine tune the air conditioning to reduce the power consumption.
As more and more things become a part of the Internet of Things, the surface area expands. They are not properly secured. And some of them may not allow you to properly secure them at all. Typically, they don’t get patched on a monthly basis, like a computer operating system. At best, it might happen once or twice a year.
If someone gets into these devices, they can then use that to get into other things. It’s a challenge and that means that you have got that much more of a surface area to cover. And all the traditional tools used to manage the IT environment do not have reach into these IoT services.
You need specific expertise and knowledge and you usually have to go the extra mile to understand how can they secured and what you need to do for security. Someone has to audit, see what the passwords of the cameras are.
Does the ICT infrastructure of the City of Boston have connections with the ICT infrastructure of the state, or other cities?
We have specific state agencies that we bridge to. Some of the buildings that we are in, are state buildings.
MBTA (Massachusetts Bay Transportation Authority) runs all the transport infrastructure in the Greater Boston area. We have inter-connectivity with them for camera sharing. We also have some connectivity to state police and some other agencies like that.
We also have connectivity with the surrounding cities. We got federal funds to interconnect the fibre-optic infrastructure of the communities surrounding Boston with ours.
There’s a federal program, E-Rate which subsidises telecommunications and Internet access for schools, libraries. We compete with the telcos, with Verizon, AT&T. and we bid against them to provide services to the schools and libraries in Boston and then we get federal reimbursement. Now we have extended that out to the surrounding communities. We are providing e-ratable ISP services and managed security services to the surrounding communities at very low rate.
We are almost giving to them for the cost of the federal reimbursement. It gives them not only the data services but some security services too, which the communities don’t have the budget for. They benefit from infrastructure investment, in security features like next-generation firewalls.
The federal reimbursement subsidises our network infrastructure. Also, this helps in our negotiations for funds with the federal government.
Boston is the 23rd biggest city in the country. But the metro region is the 10th biggest. As a group, we are much bigger, which makes it easier to get funds from the fed. We are leveraging infrastructure and spreading the funds across a larger area. And we are helping out.
[1]Mr. Rothman talked about BoNET in his presentation at the Singapore OpenGov Leadership Forum.
[2] Dark fibre is optical fibre infrastructure that is not in use. Much of the cost of installing cables comes from the civil engineering work required. Hence, the cable owners usually plan for, and install, significantly more fibre than is needed for current demand, to provide for future expansion and provide for network redundancy.
[3] In Boston, the "shadow conduit" policy demands that the first company to dig should ask other companies of their potential needs so that shadow conduits can be reserved for future users.
