Search
Close this search box.

We are creating some awesome events for you. Kindly bear with us.

EXCLUSIVE – Managing enterprise-wide IT infrastructure for the City of Boston

EXCLUSIVE - Managing enterprise-wide IT infrastructure for the City of Boston

OpenGov had the opportunity to interview Dan Rothman, Chief Technology Officer at the City of Boston. He talked about providing and maintaining all the enterprise-wide IT infrastructure for the City and discussed the consolidation of data centres and sharing infrastructure with communities surrounding Boston. He explained how the city produced near limitless bandwidth through the Boston Optical Fibre Network (BoNET), with very limited capital expenditure.  

Can you tell us about your role at the City of Boston?

I am in the Department of Innovation and Technology (DOIT), which is the enterprise IT organisation for the City of Boston. Anything that is enterprise-wide is done within our group. The overall infrastructure that supports daily communications and computing needs is supported within DOIT. This includes IT security and network, data centre operations, mainframe operations, service management, telecommunications, radio networks, video networks.

All applications, infrastructure, that are not agency-specific, are supported through the DOIT department. For instance, the enterprise financial system, the enterprise system for hiring and payroll, the CRM systems which are built into the multiple agencies, those things are done through us. Those are permanent systems with broad reach. There are other silos of IT that are specific to the agencies.

Are there instances where something that should be enterprise-wide is not?

Sure, there are instances. For example, I really think that video surveillance should be managed on an enterprise-wide basis. There should be a common set of cameras that provide multiple agencies the video infrastructure they need.

But different agencies own their own infrastructure. Our transportation division, the police department, the school system, they all have their own infrastructure. We make them interoperable and we work on sharing them. But the reality is that this stuff should be a common resource that should be deployed for multiple uses.

So, there is room for improvement in that. We have been trying for multiple users to try to get their budgets transferred into a common video infrastructure budget. But that hasn’t happened yet.

Initially there were 7 different video management systems (VMS) were being used within the city. We achieved some consolidation and we got it down to 3 systems. Then we put a system on top of that, that allowed sort of a one-stop shopping access to all the systems. It didn’t really work out very well.

We ended up trying to make it better by making a separate network architecture available for video. Maybe we could have some common architecture to make it simpler. We also built up capability within the city for other agencies to adopt an existing VMS and to scale it up. We leveraged off inexpensive storage. If they adopted this, it would be cheaper than building their own infrastructure and storage.

We were also able to set some standards. We made sure we had the capacity to get everyone into compliance. We still have 3. But we got pretty much all the outliers to combine with the one system. That same system is shared by the state government, the state transportation authority, and by a bunch of state agencies.

There are other examples. There are often outliers who have not adopted a city-wide deployed system.

The complexity of city government is such that there are lots of different kinds of agencies. Some of them are purely under city, some are hybrids, where they are quasi-independent agencies. Maybe they are not reporting directly to the mayor but to some other type of government body. So, we don’t necessarily always have a mandate to force change a lot of times. We have to cajole, we have to use a carrot and stick approach.

So sometimes we will make a facility available. We will get participation in the process from a group. We will try to get other silos of IT to participate. It’s never 100% successful. But even if we get 90% of city agencies to adopt something, it’s better than nothing.

Can you tell us about the origins and use of Boston Fiber Optic Network (BoNET)[1]?

We were dependent upon telecoms for data services. Agencies had T1s or T3s, lines that offered sufficient bandwidth but very expensive and not terribly scalable.

Around 7 years ago, the City of Boston was able to get Comcast to give them dark fibre[2] in lieu of some legal mandate for shadow conduit[3] in a construction. Legal permitting process mandated that a certain shadow conduit for our city had to be put in place, any time trenches were dug within the city.

In lieu of them doing that conduit, we said give us x number of strands of fibre in these locations. That allowed us to get this fibre network for free. Initially its use was focused on public safety and the main hub for city operations, providing connectivity to few key large buildings and the public safety infrastructure.

Basically, with that we could push as much data as we wanted. We initially threw out 1 GB, with a 2 GB backhaul. We expanded over the years. Now we are up to a 100 GB backbone, at some locations we are pushing 10 GB at the edge.

With very limited capital expenditure, we were able to produce near limitless bandwidth for the people within the city. It let us do a lot of things that wouldn’t be practical otherwise. In most cases, if you have a 1000 video cameras, you don’t want to have those going across significant lengths, because video is a monster. But having this infrastructure, we can consolidate that video.

It also lets us sort of become the Internet provider for all the city employees, for libraries, for schools, because we are able to throw as much bandwidth as we need across that network, to the end-point.

You were talking about consolidation of data centres (in a pre-interview chat). What is being done in that area?

Initially we had 8 data centres within the City of Boston footprint. These were not purpose-built data centres. They tend not to meet the highest tiers for redundancy and resilience, because they were just ordinary city buildings converted to that use.

So, we have been doing a couple of things. One, we have been trying to migrate city infrastructure into purpose built data centres, managed professionally and meeting higher standards. So, initially we moved our production environments into premier data centre space in the city, which were way better than our environment.

Our main data centre for city hall is in the floodplain, below sea level. There were concerns around that. We would probably need a multi-million Dollar investment to upgrade the environmental controls to meet our future standards, energy and many other things.

So, the next step was to do a Request for Proposal (RFP) for data centre space outside of the city’s limits, to give ourselves some geographical diversity and be able to avert concerns of regional disasters. We ended up building space within another commercial data centre, around 240 miles (386 km). That was far enough to provide geographical diversity, in terms of weather issues like hurricanes. It was also far enough to be on separate power grids. That helps improve resilience.

Once we had built that, all the other city’s data centres wanted to have presence outside of physical footprint for resilience. So, we made that available. We have been in the process of building it out, the second data centre and making space available to other agencies to collapse those 8 into 2. But it’s not a mandate. We make it available, they can adopt it or not.

How are you preparing for the future?

I don’t think we are not going to be limited by bandwidth in the
foreseeable future, at least for a decade or two.

There might be a paradigm shift in technology. But right now, technologies such as small cell are all dependent on the fibre of our backhaul. They still need to get to that fibre connectivity at some point. At the moment, a lot of the wireless technology is very vulnerable, it’s fragile. So, having that fibre pathway, which we have in place, is going to be critical for the future because there is no resilient, non-fibre based solution at this point.

In dealing with IT infrastructure, what are the primary cybersecurity concerns you face?

Today we have systems which are not traditional IT systems and the people installing them are not IT companies. The guy installing security cameras may know enough to put an IP address in there and configure it. But he doesn’t know enough to properly turn off the ports and protocols that are unnecessary. He may for convenience, leave the default passwords, so that any of his customers can get into the camera easily and tweak it. He is not an IT guy and he doesn’t know that it is bad practice.

There are similar issues with say an intelligent air conditioning system, where you are collecting power consumption data and using that to fine tune the air conditioning to reduce the power consumption.

As more and more things become a part of the Internet of Things, the surface area expands. They are not properly secured. And some of them may not allow you to properly secure them at all. Typically, they don’t get patched on a monthly basis, like a computer operating system. At best, it might happen once or twice a year.

If someone gets into these devices, they can then use that to get into other things. It’s a challenge and that means that you have got that much more of a surface area to cover. And all the traditional tools used to manage the IT environment do not have reach into these IoT services.

You need specific expertise and knowledge and you usually have to go the extra mile to understand how can they secured and what you need to do for security. Someone has to audit, see what the passwords of the cameras are.  

Does the ICT infrastructure of the City of Boston have connections with the ICT infrastructure of the state, or other cities?

We have specific state agencies that we bridge to. Some of the buildings that we are in, are state buildings.

MBTA (Massachusetts Bay Transportation Authority) runs all the transport infrastructure in the Greater Boston area. We have inter-connectivity with them for camera sharing. We also have some connectivity to state police and some other agencies like that.  

We also have connectivity with the surrounding cities. We got federal funds to interconnect the fibre-optic infrastructure of the communities surrounding Boston with ours.

There’s a federal program, E-Rate which subsidises telecommunications and Internet access for schools, libraries. We compete with the telcos, with Verizon, AT&T. and we bid against them to provide services to the schools and libraries in Boston and then we get federal reimbursement. Now we have extended that out to the surrounding communities. We are providing e-ratable ISP services and managed security services to the surrounding communities at very low rate.

We are almost giving to them for the cost of the federal reimbursement. It gives them not only the data services but some security services too, which the communities don’t have the budget for. They benefit from infrastructure investment, in security features like next-generation firewalls.

The federal reimbursement subsidises our network infrastructure.  Also, this helps in our negotiations for funds with the federal government.

Boston is the 23rd biggest city in the country. But the metro region is the 10th biggest. As a group, we are much bigger, which makes it easier to get funds from the fed. We are leveraging infrastructure and spreading the funds across a larger area. And we are helping out.

[1]Mr. Rothman talked about BoNET in his presentation at the Singapore OpenGov Leadership Forum.

[2] Dark fibre is optical fibre infrastructure that is not in use. Much of the cost of installing cables comes from the civil engineering work required. Hence, the cable owners usually plan for, and install, significantly more fibre than is needed for current demand, to provide for future expansion and provide for network redundancy.

[3] In Boston, the "shadow conduit" policy demands that the first company to dig should ask other companies of their potential needs so that shadow conduits can be reserved for future users.

PARTNER

Qlik’s vision is a data-literate world, where everyone can use data and analytics to improve decision-making and solve their most challenging problems. A private company, Qlik offers real-time data integration and analytics solutions, powered by Qlik Cloud, to close the gaps between data, insights and action. By transforming data into Active Intelligence, businesses can drive better decisions, improve revenue and profitability, and optimize customer relationships. Qlik serves more than 38,000 active customers in over 100 countries.

PARTNER

As a Titanium Black Partner of Dell Technologies, CTC Global Singapore boasts unparalleled access to resources.

Established in 1972, we bring 52 years of experience to the table, solidifying our position as a leading IT solutions provider in Singapore. With over 300 qualified IT professionals, we are dedicated to delivering integrated solutions that empower your organization in key areas such as Automation & AI, Cyber Security, App Modernization & Data Analytics, Enterprise Cloud Infrastructure, Workplace Modernization and Professional Services.

Renowned for our consulting expertise and delivering expert IT solutions, CTC Global Singapore has become the preferred IT outsourcing partner for businesses across Singapore.

PARTNER

Planview has one mission: to build the future of connected work. Our solutions enable organizations to connect the business from ideas to impact, empowering companies to accelerate the achievement of what matters most. Planview’s full spectrum of Portfolio Management and Work Management solutions creates an organizational focus on the strategic outcomes that matter and empowers teams to deliver their best work, no matter how they work. The comprehensive Planview platform and enterprise success model enables customers to deliver innovative, competitive products, services, and customer experiences. Headquartered in Austin, Texas, with locations around the world, Planview has more than 1,300 employees supporting 4,500 customers and 2.6 million users worldwide. For more information, visit www.planview.com.

SUPPORTING ORGANISATION

SIRIM is a premier industrial research and technology organisation in Malaysia, wholly-owned by the Minister​ of Finance Incorporated. With over forty years of experience and expertise, SIRIM is mandated as the machinery for research and technology development, and the national champion of quality. SIRIM has always played a major role in the development of the country’s private sector. By tapping into our expertise and knowledge base, we focus on developing new technologies and improvements in the manufacturing, technology and services sectors. We nurture Small Medium Enterprises (SME) growth with solutions for technology penetration and upgrading, making it an ideal technology partner for SMEs.

PARTNER

HashiCorp provides infrastructure automation software for multi-cloud environments, enabling enterprises to unlock a common cloud operating model to provision, secure, connect, and run any application on any infrastructure. HashiCorp tools allow organizations to deliver applications faster by helping enterprises transition from manual processes and ITIL practices to self-service automation and DevOps practices. 

PARTNER

IBM is a leading global hybrid cloud and AI, and business services provider. We help clients in more than 175 countries capitalize on insights from their data, streamline business processes, reduce costs and gain the competitive edge in their industries. Nearly 3,000 government and corporate entities in critical infrastructure areas such as financial services, telecommunications and healthcare rely on IBM’s hybrid cloud platform and Red Hat OpenShift to affect their digital transformations quickly, efficiently and securely. IBM’s breakthrough innovations in AI, quantum computing, industry-specific cloud solutions and business services deliver open and flexible options to our clients. All of this is backed by IBM’s legendary commitment to trust, transparency, responsibility, inclusivity and service.