BART, Cisco Hunting for Root Cause of Computer Network Trouble That Shut Down Service

BART is working with technology partner Cisco Systems to get to the bottom of a computer networking hardware problem that led to a total shutdown of the transit system Saturday morning.

The agency said Monday that Cisco is probing the failure of a networking switch that brought down most of the agency's computer network and made it impossible to dispatch trains for the system's normal 6 a.m. Saturday opening time.

The failure required BART employees to scramble throughout the 121-mile system to reset equipment at 46 track switches and 68 electrical substations that provide traction power for trains.

Partial service was restored at 9 a.m. -- though not to the San Mateo County stations beyond Daly City, including San Francisco International Airport. The computer problem also shut down many stations' informational signs and ticket machines, and even fouled up some of the agency's website customer alerts.

Full service resumed just before 11 a.m.

"It was a monumental task to go and reset the system and ensure we could operate safely," said Bevan Dufty, president of BART's board of directors.

After the resets, the agency said in a statement Monday, "all train systems have performed as designed without failure."

Dufty, who represents a district in San Francisco, said agency staff will report on the incident at this Thursday's board meeting.

"Cisco and BART engineers and operations are working diligently to really come up with answers," Dufty said. "I understand the desire two days later to know what took place, but I do want to give some opportunity for the technical people to come forward and talk about it."

He added that the Saturday shutdown, which appears to be the system's worst computer-related failure since a botched software upgrade caused two days of service disruptions in March 2006, highlights the need to replace the agency's "Pong-era" central train control system.

Replacing the train control system was one of the selling points for Measure RR, passed in 2016, which allows the district to sell $3.5 billion in bonds to help pay for new cars and a long list of major infrastructure upgrades.

BART says the $1.15 billion train control project, which will get $400 million from Measure RR in addition to state and federal grants and revenue from a fare increase scheduled for next year, is key to plans to increase the system's passenger capacity.

With the new system in place, the agency says it will be able to run trains closer together -- a change that will allow an increase from 24 to 30 trains per hour traveling through the Transbay Tube during the morning and evening commutes. That increased frequency, along with the availability of more cars, would result in a capacity increase of 21,000 to 30,000 transbay riders an hour, the agency says.

BART says it will pick a train control provider by the end of this year, but that the system won't be fully in place until 2026.

Dufty described BART as a system that's "racing" to replace worn-out train cars, track and other equipment, and said the computer network failure over the weekend was "unfortunately a side product" of that catch-up process.