In a newly obtained 2010 memo sent to then-CEO Nat Ford and others, Muni Chief Information Officer Travis Fox detailed exactly how the agency made its calculations.
He wrote that in the late 1990s, on-time performance was measured by observing whether a bus reached the bus stop at the scheduled time. He characterized that method as the “most appropriate way to measure the consistency of service delivered.”
But the transit agency dropped that method after voters passed Proposition E, which set goals and etched into the city’s charter the definition of “on time” as no more than one minute early or four minutes late.
In the memo, Fox outlined four accounting maneuvers that led to “a reported on-time performance figure that is 13-18 percent higher than actual schedule adherence.”
Any bus arriving up to 4 minutes and 59 seconds late was counted as on time. The liberal definition of 60 seconds was reported by The Bay Citizen last month. In his 2010 memo, Fox blamed a software program that “clips” the seconds off of time entries.
Phantom buses that don't make their scheduled runs at all were not factored into on-time performance. Yet Fox pointed out that the “resulting service gaps affect on-time performance due to longer boarding times associated with crush loads.”
Muni would count a bus as on time if it came to a stop at the time when another bus was scheduled to arrive, even if it was a different late or early bus on the same bus line. Fox wrote that accounting maneuver was used to “maximize the number of observations falling within the -1/+4 minute on-time performance window.”
Late inbound Muni trains heading to the Embarcadero Station were excluded from the calculation because “service in the tunnel cannot be controlled.” Tunnel traffic, automatically controlled by Muni computers, is a chronic source of train delays. The N-Judah, for instance, was on time 52 percent of the time during the last six months of 2011, according to a Bay Citizen analysis.
Ford, the former Muni CEO, acknowledged in an interview that the memo should have “set off alarm bells.”
“That would have been a major issue that would have had to be brought to the board,” he said.
But Ford, who resigned last year, said he didn’t recall ever reading the memo, which was also sent to four other SFMTA higher-ups, including John Haley, director of Transit Operations, and Julie Kirschbaum, manager of Operations Planning. The transit agency was in the midst of dealing with cuts to bus service, Ford said, and the memo “may have got lost in the shuffle.”
Instead, two weeks after Fox sent the memo, Ford and then-Mayor Gavin Newsom together announced that the troubled transit agency had achieved a record 75 percent on-time performance rate in the first three months of 2010.
“Improving Muni’s on-time performance is fundamental to the SFMTA’s mission,” Ford said in the news release on the achievement.
Disappointment, but no surprise
Proof that Muni’s on-time arrival rate is lower than it reported for years did not surprise passengers.
“I’ve been here for seven years, and they’re always late,” said Rex Tan, who takes a train and a bus to his job selling sunglasses at Fisherman’s Wharf. “Just last week I was late to work twice.”
Ben Kaufman, a spokesman for the San Francisco Transit Riders Union, said that it was “disappointing and frustrating that the MTA would deceive the public like that for better performance data.”
“They’re not going to get better on-time performance unless they make serious changes to the system,” Kaufman said.
The initial reaction at the transit agency to inflated on-time reports was muted. At a board meeting last month, the agency’s directors said they were not concerned when the issue first was raised by The Bay Citizen.
“We have a lot of faith in the management team here in terms of the integrity of the information,” Jerry Lee, chairman of the agency’s Policy and Governance Committee, said at the time.
The board made no effort to inquire further.
But Fox said in an interview last week that, starting in July, Muni would be reporting on-time performance without any of the maneuvers he had identified.
“There will be no quirk implementation,” he said.
Origins of the deception
It is unclear who set Muni down the path of overstating on-time performance. In his memo, Fox wrote that the methodology was “modified” by Michael Burns, the Muni chief from 1999 to 2005. And in an interview Fox said the method first was used in 2001.
Burns, however, said that he “was not involved in developing the formula at all.”
“I was not aware of the details, and I did not direct any of the calculations,” he said.
Burns said he wouldn’t have stood for it.
“I would've had them go back and use the four minutes,” he said. “ I never, nor do I now, think there’s any flexibility on what is written in charter.”
Three SFMTA employees who worked on the performance statistics at the time either refused to comment or didn’t return emails and phone calls seeking comment.
What is clear is that after voters set the 85 percent target, Muni struggled to reach it. In fiscal year 2001, the transit agency’s on-time arrival rate was 55.4 percent. The next year, the rate jumped to 69.9 percent. From then on, the rate leveled off before peaking in 2010.
Bonuses for Muni employees, including Burns, were tied to the performance standards back then. A total of 1,500 employees received incentive payments in 2001, according to meeting minutes. Burns' contract allowed him to receive up to $15,000 in bonuses tied to performance, including on-time performance, although it is not clear whether he received them.
The man behind the numbers
Fox, 40, arrived at the transit agency in 2007 with a background in performance measurement. He quickly pieced together how the transit agency calculated on-time arrivals. The issue came to the fore as the agency considered switching from a manual tally to electronic data supplied by transmitters on the buses.
In a November 2010 email to Ford, Fox wrote, “As discussed previously, this is not a matter of simply flicking the switch – we need to ensure the NextMuni data is as 'apples to apples' as possible when compared to the historical methodology.”
In his memo from earlier that year, Fox had weighed the pros and cons of switching to the unadulterated electronic data. The new system, he wrote, “would likely result in a hard-to-explain 10-15 percent drop in reporting on-time performance.”
Ultimately, it wasn’t until current Muni Chief Ed Reiskin arrived last year that the agency decided to ditch the old method, Fox said.
Soon, Fox said, the public and Muni managers will have access to plentiful data measuring real-time performance of the transit agency’s buses, trains and cable cars. One statistic the agency will highlight is the time between arrivals. Muni officials believe that it's more important that a bus comes every 10 minutes than whether it arrives exactly at 10:02 a.m. as scheduled, for example.
“There's the importance of transparency, but it’s also important that the management have the information they need to improve service,” he said.
When asked how he felt about Muni inflating numbers for all those years, Fox hesitated. Then he answered.
“I'm really happy where we are today,” he said. “I think that probably being spelled out helped the organization along in a significant way, and now we’re on the path to having much clearer reporting.”
Tan, the Muni rider who was waiting for his bus, said he hoped that Muni buses would arrive on schedule more often. When he leaves work at 10 p.m., it often takes hours to travel from Fisherman's Wharf across town to his home on Ocean Avenue.
“Sometimes I get home at midnight,” he said.
In a memo to top executives, including the CEO, Muni's chief information officer detailed the accounting maneuvers, called “quirks,” that were boosting the numbers reported to the public by as much as 18 percent.