About recent service interruptions, what we're doing to prevent similar problems in the future
BART recognizes that the service disruptions that occurred during the week of March 27th are unacceptable and we sincerely regret the inconvenience they may have caused you. What follows are some details on what happened and what BART is doing to prevent similar problems in the future.
BART suffered three computer shutdowns between Monday, March 27 and Wednesday, March 29. The shutdowns forced our central control staff to halt train service each time. We experienced two different problems; both were related to our two-and-a-half year effort to upgrade a computer system that brings train information into BART's control center.
The first two incidents were due to a problem with the latest version of software that was installed. This software was developed, configured and thoroughly tested on a simulator before being installed for revenue service. BART's practice for installing new software is that installation takes place when trains are not running, mostly on early weekend mornings. In fact, this software was installed over the weekend and ran without problems all day Sunday and through Monday morning peak commute.
Soon after the first software problems on Monday and Tuesday, BART staff began immediately working to configure a backup system that would enable a faster recovery from any future software failure. On Wednesday, work on that backup system inadvertently contributed to the failure of a piece of hardware that, in turn, created the longest delay. It appears the technicians believed they were working in a way that would not affect service.
It was a mistake to work on the computer system during revenue hours. The individuals involved did so with good intentions: to shorten the recovery time in incidents like Monday and Tuesday's shutdowns.
What are we doing to correct these types of service interruptions? First, we have reverted to an earlier version of software that has provided reliable service for many months. We will continue to use this until we are confident that the new, upgraded software can perform with the same reliability.
Second, we are adamant that we will not break established practice by working on the system during revenue hours. In all cases, BART holds its employees to the highest standards and will take appropriate action to ensure adherence to established procedures.
Finally, we are working with the manufacturer of the computer hardware that did not perform as expected to determine what corrective steps need to be taken.
BART has an excellent passenger on-time record and these service disruptions, while an anomaly, are clearly unacceptable. We fully intend to return to our exceptional standards.
We sincerely appreciate your continued support and understanding.