A man walks along an empty platform as a train stands idle at Toronto's Union Station after GO Transit and UP Express announced that trains are not running on the entire network due to a networkwide system failure on Tuesday October 3, 2023. THE CANADIAN PRESS/Chris Young

MONTREAL - The massive outage on Canadian National Railway Co. lines that delayed thousands of Toronto-area commuters during the evening rush hour Tuesday can be traced to a routine software upgrade, the company says.

"At this time, our teams have determined that this was caused by an internal systems upgrade, which affected CN's ability to connect to the internet," spokesman Jonathan Abecassis said in an interview.

"There's no indication of a cybersecurity incident whatsoever. At no time was the safety of the public compromised, and at no time was data impacted."

The digital blackout "made it impossible for CN to connect to the internet," he clarified. On top of crashing the company website, it also barred regional transport authority Metrolinx and its GO Transit trains from connecting to CN servers — essential to digital communications that direct locomotives where to move — Abecassis said. GO Transit trains run partly on CN-owned tracks.

After CN's computer system went down, a handful of rail traffic controllers began issuing orders — sometimes by pen and paper — that were then sent out to conductors via radio, phone or fax to move trains to stations or other safe areas of track, where they were halted, according to two railway sources who spoke on condition of anonymity because they were not authorized to discuss the matter.

Partial connectivity was restored at about 3:45 p.m., as some trains lurched back into motion hours behind schedule after the outage began at 12:30 p.m. System recovery was nearly complete by 8 p.m., and fully resolved overnight, Abecassis said.

Earlier, the computer malfunction forced GO Transit commuter trains in the Greater Toronto Area to stop at the nearest station, while riders crowded the city's Union Station downtown during rush hour in hopes the network would resume.

The rail link connecting the downtown core to Toronto's Pearson airport also stalled, while more than 30 Via Rail arrivals and departures at Union Station were late — 11 by more than two hours, including busy routes to or from Montreal, Ottawa and Windsor, Ont. — according to data compiled by Greg Gormick, who heads On Track Consulting.

Pierre Barrieau, who teaches transportation and urban planning at the Université de Montréal, said technical issues are the No. 2 cause of rail disruptions, after weather.

"These things are rare. However, as our technological systems become more and more complex, they become more and more fragile to these updates," he said.

As the need for upgrades rises in lockstep with security concerns over data breaches — and as the software code becomes increasingly complex — the odds of a problem increase, he said.

"Was that the best time to do the upgrades?" asked Ian Naish, a rail safety consultant and former director of rail and pipeline investigations at the Transportation Safety Board. "It just seems so unusual."

On the issue of complexity, Barrieau pointed to the three service disruptions in as many days that hit Montreal's new light-rail line during its opening week in August. Computer and track switch problems on the rail network caused drawn-out delays and forced it to deploy shuttle buses to ferry commuters between a downtown station and the city's South Shore.

On Tuesday, Barrieau noted, CN and Metrolinx were able to respond quickly.

"They were able to at least get commuter rail trains — some of them — running ... and there were no safety issues."

No CN freight trains experienced major delays, while EXO commuter trains in the Montreal area and Amtrak passenger trains from the U.S. also remained in operation, Abecassis said.

EXO trains are dispatched directly by CN, which was able to revert to analogue techniques to keep them moving, he said.

During the outage, CN worked with GO Transit to temporarily take over their train dispatching responsibilities, CN said. "This allowed for the partial resumption of GO and Via services" starting in late afternoon.

Meanwhile, IT teams at the country's largest railway worked late into the night to ensure servers came back online smoothly and the "routine upgrade" sparked no more headaches, Abecassis said.

“We're going to be working with Metrolinx to put in place processes to avoid this happening again," he said.

This report by The Canadian Press was first published Oct. 4, 2023.

Companies in this story: (TSX:CNR)

News from © The Canadian Press, 2023