Almost 15-hour network outage traced to maintenance update: Rogers

Rogers Communications says services have been restored and that networks and systems are close to fully operational with technical teams monitoring a few remaining intermittent issues following Friday’s widespread network outage.

In an open letter to customers on Saturday, Rogers President and CEO Tony Staffieri once again apologized for the service interruption while putting forth an action plan to “further strengthen the resiliency of our network.”

Staffieri tells CityNews they’ve narrowed the cause of the outage down to a network system failure following a maintenance update that was done early Friday morning which caused some routers to malfunction.

“What that did was cause traffic to overload the network making them inoperable for our customers,” he explained. “We removed and replaced that equipment and we were able to bring the networks back up.”

Staffieri says a detailed investigation is ongoing.

According to Netblocks, a United Kingdom based organization that monitors cybersecurity, at its peak, the outage knocked out around 25 per cent of Canada’s observable internet connectivity.

⚠️ Confirmed: A major internet outage has been registered across #Canada; real-time network data show national connectivity down to 75% of ordinary levels; incident impacts network operator Rogers and other ISPs, with telephony and emergency calls reported unavailable by many ???? pic.twitter.com/aJhksefYC1

— NetBlocks (@netblocks) July 8, 2022

The outage pushed businesses and organizations to notify customers that delays and service interruptions should be expected. It also caused trouble for 911 services and debit transactions.

Multiple police forces say 911 services are able to receive calls again after the more than 15-hour-long outage paralyzed phone networks across much of Canada. Staffieri says they are troubled that some customers could not reach emergency services and will be addressing that issue as an urgent priority.

Interac issued a statement on its social media platforms apologizing for the inconvenience and saying it is adding a supplier to bolster service access in the future.

Staffieri says Rogers plans to give two days of credit all customers, with more information to be made available soon. The company also warned about scam text messages being sent offering credits.

“We will apply the credit proactively to your account & no action is required,” the company said in a tweet.

We are aware of scam text messages being sent claiming to offer credits in the wake of yesterday’s service interruptions. We will apply the credit proactively to your account & no action is required. If you receive a suspicious SMS, please forward it to 7726 (SPAM).

— RogersHelps (@RogersHelps) July 9, 2022

Sharif Ahmed, the owner for Plantforsoul plant shop in Toronto’s west end, said the outage left him feeling helpless, as he turned away customers who didn’t have cash.

“It pretty much stopped my business,” he said in an interview Saturday. “Most of the people, they don’t use cash anymore, so pretty much, I sat down in my office doing nothing.”

Suddenly not being able to take card payments is a “big problem,” he said. “We just can’t stop, we’re paying rent and everything.”

At nearby Caked Coffee, owner Supreet Arora said people came in thinking it was just their wifi and cell phones that were affected — only to find out that the cafe also had no wifi.

“They came in you want to help them out, (but) there’s no wifi here,” he said in an interview Saturday.

He said he kept forgetting to tell people he was only able to take cash until he’d made their food, but served them anyways, adding that many people came back to pay on Saturday.

Far-reaching implications of Rogers outage

It’s the second time in 14 months that Rogers has faced a major network disruption. In April 2021, a software update affected wireless calls, SMS, and data services.

The widespread outage had many warning the incident is a sign that oligopolistic telecommunications companies need more competition.

“The outage is illuminating the general lack of competition in telecommunications in Canada,” said Vass Bednar, executive director of McMaster University’s master of public policy program.

The country’s telecom sector is dominated by three large carriers — Rogers, BCE Inc. and Telus Corp. — and their hold on the industry has long been a concern of academics, who have called for regulators to increase competition for mobile and internet services in Canada.

The Competition Bureau is currently fighting Rogers’ plans to purchase Shaw Communications Inc. for $26 billion despite the planned sale of its Freedom Mobile business to Quebecor Inc. because the regulator feels the deal would only bolster Rogers’ monopoly and not create a viable fourth carrier.

RELATED: A timeline of key events in the Rogers-Shaw deal

When the outage began Friday, Rogers, Shaw and the Competition Bureau had just wrapped a two-day mediation period that ended with no resolution.

The Rogers-Shaw transaction already has approval from shareholders and the Canadian Radio-television and Telecommunications Commission, but remains subject to review by the Competition Bureau and the Ministry of Innovation, Science and Economic Development.

Richard Leblanc, a professor of professor of Governance, Law and Ethics at York University, said the outage was a learning opportunity for threat actors, like Russian state-sponsored hackers, who can now see how vulnerable Canadian industry, financial institutions and health-care systems are to an attack on a telecom provider.

“This could have been catastrophic for the country if this was a threat actor,” he said in an interview Saturday.

Leblanc said the outage makes it clear that the federal government can’t just rely on telecoms companies to do the right thing.

“I think it’s time that regulators, and this includes Industry Canada, the CRTC and the Competition Tribunal begin to insist on proper, robust, independently-audited internal controls, so that you don’t have an outage like this,” he said.

Patricia Valladao, a spokeswoman for the Canadian Radio-television and Telecommunications Commission said the telecom regulator is in contact with Rogers.

“Right now, our focus is on the outage and recovering from it, when it is over, we will take all necessary actions to examine what occurred and put in place the necessary measures to prevent it from happening again,” she wrote in an email.

When everything from 911 services to GO Transit is impacted by a Rogers outage, the reach of telecommunications companies is very obvious, Bednar said.

“But unless we’re going to see people switching their providers today or new publicly run options suddenly springing up, there’s not much more that we can do right now other than perhaps factor in people’s anger and frustration, as the pending Rogers-Shaw deal is considered.”

She added that people should be compensated for the disruption.

“It’s a huge expense to Rogers, but even a modest decrease on people’s bills would acknowledge some kind of deficit.”

Beanfield, an independent fibre network operator, called the outage “every telecom provider’s nightmare,” but said it was also an example of why it has long been concerned with the lack of rivals for Rogers, Telus and BCE.

“A lack of competition and choice can lead to a building with the population of a small town going completely dark- cut off from all communications,” the company said on Twitter.

“If you can’t even get help from a neighbour, where do you go? How do you call 911?”

The business implications are likewise tremendous, the company added.

“The consequences of such an outage for the financial sector, the lack of functioning ATMs, of working bank branches, can be catastrophic,” it said.

“Not to mention the independent businesses across the country with no way of processing payment.”

Leblanc, said he expects to see class action lawsuits — and suits by individual firms — attempt to quantify the cost of the outage.

“Lawyers are good at that, so if there’s a class action, they can measure the loss of productivity, the opportunity cost of not being able to work, missed meetings, missed opportunities, missed contracts, it is significant,” he said. “If all of these millions of people lose a day of their working life, they’re not going to be made whole by a credit.”