After outage, Rogers vows to separate wireless and wireline networks; critics not impressed with CRTC response

The July 8 nationwide Rogers Communications Inc. outage that was described as both "unprecedented" by Rogers executives and "unacceptable" by a senior government minister will force Rogers to spend some $250 million to "physically separate" its wireless and wireline networks in the coming years. Rogers CEO Tony Staffieri and newly-minted chief technology officer Ron McKenzie made the pronouncement Monday at the first of two House of Commons industry committee hearings into the day-long outage that saw some 10 million people across the country lose both internet and wireless connectivity, and has sent ripples through the entire sector. "On that day, we failed to deliver on our promise to be Canada's most reliable network," Staffieri told MPs. "To guard against a system-wide outage, we will set a higher standard by physically separating our wireless and internet networks, and create an always-on network," the CEO said, adding that the process would be expensive. Innovation Minister François-Philippe Champagne told MPs that "competition promotes resilience" in response to a question about the small number of players in the telecommunications sector, but ultimately deflected questions about what impact the outage might have on Rogers' plans to acquire Shaw Communications Inc., which is still pending regulatory approval from the Competition Tribunal and the department of Innovation, Science, and Economic Development. Champagne also said that his department's decision on approval for the aspects of the Rogers-Shaw merger for which it is responsible will come after the Competition Tribunal process runs its course. Asked about what accountability the government could demand from Rogers, Champagne pointed to the demands he already issued in the days following the outage: a mutual aid agreement amongst the country's largest providers, emergency roaming, and a communications protocol, demands which Champagne said Rogers "immediately agreed to." The minister added that he is expecting the CRTC to do a "full investigation" into the outage. Whether or not the CRTC will issue a notice of consultation is an open question, and the regulator has yet to decide what it will do beyond the request for information (RFI) it issued the week following the outage, according to CRTC chair Ian Scott, who appeared before the committee in the afternoon. The CRTC is assessing the "completeness and adequacy" of Rogers' responses to the RFI, and is still determining its next steps. "We have various tools at our disposal," Scott said in French, according to simultaneous translation. "We can always look at administrative monetary penalties, that's a possibility. But we probably won't need this kind of thing," he said. The CRTC had requested that the incumbent submit a “complete and detailed” report of the outage, including timelines, root causes, numbers of customers and services affected. The CRTC also wanted to know what measures or steps were put in place after the last major outage in April of 2021, why they failed to prevent the most recent outage, and how Rogers’ staff communicated with each other in the early hours of the outage. “The network outage experienced by Rogers was simply not acceptable. We failed in our commitment to be Canada’s most reliable network. We know how much our customers rely on our networks and we sincerely apologize. Rogers is particularly troubled that some customers could not reach emergency services or receive alerts during that outage,” Rogers wrote in its July 22nd letter, signed by its chief regulatory officer and governmental affairs Ted Woodhead. Rogers said that Bell and Telus offered assistance during the outage, however, “given the nature of the issue, Rogers rapidly assessed and concluded that it was not possible to make the necessary network changes to enable our wireless customers to move to their wireless networks.” Rogers said the switch could not be made because it did not have access to its own home location register, home subscriber server, and centralized user database during the incident. “Furthermore, given the national nature of this event, no competitor’s network would have been able to handle the extra and sudden volume of wireless customers (over 10.2 million) and the related voice/data traffic surge. If not done carefully, such an attempt could have impeded the operations of the other carriers’ networks,” it wrote. The company said Canada’s big telecoms companies are exploring the transfer of customers to other networks as part of the work towards the Memorandum of Understanding for cooperation between carriers mandated by Minister Champagne. However, Rogers employees were equipped with emergency SIMs on the alternate carriers that enabled Rogers to switch carriers in its initial coordination efforts. More detailed information on the outage and steps taken by the company to resolve the issue were filed with the commission in confidence. At the early stage of the outage, many of its network employees could not connect to its IT and network systems, slowing down initial triage and restoration efforts, Rogers noted. “When it was realized that Rogers' entire core network was offline, employees started swapping out our Rogers SIM cards with our alternate carrier SIM Cards. This previously established contingency plan allowed us to begin communicating within our organization in the early hours of the outage and to start restoring services.” Rogers noted that “some minor instability issues” persisted the weekend of the outage, but is now fully operational. Rogers said the cause of the outage was due to a network system after the company updated the core IP network in the early-morning. “This caused our IP routing network to malfunction. To mitigate this, we re-established management connectivity with the routing network, disconnected the routers that were the source of the outage, resolved the errors caused by the update and redirected traffic, which allowed our network and services to progressively come back online later that day,” it wrote. Rogers said the updates began weeks in advance and was in the sixth phase of a seven-phase process. “The first five phases had proceeded without incident,” Rogers stated. The sixth phase of the upgrades began at 2:27 a.m. when network traffic is quietest, then by 4:43 a.m., a “specific coding was introduced in distribution routers which triggered the failure of the Rogers IP core network” two minutes later. “The configuration change deleted a routing filter and allowed for all possible routes to the Internet to pass through the routers. As a result, the routers immediately began propagating abnormally high volumes of routes throughout the core network. Certain network routing equipment became flooded, exceeded their capacity levels and were then unable to route traffic, causing the common core network to stop processing traffic. As a result, the Rogers network lost connectivity to the internet for all incoming and outgoing traffic for both the wireless and wireline networks for our consumer and business customers,” the company wrote. Rogers said it is conducting a full review of the outage, including an external review team, compensating its customers for five days of service and it is reaching out to the government, public, and corporate organizations that Rogers services. All of Rogers’ flanker brands, such as Fido and Chatr, were impacted. Wireless, home phone telephony, internet and television services were down during the outage. Only some wireless customers had intermittent service throughout the day such as texting and 911 access on its Global System for Mobile communications (GSM) and 3G networks. The telecommunication service providers affected by the Rogers outage was not made public. The company did state that all its international and national roaming partners would have been affected when attempting to roam on Rogers’ network as well as all third-party internet providers (TPIA) who utilize Rogers. Aside from Interac services being disrupted (much of which was redacted), Rogers said other critical infrastructure (hospitals, gas and energy providers, etc…) customers were affected but “it is not known whether these customers were fully impaired or if they had some degree of dual-carriers diversity that protected them from full disablement.” “The prioritization of service restoration was always dependent on which service was most relied upon by Canadians for emergency services. As wireless devices have become the dominant form of communicating for a vast majority of Canadians, the wireless network was the first focus of our recovery efforts. Subsequently, we focused on landline service, which remains another important method to access emergency care. We then worked to restore data services, particularly for critical care services and infrastructure,” Rogers wrote when asked how it prioritized repairs. The number of successful wireless and wireline 911 calls on July 8th, versus the daily average, were redacted from the publicly available document. Rogers did not have any available data on the number of unsuccessful calls. The 911 networks that receive calls from originating networks, operated by the three large Canadian incumbent local exchange carriers rather than Rogers, were unaffected by the outage. Rogers noted that 911 service was only restored when the network was fully restored because it was deemed to be “the fastest way” to get emergency services back online. Four emergency alerts were issued on July 8th, all pertaining to Saskatchewan, which were received but not delivered to wireless devices in Rogers’ coverage area. Rogers said it is working on emergency roaming agreements with Canada’s wireless service providers to enable customers to roam onto peer wireless networks to receive emergency alerts and other important services such as 911. Several of the company’s broadcasting services were also disrupted. Three radio stations in or near Whistler, B.C., were unable to transmit programming for approximately 22 hours. The cause of those issues were filed in confidence with the commission. CHST-FM in London, Ont., was unable to air live programming until 1 p.m. on the Friday of the outage and two stations in Tillsonburg, Ont., had similar issues which were only resolved after 10 p.m. OMNI Regional was unable to produce three of its national newscasts in Arabic, Punjabi, and Filipino, and closed captioning for live events on Sportsnet was affected. Rogers claimed the April 2021 outage and its most recent outage were caused by different issues. The former outage strictly affected the wireless network and the latter affected the core network thereby impacting wireless, wireline and Internet services. The steps the company took to prevent further outages like the one seen in April 2021 were filed in confidence but the company stated that “while these steps were broad in nature and will help prevent any type of incident, many were also focused on the particular circumstances of what happened in 2021. Unfortunately, they did not prevent the particular circumstances that resulted in the July 8th outage, although they did contribute positively to its resolution.” At the committee hearing Monday, NDP MP Brian Masse objected to the level of redaction in the Rogers submissions. "Not only are the claims of confidentiality subject to applications by the public, but they are a matter of public record. So the information can be sought, and Rogers will have to persuade the commission of the need to keep it confidential," Scott told MPs Monday. The Public Interest Advocacy Centre indicated its desire to challenge the confidentiality of Rogers' filings in a Monday morning letter to the CRTC. "We request clarification from the Commission immediately if the present set of RFIs and redacted answers is to be considered a ‘proceeding’ (we note two file numbers were assigned) and whether PIAC and others may challenge Rogers’ assertions of confidentiality and if so, what procedure should be undertaken to challenge those assertions," PIAC wrote in the letter. Scott also told MPs that there were no specific regulations that could have prevented the outage. "There's no legislative provision that is adequate to address this kind of situation," Scott said in French, according to simultaneous translation. "Specifically with respect to the outage and network reliability, I think this is a situation that can be addressed by the industry. It's in their collective interest to do so, it's in our interest and that of ISED to make sure they do so," he said. In late afternoon testimony before the committee, University of Ottawa law professor Michael Geist called the admission of Scott a "virtual shrug" and said that the investigation of the outage should extend beyond the CRTC's initial questions. "Organizations such as Interac, governments, and healthcare providers need to explain how they found themselves without redundancy plans and an appropriate backup system to address widespread network outages," Geist said, and also criticized what he called "widespread redactions" in Rogers' filings with the CRTC. Geist also argued that the Rogers-Shaw merger should be considered "dead in the water." "The last things Canadians need is an even more concentrated market," Geist said. PIAC executive director and general counsel John Lawford told MPs that the outage emphasized the need for a quality of service regime to be instituted by the CRTC for wireless and internet service. The lack of such a regime, Lawford said, is "why the commission, who you have just heard, has a hard time describing just what it is doing: to both inquire into the Rogers' outage and to ensure that it will not happen again, not only to Rogers, but to any other telecommunications service provider." – Reporting by Michael Lee-Murphy at mleemurphy@thewirereport.ca and Jenna Cocullo at jcocullo@thewirereport.ca and editing by Paul Park at ppark@thewirereport.ca

After outage, Rogers vows to separate wireless and wireline networks; critics not impressed with CRTC response

Broadcast | July 25, 2022

Unlock all the Canadian telecom, broadcasting and digital media news you need.

FREE TRIAL

SUBSCRIBE

Inside The Wire Report

Information

Hill Times Publishing

Get in Touch With Us

After outage, Rogers vows to separate wireless and wireline networks; critics not impressed with CRTC response

Broadcast | July 25, 2022

Unlock all the Canadian telecom, broadcasting and digital media news you need.

FREE TRIAL

SUBSCRIBE

Related Stories

Telus’ arguments for incumbent access to wholesale regime not supported by competitors

CRTC finalizes terms of Uvagut TV distribution

Bell won’t participate in subsidized broadband projects if wholesale access for Big Three continues, signs agreement with CNOC

CRTC announces changes to Broadband Fund

Reuse & Permissions

Inside The Wire Report

Information

Hill Times Publishing

Get in Touch With Us