After Verizon suffered a widespread, all-day outage this week that affected potentially 2 million customers, the cause still remains elusive. The company is offering a $20 credit to impacted customers, which appears to be trickling out via text message and in the myVerizon app.
Verizon hasn’t responded to a request for clarity about what happened. It did say that the problem was a “software issue” and that there was no indication of a “cybersecurity issue” in a statement to Mashable Thursday morning.
The loss in service was unique for its longevity. Unlike previous cellular outages, this one wasn’t region-specific and affected users across the US. When natural disasters take down cellular towers or hardware failures happen, the effects are felt in specific cities or areas. We saw reports (and staffers chimed in) of service being down in New York, New Jersey, Florida, Hawaii, California and other spots.
Until Verizon shares more information, we can piece together some possibilities.
Analyst Roger Entner of Recon Analytics, a telecommunications research firm, suspects that a feature update went awry, based on how devices were affected. “It looks like their 5G SA (Standalone) core went down during a minor feature change,” he wrote to .
Entner noted that the outage was limited to new high-end devices in selected markets where Verizon’s 5G SA core is deployed, which is why the outage wasn’t felt everywhere. The term 5G SA core refers to a network that uses only 5G technology and doesn’t rely on older 4G LTE infrastructure.
Entner also suggested the outage’s timing was unusual. “When carriers do massive upgrades, they do that between 2 a.m. and 4 a.m. in the morning,” he said, suggesting that it could be due to an input error or automatic failure. “A noon start for the crash indicates ‘fat fingers’ for a smaller change that cascaded through the system.”
Lee W. McKnight, an Associate Professor at Syracuse University’s School of Information Studies, believes the problem originated with a failed update to a Virtual Network Function (VNF), leading to “data overflows/cascading failures of other VNFs that were collateral damage from the first failure,” he wrote in an email exchange.
That also explains, McKnight theorizes, why many customers reported that their service resumed temporarily, only to fail again. “Like an engine stalling out, if all Verizon VNFs are not in sync [or] well orchestrated, the network is off-key,” he said.
A VNF is a virtualized service running on cloud platforms, versus on dedicated hardware. For this reason, McKnight’s proposed solution moving forward is that the big carriers need to provide more paid training for its technical staff “in what they are now: cloud software engineers,” he said.
Although we can’t confirm a specific cause for this outage, many Reddit commenters (some who claim to have worked at Verizon) lay some of the blame on Verizon’s recent layoffs that removed roughly 13,000 employees, many of them highly technical engineers with years of experience. Some discussions online speculate on what are frequent causes for network outages, like infrastructure damage, technical glitches or server failures, among others.
This is a developing story. Follow all of ‘s Verizon news for more about the network outage.
