Here’s what caused the AT&T outage that blocked 92 million phone calls (plus 25,000 attempts to reach 911)

February 22, 2024: do you remember that date? That’s when a massive AT&T outage caused the following to happen in the 12 hours that it lasted:

• More than 125 million devices were affected;
• More than 92 million voice calls were blocked;
• More than 25,000 calls to 911 call centers were obstructed.

Heck, massive doesn’t begin to describe it: these are some intergalactic numbers.After months of investigating, the FCC (Federal Communications Commission) is ready with its report on the said AT&T outage. Back when it happened, AT&T released this statement: “Based on our initial review, we believe that today’s outage was caused by the application and execution of an incorrect process used as we were expanding our network, not a cyberattack.”

Now, the FCC provides a report detailing the cause and impact of a nationwide AT&T wireless service outage on February 22, 2024. The report also contains recommendations to help prevent similar outages in the future.

– FCC Chairwoman Jessica Rosenworcel.

As soon as the outage occurred, the FCC’s Public Safety and Homeland Security Bureau launched an investigation.

Among key findings in today’s report:

• The outage affected users in all 50 states as well as Washington, D.C., Puerto Rico, and the U.S. Virgin Islands. All voice and 5G data services for AT&T wireless customers were unavailable, affecting more than 125 million devices, blocking more than 92 million voice calls, and preventing more than 25,000 calls to 911 call centers.
• Voice and 5G data services were unavailable to customers of other wireless providers that regularly use or were roaming on AT&T’s network.
• It took AT&T at least 12 hours to fully restore service.
• The outage also cut off service to devices operated by public safety users of the First Responder Network Authority (FirstNet). AT&T prioritized the restoration of FirstNet before other services but did not notify FirstNet customers of the outage until three hours after it began, and nearly one hour after service was restored.
• The incident began after AT&T implemented a network change with an equipment configuration error. The report details numerous factors that resulted in the extensive scope and duration of the outage, and it cites the corrective actions since taken by AT&T to prevent a reoccurrence.

The report stresses that this “sunny day” outage highlights the need for network operators to adhere to their internal procedures and industry best practices when implementing network changes. Implementing sufficient network controls to mitigate errors is a must, so they do not escalate and disrupt network operations.
Based on its investigation into this AT&T wireless outage, the Public Safety and Homeland Security Bureau has referred this matter to the Enforcement Bureau for potential violations of FCC rules. The FCC is also currently investigating a recently disclosed AT&T breach of consumer data and working closely with law enforcement agencies.

So, a network update caused the problem.