AT&T built an AI system to prevent network outages. It reduced customer downtime by more than 12 million hours.

AT&T is taking significant strides to ensure uninterrupted service for its vast customer base, which includes 145 million wireless and 16 million broadband users. In an industry where even a brief outage can have substantial financial repercussions, the telecommunications giant has developed an innovative AI-driven system designed to minimize disruptions and enhance service reliability. The need for such a system became evident after a major outage in February 2024, which resulted in over 92 million blocked voice calls and more than 25,000 failed 911 calls, as reported by the US Federal Communications Commission. With other telecom providers like Verizon and T-Mobile also facing similar challenges in recent years, AT&T recognized the importance of improving their response to outages. "We have to know of any network interruptions immediately and solve them, ideally before any customer feels it," stated Andy Markus, AT&T's chief data and AI officer. Beginning in 2017, AT&T initiated the development of its end-to-end incident management system (EEIM), integrating advanced technologies from companies such as MongoDB and Snowflake. Initially based on traditional machine learning, the system has progressively incorporated generative and agentic AI capabilities, significantly enhancing its operational efficiency. This advanced tool is now capable of diagnosing issues, suggesting remote fixes, or dispatching technicians when necessary, while also keeping customers informed about outages. Telecommunications outages can arise from various factors, including severe weather, technical malfunctions, and even cyberattacks. Recognizing the pressure on customer service centers during such events, AT&T aimed to create a system that responds more effectively to these challenges. By assembling a cross-functional team, the company gathered insights from IT, data, and AI departments, as well as field technicians who manage network repairs. The EEIM processes an immense volume of data—around 10 petabytes, which equates to approximately 5,000 billion pages of text. This extensive data includes network logs and incident reports, all meticulously organized to facilitate problem detection and prediction. MongoDB’s elastic database platform plays a pivotal role, allowing AT&T to scale its data management capabilities seamlessly. Additionally, Microsoft Azure, Databricks, and Snowflake are integral to the EEIM's functionality. The EEIM was first operational for broadband fiber in early 2018, with subsequent expansions to cover DSL services. A complementary application, Atlas, was launched in June 2018 to assist field technicians by utilizing AI to diagnose outages and propose solutions. Since Andy Markus joined the company in July 2020, AT&T has witnessed a significant increase in AI adoption, with around 100,000 employees now using these advanced tools. By the first quarter of 2021, AT&T introduced a proactive customer notification system, further easing customer frustrations during outages. The incorporation of generative AI features in early 2022 allowed the EEIM to analyze historical data to identify potential outage causes more effectively. Recent enhancements include the addition of AI agents capable of interacting with customers to gather details about outages and assist technicians in the field. In total, the AI-empowered EEIM has prevented 3.1 million unnecessary technician visits and has remarkably reduced customer downtime by over 12 million hours in the past year, demonstrating AT&T’s commitment to leveraging technology for improved service delivery.