Companies are hoarding AI compute because of FOMO — and they're sitting on most of it

Recent analysis reveals that many organizations are investing heavily in AI infrastructure, yet a significant portion of this computing power remains untapped. Cast AI, a platform specializing in automated cloud cost optimization, has released its 2026 State of Kubernetes Optimization Report, highlighting a striking trend among companies like BMW and Cisco. According to the report, organizations are provisioning approximately 20 times the GPU capacity they actually utilize at any moment. Data gathered from 23,000 clusters across a wide range of enterprises indicates that average GPU utilization is a mere 5%, meaning that roughly 95% of the allocated GPU capacity goes unused. Similarly, CPU utilization stands low at 8% of total capacity. The report emphasizes the financial implications of this underutilization. While an idle CPU might waste just a few cents hourly, a GPU, which is essential for tasks like machine learning and video processing, can incur losses of several dollars per hour. Given that GPUs can be up to 50 times more costly than their CPU counterparts, this inefficiency is concerning. These findings emerge as businesses scramble to acquire high-demand AI chips, particularly premium GPUs such as Nvidia's Blackwell models, which are experiencing price hikes due to overwhelming demand. Laurent Gil, CEO of Cast AI, explained the urgency driving this phenomenon. Unlike traditional cloud services that allow for flexible resource allocation, the current scarcity of GPUs leads companies to enter long-term contracts, often resulting in over-purchasing out of fear of missing out. "The act of buying has no correlation with whether you need them or not," Gil pointed out, emphasizing that companies are acquiring GPUs simply because they are available, not necessarily because they require them. Gil urges CTOs to reassess their GPU usage, suggesting they ask their teams, "We already have a few thousand of those GPUs. How are we using them?" He notes that with only 5% utilization, there could be 20 times more latent capacity available that organizations are unaware of before they consider buying additional machines.