As HBM shortages bite, AMD turns to software to stretch AI memory

As the race to develop and implement generative AI models intensifies, the focus has shifted from merely compute power to the crucial role of memory in AI infrastructure. The demand for High-Bandwidth Memory (HBM), which complements AI accelerators, has surged significantly. Major memory manufacturers like Samsung, SK Hynix, and Micron are prioritizing the production of specialized HBM, leading to a tightening supply chain that is expected to persist through at least 2027. This scarcity is creating challenges for enterprise buyers who are eager to expand their AI capabilities while managing infrastructure costs. In response to this issue, AMD has taken a strategic step by acquiring MEXT, a startup dedicated to enhancing memory efficiency. MEXT's innovative platform harnesses AI to optimize flash storage, enabling it to function more like DRAM. This advancement aims to improve memory utilization, lessen reliance on costly DRAM, and ultimately lower the expenses associated with AI infrastructure. MEXT's technology intelligently predicts the data an application will need next and manages data movement between memory and storage. According to Pareekh Jain, CEO of EIIRTrend & Pareekh Consulting, memory has become a significant bottleneck in AI systems, with HBM shortages hampering both training and inference processes. Software solutions such as memory tiering, caching, and compression could enhance efficiency by 20% to 50%, allowing customers to maximize the value of their existing memory resources. However, while MEXT's technology addresses a pressing issue, analysts advise against viewing it as a comprehensive solution to the global memory shortage. Manish Rawat, a semiconductor analyst at TechInsight, emphasizes that while MEXT's predictive memory technology enhances memory optimization, it cannot replace DRAM entirely. Flash storage, although improved, still falls short of DRAM's speed and bandwidth specifications, making it unsuitable for all workloads. MEXT's advantages are likely to be most beneficial for enterprises focused on databases, analytics, and AI inference, where access patterns are more easily predictable. In contrast, companies engaged in high-performance computing or frontier AI training may find it less effective, given their stringent requirements for memory bandwidth and low latency. AMD plans to leverage MEXT’s technology across its data center offerings, potentially enhancing its EPYC server processors, Instinct AI accelerators, and associated software. However, customers may need to wait for approximately 12 to 18 months to see tangible benefits, as the integration of memory-management software into AMD's hardware ecosystem will be a gradual process. As the competition in AI infrastructure evolves, AMD's acquisition of MEXT aims to bolster its position against rivals like Nvidia, which has established a significant lead in the market through its comprehensive platform and software ecosystem. While MEXT may not dramatically shift the competitive landscape, it could allow AMD to enhance efficiency and reduce costs for hyperscale deployments, providing a valuable advantage in an increasingly demanding AI market.