This CEO left Bloomberg to track GPUs. She explains why prices are 'going nuts.'

The GPU market is experiencing a dramatic surge in prices, reflecting the ongoing demand for AI chips and indicating that normalization is still a distant prospect. In a recent interview, Carmen Li, CEO of Silicon Data, highlighted the firm’s findings which show significant price increases across both older and newer Nvidia GPUs, essential for training and executing AI models. Silicon Data’s Neo Cloud H100 index has seen a notable rise from 2.20 to 2.64, equating to a 20% increase over the last three months. Similarly, the Neo Cloud B200 index jumped from 4.40 to 5.35, marking a 22% climb, while the Hyperscaler H100 index grew from 7.26 to 7.46, a 3% uptick. These statistics indicate that the demand for AI compute resources continues to significantly outstrip supply. Li pointed out that the current market dynamics differ from traditional patterns. Typically, prices would rise with the introduction of a new chip and then gradually decline as supply catches up. However, she noted that prices for B200s have not only stayed elevated but have also increased, revealing ongoing capacity constraints. This imbalance between demand and supply is impacting the entire AI ecosystem, including chips, memory, power, and data center space. In her interview, Li remarked, "The price is going pretty nuts," particularly referencing the rental costs of H100 GPUs. The most substantial rental price premiums are observed in hyperscaler environments, where clients pay significantly more for the benefits offered by major cloud providers like Amazon, Microsoft, Google, and Oracle. Renting an H100 from these hyperscalers can cost nearly three times more compared to specialized providers like CoreWeave. Li, who transitioned from a role as the global head of strategic alliances at Bloomberg, established Silicon Data to enhance transparency within the GPU rental market. Her company compiles extensive pricing data globally, normalizing it into indices that help stakeholders understand both immediate rental prices and long-term value. This pricing intelligence is vital as major tech firms and AI labs invest trillions in GPUs and data centers to fuel the generative AI surge. One potential concern is that should GPU prices decline due to weakened demand, AI cloud companies may face faster depreciation of these assets, negatively affecting their earnings. However, current trends show the opposite, with GPU prices on the rise and older models not depreciating significantly. Li, who also operates a business called Compute Exchange that resells refurbished GPUs, noted that refurbished H100s can retain about 85% of their value in the second year and 84% in the third. "My car depreciates a lot faster than that," she commented. At present, the demand for AI capabilities is soaring, and the available computing resources are insufficient to meet this need. This situation not only impacts infrastructure budgets but also influences the economics of AI products that rely on rented GPUs for large-scale operations.