Startups find Amazon's AI chips 'less competitive' than Nvidia GPUs, internal document shows

Amazon's ambitions in the AI chip market are facing significant hurdles, as highlighted by a recently leaked internal document. The tech giant's Trainium chips, designed to power cloud-based AI applications, have been labeled as underperforming compared to Nvidia's widely used H100 GPUs, according to findings from the AI startup Cohere. The document, dated July, indicates that Cohere experienced limited access to Trainium 2 due to frequent service interruptions, raising concerns about the chip's reliability. Similar feedback was provided by Stability AI, which noted that Trainium 2's performance on latency made it less competitive against Nvidia’s offerings. Amazon's Trainium chips are crucial for the company's strategy to reduce reliance on expensive Nvidia GPUs while providing robust AI services to cloud customers. However, if AWS clients express a preference for Nvidia hardware, it could jeopardize Amazon's profit margins in the cloud sector. Internal complaints are indicative of the challenges Amazon faces in matching Nvidia's performance, particularly among its startup clientele, historically a key market for AWS. In response to the feedback, an Amazon spokesperson acknowledged the importance of customer input in refining their chips and highlighted that Trainium and its sibling chip, Inferentia, have seen positive results with major clients like Ricoh and Datadog. They also pointed out that Trainium chips are reported to offer 30% to 40% better price performance than current-generation GPUs. Despite these assurances, the document revealed that some startups, such as Typhoon, found Nvidia's older A100 GPUs to be significantly more cost-effective than AWS's Inferentia 2 chips for specific tasks. This disparity in performance and cost has led to concerns about the adoption of Amazon’s custom AI chips, which were previously flagged as having “friction points” that deterred usage. Nvidia maintains a commanding lead in the AI chip market, holding over 78% of the share, while Amazon's chips rank sixth with only 2%. The competitive landscape is further complicated by a $38 billion partnership between AWS and OpenAI, which exclusively utilizes Nvidia GPUs, illustrating the ongoing challenges Amazon faces. Though AWS CEO Andy Jassy reported that Trainium 2 has become a multibillion-dollar business with full subscriptions, the company's market position remains precarious. Anthropic, a leading AI lab and significant Trainium user, is expected to deploy over 1 million Trainium 2 chips by year-end, potentially bolstering Amazon’s chip credibility. However, the success of this initiative is yet to be determined, especially after Anthropic's recent expansion of partnership with Google, which could dilute Amazon's influence in the AI chip sector. As AWS continues to explore its place in the competitive AI landscape, the company emphasizes its commitment to providing diverse hardware options, aiming to meet the varied needs of their customers without solely depending on Nvidia’s dominance.