Breaking down AI chips: from Nvidia GPUs to ASICs by Google and Amazon

Nvidia has recently exceeded all forecasts, reporting remarkable profits driven by its graphics processing units (GPUs) that are particularly effective for AI tasks. However, the landscape of AI chips is diversifying rapidly. Major tech giants are now developing custom application-specific integrated circuits (ASICs), including Google's Tensor Processing Units (TPUs), Amazon's Trainium, and OpenAI's upcoming collaboration with Broadcom. These specialized chips promise to be smaller, more affordable, and less dependent on Nvidia's GPUs. Daniel Newman from the Futurum Group noted that he anticipates a surge in the use of custom ASICs, potentially outpacing the GPU market growth in the coming years. Alongside GPUs and ASICs, field-programmable gate arrays (FPGAs) also play a significant role, offering flexibility as they can be reconfigured post-manufacture for various applications, including AI, networking, and signal processing. Historically, GPUs were primarily used for gaming, but their role shifted dramatically around 2012 when researchers utilized Nvidia GPUs to develop AlexNet. This innovation marked a pivotal moment in AI, showcasing the power of GPUs for training neural networks, which excel at processing large datasets through parallel computations. In modern applications, GPUs work in tandem with central processing units (CPUs) within server systems located in data centers, powering AI workloads in the cloud. While CPUs handle fewer, more powerful tasks sequentially, GPUs are built for parallel processing, making them ideal for both training AI models and inference, where the trained model makes predictions based on new data. Nvidia and its main competitor, Advanced Micro Devices (AMD), dominate the GPU market, with their products being rented by cloud service providers like Amazon, Google, and Microsoft. For instance, Anthropic's recent deal with Nvidia and Microsoft includes a significant allocation of computational capacity on Nvidia GPUs. Nvidia's GPUs can be quite expensive, costing up to $40,000, which leads many startups to rely on them despite the high costs associated with developing their own ASICs. Custom ASICs, while initially expensive to design, offer long-term efficiency for the largest cloud providers. Google pioneered the custom ASIC approach with its TPU in 2015, initially created in response to its growing AI needs. The TPU has since contributed to significant advances in AI architecture, particularly the Transformer model. Amazon Web Services (AWS) followed suit by developing its own AI chips, Inferentia and Trainium, which reportedly offer better price performance ratios compared to other hardware. AWS is also expanding its capabilities with future generations of Trainium. The complex design of ASICs often necessitates partnerships with chip design firms like Broadcom and Marvell, which provide the necessary expertise and resources. Microsoft has also launched its own in-house AI chip, Maia 100, as companies like Qualcomm and Intel focus on creating their own specialized AI solutions. A notable segment of AI chips is those designed for on-device applications, enabling devices to perform AI tasks while conserving battery life and enhancing privacy. Companies like Qualcomm, Apple, and others are integrating neural processing units (NPUs) into personal devices, allowing for immediate, low-latency AI capabilities without relying heavily on cloud computation. Despite the crowded market for AI chips, Nvidia's stronghold remains formidable due to its established developer ecosystem and long history in the field. As the demand for AI capabilities continues to grow, the competition among chip manufacturers is only expected to intensify.