DeepSeek tests “sparse attention” to slash AI processing costs

Have you ever noticed that ChatGPT tends to lag during extended conversations? This slowdown can be traced back to a critical mathematical challenge: managing long text sequences necessitates substantial computational power, even with the efficiency strategies that are already in place. While major U.S. tech firms can easily invest in additional hardware, Chinese AI startup DeepSeek faces unique pressures due to export restrictions that limit access to advanced AI chips. Consequently, the company is driven to optimize performance while using fewer resources. On Monday, DeepSeek unveiled an experimental version of its latest simulated reasoning language model, named DeepSeek-V3.2-Exp. This model introduces a novel approach called "DeepSeek Sparse Attention" (DSA). This technique is a variation on a computational method that has already been implemented by some of the leading AI models globally. Notably, OpenAI pioneered sparse transformers in 2019 for its creation of GPT-3, and Google Research explored similar methodologies in its 2020 "Reformer" models. However, the current usage of sparse attention by Western AI companies remains largely under wraps. Although sparse attention has been known for several years, DeepSeek asserts that it has achieved "fine-grained sparse attention for the first time" with its new model, which has resulted in a 50% reduction in API costs as evidence of its efficiency improvements. To appreciate the significance of DeepSeek V3.2, it’s essential to revisit the company’s recent achievements. Earlier this year, DeepSeek's R1 simulated reasoning model reportedly matched OpenAI's performance while requiring only $6 million for training. Additionally, its chat application recently topped the iPhone App Store charts, even surpassing ChatGPT. The spotlight is firmly on DeepSeek as it challenges some of America's top AI research labs.