Google says new TurboQuant compression can lower AI memory usage without sacrificing quality

Generative AI models are notorious for their substantial memory requirements, making RAM a hot commodity in the tech world. Google Research has introduced TurboQuant, an innovative compression algorithm designed to significantly decrease the memory footprint of large language models (LLMs) while enhancing processing speed and preserving accuracy. TurboQuant focuses on optimizing the key-value cache, which Google describes as a 'digital cheat sheet.' This cache plays a crucial role by storing vital information, thus eliminating the need for recalculation. While LLMs are not inherently knowledgeable, they simulate understanding through vectors that encapsulate the semantic meaning of tokenized text. The similarity of these vectors indicates conceptual relationships, and high-dimensional vectors can encapsulate intricate data, such as image pixels or extensive datasets. However, their memory consumption can slow down performance, leading to inefficiencies. To improve the compactness and efficiency of AI models, developers often resort to quantization techniques that allow for lower precision operations. Unfortunately, this often comes at the cost of output quality. In contrast, TurboQuant has shown promising early results, achieving an impressive eightfold performance boost and sixfold reduction in memory usage in certain tests—all without compromising quality. Implementing TurboQuant involves a two-step approach, starting with a unique system named PolarQuant. Traditional vector encoding relies on standard XYZ coordinates, but PolarQuant innovatively transforms these vectors into polar coordinates within a Cartesian grid. This method reduces each vector to two fundamental components: a radius, representing the strength of the core data, and a direction, indicating the data's meaning.