Some elite AI researchers say language is limiting. Here's the new kind of model they are building instead.

In a groundbreaking move, a select group of elite AI researchers is venturing beyond traditional large language models to develop what they term "world models." This initiative is spearheaded by notable computer scientists like Fei-Fei Li, a Stanford professor renowned for her creation of ImageNet, and Yann LeCun, the outgoing chief AI scientist at Meta. Unlike conventional language models that rely on statistical relationships between words, world models are designed to simulate human-like mental constructs to predict outcomes. As Fei-Fei Li explained in a recent podcast, humans navigate life, not just through language but by building complex civilizations. These models aim to replicate that intuitive understanding, allowing AI to anticipate events much like a child learns that pushing a toy car will cause it to roll. Li, alongside co-founder Jay Wright Forrester, is channeling her efforts into World Labs, which received an initial investment of $230 million from prominent venture firms. Their objective is ambitious: to transform AI from the 2D realm of pixels into rich 3D worlds, enhancing spatial intelligence akin to human capabilities. Li envisions applications for these world models across various fields, including robotics and creative industries, where limitless possibilities exist. However, the path to creating effective world models is fraught with challenges, notably the scarcity of relevant data. According to Li, while language has been carefully honed over time, spatial intelligence remains underdeveloped. She likens the difficulty of verbally conveying our 3D surroundings to the challenge AI faces in building accurate representations. LeCun, who plans to launch his own venture, Advanced Machine Intelligence, believes world models can outperform large language models by incorporating common sense, reasoning, and memory. He envisions a future where AI systems not only comprehend the physical world but can also plan intricate sequences of actions. In parallel, startups like Moonvalley are making strides in this area, developing generative AI video technologies that prioritize world modeling. Their goals align with the vision of creating systems capable of understanding and simulating the complexities of the real world, moving beyond simple visual tasks. This new frontier in AI research could redefine how machines interact with and understand their environments, paving the way for innovations that extend into humanoid robotics and beyond.