Meet Google DeepMind's SIMA 2: AI that learns to play new games on its own

On Thursday, Google DeepMind unveiled the latest iteration of its gaming-centric AI, the Scalable Instructable Multiworld Agent, or SIMA 2. This enhanced version builds upon its predecessor released in March 2024, showcasing significant improvements in reasoning, adaptability, and user interaction. SIMA 2 is designed to learn continuously, enhancing its capabilities through self-directed gameplay. DeepMind emphasized that this AI can reflect on its actions and methodically consider the steps needed to achieve its objectives. Powered by Google’s Gemini models, SIMA 2 interprets human instructions, comprehends tasks, and strategizes its next actions based on the virtual environment it encounters. The agent processes visual information from a 3D gaming world alongside user-defined goals, such as "build a shelter" or "find the red house." It subsequently deconstructs these goals into smaller, manageable tasks and executes them using controls that mimic a keyboard and mouse. One of SIMA 2’s standout features is its enhanced capability to navigate unfamiliar gaming environments. In tests conducted in settings like Minedojo, a variant of Minecraft, and ASKA, a Viking-themed survival game, SIMA 2 achieved higher success rates than its earlier version. Moreover, the system can handle multimodal prompts, including sketches, emojis, and various languages. It demonstrates the ability to transfer knowledge from one game to another; for instance, skills learned in mining within a sandbox environment can inform its approach to harvesting in a separate survival game. The second-generation agent is trained using a combination of human demonstrations and automatically generated annotations from the Gemini models. When SIMA 2 acquires a new skill in a fresh environment, that experience is incorporated back into its training pipeline, reducing reliance on human-labeled data and enabling the agent to refine its abilities over time. However, there are still notable challenges. The AI's memory of past interactions is limited, long-range reasoning involving multiple steps remains complex, and precise low-level control akin to robotic movements is not currently addressed. DeepMind clarifies that SIMA 2 is not meant to be a gaming assistant; rather, it serves as a testing platform for AI agents aimed at controlling real-world robotics. The broader vision is to develop versatile machines capable of following natural language commands and managing diverse tasks in intricate physical environments.