Apple’s upgraded AI models underwhelm on performance

Apple has unveiled enhancements to its AI models, which power various features in iOS, macOS, and other platforms. However, internal benchmarks reveal that these updated models lag behind older versions from competing tech companies, including OpenAI. In a blog post released on Monday, Apple disclosed that human evaluators rated the text produced by its latest 'Apple On-Device' model—designed to operate offline on devices like the iPhone—as 'comparable' in quality to similar models from Google and Alibaba, but not superior. Furthermore, Apple’s more advanced model, known as 'Apple Server,' which is intended for use in the company's data centers, was rated lower than OpenAI's established GPT-4o. In an intriguing twist, when assessing the models' image analysis capabilities, human reviewers favored Meta’s Llama 4 Scout over Apple Server. This finding is unexpected, as Llama 4 Scout has underperformed relative to leading models from AI organizations such as Google, Anthropic, and OpenAI in various tests. These benchmark results support ongoing reports that suggest Apple’s AI research team has been struggling to keep pace in the fiercely competitive AI landscape. Over recent years, Apple’s AI advancements have not met expectations, compounded by the indefinite delay of a much-anticipated Siri upgrade. Additionally, some customers have initiated lawsuits against Apple, claiming the company has falsely marketed AI features that are yet to be realized. The Apple On-Device model, which contains approximately 3 billion parameters, facilitates functions such as text summarization and analysis. Generally, a higher number of parameters equates to better performance in AI models. Starting Monday, third-party developers can access this model through Apple’s Foundation Models framework. Apple asserts that both the Apple On-Device and Apple Server models have enhanced tool utilization and efficiency compared to their earlier iterations and can comprehend around 15 languages, aided by a broadened training dataset that encompasses image data, PDFs, documents, manuscripts, infographics, tables, and charts.