Amazon internal review found its AI tool Q fell 'significantly' behind rivals on accuracy in its first year

Amazon's AI productivity tool, Q Business, has encountered significant challenges in its inaugural year, falling notably short in accuracy compared to its rivals, as outlined in an internal document from March. This report highlighted mixed outcomes for the tool, particularly in its ability to handle tabular data and spreadsheets, leading to dissatisfaction among major clients such as Accenture, Intuit, and Smartsheet. The document revealed that Q Business faced persistent issues with longer responses and maintaining conversational flow, stating, "We face challenges with non-text data (embedded tables and spreadsheets), accuracy evaluation methods, and currently lag behind competitors in conversational experience that customers seem to be delighted with." Launched during AWS's 2023 re:Invent conference, Q Business has been criticized for its rushed rollout, with some employees expressing concerns that its underwhelming features could lead to customer attrition. Despite these setbacks, Amazon is gearing up to release a new product called Quick, which aims to integrate Q Business with other AWS offerings. Amazon's commitment to transparency and improvement is evident, as a spokesperson referred to the March document as "outdated" and asserted that updates have addressed many accuracy issues. "Our culture demands that we remain vocally self-critical as we innovate rapidly for customers," the spokesperson emphasized. The challenges faced by AI systems, including Q Business, often include the risk of generating incorrect or fabricated responses, commonly termed "hallucinations." The internal report indicated that one of the primary obstacles for Q Business was its connectors, which struggled to reliably fetch requested information, leading to inaccuracies. Clients like Intuit reported difficulties accessing their custom metadata, while Accenture faced challenges in analyzing architectural diagrams. Additionally, Q Business's conversational capabilities were noted to significantly lag behind competitors like Perplexity, which offers richer contextual insights. Staffing shortages within the accuracy team further complicated matters, with at least six product manager changes occurring in the previous year. To combat these issues, Amazon initiated a formal accuracy program in February, leading to a series of updates that included a "hallucination mitigation" feature and a response customization tool. Recent improvements to Q Business have reportedly resulted in a more effective chat experience and enhanced query response capabilities. Customers such as Nasdaq and Jabil have shared positive feedback, indicative of the tool's growing utility. Nasdaq, for example, has utilized Q Business to simplify the creation of AI applications, thereby enhancing its regulatory compliance processes. Despite these advancements, some employees within AWS remain skeptical about Q Business's long-term prospects. Concerns about Amazon's track record with business applications persist, suggesting that its strengths lie more in cloud infrastructure than in customer-facing software solutions. In response to these doubts, an AWS spokesperson highlighted the company's successes in AI offerings, stating that Amazon continues to lead in third-party evaluations of its services.