Confidence in agentic AI: Why eval infrastructure must come first

As artificial intelligence agents make their way into real-world applications, businesses are increasingly tasked with determining their optimal roles, effective construction methods, and scalable integration. During the recent Transform 2025 event hosted by Venture Beat, industry leaders convened to share insights on leveraging AI agents to revolutionize their operations. Among the speakers were Joanne Chen from Foundation Capital, Shailesh Nalawadi of Sendbird, Thys Waanders from Cognigy, and Shawn Malhotra, CTO of Rocket Companies. Nalawadi pointed out that while the initial allure of deploying AI agents often centers around reducing labor costs, it overlooks the profound transformational benefits these technologies can bring. At Rocket, for example, AI agents have significantly boosted website conversion rates. Malhotra noted, "With our conversational AI experience, clients are three times more likely to convert when they interact through that channel." This is merely a glimpse into the capabilities of AI agents. A standout example from Rocket involved an engineer who developed an agent in just two days to automate the calculation of transfer taxes in mortgage underwriting. Malhotra emphasized the impact of this quick project, stating, "That two days of effort saved us a million dollars a year in expenses." In 2024, the company reported saving over a million hours of team member time, showcasing how AI solutions not only cut costs but also allow employees to focus on critical tasks like assisting clients with their substantial financial decisions. AI agents are enhancing the productivity of individual team members, allowing them to concentrate on high-value interactions rather than repetitive tasks. Malhotra explained, "Some of our employees managed 50% more clients last year compared to the previous year. This increase in capacity means higher throughput, more business, and improved conversion rates as our team spends more time understanding client needs rather than performing mundane tasks that AI can handle." As organizations embrace AI, there is a shift needed in mindset, from traditional software engineering to a more probabilistic approach, as highlighted by Nalawadi. This transformation requires collaboration across various teams, including product managers and UX designers. The advancements in large language models (LLMs) have improved the performance of AI agents significantly. Waanders mentioned that while earlier models required careful selection to ensure effective functionality, modern models exhibit greater reliability. However, challenges remain, such as orchestrating multiple models and managing extensive datasets to ensure responsiveness. Malhotra pointed out that orchestrating a network of agents presents an intriguing technical challenge. As companies scale up their use of AI agents, they must consider how to effectively route requests among potentially hundreds or thousands of agents that each perform distinct functions. Most companies venturing into agentic AI have begun by developing in-house solutions due to a lack of specialized tools. However, Nalawadi cautioned that organizations cannot gain a competitive edge by merely building generic AI infrastructures. The evolution of technology necessitates ongoing orchestration, debugging, and refinement beyond the initial development phase. Looking ahead, the complexity of agentic AI is poised to grow, with an increasing number of agents and use cases. Malhotra warned that this evolution will stress existing checks and balances. Businesses need to ensure robust monitoring systems are in place, particularly in regulatory contexts where human oversight is critical. To maintain confidence in the reliability of AI agents, Nalawadi stressed the importance of establishing a robust evaluation infrastructure prior to deployment. This framework should define success metrics for AI performance and provide a basis for continuous improvement. Waanders added that while unit testing is vital, the unpredictable nature of AI behavior makes it essential to simulate various scenarios to understand how agents respond under different conditions. In summary, as AI technology advances, organizations must prioritize the development of evaluation infrastructures to ensure that their AI agents can grow and adapt reliably in response to evolving challenges.