Salesforce’s new Co Act-1 agents don’t just point and click — they write code to accomplish tasks faster and with greater success rates

In an exciting development, researchers from Salesforce and the University of Southern California have unveiled a groundbreaking technique that empowers computer agents to execute code while simultaneously navigating graphical user interfaces (GUIs). This innovative approach, known as Co Act-1, integrates the efficiency of coding with traditional mouse and keyboard controls, allowing for faster workflows and a significant reduction in errors. Co Act-1 represents a leap forward in the capabilities of computer-use agents, which typically rely on vision-language models to interpret screens and perform tasks. While these agents have successfully tackled various tasks, they often struggle with lengthy, complex workflows, particularly in intricate applications like office productivity software. Tasks that require precise sequences of GUI interactions can lead to errors, such as mis-clicks, that derail progress. The researchers aimed to address these challenges by creating a hybrid system that combines the intuitive strengths of GUI manipulation with the reliability and precision of coding. Co Act-1 operates through a collaborative framework of three specialized agents: an Orchestrator, a Programmer, and a GUI Operator. The Orchestrator serves as the central planner, analyzing goals and assigning subtasks to the most suitable agent. For backend functions, the Programmer is responsible for writing and executing scripts in Python or Bash, while the GUI Operator handles tasks that necessitate visual interactions. This division of labor allows Co Act-1 to strategically bypass cumbersome GUI sequences, opting instead for direct code execution when appropriate. Testing on the OSWorld benchmark, which encompasses 369 real-world tasks, demonstrated Co Act-1's impressive success rate of 60.76%. This hybrid agent system notably excelled in scenarios where programmatic control was advantageous, such as multi-application workflows and OS-level tasks. For example, finding and compressing image files within complex folder structures can be cumbersome for GUI-dependent agents, but Co Act-1's Programmer can accomplish this efficiently with a single script. In terms of efficiency, Co Act-1 completed tasks in an average of just 10.15 steps, significantly fewer than the 15.22 steps needed by leading GUI-only agents. This reduction in steps not only accelerates task completion but also minimizes the risk of errors, showcasing the system's potential for more robust automation solutions. As enterprise environments often involve complex, multi-tool processes, the potential applications for Co Act-1 extend beyond general productivity. It can particularly benefit sectors like customer support, where agents interact with various tools, some of which may lack API access. Ran Xu, a co-author and Director of Applied AI Research at Salesforce, emphasized the technology’s versatility, suggesting it could enhance processes in sales and marketing as well. However, the transition to real-world applications raises important considerations regarding robustness and security. Ensuring that the Orchestrator makes sound decisions in unfamiliar environments is crucial, as is implementing strict access controls and sandboxing to prevent the execution of harmful code. While the results from the OSWorld benchmark are promising, the complexities of enterprise systems necessitate a human-in-the-loop approach for high-stakes operations. As the technology evolves, it will be vital to strike a balance between automation and human oversight to ensure safe and effective deployment.