AutomationBench

AutomationBench evaluates AI agents on complex, cross-application business workflows using realistic simulated SaaS environments and REST APIs.

AutomationBench is an open benchmark designed to assess AI agents' ability to execute end-to-end business workflows across various applications. It utilizes environments exposing 47 simulated SaaS applications (CRM, email, calendars, spreadsheets, etc.) and requires agents to autonomously discover and operate REST API endpoints. Tasks span six business domains: Sales, Marketing, Operations, Support, Finance, and HR, drawing on real workflow patterns from Zapier's platform, which processes over 2 billion AI tasks monthly. The benchmark grades purely on the final environment state (end-state correctness) using deterministic assertions, not on the agent's prose or subjective judgment. Even top frontier models currently score below 10%, highlighting the significant gap between current AI capabilities and the needs of complex business automation.

https://zapier.com/benchmarks