ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM

Why This Benchmark Matters Now We've grown accustomed to AI setting new high scores in 'digitally native' tasks like coding, writing, and Q&A. But when AI steps out of the lab and into the messy, complex trenches of real-world enterprise IT operations, its actual capabilities have been a black box. ITBench-AA, launched by IBM and Artificial Analysis, aims to answer this question. It doesn't test if AI can write code or chat; it tests if it can act like an experienced Site Reliability Engineer (SRE), pinpointing the root cause of a failure from a tangled web of logs, metrics, and topology maps when an alert fires. This marks a pivotal shift in AI evaluation from 'general capability' to 'professional domain实战 capability.'

What and How It Tests ITBench-AA currently focuses on SRE tasks, specifically Kubernetes incident response. Imagine an e-commerce site's frontend failing. The AI agent is 'thrown' into a sandbox containing all relevant logs, events, traces, and topology info. With no preset answers, it must explore this 'digital crime scene' like a human engineer—using shell commands—and finally submit a structured JSON diagnosis identifying the 'root-cause entities' (e.g., a specific Deployment, Service, or Pod).

Its scoring mechanism is stringent, using 'precision at full recall.' This means: First, you must find all true root-cause entities; miss any, and you score 0 for that attempt. Second, your submitted list must have no false positives. If you correctly identify one cause but also mistakenly point to an upstream distraction or co-occurring symptom, your precision drops. This rule targets the core enterprise ops need: being precise and complete—better to be cautious than wrong.

Trend Insight: The 'Last Mile' Challenge for AI Deployment The results are telling: the strongest model, Claude Opus 4.7, only scored 47%. This reveals a deeper trend: in highly specialized, context-dependent verticals with low error tolerance, current AI agents are still in a very early stage. In stark contrast to scores above 90% on benchmarks like Terminal-Bench, enterprise IT ops remains a 'tough nut to crack.'

Another counter-intuitive finding is that 'more work can mean more errors.' Google's Gemini 3.1 Pro Preview averaged 83 turns per task but scored only 30%, while the more concise Gemma model achieved 37% with 58 turns. This shows that models can fall into an 'over-analysis' trap in complex environments, misidentifying irrelevant system noise or secondary symptoms as root causes. The takeaway: For enterprise AI agents, designing efficient reasoning paths and decision boundaries may be more critical than simply scaling compute or extending thought chains.

Practical Value for Developers and Enterprises For AI developers and entrepreneurs, ITBench-AA is an excellent 'litmus test' and 'compass.' If you're building enterprise AI agents, this benchmark helps objectively assess your product's shortcomings in real ops scenarios—is it weak reasoning, or poor tool use (like shell commands)? It points to optimization directions: how to make models more 'restrained' and precise in decision-making while retaining exploratory ability.

For enterprise tech decision-makers, this report is an important 'reality check.' It shows that despite the hype around AI agents, expecting them to fully automate SRE fault diagnosis in the short term is unrealistic. A more pragmatic path is to position AI as an engineer's 'co-pilot,' initially assisting with tasks like log summarization and pattern recognition, while using this benchmark to vet the true capabilities of vendor solutions.

Counter-Intuitive Insight Most might assume that giving AI more 'thinking time' (more turns) always leads to better results. ITBench-AA's data颠覆 this: in complex diagnostic tasks, undirected deep-diving反而 increases the risk of misjudgment. This is akin to a novice doctor ordering a battery of tests for a complex case, while an expert quickly identifies the key indicators. This suggests that the core competitiveness of future advanced AI agents may not lie in 'how much they know,' but in 'knowing when to stop and how to focus.'