Artificial Analysis releases Coding Agent Index across three benchmarks

Artificial Analysis released the Coding Agent Index, a benchmark suite measuring how model-and-harness combinations perform on real-world coding tasks. The index spans three separate benchmarks designed to test different aspects of agentic coding: SWE-Bench-Pro-Hard-AA (150 realistic coding tasks sampled from Scale AI's SWE-Bench Pro, representing problems that frontier models struggle with), Terminal-Bench v2 (84 agentic terminal tasks from the Laude Institute covering system administration, cryptography, and machine learning workflows), and SWE-Atlas-QnA (124 technical questions developed by Scale AI requiring agents to explore codebases and explain code behavior and root causes).

The index compares model-and-harness pairings rather than ranking models in isolation, acknowledging that the same base model can perform very differently depending on the agent framework. A model that excels at SWE-Bench-Pro-Hard-AA with one harness might fall behind on Terminal-Bench v2 with another. The leaderboard at artificialanalysis.ai/agents/coding-agents breaks down solve rates by benchmark and pairing, helping practitioners choose stacks that ship working code in their specific contexts. The release reflects a shift in coding agents from research demos to production tooling, where developers building on Claude, GPT-4, or open-weight alternatives need apples-to-apples comparisons that go beyond single-turn code generation—tasks that mirror real repositories: ambiguous requirements, multi-step workflows, and unfamiliar codebases.