Artificial Analysis Coding Agent Index v2: what the new scores mean

What changed in v2

The index replaces DeepSWE v1.1 with Cognition’s FrontierCode v1.1. It also changes the grading of repository questions, now labeled SWE-Atlas-QnA-AA. Launch details

The three components cover different kinds of work:

Three attempts are averaged per task, not scored best-of-three. Tasks have equal weight within their component; the index weights each component equally. Scoring methodology

The headline 61.7 is a composite score, not the percentage of 290 tasks merged or your ticket’s probability of success.

FrontierCode gives zero when a required blocking check fails. Clearing those checks yields a score between 70% and 100%, depending on the remaining criteria. This is an automated assessment against maintainer-oriented requirements, not a claim that a human maintainer personally approved each patch. Methodology

For a product developer, the useful question is whether an evaluation notices the failures your existing tests miss. A patch can satisfy a narrow assertion while mishandling a neighboring state, expanding scope unnecessarily, or leaving the next developer with difficult code. A more demanding rubric is directionally useful, but your own requirements still decide whether the work ships.

A new grader can move an old score

The revised Q&A benchmark keeps its 124 questions and 1,306 criteria. Artificial Analysis reworded 127 criteria and changed the judge to GPT-6 Luna at xhigh, using five votes per criterion. Every criterion must pass for the answer to count as resolved. Its published dataset distinguishes reported runtime observations from conclusions inferred by reading code. Dataset and grading details

Across 21 configurations evaluated under both versions, Artificial Analysis reports that the mean Q&A score fell from 58.8% to 57.1%. Announcement

Read that as a warning about comparing versions. An agent’s position can change when the measuring process changes, even without a new model release. Treat a screenshot of an older index as historical context, not a clean before-and-after capability test.

There is also a practical lesson for your prompts. “Explain how this works” and “run it and demonstrate how it works” require different evidence. If a task needs an observed result, make that part of acceptance before the agent starts. A convincing explanation should not accidentally stand in for a test you needed it to run.

The cost frontier is a shortlist

Artificial Analysis identifies these four configurations on its launch cost frontier:

Source: Artificial Analysis’s October 22 announcement. This is a selected cost-quality comparison, not the complete ranking.

The costs use pay-per-token API pricing. They are not consumer subscription prices. Cache treatment affects effective spend, and the figures do not include a developer’s review time. Live benchmark and cost notes

These are model-plus-harness configurations at specified effort settings, not isolated base models. Results need not transfer unchanged to another tool. Agent settings

For a team paying through subscriptions, the immediate constraint might be usage limits or waiting time rather than a token invoice. For an API-driven workflow, the price difference may matter immediately. Neither arrangement tells you how much human cleanup the result needs.

I would use this table to choose two or three plausible configurations, then investigate the component closest to the work. Repository comprehension, terminal operation and implementation deserve separate consideration. There is no reason your best choice for a codebase question must also be your best choice for a difficult patch.

Run a comparison your repository can answer

A small, repeatable local evaluation is more useful than an afternoon of switching models until one produces an impressive answer. Here is a practical starting point, not a claim of statistical certainty.

1. Choose representative work before choosing a winner. Include a bug with a reproducible failure, a change crossing more than one module, and a question that requires tracing the repository. Add a visual task if interface quality matters to your product. Avoid selecting only tasks already known to flatter a favorite agent.

2. Write down what acceptance requires. Specify behavior, regression checks, permitted scope and any manual inspection. For a mobile layout, that might include particular viewport widths and checking the actual rendered page. For a migration, it might include rollback behavior and existing-data handling.

3. Keep the starting conditions comparable. Start from the same commit in separate worktrees. Record the agent version, model, effort, available tools, instructions and time or spending limit. If one run receives extra context or human help, record that intervention rather than treating the runs as equivalent.

4. Preserve failures and repeat trials. Keep unsuccessful attempts alongside successful ones. Record timeouts, missing evidence and changes that only appeared correct. A small trial set can reveal failure modes; it cannot support sweeping claims about every repository.

5. Review the patch separately from the explanation. Inspect the diff and run the acceptance checks. Where practical, review without seeing which configuration produced it first. Count the time spent finding problems, requesting corrections and verifying the final result.

6. Make the decision at the workflow level. Compare accepted changes, total spend and review burden. For API work, a useful internal measure is total API spend plus estimated review cost, divided by accepted changes. Keep failed attempts in the numerator. If no changes were accepted, report that failure rather than forcing a cost-per-success number.

That last measure is a proposed team metric, not Artificial Analysis’s formula. It makes the tradeoff explicit: a more expensive run can be worthwhile if it removes enough rework; a cheaper configuration can be sufficient when the task is well bounded and verification is straightforward.

Use the ranking to improve the next decision

The strongest reason to follow this release is the chance to ask better questions about agent output. Which part of the task was measured? What evidence counted? How much help did the configuration receive? What would make your team reject the result?

Codeck’s role is modest: it helps you switch between Codex and Claude Code tasks on your Mac. It does not certify the output or improve a benchmark score. Keep your comparison reproducible, your evidence easy to inspect, and your acceptance criteria independent of the agent’s confidence.

Then choose the configuration that earns its place in your workflow.