AI Agent Benchmarks Explained: How Scores Get Inflated and How to Read Them

You end up buying a number that was measured on someone else's tasks, with someone else's setup, often on their best day. AI agent benchmarks are useful, but a single score hides how it was earned.

Picture a common setup. An online store's engineering team must choose an AI agent to handle refund and order-change chats, and two vendors send slide decks. The vendors and every score in this guide are made up, but the decision is one many teams face.

Vendor A's deck shows three numbers: 52 percent on a repository issue-fixing benchmark, 61 percent on a general assistant benchmark, and 78 percent on a tool-and-user simulation benchmark. Vendor B shows one number: 71 percent on the same simulation benchmark. Vendor A's slide adds 'Number 1 on the public leaderboard'.

The team's first instinct is to pick 78 over 71. That instinct is naive, and this guide shows why. The 52 percent coding score says almost nothing about refunds. The 78 might be the best of five runs. And a refund customer gets one try, not five.

You need six words: agent, task, scaffold, harness, leaderboard and contamination. Each one is short to explain.

• Agent: a program where a language model reads a goal and calls tools, such as a search box or a database. It reads the results and repeats until done.
• Task: one test case in a benchmark, such as one bug to fix or one customer request, with a known correct outcome.
• Scaffold: the code around the model that gives it tools, writes its instructions and runs its loop. The same model in two scaffolds is two different agents.
• Harness: the script that sets up each task, runs the agent, checks the result and adds up the score.
• Leaderboard: a public table of scores on one benchmark. Always check who ran each row and how.
• Contamination: the benchmark's tasks or answers got into the model's training data, so the model may remember instead of reason.

An agent benchmark is a frozen exam for agents: a fixed list of tasks, an environment to act in, and an automatic checker. The harness runs the agent on every task, the checker marks each one pass or fail, and the score is the share that passed.

The checker is the heart of every benchmark. A human cannot read thousands of agent runs, so each benchmark picks something a script can check. That choice decides what the score really measures, and what it misses.

Public agent benchmarks mostly fall into three types. They differ in what the agent does and what the script checks. The table below sums them up, and the next three sections open each one. Task counts here come from each original paper. As of 30 October 2026, maintained benchmarks may have newer versions, so check the current release.

It gives the agent a real code repository and a real bug report, then runs tests the agent never saw. If the agent's change makes the right tests pass without breaking others, the task counts as resolved.

The best-known paper on this type, from October 2023, built 2,294 tasks from real issues and pull requests in 12 popular Python repositories. A pull request is a proposed code change that maintainers review and merge. Each task keeps the repository as it was before the human fix.

The checker is the clever part. The human fix came with new tests. The paper calls the tests that fail before the fix and pass after it fail-to-pass tests. Tests that passed before and must still pass are pass-to-pass tests. The agent's patch, its proposed code change, is applied, and both groups run.

In that first paper, the best model the authors tested resolved only 1.96 percent of the tasks. A May 2024 agent paper reached 12.5 percent on the same tasks. This type became a headline number, and also a target, as the inflation section shows.

Back to the store. Vendor A's 52 percent here says its agent can fix some Python bugs. The team wants refunds handled by chat. This number is real, but it answers a question the team did not ask.

It asks real-world questions that need browsing, reading files or several steps, and it compares the agent's final short answer with one stored answer. The answer must be a number, a few words or a short list, so a script can mark it.

The paper that introduced the best-known benchmark of this type, from November 2023, wrote 466 questions. They are easy for people and hard for software. Human respondents scored 92 percent, while a leading model with plugins scored 15 percent at the time.

Questions come in three levels. Level 1 needs no tool or one tool, and no more than five steps. Level 2 needs roughly 5 to 10 steps and several tools. Level 3 can need long chains of actions. Answers are matched almost exactly after light clean-up, called quasi exact match, so reasoning earns no partial credit.

The authors kept the answers to 300 questions private to power a leaderboard. They also warned that the benchmark may decay over time, as questions leak into training data or web pages vanish. Vendor A's 61 percent says its agent can dig up facts. It says nothing about refund policy.

They put the agent in a chat with a pretend customer played by another language model, give it real-looking tools and written policies, and check the database at the end. This is the type closest to the store's refund desk.

The paper behind this type, from June 2024, built two domains: retail with 115 tasks and airline with 50 tasks. Each task gives the simulated user a hidden script with an identity, a goal and preferences. The agent sees the chat and its tools, but not that script.

The reward is 1 only if two checks pass. First, the final database must match the expected end state. Second, the agent's replies must contain required facts, such as a specific amount. Otherwise the reward is 0.

The paper reported that even strong function-calling agents, which call tools through structured requests, finished under 50 percent of tasks. It also found them inconsistent: in retail, the chance of passing the same task in all of 8 tries was under 25 percent. That second finding needs a new metric, which is the next section.

pass@k is the chance that at least one of k tries succeeds. pass^k is the chance that all k tries succeed. With k equal to 1 they are the same number. As k grows, pass@k can only rise or stay flat, and pass^k can only fall or stay flat.

pass@k comes from code generation research. A 2021 paper on models trained on code reported it: generate several programs, and a problem counts as solved if any one passes the unit tests. Unit tests are small automatic checks written for each problem. That fits a developer who can try again and keep the version that works.

pass^k comes from the June 2024 simulation paper. It asks how reliably the agent behaves when the same request arrives again and again. That fits a refund desk, where each customer gets one try and the store needs every try to go right.

Think of a driving test you may retake. pass@k is 'passed at least once in k attempts', so a shaky driver who keeps trying will pass. pass^k is 'passed every one of k attempts', so only a steady driver scores well. Where the picture breaks: an agent's tries on one task are not fresh starts, because the same agent often fails the same task for the same reason.

Working it out on two made-up agents

Both formulas take n, the number of tries you ran on a task, and c, the number that passed. The steady agent passes every task 6 times out of 8. The split agent passes 75 tasks every time and never passes the other 25. Both score 75 percent with one try.

from math import comb

def pass_at_k(n, c, k):
    # chance that at least one of k tries passes (n tries run, c passed)
    # math.comb returns 0 when k is larger than n - c, so this gives 1.0
    return 1 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n, c, k):
    # chance that all k tries pass (n tries run, c passed)
    return comb(c, k) / comb(n, k)

steady = [(8, 6)] * 100                  # every task passes 6 of 8 tries
split = [(8, 8)] * 75 + [(8, 0)] * 25    # 75 tasks always pass, 25 never do

for k in (1, 2, 4):
    s_at = sum(pass_at_k(n, c, k) for n, c in steady) / 100
    s_hat = sum(pass_hat_k(n, c, k) for n, c in steady) / 100
    p_hat = sum(pass_hat_k(n, c, k) for n, c in split) / 100
    print(f"k={k}  steady pass@k={s_at:.3f}  steady pass^k={s_hat:.3f}  split pass^k={p_hat:.3f}")

We ran this on Python 3.13 with only the standard library. The output was three lines. With k=1, all three numbers were 0.750. With k=2, they were 0.964, 0.536 and 0.750. With k=4, they were 1.000, 0.214 and 0.750.

What just happened? Both agents look identical on a one-try leaderboard. With four tries, the steady agent's pass@k looks perfect, yet its pass^k collapses to about 21 percent. The split agent keeps 75 percent, because its failures are predictable and you can route those 25 kinds of task to a human.

A small detail an expert will ask about: with only 8 tries per task, the counting formula gives 0.214 for k=4. If the true chance were exactly 75 percent, the long-run value would be 0.75 to the power 4, which is 0.316. Both tell the same story; more tries make the estimate steadier.

Scores get inflated in five main ways. Tasks leak into training data, answers leak into tasks, tests are too weak, scaffolds differ from the product, and only the best run gets reported. Small task sets make all of these harder to spot.

None of these needs anyone to cheat on purpose. Most come from how benchmarks are built and how results get reported. The evidence below comes from published audits, which are papers where researchers re-checked benchmark results by hand or with new tools.

Contamination and leaked answers

Issue-fixing tasks come from public code, so a model may have seen the fix during training. A June 2025 audit gave models only the bug report, with no repository at all. The best results still named the buggy file's path up to 76 percent of the time on benchmark tasks.

On tasks from repositories outside the benchmark, the best accuracy was only up to 53 percent. That gap points to memory, not reasoning. The authors call it possible contamination and ask for benchmarks that resist it.

Leakage can also sit inside the task itself. An October 2024 audit read the successful patches of one agent by hand. In 32.67 percent of them, the fix was already written in the bug report or its comments. Copying a fix is not the same skill as finding one.

Weak tests and flawed checkers

The same October 2024 audit found that 31.08 percent of passed patches were suspicious, because the tests could not show the fix was correct. After removing leaked and weakly tested cases, that agent's score fell from 12.47 percent to 3.97 percent.

A March 2025 study compared passing patches with the human fixes on a human-filtered subset of 500 issue-fixing tasks. It found 7.8 percent of passing patches failed the project's full developer test suite. It estimated that these weaknesses inflate reported resolution rates by about 6 percentage points.

Checkers in other benchmark types fail too. A July 2025 paper on benchmark rigour found a tool-and-user benchmark that counted empty replies as successes. It reported that such flaws can misstate agent performance by up to 100 percent in relative terms.

Scaffold differences

A leaderboard row is a model plus a scaffold, and the scaffold can matter as much as the model. A May 2024 paper showed that designing the commands an agent uses to view, search and edit files changes how well it fixes issues.

So ask whether the score came from the same scaffold you will buy. A research scaffold may allow long runs, many retries and a large tool budget that the product does not. A July 2024 paper on agent evaluation also argued that agents should be judged on cost as well as accuracy.

Cherry-picked runs and small task sets

Agents usually sample their outputs with some randomness, so the same agent scores differently on each full run. We simulated an agent that passes every task with a 60 percent chance, on 300 tasks, run 5 times. Over 2,000 repeats, reporting only the best run added about 3.4 points on average.

Task count matters as well. With 500 tasks and a 70 percent score, the usual 95 percent margin of error is about 4 points either way. With 50 tasks, it grows to about 12.7 points. Vendor A's 78 and Vendor B's 71 could be closer than they look.

There is one more twist. An October 2025 paper that ran 21,730 agent runs across 9 models and 9 benchmarks, at a cost of about 40,000 US dollars, read the agents' logs. It found an agent that searched for the benchmark on a public site that hosts AI datasets, instead of solving the task.

Turn the score into a set of questions, then test the agent on your own tasks. A public benchmark tells you who is worth a trial; only your own evaluation tells you who to buy.

1. 1

   Match the benchmark type to your job

   The store needs refunds by chat, so only the tool-and-user simulation score matters. Set the 52 percent coding score and the 61 percent assistant score aside.

2. 2

   Ask for the exact benchmark version and subset

   Benchmarks have subsets and cleaned re-releases, such as a human-filtered subset of issue-fixing tasks. Two scores on different subsets cannot be compared.

3. 3

   Ask how many runs and how they were combined

   Ask for the mean and the spread across runs, not the best one. Ask for pass^k with k of at least 4 for anything customer-facing.

4. 4

   Check the scaffold and the cost

   Confirm that the scored agent uses the same tools, limits and settings as the product. Ask for cost per task beside the score.

5. 5

   Ask who ran it and whether logs exist

   Self-reported scores are not the same as an independent rerun. Ask for sample run logs, so you can see how passing tasks were really solved.

6. 6

   Run a small trial on your own tasks

   Write 50 to 100 real refund and order-change chats with known correct outcomes. Run each vendor's agent several times and compute pass^k yourself.

How do you know the trial worked? You should end with a table of task, tries run and tries passed for each vendor. Feed those counts into the pass_hat_k function above. If Vendor A drops from 78 to below Vendor B on your tasks, you have learned something the slide deck could not show.

Interviewers ask about benchmarks when you propose an agent, because they want to know how you would judge it. Use this frame and keep each step to a sentence or two.

1. 1

   Define a benchmark in one line

   Say it is a fixed task set, an environment and an automatic checker, and that the checker decides what the score means.

2. 2

   Name the three types

   Issue-fixing with hidden tests, assistant questions with exact answers, and tool-and-user simulations that check a database after a chat.

3. 3

   Pick the metric for the product

   Say pass@k suits a developer who can retry, and pass^k suits a customer who gets one try. Show the 0.75 vs 0.214 example.

4. 4

   List the inflation risks

   Contamination, leaked fixes, weak tests, scaffold mismatch and best-of-many runs. Mention one audit number to show you have read the evidence.

5. 5

   Close with your own evaluation

   Say you would build a small task set from real traffic, run each candidate several times and report pass^k with a margin of error.