How Public Evals Work and Fail, and What Should Change
Contents
As models ship faster, eval scores are often the first thing we learn about a new model. But how much weight should we put in them? In eight of twelve public evals studied, there’s evidence the grader failed work that should have passed. And even when the grader gets it right, many benchmarks measure what’s easy rather than what matters, and should instead focus on reliability, work velocity, multi-turn collaboration, etc.
Here, we’ll cover the common patterns and principles, the errors models make, and where current benchmarks fall short. I’ll also share what we’re getting wrong and what should change, including why average pass rate isn’t enough and why dollars are the wrong measure of cost. Finally, we’ll walk through twelve evals: four on coding and science, six on knowledge work, and two on browsing and computer use. By the end, you’ll know what to trust in a benchmark and what to question.
Patterns: How eval tasks are built and maintained #
Most eval tasks share four components:
- Instruction: A prompt that sounds like a real user’s request. The agent gets it once at the start and typically can’t ask clarifying questions.
- Environment: The workspace where the agent operates. Typically comes with the files, software, and services needed. For coding, this is usually a container with a repo at a specific git commit. For knowledge work, it might be a VM with installed software, simulated business apps, or a folder of documents.
- Grader: This decides whether the work passes. It can be deterministic code (tests and assertions over the final output or environment state), an LLM judge scoring a rubric of yes/no criteria, or a mix of both.
- Reference (optional): A human expert’s solution. Evals include this to prove a task is solvable, anchor pairwise comparisons, or establish a human baseline.
The four basic components of an eval task
In addition, eval benchmarks are increasingly maintained like software. They ship versioned releases and accept fixes through issues and pull requests. Agents’ Last Exam v1.1 retired 30 tasks and revised 57 others, and Harvey LAB’s public set has grown from about 1,250 tasks to 2,010 since launching in May 2026. Like software, these releases can also introduce breaking changes—Terminal Bench can drop saturated tasks with each version, making scores hard to compare across releases.
Principles: That shape grading and task selection #
Across the benchmarks studied, four design principles show up repeatedly:
Deterministic checks over LLM judges. Code-based checks are reproducible and won’t shift rankings when a judge model is updated. DeepSWE and AutomationBench avoid LLM judges altogether, and Agents’ Last Exam goes to great lengths to minimize them. When an eval does need a judge—for a legal memo or clinical response—it narrows the judge’s job to binary yes/no rubric items rather than a holistic score.
Graders check the outcome, not the steps. The intent is to allow any approach that produces the right result to pass. For example, Terminal Bench inspects the final state of the container and ignores the agent’s intermediate commands or console output. AutomationBench, OSWorld 2.0, and DeepSWE work the same way.
All-pass scoring is common because partial work often isn’t usable. Under all-pass scoring, a task passes only if every test or criterion passes. DeepSWE, Terminal Bench, Terminal Bench Science, AutomationBench, and Harvey LAB score this way. As Harvey puts it, “there is no partial credit for catching most of the issues”—a report that flags eight of ten risks is incomplete, not 80% useful. Nonetheless, all-pass can be brittle where a single flawed assertion or criterion fails an otherwise correct solution.
One alternative is partial credit, where each test or criterion carries a weight and tasks score from 0.0 to 1.0. FrontierSWE and HealthBench Professional use partial credit, while Agents’ Last Exam, Finance Agent, and OSWorld 2.0 report both.
Tasks are mined for difficulty to slow saturation. AutomationBench hardened easy tasks with irrelevant data and conflicting policies. HealthBench Professional made difficult tasks about 3.5x more common (than in the base set). BrowseComp kept a question only if the strongest models at the time couldn’t solve it, and Terminal Bench retires tasks once they saturate.
Failures: Skipped verification and dropped reqs #
Across the error analyses, two failure modes come up repeatedly:
Skipping verification or stopping early. On OSWorld 2.0, agents submitted their work without checking the final state. On Harvey LAB, agents that drafted without review scored lower. And on AutomationBench, models processed part of a list, summarized as if they’d finished, and incorrectly declared success.
Missing stated requirements. Instruction following was the most common failure in GDPval, and agents on OSWorld 2.0 dropped constraints explicitly stated in the prompt. Many of these misses are unforced errors that a basic verification pass should have caught.
Gaps: Failing correct work and passing wrong work #
Not every failure is the model’s fault. Many evals have two recurring grader gaps.
False negatives (failing correct work). This is the most common issue, usually caused by a check that’s stricter than what the prompt requested, or enforces an unstated requirement. Reviewers found that 41 of 65 Terminal Bench 4.0 tasks were graded on unstated requirements, and that 23 of 113 DeepSWE tasks marked correct solutions as failures. On OSWorld 2.0, an outside reviewer filed isuses that failed correct work in 23 of 108 tasks, such as failing software loads. This is amplified under all-pass scoring where a single flawed check can cause an entire task to fail.
False negatives can also come from wrong or ambiguous answer keys, where the reference answer or criterion is wrong or isn’t unique. On BrowseComp, human trainers who solved a question disagreed with the reference answer on 50 of 367 questions (14%). A later paper confirmed errors in 10 of 200 sampled questions from BrowseComp-Plus (which reuses BrowseComp’s questions). Harvey LAB removed 17 (unsolvable) criteria from the held-out set and a contributor reported that they couldn’t find the requested damages estimate in any of the source documents.
False positives (passing wrong work). Assertions and rubric items usually check that an expected point is present, not that the rest of the work is correct. On Harvey LAB, Artificial Analysis used an LLM judge to check deliverables against the source documents, and found material hallucinations in more than 60% of the results that passed every criterion. On AutomationBench, 55% of the assertions only check whether a message or record contains or excludes a string, leaving room for an agent to match the keyword while getting the rest wrong. For example, one task asks for an email with the counts of ready and not-ready audit items, and checks only that the email contains a “3” and a “4”—any email that swaps the two counts would pass.
Hot takes: How we’re thinking about evals wrongly #
Looking ahead, here’s where I believe our thinking on evals is off, and what should change. Some of these are based on the benchmarks studied while others are vibes.
We should measure reliability () in addition to the average pass rate. Most evals report average pass rate—this hides a huge difference in production. Consider two agents that both score 70%. The first solves 70% of tasks every time, while the second solves every task 70% of the time. We can trust the first on the 70% of tasks it’s reliable on, but have to check every output from the second. (popularized by -bench) distinguishes them by counting a task as solved only if the agent passes all trials. Over five trials, the first still scores 70% while the second falls to 17%.
Reliability is what matters in production workflows, where agents have to complete tasks consistently without someone checking the work. I’d much rather pay 2x for a model that finishes a task every time than 1x for one that finishes it 80% of the time and leaves me debugging the rest. The error analyses above suggest that reliability is where the gap is. On OSWorld 2.0, agents rarely fail on basic GUI control or coding—they fail by skipping verification and dropping requirements.
Cost per task is a misleading metric; the scarce resource is human attention, not token cost. Evals typically average cost across every task attempted, pass or fail, and models can fail cheaply by skipping verification or giving up early. On Agents’ Last Exam, 17% of failures were premature abandonment, where the agent stopped early or didn’t produce the required deliverables. And on FrontierSWE, one model submitted and stopped within eight hours of a 20-hour budget in 169 of 170 trials. Agents that fail or give up often can look cost-effective on a leaderboard while stalling on real work.
We shouldn’t mistake a lower dollar cost for efficiency. Saving on token costs may be token-wise and time-foolish if an engineer has to spend days reviewing outputs, hunting down silent failures, and finishing the last mile. On AutomationBench, models declared success on 72% to 91% of the failures analyzed. When we can’t tell which tasks failed, we have to review all of them. We can see this trade-off in GDPval’s analysis, where GPT-4o cost about a tenth of GPT-5 per task, but once expert time to review and redo unsatisfactory work was factored in, GPT-4o was slower than an expert working alone, while GPT-5 was faster.
By making tasks verifiable, we lose something by testing implementation more than judgment. Deterministic assertions and binary rubrics are more reliable than vague Likert scales, and I’m glad benchmarks prefer them. Nonetheless, these checks are defined by a task author who decided what good looks like. And for a check to be fair, the prompt has to state what it tests—DeepSWE reviews each task for this (prompt-verifier bijection) because a verifier that tests unstated requirements leads to false negatives. Thus, some benchmarks measures more of instruction following and how to build (implementation), and less of what to build or not build (judgment).
Some evals go further and reframe open-ended tasks around checkable targets. For example, Agents’ Last Exam asks for tasks such as reproducing an existing game—instead of designing a new one—so the output can be compared against a reference. FrontierSWE gives the agent a self-check tool that tracks the grader’s score so the agent doesn’t have to decide what to measure. This design removes the need for agents to scope an ambiguous problem, decide what matters and what doesn’t, and exercise judgment, which is often the hardest part of real work.
Eval instructions are mostly single-shot, but real work is multi-turn. Users rarely write all their requirements upfront. They start with a rough idea, discuss, review a draft, add constraints, and course-correct over several turns. Single-shot benchmarks don’t test whether models ask clarifying questions on an ambiguous request, or how they handle evolving requirements, such as resolving conflicts, remembering earlier instructions, and rethinking a design holistically instead of stapling on late changes.
The gap can be large. Laban et al. found that models scored 39% lower on average in simulated multi-turn conversations than in single-turn ones. Still, few evals test multi-turn interactions because simulating users is hard and adds noise. One exception is OSWorld 2.0, where some tasks include a simulated user the agent can ask for clarification, and others inject messages mid-task that change the requirements. Its error analysis found that agents guess instead of asking, and overlook the messages that arrive mid-task.
We should invest more in graders, not just harder tasks. Benchmark authors spend enormous effort crafting harder problems, but the graders haven’t kept up. In eight of the twelve evals studied, audits or issue reports found graders rejecting valid work, and on some, passing incorrect or hallucinated work. This matters more as models improve, where model errors shrink while grader errors stay fixed, so a growing share of the remaining failures are grader bugs. On DeepSWE v1.1, an audit found correct solutions marked as failures in at least 20% of the tasks, mostly because tests that the agent wrote or edited clashed with the grader’s hidden tests.
Fixing false negatives starts with auditing tasks where multiple frontier models fail the same check. Catching false positives is harder, because few teams scrutinize passing runs. Evals need negative tests to verify that known-bad outputs are rejected. Terminal Bench does this with a do-nothing agent and an adversarial agent (though reviewers still found that 44 of 65 tasks in version 4 could be passed by cheating). It’s also helpful to have automated audits of passing transcripts for reward hacking, like FrontierSWE’s judge panel that reads each trajectory.
Benchmark scores reflect a model-harness pair, not the model alone. In an eval, a model always runs inside a harness that defines the system prompt, tools, compaction, retry budgets, and so on. The same model likely scores differently in a different harness, and the model provider’s harness isn’t always the best one. On Terminal Bench 2.0, one model scored 57.8% on the neutral Terminus 2 harness and 52.1% on its own provider’s harness, and another scored 32.6% vs. 19.6%. On FrontierSWE, two models worked longer and scored higher on Proximal’s own Proximus harness than on Claude Code and Codex across six tasks. For builders, the takeaway is simple—evaluate models inside your own production harness.
Diving into the twelve benchmarks #
Next, we’ll go through twelve evals: four on coding and science, six on knowledge work, and two on browsing and computer use. For each, we’ll look at how tasks are built and scored, where models fall short, and where I think the gaps are.
Terminal Bench evaluates agents on work done through the command line. The paper introduces Terminal Bench (TBench) 2.0. The current version, 4.1 (Nov 2026), has 66 tasks across software, science, machine learning, operations, security, hardware, and media.
TBench 2.0 was crowdsourced from 93 contributors who wrote 229 tasks, of which 89 were selected for the final dataset. The authors estimate that an expert can finish about half the tasks in under an hour and nearly all the rest within a day. A junior engineer might need an hour to a day for most and over a week for a few. Every task got roughly three hours of human review plus an automated QA pipeline checking three things (detailed in Appendices B2 to B4):
- Specificity: Unit tests pass only if the container ends in an acceptable state, and an LLM audits for gaps between the prompt and the tests.
- Solvability: The reference solution passes all unit tests and mirrors a real workflow, while a do-nothing agent fails.
- Integrity: An adversarial agent tries to cheat, making sure there are no shortcuts like reading future git commits.
Tasks run inside containers with internet access on Harbor, the team’s open-source harness. The team built a neutral harness, Terminus 2, whose only tool is a terminal. Once the agent stops, the grading tests are copied into the working container and run. Scoring is all-pass.

When the paper was published (Jan 2026), the top score on TBench 2.0 was 63%, nearly double the top score from eight months earlier. This led the authors to expect saturation within a year. Their error analysis identified three failure modes:
- Execution: Ignoring instructions, repeating steps, or failing to stop once done.
- Coherence: Actions drifting from reasoning, losing context, or going off task.
- Verification: Stopping early without checking the work.
Potential gaps: First, many TBench 4.0 tasks are graded on unstated requirements. Epoch AI found scoring defects in 30 of the 66 tasks, 11 of which failed correct solutions. When Preference Model rebuilt 65 of the tasks, they found 41 graded on an unstated requirement—once those requirements were made explicit (and shortcuts closed), one model’s score jumped from 63% to 89%.
Second, the score is a moving target by design. TBench is now a continuous benchmark that drops saturated tasks with each release. This helps the benchmark separate frontier models but also makes it challenging to compare scores across versions. Reporting a companion score on a fixed subset of tasks could help readers track progress over time and benchmark versions.
DeepSWE evaluates coding agents on long-horizon software engineering. To avoid contamination, every task is written from scratch by software engineers rather than adapted from pull requests, and never merged back upstream. Version 1 had 113 tasks from 91 open-source repos across TypeScript, Go, Python, JavaScript, and Rust. Version 1.1 (Jun 2026) keeps the same tasks but grades the agent’s changes in a separate, fresh container, making runs more reproducible and harder to game.
Engineers came up with the problems, built the environments, wrote the tests, and reviewed each other’s tasks along four dimensions:
- Prompt-verifier bijection (specificity): The grader checks exactly what the instruction asks for, with no underspecified prompts or overly strict tests.
- Acceptance breadth: Any reasonable implementation passes, not just the reference solution.
- Realism: The prompt sounds like a real engineering request, and a maintainer would plausibly merge the result.
- Environment cleanliness: No dependency bugs, flaky tests, or infrastructure issues.
Each task also went through LLM-assisted analysis and human review. Graders were run three times to catch flakiness, and several frontier agents attempted every task so reviewers could confirm that passes were real and that failures came from the model rather than the environment.
To avoid having the harness confound results, DeepSWE standardizes on the mini-swe-agent harness (from the SWE-Bench authors). This gives every model the same bash tool and system prompt. The grader checks the results via APIs and outputs alongside the repo’s existing test suite which checks for regressions. Scoring is all-pass and doesn’t use an LLM judge, and is reported as pass@1. (Since v1.1, runs also log per-test progress though it doesn’t count toward the score).
At launch (May 2026), the top score on v1 was 81%. Output tokens, run time, and cost per trial varied by an order of magnitude across models and didn’t correlate strongly with pass rate. (See the official leaderboard for the current scores.)
Potential gaps: Despite careful review for specificity and acceptance breadth, DeepSWE still fails correct work. First, the hidden tests can break valid solutions. Epoch AI’s review of v1.1 found this failure in 18 of the 113 tasks and rated the benchmark as “Flawed.” This happens because the grader swaps in hidden test files that clash with tests the agent wrote, causing compilation errors, which ironically penalizes agents that verify their own work.
Second, the grader is often stricter than the instruction. Across dozens of failed trajectories I reviewed, a single reasonable implementation choice—like how a data schema is nested, or whether a new class is added instead of extending an existing one—failed an otherwise correct solution.
FrontierSWE evaluates agents on ultra-long-horizon engineering and research tasks, giving each run a 20-hour budget. Version 2 (Sep 2026) has 34 tasks across five categories: implementation, performance optimization, scientific computing, visual reasoning, and AI research.
Built by Proximal with external collaborators, tasks run in a containerized Linux workspace, often with GPUs (for the ML and scientific computing tasks). Version 2 is shaped by three design principles:
- Fairness: Every workspace includes a self-check tool whose output tracks the grader score, giving the agent everything it needs to earn full credit even when the prompt alone doesn’t convey it.
- Reliable grading of performance tasks: Version 1 scored on wall-clock time. This made performance evals and model rankings unstable when the evals run on different host machines. For version 2, agents optimize a weighted instruction count, which is deterministic and machine-independent.
- Preventing cheating: The grader runs in a fresh container while the agent runs as a non-root user that can’t read the scored material. The grader also runs mutated versions of the public tests so hardcoded answers fail, and a QA judge panel reviews every trajectory for suspicious behavior.
Every model runs five trials per task at maximum reasoning effort on Proximus, Proximal’s harness. Proximus is mini-swe-agent with three additions: the ability to view images, a submit tool, and compaction. Across six tasks, models worked longer and scored higher on Proximus than on Claude Code or Codex, so model scores are not compable across harnesses.
Scoring is partial credit. Version 1 allowed performance scores from zero to infinity, which confused readers and made cross-model comparisons hard. Version 2 normalizes every task between a baseline (0.0) and a full-credit target (1.0)—this is a welcome improvement. Eval runs are expensive. Most models used 100M to 500M tokens per trial, and the cheapest model that scored above 30% averaged $97 per trial.
At launch (Sep 2026), the top score on v2 was 56%, and the second-highest was 32%. (See the official leaderboard for the latest scores.) Proximal’s error analysis found three behaviors that hurt scores:
- Narrow optimizations: Tweaking for specific workloads at the expense of overall system improvements.
- Poor use of time: Submitting and stopping within eight hours, or working until the full 20-hour budget expired without ever submitting.
- Shortcuts: Hardcoding public answers or rewriting test scripts, which the grader caught and scored near zero.
Potential gaps: Two design choices come with trade-offs. First, the ready-made self-check tool—complete with validation, performance, and reference-comparison scripts—isn’t representative of real-world work. Typically, figuring out what to measure and how to measure it is half the battle. Spoon-feeding them to the agent turns the eval into a measure of how much the agent can implement in 20 hours, and less of the agent’s judgment.
Second, capping performance tasks at a fixed full-credit target can penalize exceptional solutions. An agent that exceeds the full-credit target gets the same score as one that just meets it. Scoring the share of cost saved from the baseline (e.g., 1 - 1/speedup) could fix this. A deliverable that halves the baseline’s instruction count scores 0.5, and one that cuts it by 99% would score 0.99.
Terminal Bench Science evaluates agents on real-world computational workflows run by scientists in research labs. Version 0.1 has 70 tasks across five domains: life sciences (19), physical sciences (17), mathematical sciences (17), engineering sciences (9), and earth sciences (8).
Researchers at Stanford University and the Terminal Bench team led the effort. Contributors proposed 920 tasks on GitHub, of which 70 were selected for v0.1. Task authors estimate that a domain expert would need a median of 12 hours per task. Reviewers filtered for three criteria:
- Scientifically grounded: Drawn from a real research workflow requiring domain knowledge, not a toy problem or textbook exercise.
- Objectively verifiable: Outputs can be checked programmatically, ruling out open-ended work such as hypothesis generation or literature review.
- Genuinely difficult: Frontier models can’t yet solve the task reliably.
Tasks follow the Terminal Bench setup, giving the agent open internet access and eight hours per task. The agent produces output files such as analyses, simulations, proofs, code, or data products. When the run ends, those files are copied to a fresh verifier container, where the tests run (e.g., comparing numerical results against reference values within a tolerance threshold). For the launch results, each model ran in its own coding harness (e.g., Claude Code, Codex) across three trials. Scoring is all-pass.
Scores are low by design. The team intended for frontier models to complete 10% to 20% of tasks at release. At launch (Aug 2026), the top score on v0.1 was 30%, and the second highest was 22%. Five of the nine models evaluated solved fewer than 10% of tasks. (For the latest scores, see the official leaderboard.)
Potential gaps: Low scores aren’t always the model’s fault. First, many tasks seem to fail valid work. Since launch, 106 task-fix issues have been filed against 55 of the 70 tasks—79 of them reporting bugs in the verifier or instruction. For example, one verifier rejected correct predictions simply because rows were in a different order, and one ground truth mislabeled two non-inflamed samples as inflamed.
Second, all-pass scoring hides how close a run came. Even when a task computes a continuous metric like accuracy or error, it must emit a binary reward of 0 or 1. One task passes only if the class and period of all 100 stars are correct—a reported run got 99 right and scored zero. Publishing partial credit together with all-pass would score that task as 0.99.
GDPval evaluates models on economically valuable knowledge work. The benchmark covers 44 occupations across the top nine sectors of the US GDP. It has 1,320 tasks (30 per occupation), with a public gold subset of 220 tasks (five per occupation). Tasks take an expert seven hours on average, and some take weeks.
OpenAI recruited industry professionals with at least four years of experience (14 on average) to write tasks from their own work. Each task includes the instruction prompt, reference files (e.g., spreadsheets, documents, images), and the professional’s own deliverable as the human baseline. An OpenAI model screened each task, followed by at least three rounds of human review (five on average). Reviewers checked three criteria:
- Representative: Reflects real work in the occupation.
- Feasible: Another professional can complete it with the context provided.
- Clear: Specified well enough to evaluate the deliverable.
The models received the prompt and reference files, had access to web search and a code interpreter, and ran thrice per task. Scoring uses a blind pairwise comparison. Three experts from the same occupation review the unlabeled deliverables and rate the model’s deliverable as better, equal, or worse than the human baseline. The headline metric is the win rate—the share of comparisons that the model wins or ties.
Because general knowledge work is subjective, scores can be noisy. Human expert graders agreed with each other 71% of the time, and a grader model (trained to do the same pairwise comparisons) agreed with them 66% of the time.

Win rates peaked on tasks taking an expert under two hours and fell as tasks got longer. Clustering the experts’ feedback revealed three failure modes:
- Instruction following: Missing parts of the request, ignoring reference data, or using the wrong format. This was the most common failure across models.
- Formatting: Layout and rendering errors. For example, over half of GPT-5’s PDFs had black-square artifacts before prompt tuning.
- Accuracy: Hallucinated data or calculation errors across all models.
Potential gaps: The public rubrics may reward instruction following as much as professional judgment. In the dozen tasks I sampled, roughly half of the rubric criteria check for instruction following, such as a named section, a field, or a page limit. The rest check correctness or judgment, such as reconciling a return against a warehouse report before issuing credit. This is consistent with the paper’s finding that models lost most on instruction following.
Agents’ Last Exam (ALE) evaluates agents on long-horizon, economically valuable tasks with verifiable outcomes. Version 1 has 1,490 tasks across 55 subfields in 13 industry clusters, with 152 public tasks. Version 1.1 retired 30 tasks and revised 57 of the remaining 122. (Fun fact: One of the authors regrets the name and thinks it should have been called “Agents’ First Exam,” since it’s a first attempt at real expert workflows and not the last.)
More than 250 industry experts, recruited via an advisory committee, submitted past projects that took them days or weeks. Then, an engineering team converted the projects into runnable tasks. Every task was screened for three criteria:
- Representativeness: Matches real professional practice using the software that experts actually use.
- Complexity: An end-to-end deliverable takes an expert substantial time.
- Verifiability: Outputs can be checked deterministically. Open-ended work is redesigned around a checkable target (e.g., “design a game” becomes “reproduce this game”).
Agents work inside an Ubuntu or Windows VM preloaded with professional software and input files. The agent works through the shell and desktop until it finishes or reaches the five-hour cap. ALE doesn’t fix the harness. Any agent harness, such as Claude Code or Codex, can run with its own tools, memory, and sub-agents. ALE also adds a bridge that exposes desktop actions like screenshots and clicks to CLI-first harnesses.
Scoring is partial credit (0 to 1) against a hidden reference and is 93% deterministic. By default, ALE avoids LLM judges because judge updates can silently re-rank agents. Task authors pick comparison modes that suit the deliverable: exact/hash matches for flags, field-by-field tolerances for workbooks, surface distance to a reference mesh for 3D models, held-out test suites for code, or an expert-defined acceptance window. Only creative or perceptual outputs without a code reference (like rendered scenes or animation previews) use an LLM judge, scoped to narrow yes/no questions like “Is the texture placement and orientation correct enough to pass?”
ALE reports the mean score and full-pass rate (share of tasks that earn full credit). At launch (Jun 2026, v1), the best configuration fully passed 24% of the public tasks. Mean scores were highest in computational math and agriculture/environment (55% to 85%), followed by business and legal (50% to 55%), and lowest in education (below 25%). (For the latest scores, see the official leaderboard.) The authors identified three failure modes:
- Approach errors (47%): Understood the domain but chose the wrong strategy (30%) or gave up early (17%).
- Understanding errors (31%): Lacked domain knowledge (25%) or fabricated data (6%).
- Execution errors (22%): Output had the wrong format (10%), the implementation had bugs (8%), or GUI interaction failed (4%).
Potential gaps: First, prioritizing verifiability leads to overly specified tasks. Of the 122 v1.1 public instructions, 83 use “exact” or “exactly” for file names, paths, or labels. A skeletal animation task even lists all 19 bone names that the output should have. This tests executing a comprehensive spec more than scoping a real deliverable.
Second, many of the v1.1 public tasks are academic exercises more than industry work, deviating from the representativeness goal. About half of the tasks are scientific or research computing, often on public research datasets or published papers, like reproducing a table or classifying ECG records. This measures research-style computing as much as the professional work it aimed to measure.
AutomationBench evaluates agents on cross-application workflow orchestration via REST APIs. It has 600 public tasks, 100 in each of six domains (Sales, Marketing, Operations, Support, Finance, and HR). More than 600 private tasks are held out for the leaderboard. The tasks span 47 simulated apps.
Zapier generated the tasks synthetically with LLMs (Opus 4.6, GPT 5.3, and Gemini 3), seeding them with real customer workflows that had received negative feedback on Zapier’s Agents service (i.e., hard mining). Easy tasks were hardened by adding irrelevant data, giving similar names to incorrect records, and adding conflicting business policies. The private set received additional hardening to reduce overfitting. Every task must satisfy these criteria:
- Observable outcome: Produces at least one concrete state change, like creating a record or updating a field.
- Deterministic validation: Graded by structured checks over the final state, with no free-form criteria.
- Reasoning must commit: Any judgment call must be recorded in the final state, such as a status field.
Agents work inside a simulated business whose apps (CRM, inbox, spreadsheets) are pre-populated with records. There is no agent harness. Models run directly in a tool-calling loop with two tools: Search (keyword search over API schemas) and Execute (curl-like endpoint call), meaning the agent has to discover the right endpoints itself. Runs end when the agent stops or reaches the 50-step limit, and each task is run once.

Scoring is all-pass because businesses don’t want half-finished workflows. It is also deterministic to ensure reproducibility. Assertions run exact string matches and structural checks over the final state, including negative assertions that fail agents that over-act, such as by emailing the whole company instead of the specified recipient.
At launch (Apr 2026), the best model passed 9.9% of the private tasks. Because the leaderboard scores come from the harder private set, public scores aren’t comparable with the leaderboard ones. (See the official leaderboard for current scores.) In 72% to 91% of the failures analyzed, the models falsely declared success. The paper highlights three common failure modes:
- Not finding data: Giving up when a generic search fails, or making assumptions about where the data should live (e.g., a CRM instead of a spreadsheet).
- Missing items: Only processing part of a list (e.g., a few out of 12 emails) and then summarizing as if the entire task is done.
- Dropping exact details: Paraphrasing or omitting values that the prompt required verbatim.
Potential gaps: The assertion framework can misjudge runs in two ways. First, many public tasks are graded on unstated requirements. One review flagged 108 of the 600 public tasks: 43 for missing or conflicting task information and 65 for grading defects. Another, across ~2,600 agent runs, found checks rejecting valid work in 115 tasks, such as treating “150,000” and “150000.0” as different values. Since the private leaderboard uses the same assertion framework, it possibly shares the same defects.
Second, substring matching is a weak proxy for whether a message says the right thing. In the public tasks, 55% of the ~9,600 assertions simply check whether an email, Slack message, or record contains or excludes a string. Beyond failing valid phrasing, this can lead to false positives: 94 tasks check a message for a one- or two-digit number, such as “4”, and any date or ID that contains the digit will pass. As a result, a pass or a fail may reflect the agent’s wording instead of workflow correctness.
Harvey’s Legal Agent Benchmark (LAB) evaluates agents on long-horizon legal work. The public set (v1.3.0, Oct 2026) has 2,010 tasks and roughly 114,000 criteria. This is up from about 1,250 tasks at launch in May 2026. Leaderboards run on a held-out set of 120 tasks, five in each of the selected 24 practice areas.
Harvey started from real client matters handled by practicing lawyers and broke them into discrete partner-to-associate tasks. The matter documents—contracts, emails, firm templates—are synthetic, generated from fictional people, companies, and facts. Each task is built around three properties:
- Loose instructions: Instructions average 50 words and state what’s needed without spelling out the format or style.
- Noisy matters: Key and peripheral documents are mixed together, with issues planted across several documents.
- Atomic rubrics: Expert-written binary criteria breaking down what a partner or client would scrutinize, including facts, citations, or a dollar amount.
How LAB encodes a law firm’s workflow as an agent task
Agents work in an offline sandbox with read-only matter files using lab-core, Harvey’s harness, which runs every model in the same tool-calling loop. Agents get six tools (bash, read, write, edit, glob, and grep) plus three skills for producing Word, Excel, and PowerPoint files. A run ends when the agent calls the finish tool, stops calling tools, or reaches the limit of 200 turns.
An LLM judge grades each criterion separately as pass or fail. The judge sees only the task title, deliverable files, and the criterion—reference answers and matter files are excluded. Scoring is all-pass, reflecting Harvey’s view that “All-pass grading reflects how high-stakes legal work is reviewed in practice—there is no partial credit for catching most of the issues.”
At launch (May 2026), the top model passed just 7.1% of the held-out tasks. Third-party leaderboards aren’t directly comparable: Vals AI runs Harvey’s harness and LLM judges with some changes, while Artificial Analysis uses its own harness, judges, and metric. (See the Vals AI and Artificial Analysis leaderboards for current scores.) Harvey’s analysis tied three behaviors to higher or lower scores:
- Drafting without review: Skipping validation or revision lowered scores by 1.2 points on average, while revising after a check raised scores by 1.5 points.
- Noisy tool fan-out: Making five or more tool calls in a single turn lowered scores by 0.5 points.
- Reading before drafting: Covering more than 90% of documents before drafting raised scores by 0.4 points.
Potential gaps: The rubric can misjudge deliverables in both directions. First, LAB fails faithful work when criteria can’t be satisfied from the matter files. And under all-pass scoring, one such criterion fails the whole task. Vals AI notes that Harvey removed 17 unsolvable criteria from the 120-task held-out set. In the public set, one criterion required a meeting date that appeared only in the rubric and thus can never passed, while another asked for a damages estimate that didn’t exist in any document. Unfortunately, incorrect criteria like these punish faithful agents.
Second, LAB can pass hallucinated work because the rubric only checks that an expected point is present and the judge never sees the matter files. Artificial Analysis found material hallucinations—such as a wrong party, amount, or date—in more than 60% of the deliverables that passed every criterion.
Finance Agent evaluates agents on complex analysis of public company filings. Version 2.1 (Oct 2026) has 926 tasks across nine categories of analyst work, such as comparables, precedents, earnings analysis, and financial modeling. Of these, 27 are public, 450 form a licensable private validation set, and 449 make up the private test set.
Vals AI built the tasks with financial experts, targeting the analytical depth expected of a second- or third-year investment banking analyst. Version 2 expanded v1’s 537 retrieval-heavy tasks to 927 workflow-driven ones; v2.1 refined prompts and graders after expert reviews. Each task is designed around four criteria:
- Determinism: Has a single, unambiguous correct answer.
- Multi-source synthesis: Requires chaining information across multiple filings or data sources rather than a single lookup.
- Domain specificity: Relies on implicit industry knowledge and sector conventions rather than explicit instructions.
- Forensic precision: Critical details are often buried in footnotes, caveats, or accounting policy disclosures.
Each model runs in the same tool-calling loop with six tools: SEC EDGAR search, web search, a page parser, retrieval over parsed pages (keeping long filings out of context), a calculator, and price history. The harness pins the date to March 1, 2026 and limits searches to before that date. A run ends when the agent submits its final answer; runs that exceed the two-hour limit score zero.
The harness as drawn in the v1 paper, with four of v2.1’s six tools
Three LLMs make up a jury that grades the answer against a rubric of severity-weighted checks. Key facts or numbers are flagged as dealbreakers, and failing any dealbreaker drops the task score to zero. Scoring is partial credit and the primary metric is the severity-weighted average after dealbreakers are applied. All-pass (every check passing) is reported as a secondary metric.
At launch (Oct 2026, v2.1), the top model scored 68% on partial credit and 59% on all-pass on the private test set (run on Vals AI’s harness; leaderboard). Scores were lowest on precedents (49%), financial modeling (57%), and comparables (64%). Vals AI’s error analysis from v2 and v2.1 highlighted three failure modes:
- Missing details: Rarely answering completely, often omitting core details an analyst would include.
- Imprecise calculations: Computing one step at a time over many turns, with the final result landing close to the rubric’s targets but not close enough to pass.
- Searching without resolving: Leaning heavily on exploratory web search and retrieval calls that don’t lead to correct answers.
Potential gaps: Some rubrics grade on unstated requirements. For example, one public question asks which of three companies is the most leveraged, yet its dealbreakers included the peer averages of three ratios (until a recent rubric update). An answer with the right company but without those averages would fail. Thus, a score may reflect whether the agent matched the rubric author’s conventions as much as whether it answered the question.
HealthBench Professional (HealthBench Pro; Apr 2026) evaluates models on clinical tasks. There are three use cases: care consult (45%), writing and documentation (27%), and medical research (28%). It has 525 tasks, about a third of which come from red teaming where physicians deliberately probed the model for errors.
Tasks were built by 190 physicians across 50 countries and 26 specialties while testing ChatGPT for Clinicians. They worked in two modes: good faith (routine use) and red teaming. Each task pairs a physician-model conversation (ending on a user turn) with a physician-written baseline response and a point-weighted rubric (-10 to +10 per criterion). There are three stages to building a task.
- Authorship: A physician creates a conversation, drafts an initial rubric, and rates the model’s final-turn response from 1 to 7.
- Review: Other physicians check the rubric quality and task realism; for responses rated 1 or 2 (marking a task as difficult), two physicians confirm that the response was flawed.
- Adjudication: Physicians remove subjective criteria, keeping only widely agreed standards of practice.

From 15,079 candidates, OpenAI used stratified sampling to select the final 525 tasks, making difficult tasks about 3.5x more common than in the candidate pool to preserve headroom for future models.
During evaluation, the model (or a system like ChatGPT for Clinicians, which adds evidence retrieval) reads the conversation and writes a single response. Models ran at maximum reasoning effort with the default system message, averaged over eight samples per task.
Scoring is partial credit. An LLM grader (GPT-5.4 at low reasoning effort) judges each criterion independently as met or not met. Positive criteria add points and negative ones subtract them. The task score is the points earned divided by the maximum positive points, so it can be negative. Scores are then multiplied by 100. Because longer responses have more chances to meet positive criteria, the primary score adjusts for length, dropping 1.47 points for every 500 characters above 2,000, and rising by the same below. The overall score is the mean of the adjusted task scores.
Potential gaps: Two scoring choices limit what the results actually tell us. First, the rubrics are surprisingly thin. The 525 tasks have just 1,135 criteria, about two per task. In contrast, the original HealthBench had roughly 11 per task. An independent analysis found that 47% of tasks have only a single positive criterion. Thus, a score may reflect whether a response hits the one or two things the physicians chose to check rather than overall response quality.
Second, the length adjustment uses a single 2,000-character target for every task, whether it’s a quick consult or a full clinical note. Yet the physicians’ own baseline responses range from about 600 to 3,800 characters (10th to 90th percentile). Benchmarking length against the physician’s response might be a fairer adjustment.
BrowseComp evaluates agents on browsing the web to answer challenging multi-hop questions. The benchmark (Apr 2025) has 1,266 questions across ten topics, including TV shows and movies (16%), science and technology (14%), art (10%), and history (10%).
Human trainers wrote the questions following OpenAI’s earlier SimpleQA guidelines. Each question combines several constraints (such as a date range, a person’s background, and event details) pointing to a single, short, timeless answer (a name, title, or number). Questions had to meet three criteria:
- Challenging to answer: Frontier models at the time (like GPT-4o with browsing) couldn’t solve it, and the answer didn’t appear on the first page of five simple Google searches. Also, another person couldn’t solve it within ten minutes (loosely enforced).
- Hard to solve, easy to verify: Trainers wrote inverted questions. They started from a seed (e.g., a person, event, or artifact) and combined several broad characteristics. Thus, finding the answer might require sifting through thousands of pages while verifying it takes just a few searches.
- Likely, but not guaranteed, to have a unique answer: Because inverted questions can accidentally match more than one entity, trainers added constraints until they were fairly confident only one answer would fit.

During evaluation, the agent receives the question and is instructed to respond in three parts: an explanation, an exact answer, and a confidence score between 0% and 100%. OpenAI’s simple-evals repo sends each question to a model and grades the response. Whoever runs the eval provides the search and browsing tools.
Scoring is binary accuracy. An LLM grader extracts the final answer and checks it against the reference answer (allowing for a small margin of error for numbers). Short answers were chosen to keep grading simple and cheap. At launch (Apr 2025), OpenAI’s Deep Research answered 51.5% of questions correctly, compared to 1.9% for GPT-4o with browsing. Human trainers working without AI answered 29% of questions (and gave up on the rest after two hours).
Potential gaps: Binary scoring against a single reference answer assumes the answer is both correct and unique. But this doesn’t always hold. In OpenAI’s human study, trainers who solved a question disagreed with the reference answer on 50 of 367 questions (14%). An independent audit later found errors in 10 of 200 sampled questions, six of which had more than one valid answer. Thus, as scores climb, gaps at the top may be a reflection of noisy answer keys more than agent performance.
OSWorld 2.0 evaluates agents on long-horizon, computer-use workflows. It has 108 tasks across seven professional domains, such as research and education, creative production, and engineering and computing. A task takes a skilled human a median of about 1.6 hours.

The authors built the benchmark in three stages:
- Task collection: About 90% of tasks came from a small group of trained annotators. They drafted tasks from workflows found in Reddit discussions, online tutorials, and their own work. The rest came from practitioner interviews (expensive to scale), questionnaires (low quality), and LLM proposals (shallow and unrealistic).
- Environment setup: Tasks run on an Ubuntu VM preloaded with a persona’s files, apps, and records. The VM also has self-hosted websites (email, banking, team chat) to avoid the layout drift and anti-bot blocks of live sites. Some tasks include a simulated user that the agent can ask for clarification. Others inject emails and chat messages mid-task that change the requirements (I love this).
- Annotation and quality assurance: A coding agent generated unit tests and two annotators completed each task end to end. Then, frontier agents ran it to catch reward hacking and false negatives.
Agents interact with the desktop through screenshots and mouse/keyboard actions. The repo ships the environment, a runner, and a reference agent for each model family. A run ends when the agent declares the task done or hits the 500-step budget.
After each run, the evaluator checks the final state against a rubric averaging 27 criteria. Most are code checks. And where code can’t decide (such as whether an edited image looks right), an LLM judge applies yes/no criteria that account for 11.5% of the total score. The benchmark reports two metrics: Binary completion (all-pass, the primary metric) and Partial credit (the weighted share of criteria met).
At launch (Jun 2026), the top agent completed 20.6% of the tasks, with a partial score of 54.8%. (See the official leaderboard for the latest scores.) The paper finds that agents rarely fail on basic GUI control or coding, and instead fail in these ways:
- Missing information: Dropping constraints stated in the prompt, overlooking mid-task messages, and guessing instead of asking the user for missing evidence.
- Skipping verification: Submitting without checking the final state, and spending under 7% of the budget on recovering from and repairing errors.
- Forgetting over long horizons: Holding early details only in their reasoning context and losing them before producing the final deliverable.
Potential gaps: A failed task doesn’t always mean the agent failed. First, some checks reject valid work. An outside review of v2.1 found this in 23 of the 108 tasks. In one case, a check required a free-text form field to exactly match a short name even though the task’s own website displayed a longer name. Copying that name as shown lost points, and under all-pass scoring this failed the entire task.
Second, the environment itself can be unreliable. In the paper’s behavioral analysis (Table 20), 31% to 43% of 108 trajectories were flagged for a possible mismatch between the state that the agent reached and the state that the environment or grader recorded. This can happen due to environment resets, stale sessions, or unavailable files. As a result, a score may reflect environment reliability as much as agent capability.
Wow, thank you for reading so far! Public evals are more realistic and rigorously engineered than ever, but they’re still a ways to go. When looking at eval results, we should look past the headline number—check how tasks are graded, whether the metric captures reliability, and what the error analysis reveals. Take scores on public benchmarks with a pinch of salt. Use them to narrow the models you’ll test and make sure to eval models in your own harness.
Did I miss any of your favourite benchmarks? Do you know other gaps or hot takes? Please reach out!
References #
Bigeard, Antoine, Langston Nashold, Rayan Krishnan, and Shirley Wu. “Finance Agent Benchmark: Benchmarking LLMs on Real-World Financial Research Tasks.” arXiv:2508.00828. Preprint, arXiv, May 20, 2025. https://doi.org/10.48550/arXiv.2508.00828.
DeepSWE. “DeepSWE.” Accessed November 19, 2026. https://deepswe.datacurve.ai.
“FrontierSWE.” Accessed November 19, 2026. https://www.frontierswe.com/blog/v2.
Harvey. “Introducing Harvey’s Legal Agent Benchmark.” Accessed November 19, 2026. https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark.
Hicks, Rebecca Soskin, Mikhail Trofimov, Dominick Lim, et al. “HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats.” arXiv:2604.27470. Preprint, arXiv, April 30, 2026. https://doi.org/10.48550/arXiv.2604.27470.
Merrill, Mike A., Alexander G. Shaw, Nicholas Carlini, et al. “Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces.” arXiv:2601.11868. Preprint, arXiv, January 17, 2026. https://doi.org/10.48550/arXiv.2601.11868.
Patwardhan, Tejal, Rachel Dias, Elizabeth Proehl, et al. “GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks.” arXiv:2510.04374. Preprint, arXiv, October 5, 2025. https://doi.org/10.48550/arXiv.2510.04374.
Shepard, Daniel, and Robin Salimans. “AutomationBench.” arXiv:2604.18934. Preprint, arXiv, April 21, 2026. https://doi.org/10.48550/arXiv.2604.18934.
Sun, Yiyou, Xinyang Han, Weichen Zhang, et al. “Agents’ Last Exam.” arXiv:2606.05405. Preprint, arXiv, October 25, 2026. https://doi.org/10.48550/arXiv.2606.05405.
TERMINAL-BENCH. “TERMINAL-BENCH.” Accessed November 19, 2026. https://www.tbench.ai.
Wei, Jason, Zhiqing Sun, Spencer Papay, et al. “BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents.” arXiv:2504.12516. Preprint, arXiv, April 16, 2025. https://doi.org/10.48550/arXiv.2504.12516.
Yuan, Mengqi, Zilong Zhou, Xinzhuang Xiong, et al. “OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks.” arXiv:2606.29537. Preprint, arXiv, July 13, 2026. https://doi.org/10.48550/arXiv.2606.29537.