The Benchmark Meaning Gap

Katie Fraser Mikhail Evtikhiev Sergey Titov

In any recent model comparison, you’ve probably seen a single number from a coding benchmark to represent “How good this model is at coding”. Those scores are tempting: they’re simple, they’re leaderboard-friendly, and they seem to tell a clear story about progress. 

But that story didn’t hold up when we looked more closely at benchmarks that are considered standards. The models dramatically improved on the exact tasks we trained them on, yet those gains often did not show up on other benchmarks or on slightly different tasks in the same codebase. This blog post is about our research on that meaning gap between what coding benchmarks measure and what we wish they measured – and how to improve these benchmarks. This paper will be presented by our team at the Deep Learning for Code (DL4C) workshop, co-located with the International Conference on Machine Learning (ICML) this July in Seoul.

Specifically, we will:

• Compare common benchmarks currently in use.
• Present our own empirical comparison and discuss why these common benchmarks fail.
• Propose solutions for better evaluation and argue for sustained benchmark maintenance.

Models and benchmarks

Before we begin with comparing benchmarks, we will first zoom out to define foundation models, or the systems being evaluated by the benchmarks. These are AI systems trained on massive datasets to acquire general-purpose capabilities rather than being specialized for a single task. Large language models (LLMs) are a type of foundation model – and arguably the most common type at that. Foundation models, however, can be any type of system, not limited to language.

Instead of building a new model for each use case, you start from a single, powerful foundation that can be steered toward chat, coding assistance, search, or domain-specific agents with comparatively little additional data. In creating the model, there is a preliminary, pre-training phase, where models are trained on large amounts of data, typically a blend of web text, documentation, code repositories, and other data sources. 

Then they are adapted in the post-training phase, where a variety of approaches can be applied to customize the model. These include:

• Reinforcement learning
• Time-test scaling
• Fine-tuning

Crucially, in the post-tuning phase, the model no longer trains on a large dataset, but rather on smaller sets of tokens relevant to the approach and overall goals. For more details, this paper offers a recent look at common post-training methods for LLMs.

Beyond training the models, it is essential for model creators and researchers to evaluate their capabilities, such as coding, common-sense reasoning, and natural language understanding, which covers tasks such as translation, question answering, and summarization. Benchmarks are central to the evaluation process because they turn these capabilities into quantitative scores that reveal where a model is strong and where it still struggles. Those scores then shape how researchers fine-tune and improve models, helping decide which training strategies are worth pursuing and which aren’t. However, benchmarks come in very different flavors, and each captures only a thin slice of “coding ability”. In the rest of this section, we show that benchmark results should be viewed with caution – and not necessarily as comprehensive evidence of a model’s coding capability.

Common coding benchmarks

Coding benchmarks differ a lot in what they actually test, even when they all claim to measure “code generation”. At one end, you have self-contained code tasks like HumanEval (GitHub), where the model sees a short problem and must produce a function that passes a handful of tests. There are adjacent examples like LiveCodeBench, which add self-repair, code execution, and test output prediction to code generation. At the other end are repository-level benchmarks such as SWE-bench (website), where the model is dropped into a real codebase and asked to resolve GitHub issues by navigating files, understanding context, and synthesizing patches that make the full test suite pass. We will now look at these in a little more detail.

Within the self-contained type of benchmarks, HumanEval is a small, function-level benchmark. Namely, it uses 164 hand-written programming problems, each with a prompt, a Python function signature, and hidden unit tests. It also focuses on short tasks such as write me a quicksort and produces corresponding reports, such as pass@k. Benchmarks such as HumanEval are great for quickly checking whether a model can correctly synthesize snippets of code from a description, but they tell you little about how that model behaves inside a large, messy codebase.

LiveCodeBench is also a self-contained type, but with a different emphasis: it continually collects new problems from competitive programming sites like LeetCode, AtCoder, and Codeforces, and it explicitly focuses on contamination-free, time-segmented evaluation. Beyond plain code generation, it includes scenarios like self-repair, code execution, and test output prediction, so it can tease apart different code-related capabilities instead of compressing them into a single number. Compared to HumanEval, LiveCodeBench is larger, evolving, and more holistic.

As a repository-level type of benchmark, SWE-bench sits at almost the opposite end of the spectrum. Instead of single functions, it gives the model a full repository snapshot plus a real GitHub issue and asks it to modify the code so that the project’s own tests pass again. Tasks are multi-file, involve navigating, reading unfamiliar code, and producing patches that integrate cleanly, which is much closer to real software engineering. 

However, SWE-bench also bundles many skills together: repository understanding, search, patching, and even how well an agent uses a particular scaffold or toolchain, so a good score may reflect proficiency with that specific workflow rather than broad coding ability. Additionally, SWE-bench is just asking whether one GitHub issue was fixed; compare, for example, LiveCodeBench, which gives a more holistic result. 

The evaluation approaches of these code-oriented benchmarks basically assess different capabilities, so we can expect that their strengths and scores will differ. High performance on self-contained tasks may not translate to strong results on repository-level evaluations, which additionally demand navigation of large codebases, reasoning about dependencies, and the synthesis of precise, context-aware edits. In other words, no one type of benchmark’s results can generalize to prove a model’s overall performance.

The limitations of model evaluations in previous studies

Evaluating foundation models comes with an inherent limitation related to their breadth: due to time, as well as the need to deliver concise and readable reports, there lies a natural tension in evaluating such a large scope. For a model-specific example, we can look at the Qwen3 technical report, which has three benchmarks with self-contained algorithmic tasks and another self-contained execution-reasoning-oriented benchmark. With only four benchmarks reported all of them self-contained, and potentially limited correlation across the assessed tasks, we cannot assume that the benchmarks comprehensively evaluate the model.

For post‑trained models, these limitations become even more acute. Unlike full foundation-model releases, post‑training work has to squeeze results into tight conference page limits, absorb higher compute costs for each additional evaluation, and deal with the engineering overhead of wiring up extra evaluation harnesses. In practice, that makes “one benchmark only” not an exception, but the default choice.

This leads to a predictable pattern. Namely, if a project targets repository‑level software engineering, it usually reports results only on SWE‑bench or closely related variants (like in this study or this study). If a project focuses on more general post‑training techniques, it typically evaluates on self‑contained suites like HumanEval instead (like in this study or this study). 

Unsurprisingly, with only one benchmark, it is difficult to tell whether a reported gain reflects narrow, task‑specific optimization or a genuine, broader improvement in coding ability. And it has become common practice to apply SWE-bench to measure “real-world coding capability” of a model. While there are indeed some associations between improved foundation models and improved coding capabilities, we are still missing the whole picture.

It is impossible to deeply analyze the results to see if they can generalize or even cross-check across tasks. This is because SWE-bench bundles together many factors: repository navigation, issue understanding, patch synthesis, and adherence to a specific workflow. We also know from the previous subsection that SWE-bench is only one type of benchmark, repository-level, and that it therefore misses some insights that a self-contained benchmark might catch. In this way, the current common evaluation practices are not necessarily working towards improving models, but rather optimizing for the benchmark itself based on task- and distribution-specific evaluations.

The meaning gap between benchmark measurements and capability claims

In our paper, we call the disparity between benchmark scores and a model’s actual performance a meaning gap. We argue that the meaning gap is actually a construct validity problem. This is an important concept in research and concerns test validation, or whether a test is actually evaluating what it should be evaluating. A benchmark is valuable as an evaluation method to the extent that it clearly and correctly defines the taxonomy of what it measures, which in this case is “general coding capability” (see also this paper on defining parameters when creating benchmarks). 

We recognize that there are practical limitations leading to many researchers and model creators evaluating models with a small number of benchmarks, and there is a tendency to claim that such scores reflect a model’s overall coding capability. All of which distorts the perception of a model’s performance, creating a meaning gap between performance claims and reality. On top of that, scientific papers and general audiences continue to amplify these claims, creating a never-ending and inaccurate cycle.

There are many real-world examples of the gap, such as those described above. But how can we actually find out whether benchmark gains correspond to real improvements in coding ability? One option is to test for transfer across different task types. In our study, we propose exactly this kind of stress test.

The meaning gap empirically tested: Preliminaries

To test the meaning gap, we created a benchmark suite to measure performance across task modalities, with the idea of explicitly assessing the cross-task transfer capability. In this subsection, we will describe the setup of our experiments. More technical details can be found in the following resources:

• All benchmark details: Paper appendices
• Benchmark replication package: GitHub
• Datasets: HuggingFace
• Checkpoints and logs: Zenodo

As noted above, it has become common practice to use SWE-bench as the standard benchmark. SWE-bench Verified is a subset of the full SWE-bench benchmark, and it was co-developed by the authors of SWE-bench and OpenAI. It comprises 500 problems from the full SWE-bench, all annotated by professional software developers.

The annotators verified that the issue descriptions are well-specified and unit tests are appropriately scoped. SWE-bench Verified became an academy and industry reporting standard for the SWE-bench benchmark family (although SWE-bench-lite and full SWE-bench are also used sometimes). Because of its status as a standard, we were interested in comparing the reported SWE-bench Verified scores of various models.

In our experiments, we built a new benchmark suite on top of a single, real-world Django repository and then used it to probe how well different code models and fine-tuned checkpoints actually transfer across tasks. Django was an obvious choice, as it makes up almost half of SWE-bench Verified. 

This setup gives us a simple, testable hypothesis: if SWE-bench-optimized checkpoints genuinely boost transferable, repository-level software-engineering ability, then their scores on Django-focused tasks drawn from the same codebase should be similar. By evaluating these checkpoints on these Django tasks, which are deliberately chosen to be close in distribution to SWE-bench, we can directly check whether these checkpoints also improve on tasks other than issue resolution.  

For the testing, we constructed three kinds of tests: (i) method generation, (ii) method completion, and (iii) program repair. For each of these, the task was the following:

• Method generation: Fill in a method body from its signature and docstring, given the source file (where the method body has been removed).
• Method completion: Finish a partially written method in the chat format.
• Program repair: Given a corrupted method and a failing test stack trace, restore the correct implementation.

After pairing each benchmark with a corresponding instruction-tuning dataset (same task template), the dataset contained 3,180 train samples and 359 test samples. We evaluated all tasks by running the official Django 4.0.4 test suite with deterministic, greedy decoding, and we counted a task as solved only if the tests passed on the first try. 

In addition, we evaluated the same models on LiveCodeBench, using 341 tasks from there, and again greedy decoding. We wanted to validate code-generation capabilities on self-contained tasks, to compare with our repository-level Django tests. In other words, our results compare the reported SWE-bench scores with our own test suite and the LiveCodeBench tests; these are presented below.

For the evaluations themselves, we targeted three groups: (i) foundation models, (ii) public SWE-bench–oriented checkpoints, and (iii) our own task-specific fine-tunes. The details of these categories are in the table below.

In setting up this experiment to test the meaning gap, we were careful to create a testable framework, with an emphasis on cross-task transfer evaluation. We chose a dataset and setup that allows a straightforward testing of our hypothesis.

Namely, if models with high SWE-bench Verified scores perform worse than foundation models on our suite, it shows that SWE-bench Verified is, in reality, a weak predictor and not a good indicator of general coding capability, beyond one dataset or a limited set of tasks. In the next subsection, we describe the results of our experiments.

The meaning gap empirically: Results

Our experiments show that there is indeed a meaning gap, or a difference between performance claims and reality. We can summarize the results by differentiating between the three evaluation groups, as follows:

1. Foundation models: Rankings are not always stable across tasks.
2. Public checkpoints: No cross-task transfer.
3. Our fine-tunes: Strong within-task gains, but weak cross-task transfer.

For (1) foundation models, we observed that leaderboard rankings are not stable across tasks. That is, a model that looks stronger on SWE-bench can underperform a competitor on our Django code-completion benchmark, even though both are ostensibly measuring “coding ability.”

For example, while Qwen-2.5-32B-Coder-Instruct outperforms Qwen3-32B on our code completion benchmark, it has worse SWE-bench scores. The results for this evaluation group are displayed in the table below, with relevant cells colored grey (darker for higher score; lighter for lower) to highlight the example instability.

For (2) public SWE-bench-optimized checkpoints, the pattern was even clearer. These checkpoints achieved large improvements on SWE-bench relative to their base models, but when we tested them on our Django suite and LiveCodeBench, they showed little consistent improvement and frequent regressions. The table below shows the results for this evaluation group, with cells highlighted to indicate degradations (orange) and improvements (green).

The table shows that degradations (in orange) outnumbered gains (in green), and no single checkpoint improved across all four non-SWE-bench benchmarks. Only DeepSWE-Preview (on the bottom row of the table) demonstrated improvements more often than degradations.

For (3), our own fine-tuning experiments were designed to test a more controlled scenario: we took strong base code models and fine-tuned each one separately on just a single Django task (generation, completion, or repair). These results can be seen in the table below, with the same color-coding as in the previous table.

Within the task we trained on, fine-tuning reliably boosted performance; every fine-tuned model became significantly better at its specific target modality. However, when we evaluated these same checkpoints on the other Django tasks and on LiveCodeBench, we again saw a pattern of weak or negative transfer. And improvements did not systematically carry over to untrained tasks. 

Taken together, these results support our meaning gap claim. That is, current post-training recipes and benchmark-optimized checkpoints give us highly specialized models that excel on a narrow evaluation, but those gains largely fail to translate into broader, more general coding capabilities. These results demonstrate that coding benchmarks (as they are currently reported) measure task-specific performance rather than general coding ability (as is often claimed claimed). 

How we can evaluate models more accurately

Our experiments showed that there is indeed a disparity between performance claims and reality. We think that there are numerous, complementary ways to address this meaning gap.

First off, benchmark suites provide broader coverage than a single benchmark can. Of course, this requires a higher effort by the community, especially in maintenance. We believe this is a systemic problem, however, and that there should be more institutional support for maintenance, e.g. through industry sponsorship or conference tracks rewarding ongoing maintenance.

Benchmark suites with broader coverage, of course, can also face the construct validity problem, leading again to a meaning gap. A robust benchmark suite needs to answer several questions, including:

• Which exact behaviors will count for coding capability?
• Which distributions matter, and how do we handle contamination risks?
• How can we report a performance profile instead of a single score?

Again, defining such parameters requires many more resources, but we know from other fields that this is not impossible for software engineering to achieve, too. There are already benchmarks being developed with clear taxonomies. For example, SWE Compass creates an explicit taxonomy of coding tasks and failure modes, providing a deeper analysis of model performance.

Another possible approach is a holistic open-ended evaluation, which can include unexpected tasks. Anthropic’s Project Vend is a good example of this: as the project unfolded in real time, they were able to judge outcomes and user experience instead of just optimizing for specific, predefined tasks. In other settings, this could look like model vs. model competitions with standardized rules, even potentially sponsored by the models’ developers. 

With this meaning gap research, we call the community to action: help make benchmarks more accurate – and more reliable. We should endeavor to fund alternative evaluation approaches such as those outlined above, as well as the long-term maintenance of benchmarks.

And for individual developers and small teams adopting AI: Don’t fall into the meaning gap! Treat coding benchmarks as probes of specific behaviors, not as verdicts on “how good a model is at coding.” Use them, but pair those numbers with diverse tasks, real-world checks, and an honest look at what you actually want the model to do.

It’s quite possible the tasks you care about are not covered well by any of the existing benchmarks. While creating your own benchmark is too much of a hassle, you can try various AI models on a set of your daily tasks to find which model and setup works best. This vibe-style evaluation may tell you more than a rigorous benchmark would, with tasks completely irrelevant to you.

If you’d like to try our benchmark suite yourself: