Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SWE-MERA turns freshly merged GitHub fixes into a continuously updated benchmark, and its Aider-based runs separate current LLMs by ability more cleanly than static sets.

desk verdict Useful dynamic benchmark infrastructure, but the core validation of gold patches is missing and the sample count is inconsistent. read the letter →

arxiv 2507.11059 v3 pith:XL6J5GZF submitted 2025-07-15 cs.SE cs.AIcs.CL

Pavel Adamenko , Mikhail Ivanov , Aidar Valeev , Rodion Levichev , Pavel Zadorozhny , Ivan Lopatin , Dmitry Babaev , Alena Fenogenova
show 1 more author
Valentin Malykh
This is my paper · ORCID
classification cs.SEcs.AIcs.CL
keywords SWE-MERAsoftwareengineeringbenchmarkLLMevaluationdynamicGitHubissuesdatacontaminationcodingagentspipeline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SWE-MERA sets out to fix two failures of static coding benchmarks like SWE-bench: models memorize old tasks, and leaderboards saturate once scores climb. Its answer is a monthly pipeline that mines real GitHub issues merged in the previous month, converts each issue plus its fixing pull request into a coding task, and filters candidates through Docker builds and an LLM judge. The paper reports roughly 10,000 candidate tasks, with a released set of a few hundred (the abstract says 728; the body text says 300), collected between September 2024 and June 2025. Running the Aider coding agent on the 2025 subset, the benchmark ranks DeepSeek-R1-0528 at 27.8% pass@1 and 40.2% pass@6, with smaller Qwen2.5-Coder models below larger ones, evidence the authors read as clear discriminative power among state-of-the-art models. If the pipeline works as claimed, it gives the field a contamination-resistant yardstick that never goes stale, plus a public leaderboard where submissions are checked against the same Dockerized environments.

What carries the argument

The load-bearing mechanism is the fail-to-pass test pair extracted from a merged pull request: the repository is pinned to its pre-fix commit, the PR's added tests are expected to fail there and pass once the gold patch is applied, while previously passing tests are expected to stay green. This converts a real-world bug fix into an executable grading signal without any human-written solution. The second mechanism is the LLM quality gate: Qwen3-32B scores each candidate task on four 1–10 criteria, and the bottom quartile on task correctness, test correctness, or test completeness is removed, which the paper says is crucial for keeping tasks from being trivial or unsolvable. Evaluation then runs the Aider agent with six independent tries per issue and reports pass@1 and pass@6 alongside diagnostics (file localization, patch generation, regression-test passage, and token-limit hits).

What would settle it

Take a random sample of retained tasks, rebuild each repository at its base commit, apply the gold patch, and run the FAIL_TO_PASS tests: if a non-trivial fraction does not go from failing to passing (or if the tests pass even without the gold patch), the benchmark's quality gate is not doing its job. A second check would ask human raters to score the same tasks on the four judge criteria; agreement below chance would mean the bottom-quartile filter is arbitrary.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a task-construction recipe: for every merged pull request that closes an issue in a recent Python repository, revert the repository to the pre-fix commit, take the pull request's diff as the gold patch, and use the tests introduced in that same pull request as the fail-to-pass signal. Tests that passed before and after the fix become PASS_TO_PASS guards. A seven-stage funnel filters these candidates by metadata, patch shape, Docker build success, end-to-end execution, and finally a Qwen3-32B judge that scores task correctness, test correctness, test completeness, and complexity; tasks in the bottom quartile on the first three scores are dropped. The authors claim the resulting monthly-refreshed set evaluates coding agents with clear discriminative power, evidenced by model rankings that track known capability orderings, and they position SWE-MERA as a contamination-resistant alternative to static SWE-bench-style benchmarks.

Load-bearing premise

The load-bearing premise is that the quality gate works: Appendix A's build check only requires pytest to report at least one passed test, and the final task filter is an LLM judge whose scores were never checked against human ratings—so nothing in the pipeline actually proves the gold patch turns the FAIL_TO_PASS tests from failing to passing.

Editorial extensions

If this is right

  • Each monthly wave uses issues merged in the prior month, so a model cannot simply memorize a fixed test set; the leaderboard stays readable as capability changes.
  • The reported 2025 rankings place DeepSeek-R1-0528 first at 27.8% pass@1 and 40.2% pass@6, followed by Devstral-Small-2505, with Qwen2.5-Coder models ordered by size—evidence the benchmark separates models the field expects to be separated.
  • Because the pipeline is public and runs on any GitHub repository, the same funnel can be pointed at Java, JavaScript, TypeScript, Go, or C++ repositories to extend coverage beyond Python.
  • The public leaderboard and submission workflow, with trajectory links and a time slider, make contamination inspectable: users can look at which tasks a model solved in which month.
  • The authors estimate the full historical collection could reach about 10,000 tasks, limited mainly by end-to-end execution effort rather than by data availability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The most direct stress test would be a human-validated sample: the paper's quality gate is a single LLM judge whose scores were not checked against human agreement, so the benchmark's floor is whatever that judge gets right.
  • The year-over-year drop for DeepSeek-R1 on 2024 versus 2025 tasks (50% to 40.2% pass@6) could mean the older tasks are contaminated or simply that the 2025 wave is harder; the paper does not separate these explanations, and a task-difficulty calibration would settle it.
  • Because the collection pipeline is public and monthly, a determined actor could pre-train on the latest GitHub issue/PR pairs and inflate a future leaderboard run; dynamic freshness lowers contamination risk but does not remove it.
  • The regression-tests column (patches passing the repository's original tests) is a cheap proxy for patch safety and could be promoted from a reported diagnostic into a formal filtering criterion, which the paper does not propose.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces SWE-MERA, a dynamically updated benchmark for evaluating LLMs on repository-level software engineering tasks. A seven-stage pipeline collects issue-PR pairs from GitHub, filters them by metadata, patch properties, and a Docker-based build check, and then applies an LLM-based quality filter (Qwen3-32B) to retain tasks judged as correct, well-tested, and complete. The current dataset contains 300 tasks collected over six months (with additional 2024 tasks for comparison). The authors evaluate ten recent LLMs using the Aider agent and report pass@1 and pass@6 metrics, claiming strong discriminative power and contamination resistance relative to static benchmarks such as SWE-bench.

Significance. If the benchmark's validation chain were execution-based, SWE-MERA would be a valuable community resource: it is updated monthly, targets contamination, is accompanied by a public leaderboard and a reproducible pipeline, and the evaluation spans a dozen recent models. The observed year-over-year gap for DeepSeek-R1 is an interesting empirical signal. However, the core validity claim depends on the pipeline actually verifying that gold patches resolve the stated issues, which the paper does not demonstrate. The contributions are therefore promising but not yet fully supported.

major comments (3)
  1. [Section 3.1 (Step 6) and Appendix A] The claimed 'rigorous quality validation' is not supported by the described procedure. Step 6 ('End-to-End Task Execution') delegates to Appendix A, but Appendix A's success criteria are only that pytest completes without errors and reports more than zero passed tests. The pipeline never verifies that the gold patch makes the FAIL_TO_PASS tests pass, nor that those tests fail at the base commit. Consequently, tasks can be included even when the gold patch does not resolve the issue or when the test patch does not cover the described behavior. This directly affects the soundness of all reported pass@1/pass@6 numbers. The paper should either add an execution-based check that (a) applies the gold patch to the base commit, (b) runs the FAIL_TO_PASS and PASS_TO_PASS test sets, and (c) requires the expected fail/pass transitions, or it should report the fraction of tasks that satisfy such a check and treat the metrics accordingly.
  2. [Section 3.1 (Step 7) and Appendix D] The LLM-based quality filter uses Qwen3-32B to score task correctness, test correctness, and test completeness, and the same model family is later evaluated in Section 5 (Table 2). The judge scores are not validated against human judgment nor against execution results; the prompt in Appendix D asks the model to judge whether the patch 'correctly solves' the problem and whether tests 'cover' it, but these judgments are not grounded in actual test outcomes. This creates a task-selection bias risk: tasks retained by the filter may be those that Qwen3 models find easy or well-formed, which can inflate Qwen3's ranking relative to other models. The paper should validate the judge (e.g., on a human-annotated sample) and should use a judge that is not from the same model family as the evaluated models, or at least quantify the robustness of rankings to alternative judges.
  3. [Section 5, Appendix C, and Section 8] The paper observes a substantial year-over-year drop for DeepSeek-R1-0528 (pass@6: 50.0% in 2024 vs. 40.2% in 2025) and for DeepSeek-R1-Distill-Qwen-32B (31.5% vs. 23.9%), while other models show smaller changes. This pattern is exactly what one would expect if the 2024 tasks overlap with the models' training data, which would contradict the Conclusion's claim that SWE-MERA establishes baselines 'free from the contamination issues.' If the gap is instead due to task-difficulty differences between years, the paper needs to control for that before drawing conclusions about model behavior. The authors should directly address whether this observation indicates residual contamination or an artifact of the task sampling process, and should temper the contamination-free claim accordingly.
minor comments (6)
  1. [Abstract vs. Section 6] The abstract states that '728 samples' are currently available, while the main text (Section 3.1, Section 6, Table 1) consistently reports 300 samples. This inconsistency must be resolved.
  2. [Table 1] There is a typo in the table header: 'beetween' should be 'between.'
  3. [Appendix C, Figure 5] Figure 5's caption says 'pass@1 and pass@5 metrics,' but the paper otherwise reports pass@6, and the surrounding text refers to pass@6. The caption should be corrected.
  4. [Table 2 footnote] The metric 'localize files' is used in Tables 2 and 4 but is never defined in the evaluation methodology. Please provide a precise definition (e.g., how 'correctly identifying files to fix' is determined).
  5. [Ethical Statement] The ethical statement claims that 'all prompts, solutions, and test cases ... have been constructed to avoid the unintentional inclusion of proprietary, copyrighted, or sensitive information' and are based on 'public domain resources.' This is inconsistent with the pipeline's use of real GitHub issues and PRs, which are typically under open-source licenses and may contain copyrighted code. Please rephrase to accurately describe the provenance and license considerations.
  6. [Section 6 (Malicious Software)] The security assessment identifies two repositories 'suitable for SWE benchmarking' that exhibit virus signatures, but it does not state whether these repositories were actually included in the final dataset. Please clarify their status and the policy for excluding such repositories.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: pipeline steps are externally grounded; the LLM-judge overlap is a validity risk, not a constructed equivalence.

full rationale

I find no derivation-level circularity in SWE-MERA. The claimed outputs (task sets, pass@1/pass@6 rankings, year-over-year comparisons) are not defined in terms of the inputs that produce them. Task collection uses external signals (GitHub issue-PR mappings, merged PRs, git diffs, Docker pytest runs), and evaluation uses the Aider agent against held-out patches; no equation or fitted parameter makes a predicted quantity equal to an input by construction. The nearest concern is Step 7, where 'we use the Qwen3-32B model to evaluate the description, patch, and associated tests' and filter the bottom quartile, while Section 4.1 and Table 2 later evaluate Qwen3-32B on the filtered set. That is a possible selection-bias or validity problem, but it is not circular: the LLM judge's scores do not determine the evaluated models' pass rates, and the paper's central claim is about the benchmark's discriminative power over many models, not about Qwen3-32B's absolute performance. The paper itself flags grading-quality limits: the Limitations state that dynamically generated problems 'pose challenges for ground truth and grading quality' and that 'automated correctness checks may yield false negatives.' Appendix A's success criterion ('the JSON report shows more than 0 passed tests') does not verify FAIL_TO_PASS behavior, but this is a missing-validation weakness, not an input-output equivalence. Citations to Qwen3, SWE-bench, and Aider are external and not load-bearing self-citations. The conclusion's phrase 'free from the contamination issues' is an overstatement relative to the evidence, but overstatement is not circularity. Hence score 0.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central claim rests on operational thresholds (star/fork cutoffs, file-count cap, length filter, bottom-quartile LLM score cutoff) and on untested assumptions about PR-issue mapping being valid ground truth, about the Qwen3-32B judge being reliable, and about temporal freshness preventing contamination. No new physical or algorithmic entities are introduced.

free parameters (8)
  • min_stars = 10
    Repository selection threshold (Step 1), chosen by hand; affects the repo pool.
  • min_forks = 10
    Repository selection threshold (Step 1), chosen by hand.
  • min_issue_text_length = 25 characters
    Filter in Step 3; arbitrary cutoff for issue title+body length.
  • max_source_files_modified = 15 files
    Patch filter in Step 4; arbitrary cap on PR size.
  • quality_bottom_quartile_cutoff = 25%
    Step 7: tasks in the bottom quartile of task_correctness, test_correctness, or test_completeness are dropped.
  • aider_tries = 6
    Number of independent attempts Aider gives to an LLM per issue.
  • aider_reflections = 4
    Allowed reflection loops per attempt on lint or test output.
  • context_token_limit = 32k tokens
    Token limit counted as 'token limit hit' in evaluation tables.
assumptions (6)
  • domain assumption A one-to-one mapping between a closed issue and a merged PR implies the PR patch is a correct solution to the issue.
    Step 2 assumes the merged PR's diff is the gold patch; if issues are mis-linked or a PR addresses multiple issues, tasks are invalid.
  • domain assumption Test files modified by the PR define valid PASS_TO_PASS and FAIL_TO_PASS tests.
    Step 4 retains only PRs touching both source and tests; test correctness is not verified by execution beyond the repository building with at least one passed test (Appendix A).
  • ad hoc to paper Qwen3-32B quality scores correlate with true task quality and human judgment.
    Step 7 uses Qwen3-32B as a judge without human validation; the same model family is evaluated in Section 5, creating a potential bias.
  • domain assumption Aider with six tries and four reflections is a fair, representative harness for comparing model coding ability.
    All results depend on Aider's search behavior; a different agent could reorder models.
  • domain assumption Restricting collection to recently updated issues prevents training-data contamination.
    The paper's own data show DeepSeek-R1 performs much better on 2024 tasks, suggesting contamination can persist; temporal freshness is not a guarantee.
  • domain assumption A Dockerized pytest environment with the standardized base image reproduces repository behavior sufficiently for scoring.
    Appendix A defines success as at least one passed test; flaky or environment-dependent tests could add noise to the evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks." pith.science (2026). https://pith.science/paper/XL6J5GZF

@misc{pith2026250711059,
  author       = {Pith},
  title        = {Pith review of: SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XL6J5GZF}},
  note         = {Machine review of arXiv:2507.11059}
}
read the original abstract

The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset. Recent studies have uncovered severe data contamination issues, e.g., SWE-bench reports 32.67% of successful patches involve direct solution leakage and 31.08% pass due to inadequate test cases. We introduce SWE-MERA, a dynamic, continuously updated benchmark designed to address these fundamental challenges through an automated collection of real-world GitHub issues and rigorous quality validation. Our approach implements a reliable pipeline that ensures quality while minimizing contamination risks, resulting in approximately 10,000 potential tasks with 728 samples currently available. Evaluation using the Aider coding agent demonstrates strong discriminative power in state-of-the-art models. We report performance across a dozen recent LLMs evaluated on tasks collected between September 2024 and June 2025.

Figures

Figures reproduced from arXiv: 2507.11059 by the authors.

Figure 1
Figure 1. Comparison of model pass@1/pass@6 metrics between two years. Error bars represent confidence [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Screenshot of the SWE-MERA evaluation platform web interface. Submission Workflow To participate in the eval￾uation and have your agent’s results displayed on the leaderboard, one should follow these steps: 1. Dataset Acquisition: Download the SWE￾MERA dataset from the Hugging Face reposi￾tory11 . 2. Agent Execution: Run a software engineering agent on the provided dataset. 3. Submission: Submit the results by creat… view at source ↗
Figure 3
Figure 3. Pass@1 results vs model size for all evaluated [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Pass@6 results vs model size for all evaluated [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of model pass@1 and pass@5 metrics between two years. Error bars indicate confidence in [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

    cs.SE 2026-07 conditional novelty 8.0 of 10

    RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substit...

  2. The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents

    cs.CL 2026-08 conditional novelty 5.0 of 10

    A survey of 1,547 papers defines the 'horizon gap' and documents that long-horizon agent research is converging on trajectory-level process signals instead of outcome-only scores.

Reference graph

Works this paper leans on

13 extracted references · 10 linked inside Pith · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. 2024. Swe-bench+: Enhanced coding benchmark for llms. arXiv preprint arXiv:2410.06992

    arXiv 2024

  4. [4]

    DeepSeek-AI. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . Preprint, arXiv:2501.12948

    arXiv 2025

  5. [5]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    arXiv 2024

  6. [6]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    arXiv 2024

  7. [7]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770

    arXiv 2023

  8. [8]

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139

    arXiv 2024

Show all 13 references
  1. [9]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: Embracing the power of reinforcement learning

    2025

  2. [10]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025 a . Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  3. [11]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024. Qwen2.5 technical repo...

    2024 arXiv

  4. [12]

    John Yang, Kilian Leret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. 2025 b . Swe-smith: Scaling data for software engineering agents. arXiv preprint arXiv:2504.21798

    2025 arXiv

  5. [13]

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, and 1 others. 2025. Multi-swe-bench: A multilingual benchmark for issue resolving. arXiv preprint arXiv:2504.02605

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.