{"id":"7d960533-fc06-408b-a12e-45a0cde54eec","arxiv_id":"2507.11059","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"SWE-MERA is a dynamic, GitHub-issue-based coding benchmark with a seven-stage pipeline and Aider-based evaluation of ten LLMs, though internal inconsistencies and residual contamination evidence weaken the quality claims.","lead":"SWE-MERA is a continuously updated benchmark that mines fresh GitHub bugs to test how well LLMs fix software issues, with about 300 tasks and results for ten models. It matters because static coding benchmarks leak solutions into training data, and a rotating benchmark could keep agent evaluations honest.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on unverified task validity: Appendix A never checks that gold patches make FAIL_TO_PASS tests pass, and the only quality gate is an unvalidated LLM judge.","rationale":"The reader's weakest assumption identifies exactly the load-bearing gap: the pipeline does not verify by execution that the gold patch makes FAIL_TO_PASS tests pass, and task quality is delegated to an unvalidated LLM judge. My read confirms this and adds that Appendix A's 'more than 0 passed tests' criterion is insufficient to establish that a task is solvable or that its tests are meaningful. The contamination-minimization claim is also weakened by the paper's own year-over-year DeepSeek-R1 gap (50% vs 40.2%), but that is secondary to task validity: without execution-verified tasks, even a contamination-free dataset cannot support reliable rankings. The proposed audit would settle the concern directly: if all sampled tasks pass gold-patch verification and judge scores agree with humans, the central claim holds; if not, the rankings and the 'rigorous quality validation' claim would need revision. A conditional verdict is appropriate because the pipeline is shipped, reproducible, and fixable, so the issue is one of missing verification rather than an inherent contradiction. Therefore the reader's CONDITIONAL verdict remains unchanged.","tokens_in":10303,"tokens_out":3175,"duration_ms":41209,"concrete_test":"Run an independent audit on a random sample of at least 50 of the 300 tasks: for each task, check out the base commit, apply the test patch, and require all FAIL_TO_PASS tests to fail; then apply the gold patch and require all FAIL_TO_PASS tests to pass. Also have human annotators score task correctness, test correctness, and test completeness on the same sample, and measure agreement with the Qwen3-32B scores. If any gold patch fails or any FAIL_TO_PASS test passes at base, report the fraction; if judge-human agreement is low, recompute the Table 2 ranking using only execution-verified tasks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim of 'rigorous quality validation' is not supported by the described pipeline. In Section 3, Step 6 ('End-to-End Task Execution') points to Appendix A, but Appendix A only requires that `pytest --json-report` completes and reports more than zero passed tests. It never checks that the gold patch causes the task's FAIL_TO_PASS tests to pass, nor that those tests fail at the base commit. Step 7 then filters tasks using Qwen3-32B scores for task correctness, test correctness, and test completeness, with no human validation and no comparison against execution results; Qwen3-32B is also one of the models evaluated in Table 2. As a result, tasks may be included even when the gold patch does not resolve the issue or the tests do not actually cover the described behavior. If a non-trivial fraction of tasks are invalid, the pass@1/pass@6 rankings and the claimed 'strong discriminative power' are unsound. The paper's own Limitations section concedes 'challenges for ground truth and grading quality,' but the Conclusion still claims 'reliable performance baselines that are free from the contamination issues,' which the validation chain does not establish.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SWE-MERA, a dynamically updated benchmark for evaluating LLMs on repository-level software engineering tasks. A seven-stage pipeline collects issue-PR pairs from GitHub, filters them by metadata, patch properties, and a Docker-based build check, and then applies an LLM-based quality filter (Qwen3-32B) to retain tasks judged as correct, well-tested, and complete. The current dataset contains 300 tasks collected over six months (with additional 2024 tasks for comparison). The authors evaluate ten recent LLMs using the Aider agent and report pass@1 and pass@6 metrics, claiming strong discriminative power and contamination resistance relative to static benchmarks such as SWE-bench.","tokens_in":10556,"tokens_out":5602,"duration_ms":61913,"significance":"If the benchmark's validation chain were execution-based, SWE-MERA would be a valuable community resource: it is updated monthly, targets contamination, is accompanied by a public leaderboard and a reproducible pipeline, and the evaluation spans a dozen recent models. The observed year-over-year gap for DeepSeek-R1 is an interesting empirical signal. However, the core validity claim depends on the pipeline actually verifying that gold patches resolve the stated issues, which the paper does not demonstrate. The contributions are therefore promising but not yet fully supported.","major_comments":[{"comment":"The claimed 'rigorous quality validation' is not supported by the described procedure. Step 6 ('End-to-End Task Execution') delegates to Appendix A, but Appendix A's success criteria are only that pytest completes without errors and reports more than zero passed tests. The pipeline never verifies that the gold patch makes the FAIL_TO_PASS tests pass, nor that those tests fail at the base commit. Consequently, tasks can be included even when the gold patch does not resolve the issue or when the test patch does not cover the described behavior. This directly affects the soundness of all reported pass@1/pass@6 numbers. The paper should either add an execution-based check that (a) applies the gold patch to the base commit, (b) runs the FAIL_TO_PASS and PASS_TO_PASS test sets, and (c) requires the expected fail/pass transitions, or it should report the fraction of tasks that satisfy such a check and treat the metrics accordingly.","section":"Section 3.1 (Step 6) and Appendix A"},{"comment":"The LLM-based quality filter uses Qwen3-32B to score task correctness, test correctness, and test completeness, and the same model family is later evaluated in Section 5 (Table 2). The judge scores are not validated against human judgment nor against execution results; the prompt in Appendix D asks the model to judge whether the patch 'correctly solves' the problem and whether tests 'cover' it, but these judgments are not grounded in actual test outcomes. This creates a task-selection bias risk: tasks retained by the filter may be those that Qwen3 models find easy or well-formed, which can inflate Qwen3's ranking relative to other models. The paper should validate the judge (e.g., on a human-annotated sample) and should use a judge that is not from the same model family as the evaluated models, or at least quantify the robustness of rankings to alternative judges.","section":"Section 3.1 (Step 7) and Appendix D"},{"comment":"The paper observes a substantial year-over-year drop for DeepSeek-R1-0528 (pass@6: 50.0% in 2024 vs. 40.2% in 2025) and for DeepSeek-R1-Distill-Qwen-32B (31.5% vs. 23.9%), while other models show smaller changes. This pattern is exactly what one would expect if the 2024 tasks overlap with the models' training data, which would contradict the Conclusion's claim that SWE-MERA establishes baselines 'free from the contamination issues.' If the gap is instead due to task-difficulty differences between years, the paper needs to control for that before drawing conclusions about model behavior. The authors should directly address whether this observation indicates residual contamination or an artifact of the task sampling process, and should temper the contamination-free claim accordingly.","section":"Section 5, Appendix C, and Section 8"}],"minor_comments":[{"comment":"The abstract states that '728 samples' are currently available, while the main text (Section 3.1, Section 6, Table 1) consistently reports 300 samples. This inconsistency must be resolved.","section":"Abstract vs. Section 6"},{"comment":"There is a typo in the table header: 'beetween' should be 'between.'","section":"Table 1"},{"comment":"Figure 5's caption says 'pass@1 and pass@5 metrics,' but the paper otherwise reports pass@6, and the surrounding text refers to pass@6. The caption should be corrected.","section":"Appendix C, Figure 5"},{"comment":"The metric 'localize files' is used in Tables 2 and 4 but is never defined in the evaluation methodology. Please provide a precise definition (e.g., how 'correctly identifying files to fix' is determined).","section":"Table 2 footnote"},{"comment":"The ethical statement claims that 'all prompts, solutions, and test cases ... have been constructed to avoid the unintentional inclusion of proprietary, copyrighted, or sensitive information' and are based on 'public domain resources.' This is inconsistent with the pipeline's use of real GitHub issues and PRs, which are typically under open-source licenses and may contain copyrighted code. Please rephrase to accurately describe the provenance and license considerations.","section":"Ethical Statement"},{"comment":"The security assessment identifies two repositories 'suitable for SWE benchmarking' that exhibit virus signatures, but it does not state whether these repositories were actually included in the final dataset. Please clarify their status and the policy for excluding such repositories.","section":"Section 6 (Malicious Software)"}],"recommendation":"major_revision","confidential_remarks":"The central idea is sound and the resource could be valuable, but the current manuscript overclaims validation. The missing execution-based verification of gold patches is the most serious issue; without it, the benchmark's accuracy is unestablished. The use of Qwen3-32B as both judge and evaluated model is a clear bias risk that should be addressed, even if it is not derivation-level circularity. The DeepSeek-R1 year gap deserves careful discussion because it may undermine the contamination-free claim. I would recommend major revision: the issues are fixable within the manuscript's scope by adding execution checks, validating the judge, and revising the conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful benchmark infrastructure paper with a validation gap at the center. The dynamic monthly collection of real GitHub PR-issue pairs, with a public leaderboard and contamination inspection, is a real contribution — nobody has quite put those together at repository level. The pipeline is transparent, the code and data are shipped, and the Aider-based evaluation across a dozen models is a reasonable sanity check. If you work on coding agents, this is a resource you want to know about.\n\nWhat it does well: the seven-stage funnel is clearly described, the task example is illustrative, and the platform makes contamination inspection interactive. The authors are honest in the Limitations section that ground truth and grading quality are challenging, and that dynamics do not fully eliminate contamination.\n\nThe soft spots are real and they sit on the load-bearing claim. The abstract says 728 samples; the body says 300. That mismatch should have been caught. More importantly, Appendix A's validation only checks that pytest reports at least one passed test; it never verifies the gold patch actually makes the FAIL_TO_PASS tests pass, or that they fail at the base commit. That is the standard sanity check for SWE-bench-style tasks and it's missing. Instead, task correctness is delegated to Qwen3-32B, whose scores are unvalidated and whose own family is later evaluated. That is a bias risk, not forced circularity — no ranking is constructed from the judge scores — but it is a real weakness. The year-over-year DeepSeek-R1 drop (50% to 40.2%) is consistent with contamination, and the conclusion's phrase 'free from the contamination issues' is stronger than the evidence supports.\n\nThe central idea is still sound. I would not desk-reject this. A serious referee should ask for executed verification of gold patches against FAIL_TO_PASS tests, a human-validated sample of the LLM quality gate, and a consistent sample count. With those fixed, this could be a solid benchmark paper. The author's claim of 'rigorous quality validation' is not yet supported, but the pipeline is the kind of thing the community needs.","headline":"Useful dynamic benchmark infrastructure, but the core validation of gold patches is missing and the sample count is inconsistent.","tokens_in":11114,"tokens_out":2019,"would_cite":true,"duration_ms":23015,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SWE-MERA turns freshly merged GitHub fixes into a continuously updated benchmark, and its Aider-based runs separate current LLMs by ability more cleanly than static sets.","keywords":["SWE-MERA","software engineering benchmark","LLM evaluation","dynamic benchmark","GitHub issues","data contamination","coding agents","benchmark pipeline"],"falsifier":"Take a random sample of retained tasks, rebuild each repository at its base commit, apply the gold patch, and run the FAIL_TO_PASS tests: if a non-trivial fraction does not go from failing to passing (or if the tests pass even without the gold patch), the benchmark's quality gate is not doing its job. A second check would ask human raters to score the same tasks on the four judge criteria; agreement below chance would mean the bottom-quartile filter is arbitrary.","tokens_in":10098,"feed_emoji":"🐛","tokens_out":10787,"duration_ms":115441,"temperature":0.7,"pith_summary":"SWE-MERA sets out to fix two failures of static coding benchmarks like SWE-bench: models memorize old tasks, and leaderboards saturate once scores climb. Its answer is a monthly pipeline that mines real GitHub issues merged in the previous month, converts each issue plus its fixing pull request into a coding task, and filters candidates through Docker builds and an LLM judge. The paper reports roughly 10,000 candidate tasks, with a released set of a few hundred (the abstract says 728; the body text says 300), collected between September 2024 and June 2025. Running the Aider coding agent on the 2025 subset, the benchmark ranks DeepSeek-R1-0528 at 27.8% pass@1 and 40.2% pass@6, with smaller Qwen2.5-Coder models below larger ones, evidence the authors read as clear discriminative power among state-of-the-art models. If the pipeline works as claimed, it gives the field a contamination-resistant yardstick that never goes stale, plus a public leaderboard where submissions are checked against the same Dockerized environments.","feed_headline":"Fresh GitHub bugs power a monthly LLM coding benchmark","feed_subtitle":"SWE-MERA mines recently merged fixes into new tasks each month, so model rankings reflect current skill, not memorized answers.","key_machinery":"The load-bearing mechanism is the fail-to-pass test pair extracted from a merged pull request: the repository is pinned to its pre-fix commit, the PR's added tests are expected to fail there and pass once the gold patch is applied, while previously passing tests are expected to stay green. This converts a real-world bug fix into an executable grading signal without any human-written solution. The second mechanism is the LLM quality gate: Qwen3-32B scores each candidate task on four 1–10 criteria, and the bottom quartile on task correctness, test correctness, or test completeness is removed, which the paper says is crucial for keeping tasks from being trivial or unsolvable. Evaluation then runs the Aider agent with six independent tries per issue and reports pass@1 and pass@6 alongside diagnostics (file localization, patch generation, regression-test passage, and token-limit hits).","core_discovery":"On the paper's own terms, the discovery is a task-construction recipe: for every merged pull request that closes an issue in a recent Python repository, revert the repository to the pre-fix commit, take the pull request's diff as the gold patch, and use the tests introduced in that same pull request as the fail-to-pass signal. Tests that passed before and after the fix become PASS_TO_PASS guards. A seven-stage funnel filters these candidates by metadata, patch shape, Docker build success, end-to-end execution, and finally a Qwen3-32B judge that scores task correctness, test correctness, test completeness, and complexity; tasks in the bottom quartile on the first three scores are dropped. The authors claim the resulting monthly-refreshed set evaluates coding agents with clear discriminative power, evidenced by model rankings that track known capability orderings, and they position SWE-MERA as a contamination-resistant alternative to static SWE-bench-style benchmarks.","pith_inferences":["The most direct stress test would be a human-validated sample: the paper's quality gate is a single LLM judge whose scores were not checked against human agreement, so the benchmark's floor is whatever that judge gets right.","The year-over-year drop for DeepSeek-R1 on 2024 versus 2025 tasks (50% to 40.2% pass@6) could mean the older tasks are contaminated or simply that the 2025 wave is harder; the paper does not separate these explanations, and a task-difficulty calibration would settle it.","Because the collection pipeline is public and monthly, a determined actor could pre-train on the latest GitHub issue/PR pairs and inflate a future leaderboard run; dynamic freshness lowers contamination risk but does not remove it.","The regression-tests column (patches passing the repository's original tests) is a cheap proxy for patch safety and could be promoted from a reported diagnostic into a formal filtering criterion, which the paper does not propose."],"forward_implications":["Each monthly wave uses issues merged in the prior month, so a model cannot simply memorize a fixed test set; the leaderboard stays readable as capability changes.","The reported 2025 rankings place DeepSeek-R1-0528 first at 27.8% pass@1 and 40.2% pass@6, followed by Devstral-Small-2505, with Qwen2.5-Coder models ordered by size—evidence the benchmark separates models the field expects to be separated.","Because the pipeline is public and runs on any GitHub repository, the same funnel can be pointed at Java, JavaScript, TypeScript, Go, or C++ repositories to extend coverage beyond Python.","The public leaderboard and submission workflow, with trajectory links and a time slider, make contamination inspectable: users can look at which tasks a model solved in which month.","The authors estimate the full historical collection could reach about 10,000 tasks, limited mainly by end-to-end execution effort rather than by data availability."],"supporting_citations":[{"why":"SWE-bench defines the static benchmark format and contamination problem SWE-MERA is built to replace.","marker":"(Jimenez et al., 2023)"},{"why":"SWE-Bench+ quantifies solution leakage in SWE-bench and motivates the post-cutoff, leakage-filtered collection strategy.","marker":"(Aleithan et al., 2024)"},{"why":"LiveCodeBench supplies the dynamic-update paradigm that SWE-MERA extends from algorithmic problems to repository-level issues.","marker":"(Jain et al., 2024)"},{"why":"Multi-SWE-bench establishes the multilingual benchmark direction that SWE-MERA's extensibility claims build on.","marker":"(Zan et al., 2025)"},{"why":"SWE-Gym provides an automatic task-generation baseline that SWE-MERA contrasts with its real-issue, quality-filtered pipeline.","marker":"(Pan et al., 2024)"},{"why":"Qwen3-32B is the model used both as the LLM judge in the pipeline's quality gate and as an evaluated baseline.","marker":"(Yang et al., 2025a)"}],"fun_headline_variants":["Monthly GitHub bugs refresh LLM coding benchmark","New benchmark mines fresh PRs to test LLMs","Contamination-free coding benchmark updates monthly","SWE-MERA: dynamic benchmark for LLM coding skills","Fresh fixes become new tasks for LLM coding eval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the quality gate works: Appendix A's build check only requires pytest to report at least one passed test, and the final task filter is an LLM judge whose scores were never checked against human ratings—so nothing in the pipeline actually proves the gold patch turns the FAIL_TO_PASS tests from failing to passing.","fun_headline_variants_meta":{"raw":{"variants":["Monthly GitHub bugs refresh LLM coding benchmark","New benchmark mines fresh PRs to test LLMs","Contamination-free coding benchmark updates monthly","SWE-MERA: dynamic benchmark for LLM coding skills","Fresh fixes become new tasks for LLM coding eval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1182,"prompt_tokens":895,"completion_tokens":287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":214}},"tokens_in":511,"tokens_out":287,"duration_ms":3698,"temperature":1.0,"reasoning_tokens":214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:18:22.451998+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of retained tasks, rebuild each repository at its base commit, apply the gold patch, and run the FAIL_TO_PASS tests: if a non-trivial fraction does not go from failing to passing (or if the tests pass even without the gold patch), the benchmark's quality gate is not doing its job. A second check would ask human raters to score the same tasks on the four judge criteria; agreement below chance would mean the bottom-quartile filter is arbitrary.","supporting_citations":[],"review_version":1}