Pith. sign in

REVIEW 18 cited by

SWE-Bench+: Enhanced Coding Benchmark for LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.06992 v2 pith:Z5VJIRA2 submitted 2024-10-09 cs.SE

Reem Aleithan , Haoran Xue , Mohammad Mahdi Mohajer , Elijah Nnorom , Gias Uddin , Song Wang This is my paper · ORCID
classification cs.SE
keywords swe-benchissuesdatasetpatchescodinggpt-4llmswere
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) in Software Engineering (SE) can offer assistance for coding. To facilitate a rigorous evaluation of LLMs in practical coding contexts, Carlos et al. introduced the SWE-bench dataset, which comprises 2,294 real-world GitHub issues and their corresponding pull requests, collected from 12 widely used Python repositories. Several impressive LLM-based toolkits recently are developed and evaluated on this dataset. However, a systematic evaluation of the quality of SWE-bench remains missing. In this paper, we addressed this gap by presenting an empirical analysis of the SWE-bench dataset. We conducted a manual screening of instances where SWEAgent + GPT-4 successfully resolved issues by comparing the model-generated patches with the actual pull requests. SWE-Agent+GPT-4 was at the top of SWE-bench leaderboard during the time of our study. Our analysis reveals some critical issues with the SWE-bench dataset: 1) 32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments. We refer to as solution leakage problem. 2) 31.08% of the passed patches are suspicious patches due to weak test cases, i.e., the tests were not adequate to verify the correctness of a patch. When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. We also observed that the same data quality issues also exist in the two variants of SWE-bench, i.e., SWE-bench Lite and SWE-Bench Verified. In addition, over 94% of the issues were created before LLM's knowledge cutoff dates, posing potential data leakage issues.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

    cs.SE 2026-07 conditional novelty 8.0 of 10

    RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substit...

  2. PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

    cs.SE 2026-07 conditional novelty 7.0 of 10

    PAIChecker detects PR-issue misalignment in SWE-bench-like benchmarks, which the authors find affects 13.6% of SWE-bench Verified instances and shifts 64.1% of leaderboard rankings.

  3. Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports

    cs.SE 2026-08 conditional novelty 6.0 of 10

    Introduces Active-SWE, a benchmark for proactive multi-bug fixing without issue reports, and shows state-of-the-art coding agents resolve few such bugs.

  4. Deep Interaction: An Efficient Human-AI Interaction Method for Large Reasoning Models

    cs.AI 2026-07 reject novelty 6.0 of 10

    Directly editing a large reasoning model's chain-of-thought and feeding back a distilled version of the edit improves correction success by over 25% and cuts token usage by roughly 40% versus dialogue-based correction.

  5. What Makes a Good Bug Report for an AI Agent?

    cs.SE 2026-07 conditional novelty 6.0 of 10

    AI repair agents solve bugs more reliably when reports include executable reproduction scripts, file-level localization cues, and clear structure, while longer prose reports and human-oriented steps to reproduce show ...

  6. Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    Coding benchmarks misalign with agentic software engineering because they conflate model and harness, grade against single references, and provide no component-level iteration signals.

  7. Estimating the Empowerment of Language Model Agents

    cs.AI 2025-09 conditional novelty 6.0 of 10

    EELMA estimates the mutual information between an LM agent's actions and future text states, and this 'empowerment' is shown to correlate with task performance across toy games and WebArena.

  8. SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

    cs.SE 2025-07 conditional novelty 6.0 of 10

    SWE-MERA is a dynamic, GitHub-issue-based coding benchmark with a seven-stage pipeline and Aider-based evaluation of ten LLMs, though internal inconsistencies and residual contamination evidence weaken the quality claims.

  9. UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench

    cs.SE 2025-06 conditional novelty 6.0 of 10

    UTBoost adds LLM-generated tests and a repaired log parser to SWE-Bench, uncovering 345 falsely-passing patches and causing dozens of leaderboard ranking shifts.

  10. TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?

    cs.SE 2024-12 conditional novelty 6.0 of 10

    Introduces TDD-Bench-Verified, a 449-issue benchmark for test-driven development, and Auto-TDD, an LLM pipeline that achieves a higher fail-to-pass rate (up to 23.6% with GPT-4o) than prior methods.

  11. Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications

    cs.SE 2026-08 conditional novelty 5.0 of 10

    An analysis of 2,572 tests and 10 practitioner interviews shows that agent testing is mostly unit-level, mock-heavy, and built on simple example data.

  12. The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents

    cs.CL 2026-08 conditional novelty 5.0 of 10

    A survey of 1,547 papers defines the 'horizon gap' and documents that long-horizon agent research is converging on trajectory-level process signals instead of outcome-only scores.

  13. Agentic Software Engineering: Foundational Pillars and a Research Roadmap

    cs.SE 2025-09 conditional novelty 5.0 of 10

    SASE introduces a dual-modality framework (SE for Humans and SE for Agents) with workbenches and structured artifacts to turn agentic coding into a disciplined engineering practice.

  14. What You See Is What It Does: A Structural Pattern for Legible Software

    cs.SE 2025-08 conditional novelty 5.0 of 10

    A concept-and-synchronization pattern with a DSL improves software legibility and enables LLM generation of a backend web app, demonstrated on the RealWorld benchmark.

  15. Are Large Language Models Memorizing Bug Benchmarks?

    cs.SE 2024-11 conditional novelty 5.0 of 10

    Some base LLMs, particularly codegen-multi, show strong memorization of the Defects4J bug benchmark, while newer models like LLaMa 3.1 show weaker leakage signals.

  16. Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.

  17. AI Scientists Fail Without Strong Implementation Capability

    cs.AI 2025-06 conditional novelty 4.0 of 10

    AI scientist systems can propose ideas but cannot reliably implement and verify experiments, making the implementation gap, not idea generation, the current bottleneck.

  18. Empirical Research on Utilizing LLM-based Agents for Automated Bug Fixing via LangGraph

    cs.SE 2025-01 reject novelty 3.0 of 10

    The paper demonstrates a LangGraph-orchestrated LLM bug-fixing pipeline on two toy Python examples, but provides no benchmark evidence.

Pith tools