Pith. sign in

REVIEW 5 major objections 5 minor 51 references

RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM agents lack evidence-grounded repository comprehension; the best model scores only 62.7% on RepoProbe, and checklist-based judging is about twice as stable as scalar scoring.

desk verdict RepoProbe is a genuine step forward for repository-QA evaluation, but its headline reliability claim and score ceiling rest on unvalidated checklists and a generator-judge overlap that a serious referee should push on. read the letter →

arxiv 2608.04783 v2 pith:5CGPNGMH submitted 2026-08-05 cs.SE cs.AI

Yuexi Yang , Alyssa Wu , Ji Luo , Richeng Xuan , Zhichao Hu , Yuhong Liu , Zhen Qin This is my paper · ORCID
classification cs.SEcs.AI
keywords RepositoryUnderstandingLargeLanguageModelsBenchmarkingEditBiasChecklist-BasedVerificationLLM-as-a-JudgeGitHubDiscussionsAutomatedSoftwareEngineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces RepoProbe, a benchmark for repository-level code understanding built from 500 answered GitHub Discussions across 50 large, active repositories. The benchmark asks models open-ended 'how' and 'why' questions about architecture, business logic, and implementation details, with no bug reports, stack traces, or other localization shortcuts. The central empirical claim is that state-of-the-art LLM agents perform far worse on such questions than their bug-fixing results suggest, with even the best model scoring only in the low-60% range overall and clarity scores consistently exceeding knowledge scores. The paper also claims that a new evaluation protocol, which decomposes answers into weighted atomic checklist items with discrete rubrics instead of a single scalar score, cuts the standard deviation of LLM judging roughly in half and produces actionable feedback. If correct, the work implies that defect-centric benchmarks such as SWE-bench overestimate repository understanding by letting models exploit localization cues.

What carries the argument

The Checklist-Based Verification Protocol is the central mechanism: each RepoProbe question carries a developer-accepted reference answer and a checklist of 1-4 knowledge items, with integer weights summing to 9, plus a fixed 1-point clarity item. Each item has a discrete rubric, and the LLM judge (Claude Sonnet 4.5, deliberately excluded from the evaluated models) must first state a rationale and then pick a score only from the predefined rubric values. This turns open-ended answers into a sequence of verifiable fact checks. The benchmark itself is the second piece: 500 discussion-derived Q&A pairs, curated so questions are self-contained and answerable solely from repository code, with no localization cues; edit bias is quantified by an LLM categorizer that labels failures, with manual checks on the edit-bias class.

What would settle it

Re-annotate a random sample of RepoProbe's 500 questions with independently written human checklists and reference answers derived from the repository code; if the top model's knowledge score moves by more than a few points or edit-bias's share of failures falls outside the 10.4% to 24.0% band, the headline results are artifacts of how the LLM-generated ground truth was chosen.

Watch

Extended reading notes

Core claim

RepoProbe is a benchmark of 500 repository-level Q&A pairs assembled from answered GitHub Discussions in 50 repositories created after January 2024, spanning 15 programming languages. The paper's central claim is that these open-ended 'how' and 'why' questions—which offer no bug reports or stack traces—expose a capability gap that defect-centric benchmarks hide: every evaluated frontier model scores below 63% overall, knowledge scores trail clarity scores for every model, and perfect solves are rare. The paper names and measures the failure mode 'edit bias': instead of analyzing the existing architecture, models propose new code or configuration patches, and this accounts for 10.4% to 24.0% of low-scoring cases across the top models. The paper further claims that its Checklist-Based Verification Protocol, which replaces a single holistic scalar score with rationale-first, rubric-constrained judgment of atomic facts, cuts evaluation standard deviation roughly in half and eliminates the contradictory feedback that scalar judges give.

Load-bearing premise

The load-bearing premise is that the LLM-generated checklists and reference answers, with expert human inspection applied to only a subset of the 500 samples, are correct ground truth for every question.

Editorial extensions

If this is right

  • Defect-centric benchmarks such as SWE-bench overestimate repository understanding because localization cues let models bypass architectural comprehension.
  • High code-generation scores do not imply engineering judgment: models can produce plausible patches that ignore the root cause the user actually has.
  • Checklist-based verification gives developers actionable, reproducible feedback, because they see exactly which atomic facts an answer missed instead of a scalar score that can contradict itself across runs.
  • General frontier progress does not reliably translate into repository comprehension; newer model releases sometimes score below their predecessors on RepoProbe.
  • Evidence grounding and cross-file dependency tracing, not fluent narration, are the limiting factors for repository-level Q&A, since clarity scores consistently exceed knowledge scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested corollary of the edit-bias finding: instructing an agent to first produce a file-referenced architecture summary before proposing any change should raise knowledge scores; the paper does not run this experiment.
  • The checklist protocol is portable to other open-ended software tasks such as code review, design documentation, and refactoring justification, where partial credit and evidence grounding are the core evaluation problems.
  • Because RepoProbe selects only repositories created after January 2024, the same pipeline could be re-run on newer discussions to build a living benchmark tracking whether repository comprehension improves across model generations.
  • If edit bias is a stable property, autonomous coding agents may need an explicit understand-before-edit gate in their scaffolding to avoid high-risk refactors for simple configuration requests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. RepoProbe introduces a benchmark of 500 repository-level Q&A pairs mined from GitHub Discussions across 50 repositories, paired with a checklist-based verification protocol intended to replace scalar LLM-as-a-Judge scoring. The paper evaluates 20 frontier and open-weight LLMs under a fixed Claude Code agentic scaffolding, reporting that the best models reach only the low-60% range on overall performance, that clarity scores systematically exceed knowledge scores, and that edit bias accounts for 10.4% to 24.0% of failure cases among top models. It also reports that checklist-based verification reduces judge standard deviation roughly by half relative to scalar scoring across three models and five runs. The benchmark's core idea, task source, and evaluation design are plausible, but the central validity claims rest on checklist ground truth, human validation coverage, and judge independence, each of which is currently under-supported.

Significance. If the benchmark and its verification protocol are valid, RepoProbe fills a real gap: existing bug-fix benchmarks often contain localization cues, and scalar LLM judging is known to be unstable. The discussion-derived task formulation, the multi-language and multi-theme coverage, the fixed agentic scaffolding across models, and the explicit rubric structure are genuine strengths. The paper also makes a falsifiable prediction about edit bias that could be tested by future work. However, the headline conclusions about model capability, evaluation reliability, and edit bias all inherit the validity of the checklists, and the checklists are generated by the same model family that later serves as judge, with human validation covering only an unspecified subset. These issues are load-bearing and require additional evidence before the benchmark's claims can be fully accepted.

major comments (5)
  1. [§3.2, Data Availability Statement] The paper states that 'A subset of the remaining samples was then manually inspected by expert annotators' and concludes that 'we obtained a pool of 500 validated code-understanding Q&A pairs,' but it never reports how many samples were manually inspected, how disagreements were resolved beyond 'discussion,' or whether all 500 final samples passed human validation. In addition, reference [49], the promised replication package, is listed as 'TODO: replace with the GitHub URL once the repository is public.' Because every model score, stability comparison, and edit-bias percentage is computed against these checklists, the paper needs to report the exact human-validation coverage, inter-annotator agreement, and release the dataset; otherwise the '500 validated' claim is not independently verifiable.
  2. [§5.3, Table 5] The RQ2 reliability claim is based on five evaluation runs for three models with a single judge model (Claude Sonnet 4.5) and no significance test. The observed standard deviations (1.5% to 2.4% for checklist versus 2.9% to 3.5% for scalar) are suggestive, but 'cutting this measurement error by more than half' is not established without confidence intervals, a significance test, or a bootstrap analysis. Because the checklist protocol is a core contribution, the paper should either add inferential statistics over more models/runs or soften the claim to a preliminary observation.
  3. [§3.2, §4.1, §5.1.3] There is a generator-judge overlap: Claude Sonnet 4.5 generates the checklists during dataset construction and is then used as the fixed judge for all evaluations, while being deliberately excluded from the evaluated model set. This design can make the checklist protocol appear more stable than it is because the judge may be self-consistent with its own rubric-generation behavior rather than with task-level ground truth. The paper should test this by re-judging a subset of answers with an independent judge (e.g., a different provider model or expert human annotators) and reporting agreement between judges.
  4. [§3.1, §4.1, §3.2] The checklists are derived from a single developer-accepted reference answer, and each knowledge item is described as 'a precise, atomic aspect of the reference answer.' For open-ended 'how' and 'why' repository questions, multiple technically correct answers can exist, and an answer that explains a different valid design path may receive near-zero credit simply because it does not contain the maintainer's specific facts. The answerability check verifies only that the reference answer is grounded in repository code, not that alternative correct answers are recognized by the checklist. Without a false-negative study comparing checklist scores against independent expert judgments for a sample of model outputs, the low-60% ceiling and the edit-bias percentages may conflate comprehension with convergence to the reference answer.
  5. [§5.4, Table 7] The edit-bias quantification is not fully supported. The failure-mode taxonomy is 'defined by the authors from recurring patterns,' a single LLM (Gemini 3 Flash) categorizes all sub-60% failure cases, the threshold of 60% is not justified, and manual inspection is reported only for the Edit Bias category. No inter-annotator agreement is reported, and the percentages in Table 7 have no confidence intervals. Since edit bias is a headline contribution, the paper should provide a systematic categorization protocol, report reliability of the automatic categorizer, and ideally validate on a random sample rather than only on the category of interest.
minor comments (5)
  1. [Abstract] There is a missing space in 'under-measuresEdit Bias'; this should be corrected to 'under-measures Edit Bias.'
  2. [§1, Contributions] The third bullet begins 'fluent explanations mask a lack of evidence-grounded correctness, This bias' with a comma splice; the sentence should be split or the punctuation corrected.
  3. [§4.2] The description says the implementation 'trades strict step-wise independence for efficiency'; this is an important limitation of the single-pass design and could be acknowledged more prominently, since the rationale-first constraint is presented as a strength but the items are scored jointly.
  4. [§5.1.2] The paper refers to 'Grok 4.20' in Section 5.1.2 but Table 3 lists 'Grok 4.2'; the version label should be made consistent.
  5. [§5.2, Table 3] The paper does not state how many evaluation runs were used for the main benchmark results in Table 3, unlike the five-run setting in Table 5. If the Table 3 numbers are from a single run, this should be stated explicitly, and variance estimates should be provided for at least a subset of models.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's core results are empirical measurements under an explicit reference-derived scoring protocol; the generator-judge overlap and subset-only human validation are external validity caveats, not construction-level reductions.

full rationale

RepoProbe's derivation chain is not circular. The benchmark defines correctness operationally: each checklist is derived from a developer-accepted reference answer, with knowledge items explicitly described as 'a precise, atomic aspect of the reference answer,' and the total score is the weighted sum of rubric choices. Under that explicit operationalization, the RQ1 finding that models score in the low-60% range is a measurement, not a quantity that the construction forces: nothing in the definition of the score entails any particular model's coverage level. The RQ2 stability result is also empirical: the paper ran both scalar and checklist protocols across five independent runs and reported measured standard deviations and ranges; the generator-judge overlap is a real confound that could inflate self-consistency, but the variance reduction is an observed statistic, not a logical consequence of the definitions. RQ3's edit-bias figures are post-hoc labels assigned by a separate judge model (Gemini 3 Flash) with manual inspection, so they are not fitted parameters being presented as predictions. The paper contains no load-bearing self-citation chain: the only self-reference of note is the replication package [49], which is explicitly a TODO placeholder and is not used to justify any technical claim. The paper itself flags the two main external-validity limitations: only 'a subset of the remaining samples' received expert manual inspection, and the data/code URL is not yet public. These are completeness and reproducibility gaps, not circular reductions. I therefore find no circular step and assign score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

RepoProbe has no executable oracle; correctness is defined by LLM-generated checklists anchored to developer-accepted GitHub Discussion answers. The main unstated costs are trust in Claude Sonnet 4.5's curation and judging, trust that accepted answers are correct, and trust that one fixed agentic scaffolding represents real usage. No new physical or mathematical entities are introduced.

free parameters (3)
  • Per-instance checklist knowledge-item weights = integers, sum to 9; varies per instance
    Assigned by Claude Sonnet 4.5 during checklist generation; no derivation justifies these weights and they directly determine the normalized score rate used in all results.
  • Failure threshold for edit-bias analysis = 60% score rate
    The 60% cutoff determines which answers are counted as failures in the edit-bias analysis; changing it changes the reported failure-mode distribution.
  • Repository inclusion thresholds = created >= 2024-01-01; stars >= 1000; at least 10 answered discussions
    Hand-chosen criteria define the repository pool, and therefore the benchmark composition and model rankings; no sensitivity analysis is reported.
assumptions (4)
  • domain assumption GitHub Discussions accepted answers are correct, complete ground truth for the reference answers in RepoProbe.
    The curation pipeline extracts the accepted answer as the reference answer without independent technical verification (Section 3.2).
  • domain assumption Claude Sonnet 4.5's relevance classification, context integration, and checklist generation preserve developer intent and factual correctness after partial human review.
    Most curation steps are performed by this LLM; only 'a subset' of samples is manually inspected, so fidelity for uninspected items is assumed (Section 3.2, Quality Assurance).
  • domain assumption The checklist items and weights generated by an LLM are an unbiased operationalization of answer correctness.
    Scores are computed only from these LLM-authored rubrics; no external oracle verifies the rubrics on all 500 items (Section 4.1).
  • domain assumption Findings under the fixed Claude Code agentic scaffolding transfer to other scaffolding choices.
    The paper fixes one agentic environment for all models and interprets differences as model capability; the representativeness of the scaffolding is assumed (Section 5.1.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists." pith.science (2026). https://pith.science/paper/5CGPNGMH

@misc{pith2026260804783,
  author       = {Pith},
  title        = {Pith review of: RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5CGPNGMH}},
  note         = {Machine review of arXiv:2608.04783}
}
read the original abstract

The integration of Large Language Models (LLMs) into software engineering has shifted the focus from function-level generation to repository-scale assistance. However, existing benchmarks largely rely on bug reports from GitHub Issues, which often allow models to bypass genuine understanding via pattern matching on error logs. This misalignment under-measures Edit Bias, which refers to premature generation, where models prematurely propose code modifications instead of understanding the existing repository architecture. Furthermore, current LLM-as-a-Judge scalar scoring suffers from high variance and low interpretability. This work introduces RepoProbe, a novel benchmark for evaluating repository-level code understanding through open-ended Q&A using GitHub Discussions, which focuses on open-ended architectural inquiries rather than defect reporting. To ensure rigorous evaluation, we propose a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts, thereby replacing subjective ratings with objective verification. Our evaluation of state-of-the-art (SOTA) LLMs reveals a persistent gap between high clarity and evidencegrounded technical correctness. It also quantitatively confirms the prevalence of edit bias, in which models prioritize code generation instead of architectural analysis. Finally, we demonstrate that our verification protocol significantly improves evaluation reliability compared to traditional evaluations with scalar scoring.

Figures

Figures reproduced from arXiv: 2608.04783 by the authors.

Figure 1
Figure 1. Without appropriate repository understanding, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. RepoProbe data curation pipeline from answered GitHub Discussions to self-contained, checklist-scored Q&A tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Distribution of RepoProbe questions by programming language (left), code-understanding category (center), and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Case Study: The “Leading Question” Trap. RepoProbe uses Discussion data to expose models that blindly follow user [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 3 canonical work pages

  1. [49]

    RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists

    Yuexi Yang, Alyssa Wu, Ji Luo, Richeng Xuan, Zhichao Hu, Yuhong Liu, and Zhen Qin. 2026. Replication Package for Paper "RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists". TODO: re- place with the GitHub URL once the repository is public

    2026

  2. [1]

    Anthropic. 2025. Claude Opus 4.5 System Card. Technical Report. Anthropic. https://www.anthropic.com/claude-opus-4-5-system-card

    2025

  3. [2]

    Anthropic. 2026. Claude Opus 4.6 System Card. Technical Report. Anthropic. https://anthropic.com/claude-opus-4-6-system-card

    2026

  4. [3]

    Anthropic. 2026. Claude Sonnet 4.6 System Card. Technical Report. Anthropic. https://anthropic.com/claude-sonnet-4-6-system-card

    2026

  5. [4]

    Christian Bird, Denae Ford, Tom Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowdermilk, and Idan Gazit. 2023. Taking Flight with ASE ’26, October 12–16, 2026, Munich, Germany Yuexi Yang2, *,† Alyssa Wu1, *,‡ Ji Luo1 Richeng Xuan1 Zhichao Hu1 Yuhong Liu1 Zhen Qin2,3,§ Copilot: Early insights and opportunities of AI-powered pair-programming too...

    doi:10.1145/3582083 2023

  6. [5]

    ByteDance Seed Team. 2025. Official Release of Seed1.8: A Generalized Agentic Model. Technical Report. ByteDance. https://seed.bytedance.com/en/blog/ official-release-of-seed1-8-a-generalized-agentic-model

    2025

  7. [6]

    ByteDance Seed Team. 2026. Doubao Seed 2.0. https://research.doubao.com/en/ seed2. Official model page; released Feb 14, 2026

    2026

  8. [7]

    Guoxin Chen, Fanzhe Meng, Jiale Zhao, Minghao Li, Daixuan Cheng, Huatong Song, Jie Chen, Yuzhi Lin, Hui Chen, Xin Zhao, Ruihua Song, Chang Liu, Cheng Chen, Kai Jia, and Ji-Rong Wen. 2026. BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?arXiv preprint (2026). arXiv:2603.03194. doi:10.48550/arXiv.2603.03194

    arXiv doi:10.48550/arxiv.2603.03194 2026

Show all 51 references
  1. [8]

    Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, Pengfei Gao, Ping Yang, and Shuiguang Deng. 2025. CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering. arXiv preprint (2025). arXiv:2501.03447. doi:10.48550/arXiv.2501.03447

    2025 doi arXiv

  2. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, et al. 2021. Evaluating large language models trained on code. arXiv preprint (2021). arXiv:2107.03374. doi:10.48550/arXiv.2107.03374

    2021 doi arXiv

  3. [10]

    DeepSeek-AI. 2025. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv preprint (2025). arXiv:2512.02556. doi:10.48550/arXiv.2512.02556

    2025 doi arXiv

  4. [11]

    Ramtin Ehsani, Esteban Parra, Sonia Haiduc, and Preetha Chatterjee. 2025. Hier- archical Knowledge Injection for Improving LLM-based Program Repair. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1440–1452. doi:10.1109/ASE63991.2025.00122

    2025

  5. [12]

    GLM Team. 2025. GLM-4.7. https://z.ai/blog/glm-4.7. Blog post

    2025

  6. [13]

    GLM Team. 2026. GLM-5: from Vibe Coding to Agentic Engineering. arXiv preprint (2026). arXiv:2602.15763. doi:10.48550/arXiv.2602.15763

    2026 doi arXiv

  7. [14]

    Google DeepMind. 2025. Gemini 3 Flash - Model Card. Technical Report. Google DeepMind. https://deepmind.google/models/model-cards/gemini-3-flash/

    2025

  8. [15]

    Google DeepMind. 2025. Gemini 3 Pro Model Card. Technical Report. Google DeepMind. https://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf

    2025

  9. [16]

    Google DeepMind. 2026. Gemini 3.1 Pro - Model Card. Technical Report. Google DeepMind. https://deepmind.google/models/model-cards/gemini-3-1-pro/

    2026

  10. [17]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Syn- naeve, and Sida I. Wang. 2024. CRUXEval: A Benchmark for Code Reason- ing, Understanding and Execution. arXiv preprint (2024). arXiv:2401.03065. doi:10.48550/arXiv.2401.03065

    2024 doi arXiv

  11. [18]

    Hideaki Hata, Nicole Novielli, Sebastian Baltes, Raula Gaikovina Kula, and Christoph Treude. 2022. GitHub Discussions: An exploratory study of early adop- tion. Empirical Software Engineering 27, 1, Article 3 (2022). doi:10.1007/s10664- 021-10058-6

    2022 doi

  12. [19]

    Kang He and Kaushik Roy. 2026. SWE-Adept: An LLM-Based Agentic Framework for Deep Codebase Analysis and Structured Issue Resolution. arXiv preprint (2026). arXiv:2603.01327. doi:10.48550/arXiv.2603.01327

    2026 doi arXiv

  13. [20]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real- World GitHub Issues?. In The Twelfth International Conference on Learning Representations (ICLR). doi:10.48550/arXiv.2310.06770

    2024 doi arXiv

  14. [21]

    Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2024. XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval. In Proceedings of the 62nd...

    2024

  15. [22]

    Kimi Team. 2025. Kimi K2: Open Agentic Intelligence. arXiv preprint (2025). arXiv:2507.20534. doi:10.48550/arXiv.2507.20534

    2025 doi arXiv

  16. [23]

    Kimi Team. 2026. Kimi K2.5: Visual Agentic Intelligence. arXiv preprint (feb 2026). arXiv:2602.02276. doi:10.48550/arXiv.2602.02276

    2026 doi arXiv

  17. [24]

    Yukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim. 2025. CheckEval: A reliable LLM-as-a-Judge frame- work for evaluating text generation using checklists. In Proceedings of the 2025 Conference on Empirical Methods in Natural Langua...

    2025 doi

  18. [25]

    Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, and Zhi Jin. 2025. Benchmarking long-context lan- guage models on long code understanding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistic...

    2025 doi

  19. [26]

    Jia Li, Yuxin Su, and Michael R. Lyu. 2026. From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level. arXiv preprint (2026). arXiv:2601.03731. doi:10.18653/v1/2026.acl-long.399

    2026 arXiv

  20. [27]

    Zhonghang Li, Zongwei Li, Yuxuan Chen, Han Shi, Jiawei Li, Jierun Chen, Haoli Bai, and Chao Huang. 2026. FastCode: Fast and Cost-Efficient Code Understanding and Reasoning. arXiv preprint (2026). arXiv:2603.01012. doi:10.48550/arXiv.2603. 01012

    2026 doi

  21. [28]

    Jiawei Liu et al. 2024. RepoQA: Evaluating Long Context Code Understanding. arXiv preprint (2024). arXiv:2406.06025. doi:10.48550/arXiv.2406.06025

    2024 doi arXiv

  22. [29]

    Yang Liu, Li Zhang, Fang Liu, Ping Lin, and Xinyi Li. 2026. A Scalable Benchmark for Repository-Oriented Long-Horizon Conversational Context Management. arXiv preprint (2026). arXiv:2603.06358. doi:10.48550/arXiv.2603.06358

    2026 doi

  23. [30]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shuj...

    2021 doi arXiv

  24. [31]

    Walid Maalej, Rebecca Tiarks, Tobias Roehm, and Rainer Koschke. 2014. On the Comprehension of Program Comprehension. ACM Transactions on Software Engineering and Methodology 23, 4, Article 31 (2014). doi:10.1145/2622669

    2014 doi

  25. [32]

    MiniMax AI. 2025. M2.1: Multilingual and Multi-Task Coding with Strong Gener- alization. https://huggingface.co/blog/MiniMaxAI/multilingual-and-multi-task- coding-with-strong-gen. Blog post

    2025

  26. [33]

    MiniMax AI. 2026. MiniMax M2.5: Built for Real-World Productivity. https: //www.minimax.io/news/minimax-m25. Official release page

    2026

  27. [34]

    Ferida Mohammed, Fatma Ayad, Petros Maniatis, Satish Chandra, and Elizabeth Dinella. 2026. RubberDuckBench: A Benchmark for AI Coding Assistants. In Proceedings of the 3rd International Workshop on Large Language Models for Code (LLM4Code ’26). doi:10.1145/3786181.3788710

    2026

  28. [35]

    OpenAI. 2025. Update to GPT-5 System Card: GPT-5.2. Technical Report. Ope- nAI. https://cdn.openai.com/pdf/3a4153c8-c748-4b71-8e31-aecbde944f8d/oai_ 5_2_system-card.pdf

    2025

  29. [36]

    OpenAI. 2026. GPT-5.4 Thinking System Card. Technical Report. OpenAI. https: //openai.com/index/gpt-5-4-thinking-system-card/

    2026

  30. [37]

    Zhongqiang Pan, Chuanyi Li, Wenkang Zhong, Yi Feng, Bin Luo, and Vincent Ng. 2026. RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repair. arXiv preprint (2026). arXiv:2603.01048. doi:10. 48550/arXiv.2603.01048

    2026 doi

  31. [38]

    Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen, and Xi- aodong Gu. 2026. SWE-QA: Can Language Models Answer Repository-level Code Questions?. In Findings of the Association for Computational Linguistics: ACL 2026. doi:10.18653/v1/2026.findings-acl.402

    2026 doi

  32. [39]

    Jielin Qiu, Zuxin Liu, Zhiwei Liu, Rithesh Murthy, Jianguo Zhang, Haolin Chen, Shiyu Wang, Ming Zhu, Liangwei Yang, Juntao Tan, Zhepeng Cen, Cheng Qian, Shelby Heinecke, Weiran Yao, Silvio Savarese, Caiming Xiong, and Huan Wang

  33. [40]

    Qwen Team. 2025. Qwen3-Max: Just Scale it. https://qwen.ai/blog?id=qwen3-max. Blog post

    2025

  34. [41]

    Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen. ai/blog?id=qwen3.5. Official release blog

    2026

  35. [42]

    Tobias Roehm, Rebecca Tiarks, Rainer Koschke, and Walid Maalej. 2012. How Do Professional Developers Comprehend Software?. In Proceedings of the 34th International Conference on Software Engineering (ICSE). IEEE, 255–265. doi:10. 1109/ICSE.2012.6227188

    2012

  36. [43]

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2024. A Long Way to Go: Investigating Length Correlations in RLHF. In Proceedings of the 1st Conference on Language Modeling (COLM). arXiv:2310.03716. doi:10.48550/ arXiv.2310.03716

    2024 doi arXiv

  37. [44]

    Shubham Ugare and Satish Chandra. 2026. Agentic Code Reasoning. arXiv preprint (2026). arXiv:2603.01896. doi:10.48550/arXiv.2603.01896

    2026 doi

  38. [45]

    xAI. 2026. Grok 4.20. https://docs.x.ai/developers/models. Official model docu- mentation

    2026

  39. [46]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. De- mystifying LLM-based Software Engineering Agents. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering (FSE 2025). doi:10.1145/3715754

    2025 doi

  40. [47]

    Yisen Xu, Jinqiu Yang, and Tse-Hsun Peter Chen. 2026. SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring.arXiv preprint (2026). arXiv:2602.03712. doi:10.48550/arXiv.2602.03712

    2026 doi

  41. [48]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv preprint (2024). arXiv:2405.15793. doi:10.52202/079017-1601

    2024 arXiv

  42. [50]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. 2024. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists ASE ’26, October 12–16, 2026, Munich, G...

    2024 doi

  43. [2025]

    arXiv preprint (2025)

    LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering. arXiv preprint (2025). arXiv:2509.09614. doi:10.48550/arXiv.2509.09614

    2025 doi arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.