Pith. sign in

REVIEW 3 major objections 5 minor 52 references

Turning the Tide: Repository-based Code Reflection

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes LiveRepoReflection as a harder, less contaminated benchmark for repository-based code reflection, and shows that fine-tuning on its multi-turn instruction data lifts a base code model's repair pass rate markedly.

desk verdict A useful, well-scaled benchmark with real training gains, but the LLM-generated test oracles need independent audit before the scores can be trusted. read the letter →

arxiv 2507.09866 v1 pith:I2WARCMF submitted 2025-07-14 cs.SE cs.AI

Wei Zhang , Jian Yang , Jiaxi Yang , Ya Wang , Zhoujun Li , Zeyu Cui , Binyuan Hui , Junyang Lin This is my paper
classification cs.SEcs.AI
keywords codereflectionrepository-levelgenerationmulti-fileeditingLLMbenchmarkinstructiontuningrepaircontamination-freeevaluationPass@1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that code LLMs should be judged on their ability to modify existing code inside multi-file repositories, not just generate standalone functions. To make that measurable, it introduces LiveRepoReflection, a 1,888-problem benchmark spanning six programming languages that is designed to be hard and resistant to data contamination. It also introduces RepoReflection-Instruct, a large instruction dataset built from simulated multi-turn coding dialogues, and trains RepoReflectionCoder, which outperforms its Qwen2.5-Coder base on both full-file and patch-based editing. What makes the benchmark distinctive is its claim to separate debugging ability from generation ability: Fix Weight scores on LiveRepoReflection barely correlate with those on Aider Polyglot, which the paper reads as evidence that the new benchmark challenges repair skills that older benchmarks miss.

What carries the argument

The load-bearing object is the automatic generation-and-verification pipeline: a 'creative' LLM writes problem topics and definitions, several 'reasoning' LLMs independently write unit tests and reference answers, and every test-answer pair is cross-executed in a sandbox; for each problem the pipeline keeps the unit test with the lowest pass rate and the reference answer with the highest pass rate. A difficulty filter then discards cases that ten strong LLMs solve on the first try, and eight graduate students manually review the survivors. The evaluation uses Pass@1, Pass@2, Fix Weight (the fraction of second-attempt successes that are genuine fixes) and Well-Formed format compliance, with outputs measured in both full-file and patch-based edit formats.

What would settle it

For a random sample of 100 LiveRepoReflection problems, check whether a stub implementation that ignores the problem text (for example, a function returning hardcoded values) passes the kept unit test; if more than a handful of stubs pass, the test suite does not enforce the described behavior and Pass@1 overstates actual code-reflection ability.

Watch

Extended reading notes

Core claim

The paper's central claim is that LiveRepoReflection reliably discriminates model debugging capabilities in repository-based code reflection, more so than Aider Polyglot. In head-to-head comparisons, Pass@1 scores on the two benchmarks correlate only moderately (R2 about 0.65 for full-file, 0.52 for patch-based edits), while Fix Weight alignment is nearly absent (R2 about 0.21 full-file and 0.04 patch-based), and nearly all models score lower in absolute terms on LiveRepoReflection, indicating higher difficulty. On the training side, RepoReflectionCoder, fine-tuned from Qwen2.5-Coder-32B-Instruct on RepoReflection-Instruct, improves full-file Pass@1 from 6.6 to 9.0 and patch-based Pass@1 from 6.2 to 10.5, while still trailing top closed-source models.

Load-bearing premise

The entire benchmark rests on the assumption that the LLM-generated unit tests and reference answers, after cross-execution and eight graduate-student checks, are correct and match what the problem text asks; if a kept test is trivially satisfiable, too strict, or mismatched, every reported Pass@1, Pass@2, and Fix Weight score would mis-measure code reflection.

Editorial extensions

If this is right

  • Fix Weight on LiveRepoReflection is nearly uncorrelated with Fix Weight on Aider Polyglot, so repair ability is a distinct axis of model capability that existing benchmarks underweight.
  • Fine-tuning on error-driven multi-turn repair dialogues is a concrete recipe: RepoReflectionCoder lifts full-file Pass@1 from 6.6 to 9.0 and patch-based Pass@1 from 6.2 to 10.5 over its Qwen2.5-Coder-32B-Instruct base.
  • The decontamination pipeline—fresh sources, removal of Aider-overlapping problems, and MinHash/LSH filtering—gives the leaderboard a mechanism to resist models trained on the benchmark data.
  • Reporting both edit formats reveals format effects: patch-based edits yield more uniform Pass@k gains across k, while full-file generation reaches higher absolute pass rates.
  • Because LiveRepoReflection contains over eight times as many cases as Aider Polyglot (1,888 vs. 225) with richer multi-file structure, it provides a broader testbed for repository-level code understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The difficulty filter keeps cases where most of ten strong LLMs fail even after one repair attempt, and the selected unit test is the lowest-passing one; scores on such cases may partly measure how precisely the test captures the problem text rather than code reflection itself.
  • The same automatic pipeline could be adapted to other repository tasks such as security patch generation, refactoring under lint constraints, or dependency upgrades, where 'reflection' means detecting and applying context-specific changes.
  • A stress-test extension would replace reference answers with deliberately buggy implementations and check that the kept unit tests fail them; if a stub that returns hardcoded values passes, the benchmark's difficulty claim would be weakened.
  • The lack of correlation in Fix Weight suggests that a model can score high on generation yet be weak at using its own error messages, so a practical use of this benchmark is to identify models that would underperform in agentic coding workflows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces LiveRepoReflection, a benchmark of 1,888 repository-style coding problems in six languages, built with an automated pipeline that combines LLM-generated problems, unit tests, and reference answers with sandbox cross-execution and human review. It also presents RepoReflection-Instruct, a large instruction-tuning corpus, and RepoReflectionCoder, a fine-tuned model that improves over its Qwen2.5-Coder base on both full-file and patch-based code editing. The paper reports evaluations of over 40 LLMs, including comparisons with the Aider Polyglot benchmark, and analyzes Pass@k curves and format compliance.

Significance. If the benchmark oracles are valid, LiveRepoReflection is a timely and useful contribution: it is substantially larger than Aider Polyglot, covers six languages, includes contamination-control steps, and the leaderboard provides a reproducible comparison of repository-level code reflection. The two-turn training recipe and the public dataset could benefit the code-LLM community. The paper also ships a concrete artifact (the benchmark, the instruction corpus, and the trained model) and reports machine-verified pass rates, which is a strength. However, because the central claims about difficulty, discrimination, and the benefit of RepoReflectionCoder all depend on the correctness of 1,888 LLM-generated and LLM-selected test oracles, the benchmark validity is the crux of the assessment.

major comments (3)
  1. [Section 3 and Figure 2, Step 6]
  2. [Section 6.1, Figure 5]
  3. [Sections 5.1, 5.5, 9, Table 1 caption]
minor comments (5)
  1. [Section 2, Figure 2]
  2. [Section 3, 'Selection of Executable Program']
  3. [Section 3, 'Dataset Statistics']
  4. [Section 5.2, 'Fix Weight (FW)']
  5. [References and Appendix]

Circularity Check

2 steps flagged · score 4.0 of 10

The benchmark's 'high difficulty' and 'better debugging discrimination' claims are partly restatements of the LLM pass-rate filter used to construct it; the model-training improvements are independent.

  1. self definitional [Section 3, 'Selection of Difficult Problems'; Section 6.1, 'Performance Comparison between LiveRepoReflection and Aider Polyglot Benchmark']
    "If more than half of the LLMs only get 'failure-failure' but there are still LLMs that can complete it, we think these code program cases have a high degree of difficulty and are more suitable for evaluating most LLMs. ... nearly all points fall below the y = x line, indicating consistently lower absolute pass rates on LiveRepoReflection and thus greater task difficulty."

    The benchmark's difficulty label is defined by the pass/fail outcomes of the same 10 LLMs used for selection: cases are retained as 'high degree of difficulty' when more than half of those LLMs fail both the first attempt and the repair attempt. The later comparison then treats the resulting low absolute pass rates on LiveRepoReflection as evidence of 'greater task difficulty' and of more reliable discrimination of debugging ability. Because the selection rule deliberately discarded cases with high LLM pass rates, the direction of the difficulty comparison is fixed by construction; it restates the filter rather than deriving an independent fact. The cross-benchmark R2 correlations are empirical, so the circularity is partial rather than total.

  2. self definitional [Section 5.2, 'Evaluation Metrics'; Section 6.1, 'Performance Comparison between LiveRepoReflection and Aider Polyglot Benchmark']
    "Fix Weight (FW). Defined as FW = Pass@2−Pass@1 / Pass@2, it represents the fraction of successful second-attempt fixes among all second-attempt successes... fix-weight exhibits very weak alignment ( R2 ≈ 0.21 and 0.04), underscoring divergent repair performance and demonstrating that LiveRepoReflection more reliably challenges and discriminates model debugging capabilities."

    FW is, by definition, a normalized difference of Pass@1 and Pass@2, and Pass@2 is exactly the 'failure-success' outcome used in the benchmark's difficulty filter, where each LLM has one chance to modify its answer after a failed first attempt. Using the weak correlation of FW with Aider to conclude that LiveRepoReflection 'more reliably challenges and discriminates model debugging capabilities' therefore does not validate an independent reflection construct; it re-describes the same pass-rate statistics that selected the tasks. The weak R2 comparison against Aider is an external empirical element, so this is partial rather than full circularity.

full rationale

The clearest circular step is the difficulty claim. Section 3 defines difficulty directly from the pass/fail outcomes of 10 LLMs, keeping cases where more than half fail both attempts, and Section 6.1 then reports lower pass rates on LiveRepoReflection than Aider and interprets this as 'greater task difficulty' and more reliable discrimination of debugging. That direction is fixed by the selection filter, so the high-difficulty/discrimination claim is partly by construction. Similarly, FixWeight is defined as a normalized difference of Pass@1 and Pass@2, and Pass@2 is the same 'failure-success' signal used in selection; using its cross-benchmark weak correlation as evidence of better debugging discrimination is partly a re-description of the selection statistics, though the R2 comparison to Aider adds some empirical content. The concern about LLM-generated oracles is a correctness risk rather than a circularity and does not affect this score. There is no load-bearing self-citation chain: the cited ExecRepoBench and preference-alignment works share authors but are not used to justify the benchmark's core validity. The RepoReflectionCoder improvement (full-file Pass@1 from 6.6 to 9.0; patch Pass@1 from 6.2 to 10.5) is a self-contained empirical comparison on held-out tests and is not circular. Overall score 4 reflects partial circularity in the interpretive difficulty and discrimination claims, while the benchmark construction and training results retain independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark's validity rests on LLM-generated test and answer correctness, selection-by-difficulty rules, contamination-freeness, and the choice of Pass@2/FixWeight as a reflection measure. Construction thresholds and quality weights are hand-set; no invented theoretical entities are introduced.

free parameters (4)
  • Quality scoring weights w = (0.3, 0.2, 0.2, 0.15, 0.15) = (0.3, 0.2, 0.2, 0.15, 0.15)
    Hand-chosen weights in Section 4 determine which training examples are kept for RepoReflection-Instruct; no ablation or sensitivity analysis is reported.
  • Decontamination Jaccard threshold = 0.8
    Section 4 uses Jaccard similarity greater than 0.8 to filter training data against the test set; the threshold is chosen without sensitivity analysis.
  • Difficulty retention rules = keep if more than half of 10 LLMs fail-fail; discard if all succeed
    Section 3 uses these hand-set rules to decide which generated cases enter the final 1,888-case benchmark, so they directly shape every reported score.
  • Sandbox timeout = 180 seconds
    Section 3 discards cases that exceed 180 seconds of running time, which changes the difficulty and language distribution of the benchmark.
assumptions (5)
  • domain assumption LLM-generated unit tests and reference answers are correct and aligned with the problem definition after cross-execution and manual review.
    Section 3 (Cross-execution Verification and Human Annotation); if tests or answers are wrong, Pass@1/Pass@2 do not measure code reflection.
  • domain assumption Pass/fail outcomes of 10 selected strong LLMs are a valid proxy for task difficulty and for discriminating reflection ability.
    Section 3 (Selection of Difficult Problems); the same models later appear in the leaderboard, introducing selection bias.
  • domain assumption The benchmark is contamination-free because it was compiled from repositories not seen through May 2025 and decontaminated with MinHash/LSH.
    Section 6.1; this is unverifiable from the preprint because test generation uses public code corpora that may overlap pretraining data.
  • domain assumption One error-driven retry, measured as Pass@2 and FixWeight, is a valid operationalization of code reflection.
    Section 5.2; reflection is not externally validated against human labels or another reflection benchmark.
  • domain assumption Human annotation by eight graduate students with LLM assistance is sufficient to guarantee benchmark correctness.
    Section 3 (Human Annotation); no inter-annotator agreement or independent audit is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Turning the Tide: Repository-based Code Reflection." pith.science (2026). https://pith.science/paper/I2WARCMF

@misc{pith2026250709866,
  author       = {Pith},
  title        = {Pith review of: Turning the Tide: Repository-based Code Reflection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I2WARCMF}},
  note         = {Machine review of arXiv:2507.09866}
}
abstract

Code large language models (LLMs) enhance programming by understanding and generating code across languages, offering intelligent feedback, bug detection, and code updates through reflection, improving development efficiency and accessibility. While benchmarks (e.g. HumanEval/LiveCodeBench) evaluate code generation and real-world relevance, previous works ignore the scenario of modifying code in repositories. Considering challenges remaining in improving reflection capabilities and avoiding data contamination in dynamic benchmarks, we introduce LiveRepoReflection, a challenging benchmark for evaluating code understanding and generation in multi-file repository contexts, featuring 1,888 rigorously filtered test cases across $6$ programming languages to ensure diversity, correctness, and high difficulty. Further, we create RepoReflection-Instruct, a large-scale, quality-filtered instruction-tuning dataset derived from diverse sources, used to train RepoReflectionCoder through a two-turn dialogue process involving code generation and error-driven repair. The leaderboard evaluates over 40 LLMs to reflect the model performance of repository-based code reflection.

Figures

Figures reproduced from arXiv: 2507.09866 by the authors.

Figure 1
Figure 1. Comparison between general code generation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of Construction Pipeline. We use this pipeline to generate polyglot repository code data of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Two evaluation edit format of LiveRepoRe [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Performance comparison between LiveRepoReflection and the Aider Polyglot Benchmark over multiple [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Pass@k (k=1–10) curves for nine LLMs under full-file code generation (left) and patch-based incremental [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Polyglot repository code file structure examples of LiveRepoReflection and instruction corpus of [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: System Prompt for Our Pipeline. When Creating files, maintain a consistent folder structure as shown in the examples by providing the appropriate path/to/filename and adhere to the following format: path/to/filename “‘ // entire code or file content ... “‘ - The first …
Figure 9
Figure 9. Figure 9: Format Reminder Prompt for Our Pipeline. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Sample Data Template Prompt for Our Pipeline. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Coding Program Definition Prompt for Our Pipeline. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Coding Program Topic for Our Pipeline. Please supply a comprehensive {language} unit test for this question. DO NOT include any answer or any other things at this stage. {format_reminder} Attention to follow and implement the ‘{project_name}‘ project structure, each f…
Figure 13
Figure 13. Figure 13: Unit Test Prompt for Our Pipeline. Please supply a comprehensive {language} answer and necessary dependencies for this question. {format_reminder} Attention to follow and implement the ‘{project_name}‘ project structure, each file should replace in ‘{project_name}‘ fo…
Figure 14
Figure 14. Figure 14: Reference Answer Prompt for Our Pipeline. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Optional End Suffix Prompt Examples for Our Pipeline. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: “python”: “pytest” “rust”: “cargo test – –include-ignored” “go”: “go test ./...” “javascript”: “./npm-test.sh” “cpp”: “./cpp-test.sh” “java”: “./gradlew test –no-daemon” [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 11 canonical work pages

  1. [1]

    https://aider.chat/docs/leaderboards/ Aider llm leaderboards

    Aider Team . https://aider.chat/docs/leaderboards/ Aider llm leaderboards

    work page

  2. [2]

    Gemini: A family of highly capable multimodal models

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. corr, abs/2312.11805, 2023. doi: 10.48550. arXiv preprint ARXIV.2312.11805, pages 24--28

    arXiv 2023

  3. [3]

    Anthropic. 2023. https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf Claude 2 . Technical report, Anthropic

    work page 2023

  4. [4]

    Anthropic. 2025. https://www.anthropic.com/news/claude-3-7-sonnet Claude 3.7 sonnet and claude code

    2025

  5. [5]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. http://arxiv.org/abs/2108.07732 Program synthesis with large language models . CoRR, abs/2108.07732

    arXiv 2021

  6. [6]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. https://doi.org/10.1109/TSE.2023.3267446 Multipl-e: A scalable and polyglot approach to benchmarking neural code generation . IEEE Trans...

    arXiv 2023

  7. [7]

    Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. 2024. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436

    arXiv 2024

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \' e de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad B...

    arXiv 2021

Show all 52 references
  1. [9]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2025 arXiv

  2. [10]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.139 Codebert: A pre-trained model for programming and natural languages . In Findings of ...

    2020 doi

  3. [11]

    Paul Gauthier. 2024 a . G P T code editing benchmarks --- aider.chat. https://aider.chat/docs/leaderboards/#polyglot-leaderboard. [Accessed 28-01-2025]

    2024

  4. [12]

    Paul Gauthier. 2024 b . G P T code editing benchmarks --- aider.chat. https://aider.chat/docs/benchmarks.html#the-benchmark. [Accessed 21-01-2025]

    2024

  5. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  6. [14]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024 a . http://arxiv.org/abs/2401.14196 Deepseek-coder: When the large language model meets programming -- the rise of co...

    2024 arXiv

  7. [15]

    Chawla, Olaf Wiest, and Xiangliang Zhang

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. 2024 b . https://doi.org/10.48550/ARXIV.2402.01680 Large language model based multi-agents: A survey of progress and challenges . CoRR, abs/2402.01680

    2024 doi arXiv

  8. [16]

    Siming Huang, Tianhao Cheng, J. K. Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. 2025. http://arxiv.org/abs/2411.04905 Openco...

    2025 arXiv

  9. [17]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  10. [18]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  11. [19]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. https://arxiv.org/abs/2403.07974 Livecodebench: Holistic and contamination free evaluation of large language models for code . arXiv prepr...

    2024 arXiv

  12. [20]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. https://openreview.net/forum?id=VTF8yNQM66 Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning R...

    2024

  13. [21]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy - Poirier, Jo \ a o Mont...

    2023 doi arXiv

  14. [22]

    Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, et al. 2024 a . Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310

    2024 arXiv

  15. [23]

    Siyao Liu, He Zhu, Jerry Liu, Shulin Xin, Aoyan Li, Rui Long, Li Chen, Jack Yang, Jinxiang Xia, ZY Peng, et al. 2024 b . Fullstack bench: Evaluating llms as full stack coder. arXiv preprint arXiv:2412.00535

    2024 arXiv

  16. [24]

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan - Bach Dinh Le, and David Lo. 2023. https://doi.org/10.48550/ARXIV.2307.12596 Refining chatgpt-generated code: Characterizing and mitigating code quality issues . CoRR, abs/2307.12596

    2023 doi arXiv

  17. [25]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and S...

    2021

  18. [26]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. https://doi.org/10.48550/ARXIV.2306.08568 Wizardcoder: Empowering code large language models with evol-instruct . CoRR, abs/2306.08568

    2023 doi arXiv

  19. [27]

    Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. 2025. Swe-lancer: Can frontier llms earn \ 1 million from real-world freelance software engineering? arXiv preprint arXiv:2502.12115

    2025 arXiv

  20. [28]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. https://openreview.net/pdf?id=iaYcJKpY2B\_ Codegen: An open large language model for code with multi-turn program synthesis . In The Eleventh International Confer...

    2023

  21. [29]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  22. [30]

    OpenAI. 2025. https://openai.com/index/introducing-gpt-4-5/ Introducing gpt-4.5

    2025

  23. [31]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  24. [32]

    Qwen. 2025. https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: Embracing the power of reinforcement learning

    2025

  25. [33]

    Houxing Ren, Mingjie Zhan, Zhongyuan Wu, Aojun Zhou, Junting Pan, and Hongsheng Li. 2024. http://arxiv.org/abs/2405.17057 Reflectioncoder: Learning from reflection sequence for enhanced one-off code generation

    2024 arXiv

  26. [34]

    Baptiste Rozi \` e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \' e r \' e my Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton - Ferrer, Aaron Grattafiori, Wenhan Xiong,...

    2023 doi arXiv

  27. [35]

    ByteDance Seed, :, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, Yufeng Yuan, Yu Yue, Lin Yan, Qiying Yu, Xiaochen Zuo, Chi Zhang, Ruofei Zhu, Zhecheng An, Zhihao Bai, Yu Bao, Xingyan Bin, Jiangjie Chen, Feng...

    2025

  28. [36]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://doi.org/10.485...

    2023 doi arXiv

  29. [37]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji - Rong Wen. 2023 a . https://doi.org/10.48550/ARXIV.2308.11432 A survey on large language model based autonomous agents . Co...

    2023 doi arXiv

  30. [38]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023 b . https://doi.org/10.18653/V1/2023.ACL-LONG.754 Self-instruct: Aligning language models with self-generated instructions . In Proceedings of the 61st Annual ...

    2023 doi

  31. [39]

    Joty, and Steven C

    Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021. https://doi.org/10.18653/V1/2021.EMNLP-MAIN.685 Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation . In Proceedings of the 2021 Conference on Empirical Met...

    2021 doi

  32. [40]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. https://doi.org/10.48550/ARXIV.2312.02120 Magicoder: Source code is all you need . CoRR, abs/2312.02120

    2023 doi arXiv

  33. [41]

    xAI. 2025. https://x.ai/news/grok-3 Grok 3 beta — the age of reasoning agents

    2025

  34. [42]

    Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin. 2023. https://doi.org/10.48550/ARXIV.2305.04764 Chatunitest: a chatgpt-based automated unit test generation tool . CoRR, abs/2305.04764

    2023 doi arXiv

  35. [43]

    Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. 2023. https://aclanthology.org/2023.findings-emnlp.337 Codetransocean: A comprehensive multilingual benchmark for code translation . In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapo...

    2023

  36. [44]

    An Yang, feng Li, and etc. 2024 a . Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2024 arXiv

  37. [45]

    Jian Yang, Jiaxi Yang, Ke Jin, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Binyuan Hui, and Junyang Lin. 2024 b . Evaluating and aligning codellms on human preference. arXiv preprint arXiv:2412.05210

    2024 arXiv

  38. [46]

    Jian Yang, Jiajun Zhang, Jiaxi Yang, Ke Jin, Lei Zhang, Qiyao Peng, Ken Deng, Yibo Miao, Tianyu Liu, Zeyu Cui, et al. 2024 c . Execrepobench: Multi-level executable code completion evaluation. arXiv preprint arXiv:2412.11990

    2024 arXiv

  39. [47]

    Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. 2023. https://doi.org/10.48550/ARXIV.2312.14187 Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation . CoRR, abs/2312.14187

    2023 doi arXiv

  40. [48]

    Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023. https://doi.org/10.48550/ARXIV.2303.16199 Llama-adapter: Efficient fine-tuning of language models with zero-init attention . CoRR, abs/2303.16199

    2023 doi arXiv

  41. [49]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. https://doi.org/10.48550/ARXIV.2303.17568 Codegeex: A pre-trained model for code generation with multilingual evaluations on h...

    2023 doi arXiv

  42. [50]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877

    2024 arXiv

  43. [51]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  44. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.