Pith. sign in

REVIEW 5 major objections 8 minor 1 cited by

Coding Triangle: How Does Large Language Model Understand Code?

T0 review · 5 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that a model's coding competence forms a self-consistent triangle of editorial, code, and test cases, and that this self-consistency—not lack of raw skill—makes single-model solutions narrow and error-prone.

desk verdict A useful three-axis evaluation framework for coding LLMs, but the headline self-consistency claim rests on an unvalidated LLM judge for the editorial axis. read the letter →

arxiv 2507.06138 v1 pith:33AYQCE7 submitted 2025-07-08 cs.CL cs.AI

Taolin Zhang , Zihan Ma , Maosong Cao , Junnan Liu , Songyang Zhang , Kai Chen This is my paper
classification cs.CLcs.AI
keywords CodeTriangleLLMgenerationself-consistencytestcaseeditorialanalysismodelmixturecompetitiveprogrammingdistributionshift
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what it means for a large language model to "understand" a programming problem, and answers by splitting coding competence into three measurable abilities: writing a natural-language analysis (editorial), implementing a correct program (code), and generating test cases that catch wrong submissions (cases). Testing these three dimensions on 200 competitive programming problems from AtCoder, the paper finds that models are largely self-consistent: their editorials, code, and cases reflect the same internal understanding, so failures cluster and repeated sampling repeats similar mistakes. That self-consistency is the paper's central finding: it explains why a model can be confident and wrong, why self-generated test cases can make weak solutions look correct, and why mixing several models produces more diverse solutions and test cases. The paper also finds self-inconsistency—a model may correctly judge its own bad code or generate sharp edge cases yet fail to integrate those strengths into a final solution—which it points to as an opening for self-reflection and iterative self-improvement.

What carries the argument

The load-bearing mechanism is the Code Triangle framework itself: editorial, code, and cases form the three vertices, each sampleable from human or model sources, and the six directed edges between them are probed by feeding one vertex's output into another and measuring the effect. Editorial quality is scored by an LLM judge (o3-mini) against the official editorial, code quality by Pass@1 on official test cases, and case quality by whether the model-generated cases reject all wrong human submissions. To measure diversity and distribution shift, the paper builds a performance matrix over solutions and test cases, computes cosine similarity between pass/fail vectors, and counts the size of the set of unique vectors; this construction is what makes the model-mixture and self-consistency arguments quantitative.

What would settle it

Take a random sample of 50–100 problems from the same AtCoder set, have expert competitive programmers independently score the model-generated editorials using the paper's binary rubric, and compare those scores with o3-mini's. If expert scores disagree substantially with o3-mini, or if expert-scored editorial performance no longer tracks code and case performance across difficulty levels, the paper's self-consistency conclusion would fail its weakest leg.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that LLMs exhibit a prevalent self-consistency across the three vertices of the Coding Triangle: editorial analysis, code implementation, and test case generation. Because all three dimensions arise from the same learned cognition, solutions generated by a model are highly similar to each other (pairwise pass/fail vector similarity mostly above 0.8), errors cluster around the same corner cases, and supplying a model with its own editorial or its own test cases does little to change outcomes. The paper interprets this as a distribution shift from human expertise, since human submissions show much more diverse error patterns. At the same time, the three abilities are not fully aligned: a model can recognize its own failed solutions as incorrect and use its own cases to discriminate correct from incorrect code, yet these strengths do not automatically translate into a correct final solution. The paper concludes that injecting human-generated editorials, solutions, and cases, and especially ensembling multiple models, mitigates the bias and improves robustness, while the observed self-inconsistency suggests a route toward self-reflection and self-improvement.

Load-bearing premise

The editorial score is produced by an LLM judge (o3-mini) with no human validation or calibration, so if that judge cannot reliably tell correct from incorrect analyses—or simply prefers editorials from similar training distributions—the reported consistency between editorial and code ability would be an artifact.

Editorial extensions

If this is right

  • A single model's coding ability cannot be reliably estimated from one dimension alone; editorial or case performance can be high while code still fails on medium-difficulty problems.
  • Self-validation with a model's own generated test cases is not trustworthy evidence of correctness—self-generated cases can accept a solution that official cases reject by up to a 40-percentage-point gap.
  • Mixing multiple models, or adding human-authored editorials, solutions, and test cases, measurably widens unique error coverage, so robustness can be improved without retraining.
  • The gap between editorial and code performance on medium-difficulty tasks suggests that reasoning-level interventions matter less there than implementation-level fixes.
  • A model's ability to judge its own incorrect solution is strongest on the hardest problems and weakest in the middle, pointing self-reflection efforts toward the medium-difficulty cases where false confidence peaks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the editorial dimension rests on an LLM judge without human calibration, the cleanest test of the self-consistency finding is to re-score a sample of editorials with expert humans; if human scores diverge, the editorial leg of the triangle needs re-grounding.
  • The same performance-matrix and unique-vector analysis transfers to other generation tasks, such as mathematical proof generation, bug localization, or natural-language planning, as a way to measure cognitive diversity and predict when ensembling will help.
  • The paper notes it does not explore all six directed interactions fully; a natural extension is cross-model cycles, such as using one model's cases to judge another model's code, which should break single-model self-consistency and reveal whether disagreement itself predicts correctness.
  • In deployment, the self-consistency finding implies a practical guardrail: treat a generated solution as provisional until it passes both self-generated and independent cases, and use disagreement across models as a signal to escalate to human review.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. This paper proposes the "Coding Triangle" framework for evaluating LLMs on three dimensions of programming ability: editorial analysis (natural-language problem explanation), code implementation (passing official tests), and test case generation (cases that identify incorrect human submissions), plus the six directed interactions among these dimensions. The authors evaluate four LLMs (DeepSeek-V3, Qwen2.5-72B-Instruct, Qwen2.5-Coder-32B-Instruct, QwQ-32B) on 200 AtCoder problems (ABC 175-374, difficulties A-F). Their main findings are: (i) a "prevalent self-consistency" across dimensions, with editorial scores consistently 0-20% above code scores (Figure 2); (ii) a distribution shift from human submissions, in that model-generated solutions and cases are more homogeneous than human ones, while mixing models increases diversity (Figures 3 and 12); (iii) self-inconsistencies such as self-generated cases being lenient (pass rates up to 40% higher than on official cases, Figure 7) and models being able to recognize their own mistakes on hard problems (Figure 6). The paper concludes with recommendations for self-reflection, human-information injection, and model ensembling.

Significance. If the findings hold, the Coding Triangle framework provides a useful multi-dimensional evaluation lens that goes beyond pass@1, and the documented phenomena — leniency of self-generated tests, limited benefit from self-editorials, and diversity gains from model mixtures — are falsifiable predictions that could guide self-improvement and ensembling research. The paper is honest about its own weak points: it states in Section 3.2 that editorial evaluation is a subjective LLM-as-judge assessment, and the appendix makes all prompts available. Notably, the code and case metrics (Eqs. 2 and 3) are anchored to objective ground truth (execution on official tests and human submission outcomes), so the core distribution-shift comparisons in Section 3.2 are not circular, and the solution/case-judging experiments in Sections 4.2-4.3 have objective correctness labels. However, the strength of the claims currently exceeds the rigor of the evidence: the editorial metric is unvalidated, quantitative trends lack error bars, and the case metric as defined does not penalize over-strict case sets.

major comments (5)
  1. [Section 2.2, Eq. (1)] The editorial score is a single binary judgment by o3-mini per problem per model, with no human validation, no inter-annotator agreement, and no calibration set. The central claims of Section 3.1 — that editorial and code abilities are "consistent" and that there is a persistent 0-20% gap between them (Figure 2) — rest entirely on this metric, and the objective metrics in Eqs. (2)-(3) cannot validate it: a judge that systematically over-credits fluent but flawed editorials would manufacture both the consistency and the gap. The statement in Section 3.2 that "editorial evaluation is based on subjective assessment using LLM-as-Judge" acknowledges the issue but does not resolve it. Please validate the judge against human expert annotations on a representative subset (e.g., 50-100 editorials per model) and report agreement, or restrict the three-dimensional self-consistency claim to the two objective dimensions. As a concrete test of the risk, the authors could also report the judge's score distribution and its sensitivity to paraphrased but correct alternative editorials.
  2. [Section 2.2, Eq. (3)] The case score checks consistency only on H_wrong, the set of submissions rejected by the official tests. Since every h in H_wrong has J(h, T_i) = Rejected, the condition in Eq. (3) requires only that the model-generated cases also reject all wrong submissions; a case set that additionally rejects correct submissions is never penalized. Hence Scase measures recall on wrong submissions only, and the interpretation in Section 3.1 of the case score as capturing "validation criteria, including edge scenarios and boundary conditions" (Section 2.1) is not supported as stated. The non-monotonic difficulty trend and the rebound on problems E and F in Figure 2 are also confounded by the size and composition of H_wrong per problem, which is not reported. Please either extend Eq. (3) to all human submissions (requiring both rejection of wrong and acceptance of correct solutions), or report the false-positive rate on correct submissions and the distribution of |H_wrong| per problem.
  3. [Sections 3.1-4.3 (overall)] No error bars, reruns, or statistical tests accompany any of the quantitative claims: the 0-20% editorial-code gap (Section 3.1), the >0.8 cosine-similarity threshold (Section 3.2), the judge-accuracy trends (Section 4.2, Figure 6), the "up to 5% and 40%" pass-rate gaps (Section 4.3, Figure 7), and the model-mixture improvements (Figures 3c, 3d, and 12). With roughly 200 problems split across six difficulty levels (about 33 problems per difficulty), differences of a few percentage points are within sampling noise. Bootstrap confidence intervals or seeded reruns are needed for the claimed trends to be assessable; at minimum, the per-difficulty standard errors should be reported.
  4. [Section 3.2, Figures 3a-3d] The experimental setup for the diversity analysis is underspecified. The paper does not state the number of solutions generated per model per problem (roll-outs), the sampling temperature, the number and selection of human submissions per problem, the number of test cases per problem used in the performance matrices, or the sampling procedure for the 200 problems from ABC 175-374. The unique-set and cosine-similarity metrics are sensitive to the number of samples, so the claim that "human submissions are much more diverse" (Section 3.2) could partly reflect a larger human sample rather than intrinsic diversity. These details are load-bearing for the distribution-shift and model-mixture conclusions, which are the paper's second and third contributions.
  5. [Section 4.3, Figure 7] The protocol for the Self-Direct condition is underspecified. The paper notes in Section 2.2 that directly generated input-output pairs are "often insufficient, as most generated cases are incorrect and will be filtered out," yet the headline result is that Self-Direct pass rates exceed ground-truth pass rates by up to 40%. It is not stated whether the self-generated cases were filtered (by the LLM case judge or by execution against a reference) before computing pass rates, nor how the expected outputs were validated. Without this information, the "reward hacking" interpretation in Section 6 is plausible, but the quantitative claim cannot be distinguished from an artifact of the case-filtering or output-generation procedure.
minor comments (8)
  1. [Section 3.1] The sentence "we surprisingly find that the code score can even surpass editorial and code score on the most difficult problems" is internally incoherent and should presumably read "the case score can even surpass the editorial and code scores."
  2. [Figure 6] The human-solution panels (a)-(b) report difficulties A-G while the self-generated panels (c)-(d) report A-F; the paper does not explain why the difficulty ranges differ.
  3. [Section 3.2, Eq. (4)] The paper does not state how many test cases n are used in the performance matrix per problem, nor how solutions with identical test outcomes but different runtime behavior are treated; this affects the resolution of the cosine-similarity metric.
  4. [References] Reference [4] duplicates [3] (both Austin et al., 2021) and reference [9] duplicates [8] (both Chen et al., 2021); these duplicate entries should be merged.
  5. [Section 2.2, Eqs. (2)-(3)] The paper should clarify whether T_i in Eq. (3) refers to the public sample tests or to a fuller official test set, since the interpretation of J(h, T_i) as ground truth depends on which test set is used.
  6. [Section 4.3, Figure 9] The ground-truth label for whether a model-generated case is "correct" is not defined in the text; the authors should state how case correctness was determined (e.g., by executing the official solution on the generated input).
  7. [Section 4.2, Figure 6] The text refers to "judge accuracy," but the figures show stacked distributions of true/false positives/negatives without reporting the underlying counts of judged solutions per difficulty; the counts are needed to assess the reliability of the percentages.
  8. [Overall] The paper does not state whether the evaluation code, the generated case sets, or the human-submission subsets will be released; for reproducibility, the authors should provide these artifacts or at least the sampling details.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: code and case metrics are anchored to ground truth, and the unvalidated LLM editorial judge is a measurement-validity concern rather than a circular step.

full rationale

The paper's central comparisons are not forced by construction. The code score (Eq. 2) is Pass@1 against official ground-truth test cases, and the case score (Eq. 3) requires model-generated case sets to reproduce the official verdicts on all incorrect human submissions; both are objective. The editorial score (Eq. 1) is the only metric that depends on an LLM judge (o3-mini), and it lacks human calibration, but this is a validity risk for the editorial dimension, not a circular derivation: o3-mini is external to the models being measured, no fitted parameter is renamed as a prediction, and the editorial metric is not used as an input to any subsequent computation that would make the headline self-consistency claim true by definition. The paper even limits its distribution-shift analysis (Section 3.2) to the objective code and case metrics, explicitly acknowledging that editorial evaluation is subjective. The self-consistency, diversity, and model-mixture conclusions are direct observations of pass/fail vectors, cosine similarities, and unique-set sizes, not products of a self-citation chain or an imported uniqueness theorem. No load-bearing self-citation or ansatz-smuggling step was found.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on domain assumptions about judge validity, representativeness of problems and submissions, and the appropriateness of the similarity metric. No free parameters are fitted to make a theoretical derivation work; the listed parameters are experimental design choices that could affect the results.

free parameters (2)
  • number_of_generated_test_cases_per_problem = 50
    The case generation prompt asks for 50 diverse inputs; this hand-set number determines the case score and could affect all case-related conclusions.
  • cosine_similarity_threshold = 0.8
    Used in the similarity analysis narrative to describe 'most pairs above 0.8'; not a fitted parameter but a chosen interpretive threshold without sensitivity analysis.
assumptions (4)
  • domain assumption o3-mini LLM-as-judge provides a reliable correctness score for editorials.
    Used in Eq. (1) to compute Sedi; no human validation or calibration is reported.
  • domain assumption The set of wrong human submissions per problem is representative of all possible incorrect solutions.
    Used in Eq. (3) for case score; if the human submissions are not diverse enough, the case score misestimates the model's ability to catch errors.
  • domain assumption AtCoder ABC problems A-F are a representative sample of code problems.
    The paper generalizes from 200 AtCoder problems to LLM coding ability; this set may not reflect other programming domains.
  • domain assumption Cosine similarity on binary pass/fail vectors is a meaningful diversity measure.
    The diversity and distribution shift arguments rely on this metric; no justification or robustness check is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Coding Triangle: How Does Large Language Model Understand Code?." pith.science (2026). https://pith.science/paper/33AYQCE7

@misc{pith2026250706138,
  author       = {Pith},
  title        = {Pith review of: Coding Triangle: How Does Large Language Model Understand Code?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/33AYQCE7}},
  note         = {Machine review of arXiv:2507.06138}
}
read the original abstract

Large language models (LLMs) have achieved remarkable progress in code generation, yet their true programming competence remains underexplored. We introduce the Code Triangle framework, which systematically evaluates LLMs across three fundamental dimensions: editorial analysis, code implementation, and test case generation. Through extensive experiments on competitive programming benchmarks, we reveal that while LLMs can form a self-consistent system across these dimensions, their solutions often lack the diversity and robustness of human programmers. We identify a significant distribution shift between model cognition and human expertise, with model errors tending to cluster due to training data biases and limited reasoning transfer. Our study demonstrates that incorporating human-generated editorials, solutions, and diverse test cases, as well as leveraging model mixtures, can substantially enhance both the performance and robustness of LLMs. Furthermore, we reveal both the consistency and inconsistency in the cognition of LLMs that may facilitate self-reflection and self-improvement, providing a potential direction for developing more powerful coding models.

Figures

Figures reproduced from arXiv: 2507.06138 by the authors.

Figure 1
Figure 1. The framework of Coding Triangle. Editorial, code, and cases form the three fundamental vertices of the triangle, with each vertex can be sampled from either human solutions or model predictions. These vertices are interconnected, influencing one another, and their relationships form the six directed edges of the triangle, representing the mutual interactions between Editorial, Code, and Cases. Based on our Coding T… view at source ↗
Figure 2
Figure 2. Ability analysis across different dimensions. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Similarity analysis and unique set size for error analysis of codes and cases. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Pass@1 score with self-generated and ground truth editorials. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Case score w.r.t. ground truth editorials. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Judge distribution over solutions. which may differ from its training data. However, when the model evaluates its own solutions, we find that its judge accuracy actually increases for the hardest problems. This indicates that, it is still able to recognize its own mist…
Figure 7
Figure 7. Figure 7: Pass@1 score over Self-Generated cases. A B C D E F 0 25 50 75 100 Score w/o Code w/ GT-Code (a) Qwen2.5-72B-Instruct A B C D E F 0 25 50 75 100 Score w/o Code w/ GT-Code (b) Coder-32B-Instruct A B C D E F 0 25 50 75 100 Score w/o Code w/ GT-Code (c) DeepSeek-V3 A B C …
Figure 8
Figure 8. Figure 8: Case Score w.r.t. GT Solutions. From Cases to Code: Can self-generated solution pass all the self-generated cases? In this section, we explore another aspect of self-consistency: whether self-generated solutions can fully pass test cases produced by the model itself. W…
Figure 9
Figure 9. Figure 9: Judge accuracy over cases. 6 Discussion and Conclusion Model cognition of LLMs significantly differs from human distribution. Our large-scale experiments reveal that the distribution of solutions and cases they produce significantly differs from that of real-world data…
Figure 10
Figure 10. Figure 10: Pass@1 score of Coder-32B-Instruct with editorials from other models. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Judge distribution of DeepSeek-V3 over solutions from different models. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Case score of model mixture between DeepSeek-V3 and other models. [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Honest Quorum Problem: Epistemic Byzantine Fault Tolerance for Agentic Infrastructure

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A quorum of protocol-compliant but semantically mistaken AI validators can certify an invalid transition; EBFT derives threshold conditions that bound this risk with calibrated budgets eδ and uε.

Reference graph

Works this paper leans on

30 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Claude 3.5 sonnet

    Anthropic. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet ,

  3. [4]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  4. [5]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [6]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  6. [7]

    Multipl- e: A scalable and extensible approach to benchmarking neural code generation

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl- e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227, 2022

    arXiv 2022

  7. [9]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  8. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 30 references
  1. [11]

    Competitive programming with large reasoning models

    Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807, 2025

    2025 arXiv

  2. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  3. [13]

    Codeeditorbench: Evaluating code editing capability of large language models

    Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi Li, Ruibo Liu, Yue Wang, et al. Codeeditorbench: Evaluating code editing capability of large language models. arXiv preprint arXiv:2404.03543, 2024

    2024 arXiv

  4. [14]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  5. [15]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  6. [16]

    Mistral 7b (2023)

    AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b (2023). arXiv preprint arXiv:2310.06825, 2023. 10

    2023 arXiv

  7. [17]

    Competition-level code generation with alphacode

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022

    2022

  8. [18]

    Autokaggle: A multi-agent framework for autonomous data science competitions

    Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tianyu Zheng, Xinyao Niu, Xiang Yue, Yue Wang, Jian Yang, Jiaheng Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20424, 2024

    arXiv

  9. [19]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  10. [20]

    M2rc-eval: Massively multilingual repository-level code completion evaluation

    Jiaheng Liu, Ken Deng, Congnan Liu, Jian Yang, Shukai Liu, He Zhu, Peng Zhao, Linzheng Chai, Yanan Wu, Ke Jin, et al. M2rc-eval: Massively multilingual repository-level code completion evaluation. arXiv preprint arXiv:2410.21157, 2024

    arXiv

  11. [21]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558–21572, 2023

    2023

  12. [22]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  13. [23]

    OpenAI. Gpt-4o. https://openai.com/index/hello-gpt-4o , 2024. 2024.05.13

    2024

  14. [24]

    Openai o1 system card

    OpenAI. Openai o1 system card. https://openai.com/index/ openai-o1-system-card/ , 2025

    2025

  15. [25]

    Openai o3 system card

    OpenAI. Openai o3 system card. https://openai.com/index/ o3-o4-mini-system-card/ , 2025

    2025

  16. [26]

    Openai o3-mini system card

    OpenAI. Openai o3-mini system card. https://openai.com/index/openai-o3-mini/ , 2025

    2025

  17. [27]

    Qwen3: Think deeper, act faster

    Qwen Team. Qwen3: Think deeper, act faster. https://qwenlm.github.io/blog/qwen3/, 2025

    2025

  18. [28]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/

    2025

  19. [29]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  20. [30]

    Tablebench: A comprehensive and complex benchmark for table question answering

    Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, et al. Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint arXiv:2408.09174, 2024

    2024 arXiv

  21. [31]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  22. [32]

    the fair use of a copyrighted work, including such use by ... scholarship, or research, is not an infringement of copyright

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. 11 Appendix A Dataset and License The evaluating problem dataset is collected from...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.