Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims a 7-billion-parameter judge model, trained on reasoning paths distilled from a 671-billion-parameter teacher, can judge code correctness more accurately than GPT-4o and DeepSeek-V3 without reference solutions or test cases.

desk verdict A real artifact and a useful empirical study, but the headline comparison is undercut by an internal inconsistency in Table IV and an uneven protocol; the core idea is solid enough to warrant peer review. read the letter →

arxiv 2505.19502 v1 pith:CUNN5RFA submitted 2025-05-26 cs.SE cs.AI

Guang Yang , Yu Zhou , Xiang Chen , Wei Zheng , Xing Hu , Xin Zhou , David Lo , Taolue Chen This is my paper
classification cs.SEcs.AI
keywords codegenerationevaluationLLM-as-Judgeknowledgedistillationreasoningpathsreference-freetest-freepreferenceleakageparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that code evaluation need not depend on reference solutions or executable test cases. It proposes CODE-DITING, a judge model that reads a problem description and a candidate implementation, reasons about functional consistency, and outputs a correctness verdict with an explanation. The authors compare many LLM-as-judge baselines across three datasets, then distill reasoning paths from DeepSeek-R1-671B into 1.5B and 7B models trained on 17,000 filtered samples. They report that the 7B version, using seven majority votes, outperforms GPT-4o and DeepSeek-V3-671B while using roughly one percent of their parameters, and that the 1.5B version matches models five times its size. If correct, this makes accurate, explainable, reference-free code evaluation cheap enough to run on a single GPU.

What carries the argument

The load-bearing mechanism is a three-part training-and-inference pipeline: CODEJUDGE-17K, a 17,000-sample dataset of problem, code, label, and reasoning-path triples distilled from DeepSeek-R1-671B and filtered for agreement with test cases and logical coherence; parameter-efficient fine-tuning of DeepSeek-R1-distill bases using LoRA initialized with PiSSA, which aligns the low-rank updates with the pretrained model's principal singular directions; and majority voting over seven independent inferences at temperature 0.6 during evaluation. The reasoning path is the central object: it is what makes the distilled judge explainable and is the main factor isolated in the ablations, with data filtering and PiSSA each contributing further gains.

What would settle it

Run all baselines with the same k=7 majority-voting budget and pick all prompts and k values on a development split before seeing the test sets; if GPT-4o or DeepSeek-V3 then matches or beats CODE-DITING 7B on any of HumanEval-Judge, MBPP-Judge, or BigCodeBench-Judge, the paper's 'surpasses' claim for that comparison fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that explicit reasoning paths, not just labels, are the transferable ingredient that lets a small model judge code. CODE-DITING is a reference-free and test-free metric: given only a problem description and generated code, it produces a functional-correctness judgment plus a reasoning trace. The method builds the CODEJUDGE-17K dataset by having DeepSeek-R1-671B judge solutions under a simple Vanilla prompt, keeping only judgments that agree with test-case labels, filtering incoherent reasoning with a discriminator, and balancing classes; it then fine-tunes DeepSeek-R1-distill 1.5B and 7B bases with PiSSA-initialized LoRA and applies majority voting with k=7 at inference. The paper reports average accuracy of 0.806 for CODE-DITING 7B across HumanEval-Judge, MBPP-Judge, and BigCodeBench-Judge, above GPT-4o's 0.795 and DeepSeek-V3's 0.781, and second only to DeepSeek-R1-671B's 0.834.

Load-bearing premise

The headline claim that CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 rests on a protocol where CODE-DITING gets seven majority votes per sample while baselines judge once, and where each baseline's best prompting method was selected from the same test sets; equalizing the vote budget or choosing k before seeing test performance could erode the margin.

Editorial extensions

If this is right

  • Code evaluation can be decoupled from test execution: once a judge like CODE-DITING is trusted, new problems do not need hand-written test suites or reference implementations to be scored, removing a major bottleneck in scaling code-generation benchmarks.
  • Explainable judgments become a practical default: every verdict comes with a reasoning trace, so users can audit why a candidate was accepted or rejected instead of trusting an opaque score.
  • The reported parameter efficiency changes the deployment calculus: a 7B judge that rivals much larger models can run on a single GPU, making large-scale evaluation of code generators affordable for small teams and offline settings.
  • If the preference-leakage results generalize, judges trained on one family of code generators can fairly score code from other families, reducing a known source of bias in LLM-as-judge evaluation.
  • The authors point toward using CODE-DITING as a reward environment for reinforcement-learning training of code generators; an accurate reference-free judge would supply a dense, continuous correctness signal where execution is unavailable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The headline comparison is sensitive to the inference budget, because CODE-DITING receives k=7 majority votes while each baseline appears to receive a single judgment, and the optimal k and best baseline prompts were selected from the same test sets; equalizing the vote budget or fixing k before seeing test performance could reduce or reverse the reported margins.
  • Editorial inference: The three judge benchmarks are all function-level, English, high-coverage Python-style tasks, so the distillation pipeline has not been shown to transfer to repository-scale code, multi-file patches, or other programming languages; the generality of the functional-alignment claim remains open.
  • Editorial inference: A direct stress test would be to use CODE-DITING as a ranking function on a hidden pool of generated programs and compare its rankings against execution-based pass@k; a genuine functional-alignment metric should correlate with execution on unseen problems, not only on the curated judge sets.
  • Editorial inference: Majority voting at k=7 improves accuracy only when single-inference accuracy exceeds 0.5; for especially hard samples where accuracy is below chance, voting can consolidate the wrong answer, so reporting per-difficulty voting behavior would clarify where the method helps and where it hurts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CODE-DITING, a reference-free and test-free LLM-as-Judge method for code evaluation. It first conducts an empirical study comparing general-purpose and reasoning-focused LLMs with several prompting strategies on three newly curated judge datasets (HumanEval-Judge, MBPP-Judge, BigCodeBench-Judge). It then distills reasoning paths from DeepSeek-R1-671B into compact 1.5B and 7B models using PiSSA-initialized LoRA, and applies majority voting (k=7) at inference. The authors report that CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 (671B) across all three datasets while falling short only of DeepSeek-R1 671B, and that CODE-DITING 1.5B outperforms models of comparable size. Additional experiments address ablations and robustness to preference leakage.

Significance. If the reported results hold, the work would be a practical contribution: it demonstrates that a 7B open-weight judge can approach or match much larger closed-source judges for functional code evaluation, with a released dataset, model weights, and a reproducible distillation pipeline. The paper also provides a useful systematic comparison of existing LLM-as-Judge prompting methods, and the explicit checks for preference leakage are a positive methodological step. However, the headline claim is not supported by the paper's own Table IV, and the comparison protocol contains selection asymmetries that could materially reduce the reported margins. With a corrected claim and a fairer protocol, the contribution is potentially valuable to the code evaluation community.

major comments (3)
  1. [Section V-A, Table IV] The claim that CODE-DITING 7B 'outperforms both (closed-source) GPT-4o and DeepSeek-V3 (671B) across all three datasets' is directly contradicted by Table IV. On MBPP-Judge, GPT-4o achieves Acc 0.807 vs. CODE-DITING 7B's 0.806; on HumanEval-Judge, DeepSeek-V3 achieves Acc 0.884 vs. CODE-DITING 7B's 0.883. Thus CODE-DITING 7B does not win per-dataset comparisons against both baselines on all three datasets. The aggregate averages do favor CODE-DITING (0.806 vs. 0.795 and 0.781), so the claim should be revised to an aggregate-level statement or supported with a statistical test that justifies the per-dataset wording.
  2. [Section IV-C and Section V-B(3) (Figure 4)] The comparison in Table IV is asymmetric in inference budget: CODE-DITING uses k=7 majority voting at temperature 0.6, while all baselines receive a single inference at temperature 0.0 (Section III-D). Moreover, k=7 is not fixed a priori; it is selected as the 'optimal' value from the same three evaluation benchmarks by examining F1 across k values (Section V-B(3), Figure 4). This is a test-set-based hyperparameter selection for the proposed method only, which inflates its reported advantage. The authors should either give baselines the same seven-inference budget, or fix k on a validation set before evaluating on the judge benchmarks, and should report variance or confidence intervals across multiple runs.
  3. [Section V-A and Table III/Table IV] The best prompt for each baseline is selected post hoc on the same test benchmarks: Table III evaluates all prompting methods and Table IV then reports the 'most effective prompt' for each model. This constitutes selection on the test set and can overstate the relative performance of the proposed method, especially when the reported margins are small (e.g., 0.011 average accuracy over GPT-4o). The authors should either define the prompt choice in advance, use a separate validation split, or report results for all prompts without selection to support a fair comparison.
minor comments (4)
  1. [Section III-D] The text refers to 'DeepSeek-v3-67B' and 'DeepSeek-r1-67B' but the model names in the abstract and Table IV indicate 671B; please correct the typos.
  2. [Section IV-C] The sentence 'Note that T is set 7 based on RQ3 findings (Section V-C)' is confusing: the majority-vote count is determined in Section V-B(3) (RQ2, inference component), not in Section V-C (RQ3, preference leakage). Also, the symbol T was already used for test cases in Section II-A; consider using k consistently for the vote count.
  3. [Abstract and Section V-A] The abstract states that 'CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 671B' without the per-dataset qualifier, while Section V-A claims 'across all three datasets'; the two statements should be aligned and made consistent with Table IV.
  4. [Section V-B(3)] The ablation for the inference component reports F1 improvements with k but does not report the corresponding accuracy or MCC values, nor any variance across the multiple inference runs; adding these would strengthen the choice of k=7.

Circularity Check

1 steps flagged · score 4.0 of 10

The k=7 majority-vote configuration is selected on the same three test benchmarks used for the headline comparison, so the 'outperforms GPT-4o/DeepSeek-V3' claim is partly a post-selected test-set result; no equation-level circularity appears elsewhere.

  1. fitted input called prediction [Section V-B(3) (Inference Component ablation) and Section IV-C]
    "In our experiments, we perform T = 7 independent inferences for each test sample and use majority voting to determine the final judgment result. ... By analyzing both the performance improvements and computational overhead across different k values, we identify k=7 as the optimal one."

    The same three judge datasets used for the headline Table IV comparison (HumanEval-Judge, MBPP-Judge, BigCodeBench-Judge) are swept over k in the inference-component ablation to pick k=7, and the central claim that CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 is then reported at that selected value. The reported margins are therefore a post-selected maximum on the test benchmark rather than an independent prediction, and the paper's own Table IV actually shows per-dataset losses (MBPP-Judge vs GPT-4o: 0.806 vs 0.807; HumanEval-Judge vs DeepSeek-V3: 0.883 vs 0.884). This is a fitted inference parameter being presented as an optimal configuration whose resulting performance is then used as evidence for the headline claim.

full rationale

The only circularity-adjacent issue is the selection of k=7. Section V-B(3) tunes the number of majority-vote passes on the same evaluation benchmarks that produce the headline comparison in Table IV, so the reported 'optimal' result is not an independent test-set prediction. This is a methodological selection-coupling problem rather than an equation-level self-derivation. I found no evidence that the distillation pipeline is circular: CODE-DITING is trained on test-case-labeled samples from KodCode, OpenCoder, and CodeHarmony, with teacher reasoning paths from DeepSeek-R1 filtered by agreement with those external labels, and it is evaluated on the separately curated HumanEval-Judge, MBPP-Judge, and BigCodeBench-Judge datasets. The majority-vote formula in Section IV-C is a standard binomial/Law-of-Large-Numbers calculation, not a result derived from the paper's own fitted values. Self-citations such as CodeHarmony [44] are used as one of several training-data sources and are not load-bearing for the central claim. The abstract and Section V-A statement that CODE-DITING 7B 'outperforms both GPT-4o and DeepSeek-V3 across all three datasets' is not supported by Table IV on a per-dataset basis, but that is an internal inconsistency in the reported results, not circularity. Score 4 reflects the test-set-dependent k selection partially compromising the headline, while the model-training and distillation content retains independent empirical substance.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central external inputs are: reliable test-based labels, faithful teacher reasoning after DeepSeek-V3 filtering, and a comparison protocol that gives CODE-DITING k=7 votes per sample. Everything else is standard supervised fine-tuning and inference.

free parameters (4)
  • Majority vote count k = 7
    Selected in Section V-C from the F1-versus-latency trade-off on the evaluation benchmarks; the headline results in Table IV use k=7 rather than k=1.
  • Temperature for reasoning models = 0.6
    Set in Section III-D to promote exploratory reasoning; affects the stochasticity of judgments and the majority vote outcome.
  • Temperature for general models = 0.0
    Set in Section III-D for deterministic outputs; not tuned per model.
  • Class balance in CODEJUDGE-17K = 1:1 positive:negative
    Downsampling in Section IV-A4 to fix the imbalance where correct samples are overrepresented; changes the training distribution.
assumptions (4)
  • domain assumption Expanded test suites (HumanEval-plus, MBPP-plus, BigCodeBench) are sufficient ground-truth oracles for functional correctness of arbitrary code samples.
    Section III-A uses pass@1 on these suites to label all samples, with manual review only for samples that pass the original but fail the enhanced tests.
  • domain assumption DeepSeek-V3 is a reliable discriminator of hallucinated or logically inconsistent reasoning paths produced by DeepSeek-R1.
    Section IV-A4 uses DeepSeek-V3 to filter reasoning paths without independent verification of the explanation quality.
  • standard math Majority vote improves accuracy when single-inference accuracy exceeds 0.5.
    Section IV-C invokes the binomial distribution and the law of large numbers; this is a theorem, not an empirical assumption.
  • domain assumption PiSSA initialization improves performance over standard LoRA in low-rank fine-tuning.
    Section IV-B3 relies on the PiSSA paper [23]; the paper's own ablation confirms the gain on their task, but the general claim is imported from prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation." pith.science (2026). https://pith.science/paper/CUNN5RFA

@misc{pith2026250519502,
  author       = {Pith},
  title        = {Pith review of: CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CUNN5RFA}},
  note         = {Machine review of arXiv:2505.19502}
}
read the original abstract

Trustworthy evaluation methods for code snippets play a crucial role in neural code generation. Traditional methods, which either rely on reference solutions or require executable test cases, have inherent limitation in flexibility and scalability. The recent LLM-as-Judge methodology offers a promising alternative by directly evaluating functional consistency between the problem description and the generated code. To systematically understand the landscape of these LLM-as-Judge methods, we conduct a comprehensive empirical study across three diverse datasets. Our investigation reveals the pros and cons of two categories of LLM-as-Judge methods: the methods based on general foundation models can achieve good performance but require complex prompts and lack explainability, while the methods based on reasoning foundation models provide better explainability with simpler prompts but demand substantial computational resources due to their large parameter sizes. To address these limitations, we propose CODE-DITING, a novel code evaluation method that balances accuracy, efficiency and explainability. We develop a data distillation framework that effectively transfers reasoning capabilities from DeepSeek-R1671B to our CODE-DITING 1.5B and 7B models, significantly enhancing evaluation explainability and reducing the computational cost. With the majority vote strategy in the inference process, CODE-DITING 1.5B outperforms all models with the same magnitude of parameters and achieves performance which would normally exhibit in a model with 5 times of parameter scale. CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 671B, even though it only uses 1% of the parameter volume of these large models. Further experiments show that CODEDITING is robust to preference leakage and can serve as a promising alternative for code evaluation.

Figures

Figures reproduced from arXiv: 2505.19502 by the authors.

Figure 1
Figure 1. The overall method of CODE-DITING. algorithmic challenges to complex system implementations. Each sample is accompanied by a detailed reasoning path that explains the judgment process, making the dataset valuable for training explainable code judgment models. B. Model Training To transfer reasoning capabilities to smaller models while maintaining efficiency, we train the model in three stages. 1. Knowledge Injection… view at source ↗
Figure 2
Figure 2. Ablation Study (F1 Score) of Data Filtering Component [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Ablation Study (F1 Score) of PiSSA Component [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks

    cs.CL 2025-07 conditional novelty 7.0 of 10

    A 5,352-pair benchmark shows thinking LLMs outperform non-thinking ones as code judges, but position bias and formatting sensitivity remain strong.

Reference graph

Works this paper leans on

46 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llm-based multi-agent systems for software engineering: Literature review, vision and the road ahead,

    J. He, C. Treude, and D. Lo, “Llm-based multi-agent systems for software engineering: Literature review, vision and the road ahead,” ACM Transactions on Software Engineering and Methodology , 2024

    2024

  2. [2]

    Efficient and green large language models for software engineering: Vision and the road ahead,

    J. Shi, Z. Yang, and D. Lo, “Efficient and green large language models for software engineering: Vision and the road ahead,” ACM Transactions on Software Engineering and Methodology , 2024

    2024

  3. [3]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 8, pp. 1–79, 2024

    2024

  4. [4]

    Exploring the capabilities of llms for code change related tasks,

    L. Fan, J. Liu, Z. Liu, D. Lo, X. Xia, and S. Li, “Exploring the capabilities of llms for code change related tasks,” ACM Transactions on Software Engineering and Methodology , 2024

    work page 2024

  5. [5]

    Chain- of-thought in neural code generation: From and for lightweight language models,

    G. Yang, Y . Zhou, X. Chen, X. Zhang, T. Y . Zhuo, and T. Chen, “Chain- of-thought in neural code generation: From and for lightweight language models,” IEEE Transactions on Software Engineering , 2024

    2024

  6. [6]

    An empirical study of retrieval-augmented code generation: Challenges and opportunities,

    Z. Yang, S. Chen, C. Gao, Z. Li, X. Hu, K. Liu, and X. Xia, “An empirical study of retrieval-augmented code generation: Challenges and opportunities,” ACM Transactions on Software Engineering and Methodology, 2025

    2025

  7. [7]

    A review on code generation with llms: Application and evaluation,

    J. Wang and Y . Chen, “A review on code generation with llms: Application and evaluation,” in 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI) . IEEE, 2023, pp. 284–289

    2023

  8. [8]

    Towards more realistic evaluation of llm-based code generation: an experimental study and beyond,

    D. Zheng, Y . Wang, E. Shi, R. Zhang, Y . Ma, H. Zhang, and Z. Zheng, “Towards more realistic evaluation of llm-based code generation: an experimental study and beyond,” arXiv preprint arXiv:2406.06918 , 2024

    arXiv 2024

Show all 46 references
  1. [9]

    Assessing and improving syntactic adversarial robustness of pre-trained models for code translation,

    G. Yang, Y . Zhou, X. Zhang, X. Chen, T. Han, and T. Chen, “Assessing and improving syntactic adversarial robustness of pre-trained models for code translation,” Information and Software Technology, vol. 181, p. 107699, 2025

    2025

  2. [10]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  3. [11]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  4. [12]

    chrf: character n-gram f-score for automatic mt evalu- ation,

    M. Popovi ´c, “chrf: character n-gram f-score for automatic mt evalu- ation,” in Proceedings of the tenth workshop on statistical machine translation, 2015, pp. 392–395

    2015

  5. [13]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  6. [14]

    Exploit- gen: Template-augmented exploit code generation based on codebert,

    G. Yang, Y . Zhou, X. Chen, X. Zhang, T. Han, and T. Chen, “Exploit- gen: Template-augmented exploit code generation based on codebert,” Journal of Systems and Software , vol. 197, p. 111577, 2023

    2023

  7. [15]

    Are nlp metrics suitable for evaluating gen- erated code?

    R. Takaichi, Y . Higo, S. Matsumoto, S. Kusumoto, T. Kurabayashi, H. Kirinuki, and H. Tanno, “Are nlp metrics suitable for evaluating gen- erated code?” in International Conference on Product-Focused Software Process Improvement. Springer, 2022, pp. 531–537

    2022

  8. [16]

    On the limitations of embedding based methods for mea- suring functional correctness for code generation,

    A. Naik, “On the limitations of embedding based methods for mea- suring functional correctness for code generation,” arXiv preprint arXiv:2405.01580, 2024

    2024 arXiv

  9. [17]

    Llms- as-judges: a comprehensive survey on llm-based evaluation methods,

    H. Li, Q. Dong, J. Chen, H. Su, Y . Zhou, Q. Ai, Z. Ye, and Y . Liu, “Llms- as-judges: a comprehensive survey on llm-based evaluation methods,” arXiv preprint arXiv:2412.05579 , 2024

    2024 arXiv

  10. [18]

    A survey on llm-as-a-judge,

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y . Shen, S. Ma, H. Liu et al. , “A survey on llm-as-a-judge,” arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  11. [19]

    From generation to judg- ment: Opportunities and challenges of llm-as-a-judge,

    D. Li, B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhat- tacharjee, Y . Jiang, C. Chen, T. Wu et al. , “From generation to judg- ment: Opportunities and challenges of llm-as-a-judge,” arXiv preprint arXiv:2411.16594, 2024

    2024

  12. [20]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  13. [21]

    Fight fire with fire: How much can we trust chatgpt on source code-related tasks?

    X. Yu, L. Liu, X. Hu, J. W. Keung, J. Liu, and X. Xia, “Fight fire with fire: How much can we trust chatgpt on source code-related tasks?” IEEE Transactions on Software Engineering , 2024

    2024

  14. [22]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    2025 arXiv

  15. [23]

    Pissa: Principal singular values and singular vectors adaptation of large language models,

    F. Meng, Z. Wang, and M. Zhang, “Pissa: Principal singular values and singular vectors adaptation of large language models,” Advances in Neural Information Processing Systems , vol. 37, pp. 121 038–121 072, 2024

    2024

  16. [24]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  17. [25]

    Preference leakage: A contamination problem in llm-as-a-judge,

    D. Li, R. Sun, Y . Huang, M. Zhong, B. Jiang, J. Han, X. Zhang, W. Wang, and H. Liu, “Preference leakage: A contamination problem in llm-as-a-judge,” arXiv preprint arXiv:2502.01534 , 2025

    2025

  18. [26]

    Who evaluates the evaluators? on automatic metrics for assessing ai-based offensive code generators,

    P. Liguori, C. Improta, R. Natella, B. Cukic, and D. Cotroneo, “Who evaluates the evaluators? on automatic metrics for assessing ai-based offensive code generators,” Expert Systems with Applications , vol. 225, p. 120073, 2023

    2023

  19. [27]

    Crystalbleu: precisely and efficiently mea- suring the similarity of code,

    A. Eghbali and M. Pradel, “Crystalbleu: precisely and efficiently mea- suring the similarity of code,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–12

    2022

  20. [28]

    Codebleu: a method for automatic evaluation of code synthesis,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” arXiv preprint arXiv:2009.10297 , 2020

    2009 arXiv

  21. [29]

    Codebertscore: Eval- uating code generation with pretrained models of code,

    S. Zhou, U. Alon, S. Agarwal, and G. Neubig, “Codebertscore: Eval- uating code generation with pretrained models of code,” in The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  22. [30]

    Codescore: Eval- uating code generation by learning code execution,

    Y . Dong, J. Ding, X. Jiang, G. Li, Z. Li, and Z. Jin, “Codescore: Eval- uating code generation by learning code execution,” ACM Transactions on Software Engineering and Methodology , vol. 34, no. 3, pp. 1–22, 2025

    2025

  23. [31]

    Codescore-r: An auto- mated robustness metric for assessing the functionalcorrectness of code synthesis,

    G. Yang, Y . Zhou, X. Chen, and X. Zhang, “Codescore-r: An auto- mated robustness metric for assessing the functionalcorrectness of code synthesis,” arXiv preprint arXiv:2406.06902 , 2024

    2024 arXiv

  24. [32]

    Ice-score: Instructing large language models to evaluate code,

    T. Y . Zhuo, “Ice-score: Instructing large language models to evaluate code,” in Findings of the Association for Computational Linguistics: EACL 2024, 2024, pp. 2232–2242

    2024

  25. [33]

    Codejudge: Evaluating code generation with large language models,

    W. Tong and T. Zhang, “Codejudge: Evaluating code generation with large language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , 2024, pp. 20 032– 20 051

    2024

  26. [34]

    Benchmarks and metrics for evalua- tions of code generation: A critical review,

    D. G. Paul, H. Zhu, and I. Bayley, “Benchmarks and metrics for evalua- tions of code generation: A critical review,” in 2024 IEEE International Conference on Artificial Intelligence Testing (AITest) . IEEE, 2024, pp. 87–94

    2024

  27. [35]

    From code to courtroom: Llms as the new software judges,

    J. He, J. Shi, T. Y . Zhuo, C. Treude, J. Sun, Z. Xing, X. Du, and D. Lo, “From code to courtroom: Llms as the new software judges,” arXiv preprint arXiv:2503.02246, 2025

    2025 arXiv

  28. [36]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023. [Online]. Available: https://openrevie...

    2023

  29. [37]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,

    T. Y . Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul et al. , “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” arXiv preprint arXiv:2406.15877 , 2024

    2024 arXiv

  30. [38]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu et al. , “Qwen2. 5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  31. [39]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  32. [40]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  33. [41]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    2023

  34. [42]

    Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding,

    Z. Xu, Y . Liu, Y . Yin, M. Zhou, and R. Poovendran, “Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding,” arXiv preprint arXiv:2503.02951, 2025

    2025 arXiv

  35. [43]

    Opencoder: The open cookbook for top-tier code large language models,

    S. Huang, T. Cheng, J. K. Liu, J. Hao, L. Song, Y . Xu, J. Yang, J. Liu, C. Zhang, L. Chai et al., “Opencoder: The open cookbook for top-tier code large language models,” arXiv preprint arXiv:2411.04905 , 2024

    2024 arXiv

  36. [44]

    Less is more: Towards green code large language models via unified structural pruning,

    G. Yang, Y . Zhou, X. Zhang, W. Cheng, K. Liu, X. Chen, T. Y . Zhuo, and T. Chen, “Less is more: Towards green code large language models via unified structural pruning,” arXiv preprint arXiv:2412.15921 , 2024

    2024 arXiv

  37. [45]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1026–1034

    2015

  38. [46]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement , vol. 20, no. 1, pp. 37–46, 1960

    1960

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.