Pith. sign in

REVIEW 4 major objections 4 minor 21 references

A Code Comprehension Benchmark for Large Language Models for Code

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning code models on rubric-graded semantic tasks raises grading accuracy from 70% to 83.47%.

desk verdict CS101-Gold is a real asset, but the fine-tuning claims rest on incompatible metrics and an unspecified split. read the letter →

arxiv 2507.10641 v1 pith:ZDQ6BVIT submitted 2025-07-14 cs.SE cs.AIcs.CL

Jayant Havare , Saurav Chaudhary , Ganesh Ramakrishnan , Kaushik Maharajan , Srikanth Tamilselvam This is my paper
classification cs.SEcs.AIcs.CL
keywords codecomprehensionlargelanguagemodelssubjectivitygradingrubric-basedevaluationmulti-taskfine-tuningdirectpreferenceoptimizationCS101-Goldsemantics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that code comprehension in large language models is not fixed at pretraining: it can be substantially improved by fine-tuning on semantically rich tasks, especially rubric-based grading of student code. The authors built a six-task benchmark around the Subjectivity Grading Task and report that QWQ-32B's micro-accuracy climbs from 70% to 83.47% after multi-task fine-tuning, while a DPO-finetuned Codestral-22B reaches 87.66%. If true, this matters because debugging, refactoring, code review, and grading all require understanding what code does, not just predicting the next token.

What carries the argument

The load-bearing mechanism is the CS101-Gold dataset, a set of 27,699 tuples of the form (problem statement, student code submission, rubric criterion, human-assigned rating) drawn from an introductory programming course. It defines the Subjectivity Grading Task and is also used to derive the Test Case Task and the Code Comparison Task. Around this core, the paper runs multi-task fine-tuning that combines grading with question answering, code search, test-case prediction, bug fixing, and code comparison, and it applies DPO (direct preference optimization) to one model. The rubric's discrete ratings force the model to commit to fine-grained semantic judgments about each submission, which is what makes the task a probe of comprehension rather than generation.

What would settle it

Fine-tune on CS101-Gold, then evaluate SGT on a new set of programming problems with fresh rubrics and never-before-seen submissions; if accuracy drops to or near the pre-fine-tuning baseline, the claimed comprehension gain is not real. A negative result would also follow from finding any test datapoint whose problem statement or rubric criterion already appears in the fine-tuning pool.

Watch

Extended reading notes

Core claim

The central claim is that fine-tuning language models on code-comprehension tasks produces genuine gains in semantic understanding of code, beyond the syntactic pattern matching learned by next-token pretraining. The evidence is the Subjectivity Grading Task: given a programming problem, a student submission, and a rubric criterion, the model must choose the rating a human grader would assign. On this task QWQ-32B goes from 70% to 83.47% micro-accuracy after training on the benchmark suite, and the best result, 87.66% micro-accuracy, comes from a DPO-finetuned Codestral-22B trained on the Bug Fix Task. The authors read these improvements as showing that semantic supervision teaches models to analyze structure and behavior, not merely to match surface syntax.

Load-bearing premise

The load-bearing premise is that SGT gains measured after fine-tuning on tasks derived from the same CS101-Gold submissions and rubrics reflect general code comprehension; if test problems overlap with training data, the 70%-to-83.47% jump could be memorization rather than understanding.

Editorial extensions

If this is right

  • Code comprehension is trainable: fine-tuning on rubric-graded code improves semantic analysis without changing the model architecture or its pretraining objective.
  • The improvement transfers across the suite: gains on the grading task appear alongside better code search, question answering, test-case prediction, and bug fixing.
  • Models with weaker initial semantic accuracy show the largest absolute gains, while the strongest model stays close to its high baseline.
  • DPO on bug-fix data can exceed plain supervised fine-tuning on the grading task, yielding the best reported SGT accuracy of 87.66%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gain may partly reflect distribution familiarity: the Test Case and Code Comparison tasks are built from the same CS101-Gold submissions and rubrics as the Subjectivity Grading Task, so a held-out split of entire problems and criteria is needed to separate memorization from comprehension.
  • A direct test of the claim would fine-tune on CS101-Gold and then evaluate SGT on a fresh set of problems with new rubrics and never-before-seen submissions.
  • Because human rubric ratings are noisy, comparing model predictions with human inter-annotator agreement would clarify whether the 87.66% ceiling means full comprehension or mastery of the grading rubric's pattern.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CS101-Gold, a dataset of 27 programming problems, 210 rubric criteria, and 3,725 student submissions from an introductory programming course, and defines five code comprehension tasks: Subjectivity Grading (SGT), Code Question-Answering (QAT), Code Search (CST), Test Case Prediction (TCT), Bug Fix (BFT), and Code Comparison (CCT). The central claim is that fine-tuning three LLMs (QWQ-32B, Codestral-22B, Granite-8B) on these tasks, particularly SGT, improves general code comprehension, as measured by SGT micro-accuracy, with QWQ-32B rising from 70% to 83.47% and a DPO-finetuned Codestral-22B reaching 87.66%. The paper also proposes three evaluation dimensions (pre-hoc/post-hoc, intrinsic/extrinsic, abstractive/extractive) and reports qualitative examples of improved error detection.

Significance. If the central claim held, the work would be a useful contribution: CS101-Gold is a real, rubric-graded educational dataset, and the paper evaluates models across three sizes. However, the quantitative evidence is compromised by (i) comparison of incompatible metrics across baseline and fine-tuned conditions, (ii) an internal contradiction in the TCT results, and (iii) the absence of any held-out split between the data used for fine-tuning and the SGT evaluation set. Because the fine-tuning datasets for TCT, CCT, and BFT are explicitly derived from the same CS101-Gold pool as SGT, the headline gains may reflect distribution memorization rather than improved comprehension. The paper's contributions are therefore not yet substantiated; with proper held-out evaluation and metric alignment, the benchmark could have value.

major comments (4)
  1. [5.1, Table 1] The baseline and fine-tuned columns in Table 1 are not comparable. The baseline entries for CST and QAT are ROUGE scores and the baseline for BFT is a CodeBLEU score, while the entries in the '+SGT' columns are micro-accuracy and micro-F1 on the Subjectivity Grading Task. Section 5.1 then reads, for QWQ-32B, 'the baseline ROUGE score was 0.49. After finetuning with the Subjectivity Grading Task, micro-accuracy ... improved to 75.56%,' which implies a cross-metric comparison. To support the claim that fine-tuning on these tasks improves SGT, the paper should compare SGT micro-accuracy before fine-tuning (70% for QWQ-32B) with SGT micro-accuracy after each combined fine-tuning; as written, the reported baselines do not establish the claimed improvement.
  2. [5.1, Table 1] For QWQ-32B, Table 1 reports a standalone TCT accuracy of 82.97% and a TCT+SGT micro-accuracy of 82.36%, which is a decrease. The prose in Section 5.1 states that 'micro-accuracy increased to 82.36% ... suggesting a positive impact on grading task.' This is internally inconsistent and the direction of the effect is misreported.
  3. [3.4, 3.5, 3.6, 5.1] The fine-tuning datasets overlap with the SGT evaluation set in a way that undermines the generalization claim. Section 3.4 states that TCT uses 'the same as the Subjectivity Grading Task' dataset; Section 3.6 states that CCT training data is 'derived from the Subjectivity Grading Task’s dataset'; and Section 3.5 builds BFT from CS101 submissions plus synthetic pairs. The SGT results in Table 1 and Section 5.1 are evaluated on CS101-Gold, but no train/test split, held-out problems, held-out criteria, or contamination control is described anywhere in the paper. With only 27 distinct programming problems, a model could memorize per-problem and per-criterion rating patterns and achieve the reported 70% to 83.47% gain without improving general code comprehension.
  4. [Abstract, 4, 5.1] The abstract and conclusion attribute the best SGT result (87.66%) to a 'DPO-finetuned Codestral-22B,' but Section 4 describes only supervised fine-tuning and Section 5.1 attributes this number to fine-tuning on the Bug Fix Task. No DPO procedure, dataset, or hyperparameters are described, so this result is not reproducible and its relationship to the rest of the experiments is unclear.
minor comments (4)
  1. [References] References [3] and [4] are cited to the scikit-learn F1-score documentation and a pypi package with 'Anonymous' as author; these should be replaced by citable publications.
  2. [Table 1] Table 1 is formatted as plain text and is difficult to parse; the column boundaries should be made explicit in the final version.
  3. [5.1, Table 1] Section 5.1 contains several numeric inconsistencies with Table 1: for Granite-8B, the text reports a Test Case Prediction micro-accuracy of 76.1% and a QAT micro-F1 of 70.69%, while Table 1 lists 75.69% and 70.04% respectively.
  4. [4.2] No dataset or code release link is provided despite the statement in Section 4.2 that the dataset is 'being published in this work'; an artifact should be made available for the benchmark to be usable.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline SGT gains reduce to fine-tuning/evaluation overlap: CCT, TCT, and BFT are built from the same CS101-Gold/SGT pool, and no disjoint held-out split is stated.

  1. fitted input called prediction [Sec. 3.6 (Code Comparison Task) and Sec. 5.1 (Quantitative Analysis)]
    "For training the model, we construct the training data indigenously from our own graded dataset and do supervised finetuning on a pre-trained LLM. The dataset used is derived from the Subjectivity Grading Task’s dataset and contains 748,940 datapoints. ... Finally, after training on the Code Comparison Task, the model achiveved 83.47% of micro-accuracy and 81.31% of micro-f1 score."

    The paper's central evidence that fine-tuning improves code comprehension is the SGT gain from 70% to 83.47% after training on CCT. But CCT is explicitly 'derived from the Subjectivity Grading Task’s dataset'. With only 27 problems, 210 criteria, 3,725 submissions, and 27,699 SGT datapoints, and with no stated train/eval split, the model can memorize problem- and criterion-specific rating patterns during CCT fine-tuning and then score higher on SGT without generalizing to unseen code. The reported improvement is therefore compatible with distribution memorization rather than evidence of general comprehension.

  2. fitted input called prediction [Sec. 3.5 (Bug Fix Task) and Sec. 5.1 (Quantitative Analysis)]
    "To construct a realistic and diverse code-repair dataset, we first transform our rubric-based subjective grading dataset into a paired buggy–fixed dataset ... resulting in a dataset of 5,336 submissions. ... we also evaluated this model on Bug Fix on a proposed dataset (Sec. 33) which we curated from the Subjectivity Grading dataset ... we noticed a significant jump of 18% where initially the model gave an accuracy of 20% and after finetuning it reached 38%."

    The 20% to 38% bug-fix improvement is also presented as evidence of enhanced comprehension, yet the fine-tuning data is constructed by transforming the same rubric-based subjective grading dataset, and the evaluation uses a proposed dataset curated from that same Subjectivity Grading dataset. No disjoint held-out split or contamination control is described, so the improvement can be explained by the model fitting the same code, problem, and error-pattern distribution on which it was trained, rather than by a general improvement in code comprehension.

full rationale

The paper's headline claim is that fine-tuning on semantically rich tasks improves code comprehension, with the strongest numbers being QWQ-32B SGT accuracy rising from 70% to 83.47% after CCT training and Codestral-22B reaching 87.66% after BFT training. The circularity concern is that the fine-tuning datasets for CCT, TCT, and BFT are built from the same CS101-Gold/SGT pool used for the SGT evaluation: TCT is described as using 'the same as the Subjectivity Grading Task' dataset, CCT is 'derived from the Subjectivity Grading Task’s dataset', and BFT is a transformation of the 'rubric-based subjective grading dataset'. Because the paper never states a held-out or disjoint split for CS101-Gold, the reported SGT gains can be accounted for by memorizing the 27 problems, 210 criteria, and 3,725 submissions present in the fine-tuning distribution. This is a fitted-input-called-prediction pattern for the central quantitative claim, warranting a score of 6. The paper is not fully circular overall: it also evaluates external datasets such as CodeQA, CodeSearchNet, and Tufano's bug-fix data, and it does not rely on self-citations or imported uniqueness theorems. However, those external results do not rescue the headline SGT generalization claim, which is the load-bearing evidence for the conclusion that fine-tuning produces 'an enhancement in code comprehension ability'.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted in the classical sense, though all training hyperparameters are undisclosed. The load-bearing assumptions concern the validity of rubric labels, the construct validity of the task suite, and the absence of contamination control between fine-tuning and evaluation. No new physical or theoretical entities are introduced.

assumptions (3)
  • domain assumption Rubric ratings in CS101-Gold are treated as ground truth labels for code comprehension quality.
    Invoked in Sec. 3.1; the entire SGT signal depends on TA and instructor ratings being correct and consistent, but no inter-annotator agreement or rubric validation is reported.
  • domain assumption The six tasks (SGT, QAT, CST, TCT, BFT, CCT) are assumed to collectively measure code comprehension.
    Sec. 3 defines these tasks as comprehension dimensions without construct validation or correlation to external measures of code understanding.
  • ad hoc to paper Fine-tuning on datasets derived from CS101-Gold and evaluating on a subset of CS101-Gold measures generalizable comprehension.
    The paper does not describe a held-out split or contamination control. This assumption is load-bearing for the headline accuracy gains and is likely violated given CCT and TCT are constructed from the same pool.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Code Comprehension Benchmark for Large Language Models for Code." pith.science (2026). https://pith.science/paper/ZDQ6BVIT

@misc{pith2026250710641,
  author       = {Pith},
  title        = {Pith review of: A Code Comprehension Benchmark for Large Language Models for Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZDQ6BVIT}},
  note         = {Machine review of arXiv:2507.10641}
}
read the original abstract

Large Language Models have shown impressive capabilities in coding tasks like code generation and code completion, as they have been trained on a large amount of code data. Also, since one of the core pretraining objectives is Next Token Prediction, these models tends to learn surface-level syntactic patterns in code. However, this does not guarantee code comprehension ability i.e. the ability to capture the semantics of the code. In our opinion, this is the reason why these models often underperform on tasks that require deeper semantic understanding, such as code debugging and code optimization. To address this, we propose fine-tuning these models specifically for code comprehension tasks using large-scale datasets, enabling them to develop a more robust understanding of code semantics. We evaluate three code models of varying sizes on a suite of code comprehension tasks designed to assess semantic understanding beyond surface-level syntactic pattern matching. In particular, we analyze performance on the Subjectivity Grading Task and observe that model performance improves after fine-tuning on relevant downstream tasks. The most significant improvement is seen in the QWQ-32B model, where accuracy increases from 70% to 83.47%. A similar or explainable trend is observed across other models, clearly indicating an enhancement in code comprehension ability. Among the models studied, the DPO-fine-tuned Codestral-22B achieves the highest micro-accuracy of 87.66% on the Subjectivity Grading Task.

Figures

Figures reproduced from arXiv: 2507.10641 by the authors.

Figure 1
Figure 1. Semantics-based Tasks following code and refactor it to improve readability. Do not change its functionality or output. Ensure that variable names, comments are consistent." However, in the refactored version, the model altered the order of operations within the loop. Specifically, it updated x1 before calculating the new value of x2, thereby breaking the iterative dependency that is critical for convergence. So the… view at source ↗
Figure 2
Figure 2. Qualitative examples (only the necessary parts of the code and response are shown) [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 16 canonical work pages

  1. [1]

    Supporting code comprehension via annotations: Right information at the right time and place

    Marjan Adeli et al. “Supporting code comprehension via annotations: Right information at the right time and place”. In:2020 IEEE symposium on visual languages and human-centric computing (VL/HCC). IEEE. 2020, pp. 1–10

    work page 2020

  2. [2]

    SweLL Benchmark: Semantic Code Evaluation Beyond Generation

    RunLoop AI. SweLL Benchmark: Semantic Code Evaluation Beyond Generation. https : / / www . runloop . ai / blog / understanding - llm - code - benchmarks - from - humaneval - to - swe - bench. Accessed: 2025-07-13. 2024. 9

    work page 2025

  3. [3]

    Anonymous.Micro F1 Score.https://scikit-learn.org/1.5/modules/generated/sklearn.metrics.f1_score.html

    work page

  4. [4]

    ROUGE Score

    Anonymous. ROUGE Score. https://pypi.org/project/rouge-score/

    work page

  5. [5]

    ProgramSynthesiswithLargeLanguageModels

    JacobAustinetal.“ProgramSynthesiswithLargeLanguageModels”.In: arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  6. [6]

    Neural code comprehension: A learnable representation of code semantics

    Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. “Neural code comprehension: A learnable representation of code semantics”. In:Advances in neural information processing systems31 (2018)

    work page 2018

  7. [7]

    Fold2vec: Towards a statement-based representation of code for code comprehension

    Francesco Bertolotti and Walter Cazzola. “Fold2vec: Towards a statement-based representation of code for code comprehension”. In:ACM Transactions on Software Engineering and Methodology32.1 (2023), pp. 1–31

    work page 2023

  8. [8]

    EvaluatingLargeLanguageModelsTrainedonCode

    MarkChenetal.“EvaluatingLargeLanguageModelsTrainedonCode”.In: arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

Show all 21 references
  1. [9]

    Z3: An efficient SMT solver

    Leonardo De Moura and Nikolaj Bjørner. “Z3: An efficient SMT solver”. In:International conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer. 2008, pp. 337–340

    2008

  2. [10]

    A comprehensive review on software comprehension models

    Anett Fekete and Zoltán Porkoláb. “A comprehensive review on software comprehension models”. In: Annales Mathematicae et Informaticae. Vol. 51. Liceum University Press. 2020, pp. 103–111

    2020

  3. [11]

    Neural semantic parsing

    Matt Gardner et al. “Neural semantic parsing”. In:Proceedings of the 56th annual meeting of the association for computational linguistics: tutorial abstracts. 2018, pp. 17–18

    2018

  4. [12]

    A formalism for dependency grammar based on tree adjoining grammar

    Aravind Joshi and Owen Rambow. “A formalism for dependency grammar based on tree adjoining grammar”. In: Proceedings of the Conference on Meaning-text Theory. MTT Paris, France. 2003, pp. 207–216

    2003

  5. [13]

    Starcoder: may the source be with you!

    Raymond Li et al. “Starcoder: may the source be with you!” In:arXiv preprint arXiv:2305.06161 (2023)

    2023 arXiv

  6. [14]

    Competition-level code generation with alphacode

    Yujia Li et al. “Competition-level code generation with alphacode”. In:Science 378.6624 (2022), pp. 1092–1097

    2022

  7. [15]

    LiveCodeBench: A Contamination-Free Evaluation for LLMs on Competitive Programming

    Yucheng Lu, Ruiqi Zhong, and Danqi Chen. “LiveCodeBench: A Contamination-Free Evaluation for LLMs on Competitive Programming”. In:arXiv preprint arXiv:2403.07974(2024). url: https: //arxiv.org/abs/2403.07974

    2024 arXiv

  8. [16]

    Studying the usage of text-to-text transfer transformer to support code- related tasks

    Antonio Mastropaolo et al. “Studying the usage of text-to-text transfer transformer to support code- related tasks”. In:2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE. 2021, pp. 336–347

    2021

  9. [17]

    Barriers for students during code change com- prehension

    Justin Middleton, John-Paul Ore, and Kathryn T Stolee. “Barriers for students during code change com- prehension”. In:Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 2024, pp. 1–13

    2024

  10. [18]

    Codebleu: a method for automatic evaluation of code synthesis

    Shuo Ren et al. “Codebleu: a method for automatic evaluation of code synthesis”. In:arXiv preprint arXiv:2009.10297 (2020)

    2020 arXiv

  11. [19]

    An empirical approach to understand the role of emotions in code comprehension

    Divjot Singh, Ashutosh Mishra, and Ashutosh Aggarwal. “An empirical approach to understand the role of emotions in code comprehension”. In:Journal of Computer Languages79 (2024), p. 101269

    2024

  12. [20]

    An empirical study on learning bug-fixing patches in the wild via neural machine translation

    Michele Tufano et al. “An empirical study on learning bug-fixing patches in the wild via neural machine translation”. In:ACM Transactions on Software Engineering and Methodology (TOSEM). Vol. 28. 4. ACM New York, NY, USA, 2019, pp. 1–29

    2019

  13. [21]

    https: //github.com/tongye98/Awesome-Code-Benchmark

    Tong Ye et al.HumanEvalPack: A Multilingual and Diagnostic Expansion of HumanEval. https: //github.com/tongye98/Awesome-Code-Benchmark. Accessed: 2025-07-13. 2024. 10

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.