Pith. sign in

REVIEW 2 major objections 5 minor 21 references

Course-aware LLM style feedback for novice programmers shows promise as a pre-submission revision aid, but the current prototype is not yet trustworthy enough to deploy as-is.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A formative expert review of CodeStylist, an LLM-based code-style feedback tool for early programming courses, found modest perceived utility but limited trust in output reliability.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Honest formative review of an LLM style tool; the course-aware claim needs validation of the manually-entered course config, but the design account deserves peer review. the 2 major comments →

arxiv 2608.00839 v1 pith:BMSS5EQK submitted 2026-08-01 cs.CY cs.HC

CodeStylist: Supporting Early Undergraduate Programmers with Course-Aware Code Style Feedback

Ethan Dickey , Libra Vento , Peter Kurto , Andres Bejarano This is my paper
classification cs.CY cs.HC
keywords computing educationcode style feedbacklarge language modelsformative feedbackundergraduate programmingexpert reviewcourse-specific standardspre-submission revision
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that course-aware, LLM-generated code style feedback is a promising pre-submission revision aid for early undergraduate programmers, while carefully bounding that claim to formative expert judgment rather than measured learning gains. It presents CodeStylist, a web application that turns an instructor-provided coding standard into file- and line-localized style suggestions, and reports a review by 18 instructional staff from one introductory C course. Staff saw real value in making implicit style expectations visible and expected students to use the tool regularly for style checking, but rated perceived correctness around 60%, gave learning impact a below-neutral score, and named reliability as the main adoption barrier. The paper concludes that future versions should pair deterministic rule checks with LLM explanations, rule citations, and stronger verification, rather than relying on the LLM alone.

Core claim

The central claim is that instructional staff find course-aware style feedback useful as a pre-submission aid specifically because it makes local, often implicit coding standards explicit and inspectable, but that the current LLM-only prototype does not yet earn enough trust to serve as a learning intervention. Evidence comes from 18 instructional staff who explored the prototype with self-selected code artifacts: 17 of 18 anticipated students would use the tool for style checking, often at least weekly, while perceived correctness averaged 60.7%, response helpfulness averaged 3.50/5, and anticipated student learning averaged 2.61/5. The authors interpret these patterns as showing a real ins

What carries the argument

CodeStylist is a web application that numbers lines in uploaded source files, sends the code plus a selected course-specific coding standard to GPT-4o, and renders structured suggestions organized by file and line range, each with an issue, suggestion, and explanation. The load-bearing design commitments are course-specificity (the standard is an explicit configuration artifact), project realism (multi-file submissions), explanatory feedback, and localization (file/line anchoring so students can inspect and verify). The paper argues that separating objective detection from explanation, citing specific rules, and making verification easier than compliance are the mechanisms needed to move fro

Load-bearing premise

The conclusions rest on 18 instructional staff from a single course trying the tool on code artifacts they chose themselves, with no common benchmark; if their judgments do not track actual feedback quality or real student behavior, the central claim about promise and adoption barriers is weakly supported.

What would settle it

Run the current LLM-only prototype and a hybrid deterministic-rule version over the same representative set of student submissions, ask instructional staff to rate correctness and revise code accordingly, and measure agreement and revision accuracy; if the hybrid does not improve perceived correctness or produce correct student edits over the LLM-only version, the paper's central redesign claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is right, instructional staff in courses with explicit local coding standards will accept course-aware style feedback as a routine pre-submission aid, even before it is reliable enough to be authoritative.
  • The below-neutral learning ratings imply that deployment should focus on revision support (catching likely violations before submission) rather than on teaching transferable style judgment.
  • The recommended hybrid architecture—deterministic checks for objective rules, LLM for explanation and prioritization—would reduce hallucinated or inconsistent output while retaining contextualized, localized explanations.
  • Rule citation, exact line highlighting, and split-view code display become prerequisites for trust, since they let students verify suggestions against the official standard rather than defer to generated text.
  • Expanding the tool beyond style coaching (for example, into syntax debugging) is not supported by the current evidence and would likely increase misuse and disappointment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the GTA/UTA split reflects role-specific tolerance for imperfect output; a testable extension would give both groups identical student submissions and compare which suggestions they judge correct and which they would act on.
  • Because the review measured perception, not behavior, the strongest next test is whether line-localized, rule-cited feedback changes what students actually edit—if students fix only the suggested lines and never generalize, the tool could enforce compliance without building judgment.
  • The transfer conditions the staff named (early courses, newly learned languages, visibly graded style, weak existing linter support) suggest the tool's value may be highest exactly where deterministic checks are least complete, which is where LLM unreliability also hurts most.
  • A hybrid deterministic-plus-LLM version could be evaluated against the current prototype on the same self-selected artifacts; if staff correctness ratings do not improve materially, the reliability barrier is not solely a matter of missing rule checks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents CodeStylist, a web application that aims to provide course-standard-aware code style feedback for early undergraduate programming courses. The tool is a prototype that sends a line-numbered, multi-file submission together with a manually entered course coding standard to GPT-4o, and returns file- and line-localized suggestions. The authors report a formative expert review with 18 instructional staff from one course, who explored the prototype with self-selected code artifacts and completed a survey. Quantitative results are modest: perceived correctness averaged 60.7%, response helpfulness 3.50/5, response usefulness 3.33/5, and anticipated student learning 2.61/5. Despite this, 17/18 respondents expected students to use the tool for style checking, often at least weekly. Open-ended responses indicated that staff valued the tool for making implicit standards visible but were concerned about reliability, overreliance, and latency/cost. The authors conclude that course-aware style feedback is promising as a pre-submission revision aid, and that future versions should combine deterministic rule checks with LLM-generated explanations and stronger verification.

Significance. If the result holds, the paper provides one of the first empirical accounts of a course-aware LLM-based style feedback tool in early programming education. The strengths are the transparent reporting of descriptive statistics (means, medians, ranges, and category counts), the explicit refusal to over-interpret a small purposive sample, and the honest characterization of the qualitative analysis as lightweight descriptive coding. The design implications (hybrid deterministic+LLM architecture, rule citation, and verification support) are actionable and well grounded in the open-ended responses. However, the central 'course-aware' claim is undermined by the lack of any validation of the course configuration used in the evaluation, and the reliance on self-selected artifacts makes the quantitative ratings difficult to interpret. These issues are load-bearing for the paper's stated contribution and need to be addressed before the claims can be accepted.

major comments (2)
  1. [III.B / VI.E] Section III.B states that the course configuration was 'manually entered from course documents' and is available in the repository, but the paper provides no evidence that this configuration faithfully encodes the CS240 coding standard. There is no coverage check, no comparison of the tool's suggestions against ground-truth standard violations, and no instructor audit of the configuration. Consequently, the expert review may not have tested 'course-aware' feedback at all; it may have tested generic LLM prompting with a superficial course label. The open-ended comment in Section V.C that the tool was 'not accurate towards the actual code standard it was tested against' is consistent with a configuration error rather than generic LLM unreliability. Because the central contribution is specifically course specificity, this construct-validity threat is load-bearing. Please validate the config
  2. [IV.B / VI.E] Section IV.B and VI.E describe participants using self-selected code artifacts with no common benchmark and no coding of artifact characteristics. The paper acknowledges this in VI.E, but it remains a substantive threat to the quantitative ratings: the average perceived correctness of 60.7% and the helpfulness/usefulness scores are based on heterogeneous, uncontrolled exploration. The central conclusion that the tool is 'promising as a pre-submission revision aid' is therefore supported only by expert opinion on unrepresentative inputs. The abstract and conclusion state this more strongly than the evidence allows. Either a small standardized benchmark should be added (even for a subset of participants) or the claim should be tempered to 'promising in the view of instructional staff under exploratory use.'
minor comments (5)
  1. [Footnote 1] The repository is referenced as 'CodeStylist project repository' without a URL or identifier; include a working link or DOI to support reproducibility.
  2. [Introduction] There are minor typos: 'This paper presentsCodeStylist' and 'we usecode style' are missing spaces.
  3. [Table I] The 'Instr.' column presents a single respondent; the caption should explicitly note this is one individual and not a subgroup, to avoid implying meaningful comparison.
  4. [III.C] The claim that the tool was 'configured against multiple early-course standards' is not supported elsewhere in the paper; either provide details or remove this assertion.
  5. [III.B / Acknowledgements] The evaluation uses GPT-4o for feedback generation, while the acknowledgments mention a different model for editing; state the model version used for feedback generation explicitly in the methods to avoid ambiguity.

Circularity Check

0 steps flagged

No significant circularity; the expert-review findings are empirical judgments, not derivations from fitted inputs or self-citations.

full rationale

The paper's central contribution is a formative expert review: 18 instructional staff explored a prototype and provided subjective ratings and open-ended comments. These results are reported as perceived usefulness, trust, and anticipated use, not as predictions derived from the tool's configuration or from prior work. The course-specific standard is manually entered from course documents, but it is an input to the tool, not a parameter fitted to the review outcomes; the paper explicitly frames the correctness item as subjective judgment rather than benchmark accuracy. The only self-citation, reference [6] by two of the authors, is used to motivate scaffolded GenAI adoption and is not load-bearing for the evaluated claims; removing it would not change the empirical results or the interpretation. The paper also transparently acknowledges the main threats to validity, including self-selected artifacts, a single-course purposive sample, and the absence of student log or revision data. Because there is no derivation chain in which an output is equivalent by construction to an input, no circular step can be exhibited.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

The paper introduces no free parameters in the sense of fitted numerical constants. Its assumptions are methodological: unvalidated survey items, transferability of staff judgments to classroom use, and author-performed descriptive coding. The prototype itself is the invented artifact under evaluation, and independent evidence of its effectiveness is absent, as the paper repeatedly acknowledges.

axioms (3)
  • domain assumption The survey instrument and Likert-scale items measure the constructs the paper claims (perceived correctness, helpfulness, usefulness, learning).
    Invoked implicitly throughout Section IV.C. The paper does not report validation of the survey items. If respondents interpreted 'perceived correctness' or 'helpfulness' differently, the quantitative claims shift meaning.
  • domain assumption Instructional staff judgments from a 40-minute exploratory session transfer to real classroom use.
    Explicitly discussed in Section VI.E as a limitation. The paper's interpretation of promise and adoption barriers depends on this transfer assumption, which is plausible but unverified.
  • domain assumption Descriptive coding of open-ended responses by the authors provides a non-distorted summary of respondent concerns.
    Invoked in Section IV.D. No inter-rater reliability or audit trail is reported, so the thematic counts could be influenced by author expectations.
invented entities (1)
  • CodeStylist prototype no independent evidence
    purpose: A web tool that translates a course coding standard into line-localized LLM-generated style feedback for pre-submission revision.
    The prototype is the subject of the study, not a theoretical entity. Its effectiveness is asserted only through subjective ratings, with no external benchmark, student outcome data, or blinded evaluation.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeStylist: Supporting Early Undergraduate Programmers with Course-Aware Code Style Feedback." pith.science (2026). https://pith.science/paper/BMSS5EQK

@misc{pith2026260800839,
  author       = {Pith},
  title        = {Pith review of: CodeStylist: Supporting Early Undergraduate Programmers with Course-Aware Code Style Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BMSS5EQK}},
  note         = {Machine review of arXiv:2608.00839}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This innovative practice full paper presents CodeStylist, a web application that provides course-standard-aware code style feedback for early undergraduate programming courses. CodeStylist addresses a common instructional gap: students are expected to follow local conventions for naming, formatting, comments, organization, and readability, but feedback on these expectations is often delayed or inconsistent. Unlike generic linters or general-purpose LLM prompts, CodeStylist supports course-specific standards, multi-file submissions, and file- and line-localized explanations intended to guide revision rather than grade correctness. We report a formative expert review with 18 instructional staff from one early undergraduate programming course. Participants explored the prototype using self-selected code artifacts and completed a survey about response quality, anticipated student use, and redesign priorities. Ratings indicated modest perceived utility but limited trust: perceived correctness averaged 60.7%, response helpfulness averaged 3.50/5, response usefulness averaged 3.33/5, and anticipated student learning averaged 2.61/5. Despite these concerns, 17/18 respondents expected students to use the tool primarily for style checking, often at least weekly. Open-ended feedback showed that respondents valued CodeStylist for making implicit course standards more visible, but were concerned about unreliable output, overreliance, and latency or cost. We interpret these findings as evidence that course-aware style feedback is promising as a pre-submission revision aid, but that future versions should combine deterministic rule checks with LLM-generated explanations, rule citations, and stronger verification support.

Figures

Figures reproduced from arXiv: 2608.00839 by Andres Bejarano, Ethan Dickey, Libra Vento, Peter Kurto.

Figure 1
Figure 1. Figure 1: CodeStylist interface during formative review. Users select a course-specific coding standard and programming language, inspect source code with [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages

  1. [1]

    Automated grading and feedback tools for programming education: A systematic review,

    M. Messer, N. C. C. Brown, M. K ¨olling, and M. Shi, “Automated grading and feedback tools for programming education: A systematic review,”ACM Transactions on Computing Education, vol. 24, no. 1, pp. 10:1–10:43, 2024

    work page 2024

  2. [2]

    How teachers would help students to improve their code,

    H. Keuning, B. Heeren, and J. Jeuring, “How teachers would help students to improve their code,” inProceedings of the 2019 ACM Con- ference on Innovation and Technology in Computer Science Education, ser. ITiCSE ’19. New York, NY , USA: Association for Computing Machinery, 2019, p. 119–125

    work page 2019

  3. [3]

    Exploring the responses of large language models to beginner programmers’ help requests,

    A. Hellas, J. Leinonen, S. Sarsa, C. Koutcheme, L. Kujanp ¨a¨a, and J. Sorva, “Exploring the responses of large language models to beginner programmers’ help requests,” inProceedings of the 2023 ACM Confer- ence on International Computing Education Research - Volume 1, ser. ICER ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 93–105

    work page 2023

  4. [4]

    Exploring the potential of large language models to generate formative programming feedback,

    N. Kiesler, D. Lohr, and H. Keuning, “Exploring the potential of large language models to generate formative programming feedback,” in2023 IEEE Frontiers in Education Conference (FIE), 2023, pp. 1–5

    work page 2023

  5. [5]

    Feedback-generation for pro- gramming exercises with GPT-4,

    I. Azaiz, N. Kiesler, and S. Strickroth, “Feedback-generation for pro- gramming exercises with GPT-4,” inProceedings of the 2024 on Innovation and Technology in Computer Science Education V . 1, ser. ITiCSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 31–37

    work page 2024

  6. [6]

    AI-Lab: A framework for intro- ducing generative artificial intelligence tools in computer programming courses,

    E. Dickey, A. Bejarano, and C. Garg, “AI-Lab: A framework for intro- ducing generative artificial intelligence tools in computer programming courses,”SN Computer Science, vol. 5, no. 6, p. 720, 2024

    work page 2024

  7. [7]

    CSM: A code style model for computing educators,

    D. Kirk, A. Luxton-Reilly, and E. Tempero, “CSM: A code style model for computing educators,”ACM Trans. Comput. Educ., vol. 25, no. 1, Apr. 2025

    work page 2025

  8. [8]

    Novice Java programming mistakes: Large-scale data vs. educator beliefs,

    N. C. C. Brown and A. Altadmri, “Novice Java programming mistakes: Large-scale data vs. educator beliefs,”ACM Trans. Comput. Educ., vol. 17, no. 2, May 2017

    work page 2017

  9. [9]

    Linking code readability, structure, and comprehension among novices: It’s complicated,

    E. S. Wiese, A. N. Rafferty, and A. Fox, “Linking code readability, structure, and comprehension among novices: It’s complicated,” in2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET), 2019, pp. 84–94

    work page 2019

  10. [10]

    AI teaches the art of elegant coding: Timely, fair, and helpful style feedback in a global course,

    J. Woodrow, A. Malik, and C. Piech, “AI teaches the art of elegant coding: Timely, fair, and helpful style feedback in a global course,” inProceedings of the 55th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 1442–1448

    work page 2024

  11. [11]

    Static analyses in Python programming courses,

    D. Liu and A. Petersen, “Static analyses in Python programming courses,” inProceedings of the 50th ACM Technical Symposium on Computer Science Education, ser. SIGCSE ’19. New York, NY , USA: Association for Computing Machinery, 2019, p. 666–671

    work page 2019

  12. [12]

    Eastwood-tidy: C linting for automated code style assessment in programming courses,

    R. Hart, B. Hays, C. McMillin, E. K. Rezig, G. Rodriguez-Rivera, and J. A. Turkstra, “Eastwood-tidy: C linting for automated code style assessment in programming courses,” inProceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE

    work page

  13. [13]

    AutoStyle: Toward coding style feedback at scale,

    J. B. Moghadam, R. R. Choudhury, H. Yin, and A. Fox, “AutoStyle: Toward coding style feedback at scale,” inProceedings of the Sec- ond (2015) ACM Conference on Learning @ Scale, ser. L@S ’15. New York, NY , USA: Association for Computing Machinery, 2015, p. 261–266

    work page 2015

  14. [14]

    Scale-driven automatic hint generation for coding style,

    R. Roy Choudhury, H. Yin, and A. Fox, “Scale-driven automatic hint generation for coding style,” inProceedings of the 13th International Conference on Intelligent Tutoring Systems - Volume 9684, ser. ITS

    work page

  15. [15]

    EduLint: a versatile tool for code quality feedback,

    A. ˇRecht´aˇckov´a and R. Pel ´anek, “EduLint: a versatile tool for code quality feedback,” inProceedings of the 57th ACM Technical Symposium on Computer Science Education V .1, ser. SIGCSE TS 2026. New York, NY , USA: Association for Computing Machinery, 2026, p. 922–928

    work page 2026

  16. [16]

    Generation probabilities are not enough: Uncertainty highlighting in AI code completions,

    H. Vasconcelos, G. Bansal, A. Fourney, Q. V . Liao, and J. Wort- man Vaughan, “Generation probabilities are not enough: Uncertainty highlighting in AI code completions,”ACM Trans. Comput.-Hum. In- teract., vol. 32, no. 1, Apr. 2025

    work page 2025

  17. [17]

    Automating human tutor-style program- ming feedback: Leveraging GPT-4 tutor model for hint generation and GPT-3.5 student model for hint validation,

    T. Phung, V .-A. P ˘adurean, A. Singh, C. Brooks, J. Cambronero, S. Gul- wani, A. Singla, and G. Soares, “Automating human tutor-style program- ming feedback: Leveraging GPT-4 tutor model for hint generation and GPT-3.5 student model for hint validation,” inProceedings of the 14th Learning Analytics and Knowledge Conference, ser. LAK ’24. New York, NY , U...

    work page 2024

  18. [18]

    Teaching students to recognize and implement good coding style,

    E. S. Wiese, M. Yen, A. Chen, L. A. Santos, and A. Fox, “Teaching students to recognize and implement good coding style,” inProceedings of the Fourth (2017) ACM Conference on Learning @ Scale, ser. L@S ’17. New York, NY , USA: Association for Computing Machinery, 2017, p. 41–50

    work page 2017

  19. [19]

    Learning with style: Improving student code-style through better automated feedback,

    L. Saliba, E. Shioji, E. Oliveira, S. Cohney, and J. Qi, “Learning with style: Improving student code-style through better automated feedback,” inProceedings of the 55th ACM Technical Symposium on Computer Science Education V . 1, ser. SIGCSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 1175–1181

    work page 2024

  20. [2016]

    Berlin, Heidelberg: Springer-Verlag, 2016, p. 122–132

    work page 2016

  21. [2023]

    New York, NY , USA: Association for Computing Machinery, 2023, p. 799–805

    work page 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.