Pith. sign in

REVIEW 3 major objections 5 minor 65 references

ParaStudent: Generating and Evaluating Realistic Student Code by Teaching LLMs to Struggle

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning a 7B code model on real student submissions makes it generate code that matches real students' errors, style, and step-by-step progress far better than prompting GPT-4.1 or an instruct model.

desk verdict Fine-tuning on real student code beats prompting on student-likeness metrics, but the largest gains are on problems seen in training, so the memorization confound needs ruling out before the headline claim is fully safe. read the letter →

arxiv 2507.12674 v2 pith:Y6ZCZXS7 submitted 2025-07-16 cs.CY cs.AIcs.SE

Mihran Miroyan , Rose Niousha , Joseph E. Gonzalez , Gireeja Ranade , Narges Norouzi This is my paper · ORCID
classification cs.CYcs.AIcs.SE
keywords studentcodegenerationLLMfine-tuningintroductoryprogrammingeducationembeddingserrortypedistributionsstylemetricslearningtrajectoriesintelligenttutoringsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ParaStudent claims that the way to make an LLM produce believable student code is not to prompt it to 'be a student' but to fine-tune it on the timestamped submissions real students actually wrote. The authors fine-tune Qwen-2.5 Coder 7B on 244,483 submissions from an introductory Python course and compare it with Qwen-2.5 Coder 7B Instruct and GPT-4.1 under prompting. Across semantic (code-embedding), functional (error type, pass rate), and stylistic (PEP 8, AST structure, verbosity) metrics, plus progress metrics in high-resolution streams, the fine-tuned model (qwen-student) sits closest to real student code on familiar problems and tracks students' pass-rate and edit-size curves. The result matters because tutoring systems, autograders, and educational benchmarks need realistic stand-ins for learners, and the paper shows that such stand-ins can be learned from logs rather than handcrafted.

What carries the argument

The load-bearing mechanism is supervised fine-tuning over submission streams, which trains the model on all attempts rather than only final answers. The evaluation machinery has four parts: SFR-Embedding-Code-400M code embeddings for cosine similarity, KNN distance, and coverage; autograder error categories (no_error, logical, runtime, compile) plus pass@1; style features (verbosity, AST depth, width, and node count, and PEP 8 violations aggregated as the first principal component); and progress metrics (doctest pass-rate change, style-score change, and Levenshtein edit distance between consecutive submissions). Experiments run at two temporal resolutions: low-resolution first/middle/last snapshots, and high-resolution next-step generation conditioned on the prior 1 or 3 attempts, with an optional student-context branch that feeds in a prior problem's submission at the same relative position.

What would settle it

Give experienced CS educators a blind mix of real and qwen-student-generated submissions from the same problems and ask them to pick the real ones; if they can reliably identify generated code, or if qwen-student's generated bugs share no more failing-test-case overlap with the matched student's bugs than random code does, the claim of trajectory-level realism fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that novice 'struggle' is a learnable distribution, not a promptable pose. Fine-tuning on real submission streams makes the model reproduce the signature of learning: early attempts with mixed logical, runtime, and compile errors; pass rates that rise gradually rather than jumping to 100 percent; style that stays verbose and PEP 8-violating; and edits between attempts that are small in Levenshtein distance rather than large rewrites. On the test set for new students solving familiar problems, qwen-student's average KNN distance to student code drops to 0.058 with 71.9 percent coverage, improving over the instruction-tuned baseline by 0.021 in distance and 15.6 percent in coverage, and its final style score (0.41) is much closer to the student mean (0.89) than GPT-4.1's (-0.96). The same signals weaken on entirely new problems, where qwen-student underpredicts final-stage correctness and loses coverage, so the paper's claim is specifically that fine-tuning transfers across students on familiar problems, with generalization to new problems remaining an open limitation.

Load-bearing premise

The load-bearing premise is that the chosen metrics—embedding proximity, error-type distributions, style statistics, and edit distance—capture genuine student-likeness, and that four semesters of one Python course represent student code generally; if either fails, the fine-tuned model could win the comparison without truly simulating learners.

Editorial extensions

If this is right

  • qwen-student can generate synthetic student submissions on familiar problems whose error mix, style statistics, and pass-rate curve match real logs, so it can expand scarce student datasets or benchmark educational models.
  • Instruction-tuned and proprietary prompting models default to correct, clean code, so any system that relies on them for student simulation will overestimate learner skill.
  • Evaluating student-code simulators requires stream-level metrics; comparing final code or pass rate alone would rank GPT-4.1 as best despite its non-studentlike trajectory.
  • Fine-tuning on intermediate attempts is what produces incremental edit behavior; conditioning on prior attempts reproduces small Levenshtein steps, matching how students revise.
  • On unseen problems the model's learning-progress signals degrade, so realistic simulation is currently tied to problem familiarity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's results, the same recipe could generate synthetic 'misconception-consistent' buggy submissions for training autograder feedback or misconception detectors, but only if the generated bugs are checked for semantic root-cause overlap with real novice bugs.
  • The paper's reliance on embedding similarity leaves open that a model could rank as student-like by imitating surface style while being wrong in the wrong way; a direct behavioral test comparing failing-test-case overlap between generated and real submissions would settle whether trajectory alignment is pedagogical or merely statistical.
  • Because fine-tuned models memorize student code, public release of such a simulator would need differential privacy or data filtering, a step the paper itself flags in its limitations.
  • Cross-course transfer is untested; the framework's success on one Python course suggests that each new course or language would need its own fine-tuning data rather than a universal student simulator.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ParaStudent, a framework for generating 'student-like' code from LLMs. Using 689,023 timestamped submissions from an introductory programming course over four semesters, the authors fine-tune Qwen-2.5 Coder 7B with LoRA on student streams and compare it with Qwen-2.5 Coder 7B Instruct and GPT-4.1 under prompting. They define low-resolution (start/middle/end) and high-resolution (next-step conditioned on prior attempts) experiments, and evaluate along embedding-based semantic similarity, functionality (autograder error types and pass rate), style (verbosity, AST metrics, PEP 8 violations), and progress (edit distance, pass-rate/style evolution). The main claim is that fine-tuning (qwen-student) yields code that is more aligned with real student code and trajectories than prompting baselines, especially on test_NS_OP (new students, old problems), and that it also better matches pass-rate and style MAE on new problems (test_NS_NP). The paper includes ablations across model families and an appendix with extended results.

Significance. If the central claim survives scrutiny, the paper makes a useful contribution to AI-in-education: a reproducible recipe for generating synthetic student code with plausible error patterns and incremental edits, plus a multi-dimensional evaluation suite. Strengths include the large real dataset, temporal train/test split by semester, separate in-distribution and out-of-distribution test sets, transparent compute reporting, and public code release. The paper also honestly acknowledges limits (single course, strong supervision in high-resolution setting, lack of privacy guarantees). The main risk is that the headline results on old problems may partly reflect retrieval of training submissions rather than learned student behavior; if the authors can control for that, the framework's value is substantially established.

major comments (3)
  1. [§3 and §5.1, Table 1] The test_NS_OP split is defined as new students on problems that also appear in the training set, and qwen-student is fine-tuned on submissions to those same problems. The largest gains are reported on this split (e.g., Table 1a: first-stage KNN distance 0.054 vs 0.080 and coverage 77.8% vs 44.4% for qwen-student vs qwen-inst, context F). Since the problem statements, fixed code, and skeleton code recur across semesters, a model that emits near-verbatim training submissions for the same problem would score as highly student-like under the KNN and coverage metrics without acquiring any general notion of student behavior. The paper does not report any memorization or training-set-overlap analysis, and Section 8 lists privacy but not this evaluation confound. Please add (a) the fraction of generated submissions that are exact or near-duplicates (e.g., by normalized edit distance or n-gram overlap) of training submissions for the same problem, (b) a re-evaluation on a version of test_NS_OP with such near-duplicates removed, or (c) a held-out-problem analogue of the old-problem setting. If the advantage persists after removing near-duplicates, the central claim is supported; if not, the headline comparison is largely retrieval.
  2. [§5, Tables 1–3] The abstract and Section 6 state that fine-tuning 'significantly improves' alignment, but no statistical significance tests are reported for the distribution-level embedding metrics. Table 1 reports KNN coverage in percentage increments that suggest small cell sizes (e.g., a 2.2% step implies about 45 observations), Tables 2–3 report standard deviations but no confidence intervals or p-values, and there are no multiple-comparison corrections across stages, contexts, and test sets. Without uncertainty quantification, differences such as the test_NS_NP first-stage distance (0.073 vs 0.072 for qwen-student vs gpt-4.1) or the coverage gaps at several cells cannot be distinguished from noise. Please provide bootstrap confidence intervals or permutation tests for each metric and test set, or rephrase 'significantly' as descriptive rather than inferential.
  3. [§4.3.1 and Figure 1] The embedding metrics are built on SFR-Embedding-Code-400M, a code-retrieval model, and the paper treats low KNN distance and high coverage as evidence of 'student-like' semantics. This is a plausible proxy, but the manuscript does not validate that proximity in this embedding space corresponds to behavioral or pedagogical realism rather than to surface similarity. Since the memorization confound in comment 1 operates through exactly this metric, the paper would be substantially strengthened by a small validation study: for example, showing that the embedding metric separates student code from expert/LLM code on held-out problems, or correlating embedding distances with human judgments of student-likeness on a sample of outputs. As written, the semantic dimension of the evaluation is not independently grounded.
minor comments (5)
  1. [§5.1, first paragraph] The sentence 'under in-distribution (test_NS_OP) and out-of-distribution (test_NS_OP)' should read 'test_NS_OP' and 'test_NS_NP', respectively.
  2. [Table 3 caption] The caption 'qwen-student generates code closest to that of students across all metrics for both test scenarios' is accurate for Table 3, but Table 1b shows gpt-4.1 achieving lower KNN distance on the first and middle no-context stages; consider qualifying the caption to avoid overgeneralization.
  3. [§4.1 and §4.3.1] The symbol k is used both for the number of prior attempts (k ∈ {1,3}) and for the KNN neighborhood size (k=3 and k=10); please rename one of them to avoid confusion.
  4. [Figure 1 caption] The caption introduces a feature space Φ but the main text never defines Φ; either define the notation or describe the space informally in the caption.
  5. [Table 1] Please report the number of student-problem pairs contributing to each cell so that the coverage percentages can be interpreted in light of the small sample sizes.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the fine-tuning advantage is measured against held-out students, so the central claim is empirical rather than definitional.

full rationale

The paper's central claim is that fine-tuning Qwen-2.5 Coder 7B on real student submissions yields code more aligned with real student code than prompting-only baselines. The evaluation is not circular: qwen-student is fine-tuned on training submissions (244,483 submissions from Spring 2021 and Fall 2021) and evaluated on held-out test students from Spring 2022 and Fall 2022, with separate test subsets test_NS_OP (new students, old problems) and test_NS_NP (new students, new problems). The embedding, functionality, and style metrics are all computed against real student code that was not used for fine-tuning, so the comparison is an empirical generalization claim rather than a definitional reduction. The closest concern is that test_NS_OP reuses problems seen in training, so a model could in principle score well by retrieving near-exact training submissions; but this is a potential validity or memorization threat, not circularity by construction, and the paper's own limitations explicitly acknowledge the single-course scope and the strong-supervision regime in Experiment 2. The few self-citations in related work (Packer et al., Patil et al., with co-author Joseph E. Gonzalez) are contextual and not load-bearing for any derivation. No equation, metric, or fitted parameter is defined in terms of the quantity it is used to predict.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted to the test set; the model is trained on a train split and evaluated on held-out semesters. The main assumptions are about the validity of the evaluation metrics and the representativeness of the course data.

free parameters (3)
  • k in KNN metrics = k=3 for distance, k=10 for coverage
    The choice of k affects the reported distances and coverage, and no sensitivity analysis is provided.
  • LoRA hyperparameters = r=16, alpha=32, dropout=0.05
    These are standard values, but the paper does not ablate them; they are chosen by hand following common practice.
  • Number of prior attempts k in high-resolution experiment = 1 and 3
    The choice of k is an experimental design decision that affects the condition; the paper reports both, so it is not a fitted value but a selected operating point.
assumptions (3)
  • domain assumption SFR-Embedding-Code-400M provides a meaningful representation of code semantics for measuring student-likeness.
    The paper uses embedding similarity as a primary semantic metric without validating that embedding proximity corresponds to human-judged student-likeness (Section 4.3.1).
  • domain assumption The autograder error categories (no_error, logical, runtime, compile) exhaustively capture the functional state of student code.
    Section 4.3.2 assumes these four categories are sufficient to compare model and student code functionality.
  • domain assumption Student code from Spring 2021 and Fall 2021 is representative of student code in Spring 2022 and Fall 2022 for the same problems.
    This is necessary for the test_NS_OP split to measure generalization to unseen students; course or semester effects could weaken this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ParaStudent: Generating and Evaluating Realistic Student Code by Teaching LLMs to Struggle." pith.science (2026). https://pith.science/paper/Y6ZCZXS7

@misc{pith2026250712674,
  author       = {Pith},
  title        = {Pith review of: ParaStudent: Generating and Evaluating Realistic Student Code by Teaching LLMs to Struggle},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y6ZCZXS7}},
  note         = {Machine review of arXiv:2507.12674}
}
read the original abstract

Large Language Models (LLMs) have shown strong performance on programming tasks, but can they generate student-like code like real students - imperfect, iterative, and stylistically diverse? We present ParaStudent, a systematic study of LLM-based "student-like" code generation in an introductory programming course setting. Using a dataset of timestamped student submissions across multiple semesters, we design low- and high-resolution experiments to model student progress and evaluate code outputs along semantic, functional, and stylistic dimensions. Our results show that fine-tuning significantly improves alignment with real student trajectories and captures error patterns, incremental improvements, and stylistic variations more faithfully. This study shows that modeling realistic student code requires capturing learning dynamics through context-aware generation, temporal modeling, and multi-dimensional evaluation. Code for experiments and evaluation is available at https://github.com/mmiroyan/ParaStudent.

Figures

Figures reproduced from arXiv: 2507.12674 by the authors.

Figure 1
Figure 1. ParaStudent Trajectories in Multi-dimensional Feature Space Φ. We embed sequences of code submissions from real students and LLMs into a shared feature space Φ, defined by a combination of code embeddings, functionality metrics, and style features. Each trajectory illustrates a student’s or model’s code progression over time. Compared to instruction-tuned or proprietary models, the fine-tuned model (qwen-student) tr… view at source ↗
Figure 2
Figure 2. Experiment 1: Code embeddings across three submission stages (first, middle, last) with (bottom) and without (top) context for student (black squares), qwen-student (orange circles), qwen-inst (green crosses), and gpt-4.1 (purple crosses) code submissions. 1024-dimensional embeddings are projected onto a 2D plane using PCA for visualization. qwen-student better matches student code distribution under test_NS_OP sett… view at source ↗
Figure 3
Figure 3. Experiment 1: Error type distributions across stages (first, middle, last) and test sets (test_NS_OP, test_NS_NP) under with-context settings. gpt-4.1 generates mostly functional code without errors. Error type distribution of qwen-student is close to that of student code on test_NS_OP, but the gap increases on test_NS_NP with the model generating erroneous code at the last stage. across all stages, with diverse err… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Experiment 2: Progression of pass rate (top), style score (middle), and code edits (bottom) across normalized submission steps when student context is provided. qwen-student aligns the closest with that of the student curve in all metrics. prev_num is the number of pri…
Figure 5
Figure 5. Figure 5: Prompt templates for fine-tuned models (Qwen 2.5 Coder 3B, 7B and Llama 3.1 8B). Top left (Experiment 1 without context), top right (Experiment 1 with context), bottom left (Experiment 2 without context), bottom right (Experiment 2 with context). Same prompt templates …
Figure 6
Figure 6. Figure 6: Prompt templates for prompting models (Qwen 2.5 Coder 7B Instruct and GPT-4.1). Top left (Experiment 1 without context), top right (Experiment 1 with context), bottom left (Experiment 2 without context), bottom right (Experiment 2 with context). No Error Logical Error …
Figure 7
Figure 7. Figure 7: Experiment 1: Error type distributions across stages (first, middle, last) and test sets (test_NS_OP, test_NS_NP) when context is not provided. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Experiment 2: Progression of pass rate (top), style score (middle), and code edits (bottom) across normalized submission steps when student context is not provided. No Error Logical Error Runtime Error 0 0.2 0.4 0.6 0.8 1 No Error Logical Error Runtime Error No Error L…
Figure 9
Figure 9. Figure 9: Experiment 1: Error type distributions across stages (first, middle, last) and test sets (test_NS_OP, test_NS_NP) across ablation models. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Experiment 1: Code embeddings across three submission stages (first, middle, last) with (bottom) and without (top) context for student (black squares), qwen-student (orange circles), qwen-inst (green crosses), and gpt-4.1 (purple crosses) code submissions for differen…
Figure 11
Figure 11. Figure 11: Experiment 1: Error type distributions across stages (first, middle, last) on different test sets. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Experiment 2: Progression of pass rate (top), style score (middle), and code edits (bottom) across normalized submission steps when student context is not provided for different test sets. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 28 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [4]

    Amjad Altadmri and Neil C.C. Brown. 2015. https://doi.org/10.1145/2676723.2677258 37 million compilations: Investigating novice programming mistakes in large-scale student data . In Proceedings of the 46th ACM Technical Symposium on Computer Science Education, SIGCSE '15, page 522–527. Association for Computing Machinery

    work page arXiv 2015

  4. [5]

    Anthropic. 2025. https://www.anthropic.com/news/claude-3-7-sonnet Claude 3.7 sonnet and claude code

    work page 2025

  5. [6]

    Benjamin S Bloom. 1984. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational researcher, 13(6):4--16

    work page 1984

  6. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021 b . Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  7. [9]

    Xinyun Chen, Maxwell Lin, Nathanael Sch \"a rli, and Denny Zhou. 2024. https://openreview.net/forum?id=KuPixIqPiq Teaching large language models to self-debug . In The Twelfth International Conference on Learning Representations

    work page 2024

  8. [10]

    Albert T Corbett, Kenneth R Koedinger, and John R Anderson. 1997. Intelligent tutoring systems. In Handbook of human-computer interaction, pages 849--874. Elsevier

    work page 1997

Show all 65 references
  1. [11]

    Rowe, and Nasser Giacaman

    Giuseppe De Ruvo, Ewan Tempero, Andrew Luxton-Reilly, Gerard B. Rowe, and Nasser Giacaman. 2018. https://doi.org/10.1145/3160489.3160500 Understanding semantic style by analysing student code . In Proceedings of the 20th Australasian Computing Education Conference, ACE '18, pa...

    2018

  2. [12]

    Google Deepmind. 2025. https://deepmind.google/technologies/gemini/pro/ Gemini 2.5 pro

    2025

  3. [13]

    Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, and 21 others

    DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, and 21 others. 2024. https://doi.org/10.48550/arXiv.2406.1193...

    2024 doi arXiv

  4. [14]

    Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. https://doi.org/10.1145/1999747.1999807 Understanding the syntax barrier for novices . In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education, ITiC...

    2011

  5. [15]

    Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Xuanjing Huang, and Tao Gui. 2024. https://doi.org/10.48550/arXiv.2402.01391 Stepcoder: Improve code ...

    2024 doi arXiv

  6. [16]

    Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Ré, and Azalia Mirhoseini. 2025. https://arxiv.org/abs/2501.14723 Codemonkeys: Scaling test-time compute for software engineering . Preprint, arXiv:2501.14723

    2025 arXiv

  7. [17]

    Andrew Ettles, Andrew Luxton-Reilly, and Paul Denny. 2018. https://doi.org/10.1145/3160489.3160493 Common logic errors made by novice programmers . In Proceedings of the 20th Australasian Computing Education Conference, ACE '18, page 83–89. Association for Computing Machinery

    2018

  8. [18]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  9. [19]

    Xiaodong Gu, Meng Chen, Yalan Lin, Yuhan Hu, Hongyu Zhang, Chengcheng Wan, Zhao Wei, Yong Xu, and Juhong Wang. 2025. https://doi.org/10.1145/3697012 On the effectiveness of large language models in domain-specific code generation . ACM Trans. Softw. Eng. Methodol., 34(3)

    2025 doi

  10. [20]

    Goodman, and Emma Brunskill

    Joy He-Yueya, Noah D. Goodman, and Emma Brunskill. 2024. https://doi.org/10.5281/zenodo.12729776 Evaluating and optimizing educational content with large language model judgments . In Proceedings of the 17th International Conference on Educational Data Mining, pages 68--82. In...

    2024 doi

  11. [21]

    Samuel Holt, Max Ruiz Luyten, and Mihaela van der Schaar. 2024. https://openreview.net/forum?id=EhrzQwsV4K L2 MAC : Large language model automatic computer for extensive code generation . In The Twelfth International Conference on Learning Representations

    2024

  12. [22]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  13. [23]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  14. [24]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024 a . https://arxiv.org/abs/2406.00515 A survey on large language models for code generation . Preprint, arXiv:2406.00515

    2024 arXiv

  15. [25]

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024 b . https://doi.org/10.1145/3672456 Self-planning code generation with large language models . ACM Trans. Softw. Eng. Methodol., 33(7)

    2024 doi

  16. [26]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. https://openreview.net/forum?id=VTF8yNQM66 SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Re...

    2024

  17. [27]

    Mohammad Khalil, Farhad Vadiee, Ronas Shakya, and Qinyi Liu. 2025. https://doi.org/10.1145/3706468.3706523 Creating artificial students that never existed: Leveraging large language models and ctgans for synthetic data generation . In Proceedings of the 15th International Lear...

    2025

  18. [28]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/8636419dea1aa9fbd25fc4248e702da4-Paper-Conference.pdf Coderl: Mastering code generation through pretrained models and deep rei...

    2022

  19. [29]

    Juho Leinonen, Paul Denny, Olli Kiljunen, Stephen MacNeil, Sami Sarsa, and Arto Hellas. 2025. https://doi.org/10.1145/3716640.3716647 Llm-itation is the sincerest form of data: Generating synthetic buggy code submissions for computing education . In Proceedings of the 27th Aus...

    2025

  20. [30]

    Jia Li, Chongyang Tao, Jia Li, Ge Li, Zhi Jin, Huangzhao Zhang, Zheng Fang, and Fang Liu. 2025. https://doi.org/10.1145/3715908 Large language model-aware in-context learning for code generation . ACM Trans. Softw. Eng. Methodol

    2025 doi

  21. [31]

    Junjie Li, Fazle Rabbi, Cheng Cheng, Aseem Sangalay, Yuan Tian, and Jinqiu Yang. 2024 a . https://arxiv.org/abs/2408.09078 An exploratory study on fine-tuning large language models for secure code generation . Preprint, arXiv:2408.09078

    2024

  22. [32]

    Xiangyang Li, Kuicai Dong, Yi Quan Lee, Wei Xia, Yichun Yin, Hao Zhang, Yong Liu, Yasheng Wang, and Ruiming Tang. 2024 b . https://doi.org/10.48550/arXiv.2407.02883 Coir: A comprehensive benchmark for code information retrieval models . CoRR, abs/2407.02883

    2024 doi arXiv

  23. [33]

    Jiate Liu, Yiqin Zhu, Kaiwen Xiao, QIANG FU, Xiao Han, Yang Wei, and Deheng Ye. 2023. https://openreview.net/forum?id=hjYmsV6nXZ RLTF : Reinforcement learning from unit test feedback . Transactions on Machine Learning Research

    2023

  24. [34]

    Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024 a . https://openreview.net/forum?id=IBCBMeAhmC Evaluating language models for efficient code generation . In First Conference on Language Modeling

    2024

  25. [35]

    Ye Liu, Rui Meng, Shafiq Joty, Silvio Savarese, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. 2024 b . https://arxiv.org/abs/2411.12644 Codexembed: A generalist embedding model family for multiligual and multi-task code retrieval . Preprint, arXiv:2411.12644

    2024 arXiv

  26. [36]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, a...

    2024 arXiv

  27. [37]

    Xinyi Lu and Xu Wang. 2024. https://doi.org/10.1145/3657604.3662031 Generative students: Using llm-simulated student profiles to support question item evaluation . In Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S '24, page 16–27. Association for Computing...

    2024

  28. [38]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2024. https://openreview.net/forum?id=UnUwSIgK5W Wizardcoder: Empowering code large language models with evol-instruct . In The Twelfth International Confe...

    2024

  29. [39]

    Zeyuan Ma, Hongshu Guo, Jiacheng Chen, Guojun Peng, Zhiguang Cao, Yining Ma, and Yue-Jiao Gong. 2024. https://doi.org/10.48550/arXiv.2403.01131 Llamoco: Instruction tuning of large language models for optimization code generation . CoRR, abs/2403.01131

    2024 doi arXiv

  30. [40]

    Stephen MacNeil, Magdalena Rogalska, Juho Leinonen, Paul Denny, Arto Hellas, and Xandria Crosland. 2024. https://doi.org/10.1145/3649165.3690100 Synthetic students: A comparative study of bug distribution between large language models and computing students . In Proceedings of...

    2024

  31. [41]

    Markel, Steven G

    Julia M. Markel, Steven G. Opferman, James A. Landay, and Chris Piech. 2023. https://doi.org/10.1145/3573051.3593393 Gpteach: Interactive ta training with gpt-based students . In Proceedings of the Tenth ACM Conference on Learning @ Scale, L@S '23, page 226–236. Association fo...

    2023

  32. [42]

    Nikolaos Nikolaidis, Karolos Flamos, Khanak Gulati, Daniel Feitosa, Apostolos Ampatzoglou, and Alexander Chatzigeorgiou. 2024. https://doi.org/10.1109/SANER-C62648.2024.00018 A comparison of the effectiveness of chatgpt and co-pilot for generating quality python code solutions...

    2024

  33. [43]

    Robert E. Noonan. 1985. https://doi.org/10.1016/0096-0551(85)90018-9 An algorithm for generating abstract syntax trees . Comput. Lang., 10(3–4):225–236

    1985 doi

  34. [44]

    Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama

    Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023. https://doi.org/10.48550/arXiv.2306.09896 Demystifying gpt self-repair for code generation . CoRR, abs/2306.09896

    2023 doi arXiv

  35. [45]

    OpenAI. 2025. https://openai.com/index/gpt-4-1/ Introducing gpt-4.1 in the api

    2025

  36. [46]

    Patil, Ion Stoica, and Joseph E

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. https://arxiv.org/abs/2310.08560 Memgpt: Towards llms as operating systems . Preprint, arXiv:2310.08560

    2024 arXiv

  37. [47]

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. https://arxiv.org/abs/2412.21139 Training software engineering agents and verifiers with swe-gym . Preprint, arXiv:2412.21139

    2024 arXiv

  38. [48]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/e4c61f578ff07830f5c37378dd3ecb0d-Paper-Conference.pdf Gorilla: Large language model connected with massive apis . In Advances in Neural Informati...

    2024

  39. [49]

    Thai Tang Quoc, Duc Ha Minh, Tho Quan Thanh, and Anh Nguyen-Duc. 2024. https://arxiv.org/abs/2408.15658 An empirical study on self-correcting large language models for data science code generation . Preprint, arXiv:2408.15658

    2024 arXiv

  40. [50]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024 arXiv

  41. [51]

    Ben Shi, Michael Tang, Karthik R Narasimhan, and Shunyu Yao. 2024. https://openreview.net/forum?id=kGa4fMtP9l Can language models solve olympiad programming? In First Conference on Language Modeling

    2024

  42. [52]

    Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. 2023. https://openreview.net/forum?id=0XBuaxqEcG Execution-based code generation using deep reinforcement learning . Transactions on Machine Learning Research

    2023

  43. [53]

    Zhihong Sun, Chen Lyu, Bolun Li, Yao Wan, Hongyu Zhang, Ge Li, and Zhi Jin. 2024. https://aclanthology.org/2024.lrec-main.521/ Enhancing code generation performance of smaller models by distilling the reasoning ability of LLM s . In Proceedings of the 2024 Joint International ...

    2024

  44. [54]

    Yun-Da Tsai, Mingjie Liu, and Haoxing Ren. 2024. https://doi.org/10.48550/arXiv.2407.05040 Code less, align more: Efficient llm fine-tuning for code generation with data pruning . CoRR, abs/2407.05040

    2024 doi arXiv

  45. [55]

    Guido van Rossum, Barry Warsaw, and Alyssa Coghlan. 2025. https://peps.python.org/pep-0008/ Pep 8 -- style guide for python code

    2025

  46. [56]

    Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, and Houari Sahraoui. 2025. https://doi.org/10.1145/3714461 Exploring parameter-efficient fine-tuning techniques for code generation with large language models . ACM Trans. Softw. Eng. Methodol

    2025 doi

  47. [57]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020 arXiv

  48. [58]

    Man Fai Wong and Chee Wei Tan. 2024. https://doi.org/10.1109/CAI59869.2024.00037 Aligning crowd-sourced human feedback for code generation with bayesian inference . In 2024 IEEE Conference on Artificial Intelligence (CAI), pages 158--163

    2024

  49. [59]

    Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, and Shuiguang Deng. 2024. https://doi.org/10.18653/v1/2024.acl-long.301 C ode S cope: An execution-based multilingual multitask multidimensional benchmark f...

    2024 doi

  50. [60]

    Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Terry Yue Zhuo, and Taolue Chen. 2024 a . https://doi.org/10.1109/TSE.2024.3440503 Chain-of-thought in neural code generation: From and for lightweight language models . IEEE Transactions on Software Engineering, 50(9):2437--2457

    2024

  51. [61]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024 b . https://proceedings.neurips.cc/paper_files/paper/2024/file/5a7c947568c1b1328ccc5230172e1e7c-Paper-Conference.pdf Swe-agent: Agent-computer interfaces enable ...

    2024

  52. [62]

    Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. 2022. https://openreview.net/forum?id=Q42f0dfjECO Differentially private fine-tuning of lang...

    2022

  53. [63]

    Murong Yue, Wijdane Mifdal, Yixuan Zhang, Jennifer Suh, and Ziyu Yao. 2024. https://doi.org/10.48550/arXiv.2404.06711 Mathvc: An llm-simulated multi-character virtual classroom for mathematics education . CoRR, abs/2404.06711

    2024 doi

  54. [64]

    Tenenbaum, and Chuang Gan

    Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. 2023. https://openreview.net/forum?id=Lr8cOOtYbfL Planning with large language models for code generation . In The Eleventh International Conference on Learning Representations

    2023

  55. [65]

    Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2024. https://openreview.net/forum?id=diXvBHiRyE Beyond correctness: Benchmarking multi-dimensional code generation for large language models

    2024

  56. [66]

    Li Zhong, Zilong Wang, and Jingbo Shang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.49 Debug like a human: A large language model debugger via verifying runtime execution step by step . In Findings of the Association for Computational Linguistics: ACL 2024, pages 851-...

    2024 doi

  57. [67]

    Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. 2024. https://arxiv.org/abs/2401.00788 Astraios: Parameter-efficient instruction tuning code large language models . Preprint, arXiv:2401.00788

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.