Pith. sign in

REVIEW 4 major objections 4 minor 7 cited by

Watching gameplay video alongside block code lets an LLM diagnose and repair Scratch bugs perfectly on a ten-project test set, without any project description.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A multimodal LLM system that diagnoses Scratch bugs from code plus gameplay video reported 100% repair success on ten curated tasks, far above text-only ChatGPT baselines.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Good idea, weak isolation: the video-as-specification claim is plausible but the evaluation never ablates video away from the rest of the pipeline. the 4 major comments →

arxiv 2509.11065 v1 pith:P6KZ22RQ submitted 2025-09-14 cs.SE cs.PL

ViScratch: Using Large Language Models and Gameplay Videos for Automated Feedback in Scratch

Yuan Si , Daming Li , Hanyuan Shi , Jialu Zhang This is my paper
classification cs.SE cs.PL
keywords Scratch debugginggameplay videomultimodal LLMvision-language modelautomated program repairvisual programmingblock-based programmingAST-level repair
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that in Scratch, correctness is perceptual: many bugs—flickering sprites, missed collisions, race-conditioned score skips—only show up in rendered gameplay, not in block inspection. To exploit this, it builds ViScratch, a two-stage pipeline that feeds a vision-language model both the project's code and its gameplay video, aligns observed behavior with code structure, proposes a single critical bug, and repairs it via minimal AST edits verified by execution in the Scratch virtual machine. In a study of 50 real forum bugs, 70% required video for diagnosis. On ten curated projects (30 trials), ViScratch achieved a 100% success rate, outperforming human debuggers given code plus descriptions plus video, and outperforming LLM baselines given code and descriptions, all without ever seeing a project description. If the results hold, video is a first-class specification for visual programming, and automated feedback for open-ended learner projects becomes feasible without expert-authored tests or rules.

Core claim

The paper's central claim is that gameplay video is a necessary, first-class debugging signal in Scratch, not a convenient extra. A survey of 50 forum-reported bugs shows 70% (35/50) are only diagnosable from visual symptoms, and all 17 clone-lifecycle cases require video. On this basis, the paper builds ViScratch, which takes the project's block code, JSON, media assets, and a normalized gameplay video; a vision-language model aligns observed sprite behavior with the AST, isolates one critical bug, and proposes two to three minimal fixes. The repair module turns the chosen fix into atomic AST edits, and the Scratch virtual machine re-executes the patched project to verify the defect is gone

What carries the argument

The carrying mechanism is the diagnose–repair–verify loop. A vision-language model is prompted with the gameplay video and the AST of the buggy project, guided by a curated taxonomy of Scratch bug patterns (broadcast mismatch, clone lifecycle, layering/visibility, collision/bounds, scene transitions, audio–visual timing). It must output exactly one critical bug and two to three fix options. The repair module then translates the chosen fix into atomic JSON/AST edits (insert, replace, delete) under a minimal-edit policy that forbids new features or wholesale rewrites. The patched project is repackaged and executed in the Scratch virtual machine; if the defect persists, the failure is logged an

Load-bearing premise

The 100% figure rests on the assumption that the ten curated projects—each with exactly one bug fitting the paper's taxonomy—represent the variety of real learner bugs, and that the authors' manual judgment of 'bug correctly identified' and 'fix valid' is an unbiased oracle; the verification module's logic is not specified, so it could be checking the same thing the diagnosing LLM already decided.

What would settle it

Take a random sample of 100 Scratch projects with real learner bugs (not filtered by 'exactly one bug' or taxonomy fit), run ViScratch's exact pipeline, and have an independent group of Scratch teachers label whether the identified bug is the actual root cause and whether the fix resolves it without introducing new problems. If the success rate falls well below 100%, the perfection claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Gameplay video can serve as a first-class debugging signal for visual programming, enabling automated feedback without needing project descriptions or instructor-authored tests.
  • Automated repair in block-based languages can be made reliable by constraining LLMs to minimal AST edits and verifying each patch by execution in the virtual machine.
  • The passive, on-demand design means learners can keep their normal Scratch workflow; the tool intervenes only when asked.
  • The same pipeline generalizes, in principle, to other visual programming environments where correctness is judged by what the learner sees on screen.
  • The reported 100% success rate, if it holds beyond the curated set, would give classroom deployment a low-cost per-project fix (about 52 seconds and $0.12 per project).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the video-plus-code approach to other block-based or visual environments (App Inventor, GDevelop, game engines) where behavior is perceptual; the taxonomy would need to be rebuilt, but the diagnose-repair-verify loop is language agnostic.
  • The 100% success on ten clean, single-bug projects is likely to degrade on real projects with multiple interacting bugs or bugs outside the taxonomy; the useful claim to test is not perfect accuracy but whether video consistently beats code-only input.
  • Video could also be used for hint generation beyond fixes: explaining to a learner why the observed behavior is wrong, by pointing to the exact frame where the symptom appears.
  • The paper leaves the verification module's internals unspecified; if it were shown to rely on the same LLM that diagnosed the bug, the measured success rate would conflate diagnosis with patch validation, so a blind check by independent execution would be a stronger test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ViScratch, a multimodal debugging system for Scratch that combines the project's block code, JSON representation, media assets, and a generated gameplay video. A vision-language model (Gemini 2.5 Pro) diagnoses one critical bug, proposes minimal AST-level repairs, and an iterative repair-and-verify loop checks patches in a Scratch VM. The central empirical claim is that gameplay video is a crucial debugging signal: on 10 curated projects with exactly one bug, ViScratch achieves a reported 100% success rate in bug identification and repair, outperforming human participants and four ChatGPT/GPT-4o prompt variants. The paper also reports a human study showing video improves human debugging success (41.2% to 68.6%) and an ablation varying the LLM backbone (Gemini, Qwen3-Max, GLM-4V).

Significance. If the central claim held, this would be a useful advance: it proposes a practical way to treat visual runtime behavior as a first-class debugging signal in block-based programming, where correctness is inherently perceptual. The human study is a valuable positive control, and the design principle of minimal, verified repairs is pedagogically sound. The system is described with enough detail to be reimplemented, and the cost/runtime figures (52.2 s, $0.1185/project) are concrete. However, the evidence as presented does not isolate the contribution of video to ViScratch's success, and the verification and evaluation protocols are under-specified. The headline 100% figure is therefore not yet established as a measure of the video signal.

major comments (4)
  1. [§5.1, §5.2, Table 3] The abstract and Section 5.2 claim that 'gameplay video is a crucial debugging signal' for ViScratch, but the comparison confounds video with the whole pipeline. ViScratch differs from the ChatGPT baselines not only by having video, but also by using Gemini 2.5 Pro, a taxonomy-primed prompt, AST-based repair, and a verification loop. The ablation in Section 5.3 varies only the LLM while keeping video in all conditions. A ViScratch condition with video removed (and ideally with/without project descriptions) is necessary to attribute the 100% success to video. Without it, the result may largely reflect the other components.
  2. [§4.5, Algorithm 1 line 14] The verification module is under-specified. The text says it 'loads the repaired project and checks whether the defect is resolved' and gives a score-update example, but does not state whether the check is a scripted VM assertion, an LLM call, or a human visual judgment. If the same LLM or a human who knows the expected fix performs verification, the reported '100% verified' repairs are not an independent automated outcome. Section 5.1's statement that 'we manually assessed the results' makes this ambiguity concrete. Please clarify what is automated in verification and how the verifier avoids leaking the diagnosis.
  3. [§3.2, §4.3, §5.1] The evaluation risks circularity. The ten projects were selected for 'the presence of exactly one bug which falls into the patterns outlined in Section 3', while the reasoning prompt (Figure 6) is primed with a bug taxonomy derived from the same forum study that motivates the system. Consequently, the 100% success rate may reflect the model's ability to match a known, supplied taxonomy rather than a general debugging capability. Additionally, the manual assessment of 'bug correctly identified' and 'fix valid' is not described as blinded or independently annotated. Please report inter-rater agreement, use a less curated or randomly sampled bug set, or ablate the taxonomy prompt.
  4. [§5.2 vs §5.1] The comparison in Section 5.2 refers to 'the strongest human configuration (Blocks + Project Description + Video)', but the human procedure described in Section 5.1 only provides Blocks first and then Blocks + Video, with no project description. This is an inconsistency in reporting the headline risk ratio (1.46). If humans did receive descriptions, the procedure must be corrected; if they did not, the configuration label must be fixed. As written, it is unclear which human configuration produced the reported comparison.
minor comments (4)
  1. [§3.3 / Table 1] Finding 1 says there are 17 clone-related cases, but the rows 'Missing clone operation' (10) and 'Recursive cloning' (8) sum to 18. Please reconcile the count.
  2. [References] References [19] and [20] appear to be duplicates of the same ICSE paper ('On the Applicability of Language Models to Block-Based Programs') with different author lists. Please verify.
  3. [§5.2] The 100% success rate is reported over '30 rounds of trials' (10 projects × 3 rounds). The paper should state clearly how a 'round' is defined for ViScratch (e.g., whether a round is a single diagnosis attempt or allows iteration within the repair loop), since this affects the interpretation of the success rate.
  4. [Table 3b] Some human success percentages are not multiples of 20% despite '5 participants each' (e.g., Tester 8 shows 33% in the Blocks+Video column). Clarify the per-participant denominators.

Circularity Check

0 steps flagged

No significant circularity; the central claim is empirical and not equivalent to its inputs.

full rationale

ViScratch's central claim is an empirical evaluation result (100% success on 10 projects), not a mathematical derivation from first principles. The paper contains no equations that reduce an output to an input, and no parameter is fitted to the evaluation set and then reported as a prediction. The bug taxonomy in Section 3 was derived from a separate corpus of 50 forum posts; Section 4.3 primes the reasoning prompt with that taxonomy, and Section 5.1 selects evaluation projects that fall into those patterns. This overlap is a genuine threat to the independence of the evaluation and to external validity, but it is not circular: the taxonomy was constructed from data other than the evaluation projects, and ViScratch still had to identify the specific bug and generate a verified fix for each project. The absence of a video-free ViScratch ablation means the causal role of video is under-supported (the 100% result could be driven by the AST repair loop or verification rather than video alone), but that is a confounding/ablation gap, not a circularity. Self-citations such as PyDex [44] appear only in related work and are not load-bearing. The verification module is under-specified, and the paper itself acknowledges that correctness validation relies on perceptual video playback; these are validity limitations, not circular derivations. No quoted passage exhibits a reduction of a claimed result to its own definition or to a self-citation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new entities are postulated. The central result depends on domain assumptions about perceptual correctness, the representativeness of the curated taxonomy, the independence of the VM verifier, and the reliability of manual assessment. The free parameters listed are evaluation design choices that directly shape the reported success rates.

free parameters (3)
  • LLM temperature = 1.0
    Set to 1.0 based on preliminary experiments (Section 5.1, Implementation); no controlled sweep is reported.
  • trials per round = 3
    A round is counted as successful if at least one of three trials succeeds (Section 5.1, Experiment Procedure), which inflates per-project success rates relative to pass@1.
  • iteration cap K for repair-and-verify loop = 3 (main evaluation), 5 (ablation)
    Algorithm 1 caps attempts at K; the main evaluation notes three or fewer iterations often suffice and the ablation allows up to five attempts. This bound directly determines the pass@k success metric.
axioms (5)
  • domain assumption Scratch correctness is perceptual and gameplay video is the de facto ground-truth oracle for program behavior
    Introduced in Sections 2 and 3; the entire design and evaluation treat video as the correctness signal, a view the authors argue for but do not independently validate.
  • domain assumption The 50-post forum survey yields a bug taxonomy that covers recurring bug patterns in real learner projects
    Section 3.2; the taxonomy is derived from 50 selected posts and is used both to motivate the system and to prime the diagnostic prompt.
  • domain assumption The Scratch VM execution in the verification module faithfully reflects runtime behavior and the resolution check is independent of the diagnosing LLM
    Section 4.5; the verifier 'checks whether the defect is resolved' but the checking mechanism is not specified.
  • domain assumption Manual assessment by the authors of bug identification and fix validity is an unbiased oracle
    Section 5.2; all outputs were manually assessed with no reported inter-rater reliability or blinding.
  • domain assumption Gemini 2.5 Pro can reliably align gameplay video with code within the prompt constraints
    Section 4.3; the behavior-mapping step depends on the model's multimodal ability, and the ablation (Section 5.3) shows this is model-dependent, with Qwen3-Max failing on 8 of 10 projects.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ViScratch: Using Large Language Models and Gameplay Videos for Automated Feedback in Scratch." pith.science (2026). https://pith.science/paper/P6KZ22RQ

@misc{pith2026250911065,
  author       = {Pith},
  title        = {Pith review of: ViScratch: Using Large Language Models and Gameplay Videos for Automated Feedback in Scratch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P6KZ22RQ}},
  note         = {Machine review of arXiv:2509.11065}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Block-based programming environments such as Scratch are increasingly popular in programming education, in particular for young learners. While the use of blocks helps prevent syntax errors, semantic bugs remain common and difficult to debug. Existing tools for Scratch debugging rely heavily on predefined rules or user manual inputs, and crucially, they ignore the platform's inherently visual nature. We introduce ViScratch, the first multimodal feedback generation system for Scratch that leverages both the project's block code and its generated gameplay video to diagnose and repair bugs. ViScratch uses a two-stage pipeline: a vision-language model first aligns visual symptoms with code structure to identify a single critical issue, then proposes minimal, abstract syntax tree level repairs that are verified via execution in the Scratch virtual machine. We evaluate ViScratch on a set of real-world Scratch projects against state-of-the-art LLM-based tools and human testers. Results show that gameplay video is a crucial debugging signal: ViScratch substantially outperforms prior tools in both bug identification and repair quality, even without access to project descriptions or goals. This work demonstrates that video can serve as a first-class specification in visual programming environments, opening new directions for LLM-based debugging beyond symbolic code alone.

Figures

Figures reproduced from arXiv: 2509.11065 by Daming Li, Hanyuan Shi, Jialu Zhang, Yuan Si.

Figure 1
Figure 1. Figure 1: An example Scratch project simulating a Mario game, source: [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Strobe flicker caused by two concurrent broadcast handlers that show and hide the sprite every 0.01s [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Race condition caused by splitting sprite position update and score change into two concurrent forever [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Duplicate counting caused by global broadcast in a Cat Catcher game. Clicking a sprite broadcasts [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: ViScratch system pipeline. Given a learner’s Scratch buggy code and its corresponding gameplay video, the reasoning module jointly analyzes code and visual behavior to identify the root cause of failure and propose candidate fixes. When applicable, optional user hints and retry history are consulted to refine the diagnosis. The selected fix is passed to the repair module, which applies edits and produces c… view at source ↗
Figure 6
Figure 6. Figure 6: Prompt for reasoning module: aligning video and code to identify the bug and propose fix options in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt for repair module: enforcing JSON edits in [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Experiment Procedure for each project. There are three test groups: human, ChatGPT, and [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Checked Program Recovery from Execution Video: A Sound Oracle for Untrusted Generators

    cs.SE 2026-07 conditional novelty 7.0

    Vid2Prog recovers Scratch programs from execution videos via a sound oracle that certifies lens-equivalence with zero false accepts on 246 test pairs and 80% certificate rate for in-vocabulary cases while abstaining o...

  2. SchedCheck: Schedule-Robustness Analysis for Event-Driven Block Programs

    cs.SE 2026-07 conditional novelty 7.0

    SchedCheck performs partial-order exploration over dependence-equivalence classes of schedules on the Scratch VM to detect and localize schedule-sensitive behaviors, reporting 17-21% of real concurrent projects affected.

  3. Certificate-Carrying Transformation of Event-Driven Block Programs

    cs.PL 2026-07 accept novelty 7.0 full

    A certificate-carrying rewriting system for Scratch-like languages uses a trusted checker to verify optimizer rewrites by recomputing preservation conditions, with a Lean-mechanized cooperative-frame refinement theore...

  4. NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming

    cs.CL 2026-06 unverdicted novelty 7.0

    NL2Scratch supplies an executable benchmark of 311,648 NL-Scratch pairs and the SAC metric, showing LLMs with high lexical F1 often fail semantic alignment on actions, conditions, and numbers.

  5. Content Hidden Behind Execution: Analyzing Public Scratch Projects at Runtime

    cs.CY 2026-07 conditional novelty 6.0

    In a targeted 500-project Scratch audit, 93% of safety-relevant signals required runtime exploration and 77% needed interaction, progression, failure states, or code/asset inspection.

  6. Fixed-Set Robustness in Programming by Example: Example Corruption and Semantic Partition Recovery

    cs.LG 2026-07 conditional novelty 6.0

    The paper formalizes fixed-set worst-case corruption in PBE, implements corruption searches on a string DSL, and shows VPA recovers some margin-1 tasks but fails on public SyGuS where vote margins are near one.

  7. Raven: Rethinking Automated Assessment for Scratch Programs via Video-Grounded Evaluation

    cs.SE 2026-04 unverdicted novelty 6.0

    Raven automates Scratch program assessment by having instructors specify task-level video generation rules and using LLMs to analyze resulting videos for behavioral compliance, outperforming prior tools on real studen...

Reference graph

Works this paper leans on

47 extracted references · 1 canonical work pages · cited by 7 Pith papers

  1. [1]

    Zhipu AI Open Platform

    Zhipu AI Open Platform 2025.GLM-4V-Plus-0111. Zhipu AI Open Platform. https://bigmodel.cn/dev/howuse/vlm/GLM- 4V-Plus-0111

    2025

  2. [2]

    OpenAI 2025.GPT-4o model | OpenAI API. OpenAI. https://platform.openai.com/docs/models/gpt-4o

    2025

  3. [3]

    Barr, Yuriy Brun, Premkumar T

    Earl T. Barr, Yuriy Brun, Premkumar T. Devanbu, Mark Harman, and Federica Sarro. 2014. The Plastic Surgery Hypothesis. InFSE. 306–317. doi:10.1145/2635868.2635898

    arXiv 2014

  4. [4]

    Brennan Boe, Caitlin Hill, Michelle Len, Gina Dreschler, Philip Conrad, and Diana Franklin. 2013. Hairball: Lint-Inspired Static Analysis of Scratch Projects. InSIGCSE. 215–220. doi:10.1145/2445196.2445265

    arXiv 2013

  5. [5]

    Liuqing Chen, Shuhong Xiao, Yunnong Chen, Yaxuan Song, Ruoyu Wu, and Lingyun Sun. 2024. ChatScratch: An AI-augmented system toward autonomous visual programming learning for children aged 6-12. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–19

    2024

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374. https://arxiv.org/abs/2107.03374

    Pith/arXiv arXiv 2021

  7. [7]

    Yunnong Chen, Shuhong Xiao, Yaxuan Song, Zejian Li, Lingyun Sun, and Liuqing Chen. 2025. MindScratch: A Visual Programming Support Tool for Classroom Learning Based on Multimodal Generative AI.International Journal of Human–Computer Interaction(2025), 1–19

    2025

  8. [8]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, et al. 2025. Gemini 2.5: Pushing the Frontier with Advanced Rea- soning, Multimodality, Long Context, and Next Generation Agentic Capabilities.arXiv preprint(2025). arXiv:2507.06261 doi:10.48550/arXiv.2507.06261 Accessed: 2025-09-09

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.06261 2025

  9. [9]

    Adina Deiner, Patric Feldmeier, Gordon Fraser, Sebastian Schweikl, and Wengran Wang. 2023. Automated Test Generation for Scratch Programs.Empirical Software Engineering28, 79 (2023). doi:10.1007/s10664-022-10255-x

    doi:10.1007/s10664-022-10255-x 2023

  10. [10]

    Adina Deiner and Gordon Fraser. 2024. NuzzleBug: Debugging Block-Based Programs in Scratch. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE ’24). 1—2. doi:10.1145/3597503.3623331

    arXiv 2024

  11. [11]

    Stefania Druga and Amy J Ko. 2025. Scratch Copilot: Supporting Youth Creative Coding with AI. InProceedings of the 24th Interaction Design and Children. 140–153

    2025

  12. [12]

    Stefania Druga and Nancy Otero. 2023. Scratch Copilot Evaluation: Assessing AI-Assisted Creative Coding for Families. InarXiv preprint. arXiv:2305.10417

    Pith/arXiv arXiv 2023

  13. [13]

    Benedikt Fein, Florian Obermüller, and Gordon Fraser. 2022. CATNIP: An Automated Hint Generation Tool for Scratch. InITiCSE. 124–130

    2022

  14. [14]

    Christoph Frädrich, Florian Obermüller, Nina Körber, Ute Heuer, and Gordon Fraser. 2020. Common Bugs in Scratch Programs. InITiCSE. 89–95. doi:10.1145/3341525.3387389

    arXiv 2020

  15. [15]

    Gordon Fraser, Ute Heuer, Nina Körber, Florian Obermüller, and Ewald Wasmeier. 2021. LitterBox: A Linter for Scratch Programs.Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET)(2021), 182–188. doi:10.1109/ICSE-SEET52601.2021.00028

    arXiv 2021

  16. [16]

    GLM-V Team. 2025. GLM-4.1V-Thinking and GLM-4.5V: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning.arXiv preprint(2025). arXiv:2507.01006 https://arxiv.org/abs/2507.01006

    Pith/arXiv arXiv 2025

  17. [17]

    Katharina Götz, Patric Feldmeier, and Gordon Fraser. 2022. Model-based Testing of Scratch Programs. InIEEE ICST. 411–421

    2022

  18. [18]

    Holtschulte, Edward K

    Claire Le Goues, Neal J. Holtschulte, Edward K. Smith, Yuriy Brun, Premkumar T. Devanbu, Stephanie Forrest, and Westley Weimer. 2015. The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs.IEEE Transactions on Software Engineering (TSE)(2015), 1236–1256. doi:10.1109/TSE.2015.2454513

    arXiv 2015

  19. [19]

    Clegg, Florian Obermüller, Gordon Fraser, René Just, and Phil McMinn

    Erik Griebl, Benedikt S. Clegg, Florian Obermüller, Gordon Fraser, René Just, and Phil McMinn. 2023. On the Applicability of Language Models to Block-Based Programs. InICSE. 2374–2386

    2023

  20. [20]

    Elisabeth Griebl, Benedikt Fein, Florian Obermüller, Gordon Fraser, and René Just. 2023. On the Applicability of Language Models to Block-Based Programs. InProceedings of the International Conference on Software Engineering (ICSE). 2374–2386

    2023

  21. [21]

    Glassman

    Jingmei Hu, Priyan Vaithilingam, Stephen Chong, Margo Seltzer, and Elena L. Glassman. 2021. Assuage: Assembly Synthesis Using A Guided Exploration. InThe 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA)(UIST ’21). Association for Computing Machinery, New York, NY, USA, 134–148. doi:10 .1145/ 3472749.3474740

    arXiv 2021

  22. [22]

    Kai Huang, Jian Zhang, Xiaofei Xie, and Chunyang Chen. 2025. Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing.arXiv(2025). doi:10.48550/arXiv.2506.16136

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.16136 2025

  23. [23]

    David E. Johnson. 2016. ITCH: Individual Testing of Computer Homework for Scratch Assignments. InSIGCSE. 223–227. doi:10.1145/2839509.2844600

    arXiv 2016

  24. [24]

    Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2019. A Systematic Literature Review of Automated Feedback Generation for Programming Exercises.ACM Transactions on Computing Education19, 1 (2019), 3:1–3:43. , Vol. 1, No. 1, Article . Publication date: September 2025. 20 Yuan Si, Daming Li, Hanyuan Shi, and Jialu Zhang

    2019

  25. [25]

    Ko and Brad A

    Andrew J. Ko and Brad A. Myers. 2004. Designing the Whyline: A Debugging Interface for Asking Questions about Program Behavior. InCHI. 151–158. doi:10.1145/985692.985712

    arXiv 2004

  26. [26]

    Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes.Proceedings of the 23rd International Conference on Computing Education Research (Koli Calling 2023)(2023). doi:10.1145/3631802.3631830

    arXiv 2023

  27. [27]

    John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn Eastmond. 2010. The scratch programming language and environment.ACM Transactions on Computing Education (TOCE)10, 4 (2010), 1–15

    2010

  28. [28]

    Jesús Moreno-León and Gregorio Robles. 2015. Dr. Scratch: A Web Tool to Automatically Evaluate Scratch Projects. In WiPSCE. 132–133. doi:10.1145/2818314.2818338

    arXiv 2015

  29. [29]

    OpenAI. 2024. GPT-4o System Card.arXiv preprint(2024). arXiv:2410.21276 doi:10 .48550/arXiv.2410.21276 Accessed: 2025-09-09

    Pith/arXiv arXiv 2024

  30. [30]

    Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models.Proceedings of the 16th International Conference on Educational Data Mining (EDM 2023)(2023), 370–377. doi:10.5281/zenodo.8115653

    doi:10.5281/zenodo.8115653 2023

  31. [31]

    Price, Yihuan Dong, and Dragan Lipovac

    Thomas W. Price, Yihuan Dong, and Dragan Lipovac. 2017. iSnap: Towards Intelligent Tutoring in Novice Programming Environments. InSIGCSE. 483–488. doi:10.1145/3017680.3017762

    arXiv 2017

  32. [32]

    Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Evelyn Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian Silverman, et al. 2009. Scratch: programming for all.Commun. ACM52, 11 (2009), 60–67

    2009

  33. [33]

    Koedinger

    Kelly Rivers and Kenneth R. Koedinger. 2017. Data-Driven Hint Generation in Vast Solution Spaces: A Self-Improving Python Programming Tutor.International Journal of Artificial Intelligence in Education27, 1 (2017), 37–64

    2017

  34. [34]

    Mark Santolucito, Jialu Zhang, Ennan Zhai, Jürgen Cito, and Ruzica Piskac. 2022. Learning CI Configuration Correctness for Early Build Feedback. InIEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022, Honolulu, HI, USA, March 15-18, 2022. IEEE, 1006–1017. doi:10.1109/SANER53432.2022.00118

    arXiv 2022

  35. [35]

    Sebastian Schweikl and Gordon Fraser. 2025. RePurr: Automated Repair of Block-Based Learners’ Programs.Proceedings of the ACM on Software Engineering2, FSE (2025), 1475–1498

    2025

  36. [37]

    Yuling Shi, Hongyu Zhang, Chengcheng Wan, and Xiaodong Gu. 2025. Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmers. In47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 1628–1639. doi:10.1109/ICSE55347.2025.00005

    arXiv 2025

  37. [38]

    Da Song, Zijie Zhou, Zhijie Wang, Yuheng Huang, Shengmai Chen, Bonan Kou, Lei Ma, and Tianyi Zhang. 2023. An Empirical Study of Code Generation Errors made by Large Language Models. InMAPS Workshop @ NeurIPS

    2023

  38. [39]

    Andreas Stahlbauer, Michael Kreis, and Gordon Fraser. 2019. Testing Scratch Programs Automatically. InESEC/FSE. 165–175

    2019

  39. [40]

    Niko Strijbol, Robbe De Proft, Klaas Goethals, Bart Mesuere, Peter Dawyndt, and Christophe Scholliers. 2024. Blink: An educational software debugger for Scratch.SoftwareX25 (2024), 101617. doi:10.1016/j.softx.2023.101617

    arXiv 2024

  40. [41]

    Peeratham Techapalokul and Eli Tilevich. 2017. Quality Hound—An Online Code Smell Analyzer for Scratch Programs. InIEEE VL/HCC. 277–281

    2017

  41. [42]

    Chunqiu Steven Xia and Lingming Zhang. 2024. Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’24). ACM, 819–831. doi:10.1145/3650212.3680323

    arXiv 2024

  42. [43]

    An Yang, Anfeng Li, Baosong Yang, et al. 2025. Qwen3 Technical Report.arXiv preprint(2025). arXiv:2505.09388 https://arxiv.org/abs/2505.09388 Accessed: 2025-09-09

    Pith/arXiv arXiv 2025

  43. [44]

    Jialu Zhang, José Pablo Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Verbruggen

  44. [45]

    Jialu Zhang, De Li, John Charles Kolesar, Hanyuan Shi, and Ruzica Piskac. 2022. Automated Feedback Generation for Competition-Level Code. In37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022. ACM, 13:1–13:13. doi:10.1145/3551349.3560425

    arXiv 2022

  45. [46]

    Jialu Zhang, Todd Mytkowicz, Mike Kaufman, Ruzica Piskac, and Shuvendu K. Lahiri. 2022. Using pre-trained language models to resolve textual and semantic merge conflicts (experience paper). InISSTA ’22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, South Korea, July 18 - 22, 2022, Sukyoung Ryu and Yannis Smaragd...

    arXiv 2022

  46. [47]

    Jialu Zhang, Ruzica Piskac, Ennan Zhai, and Tianyin Xu. 2021. Static detection of silent misconfigurations with deep interaction analysis.Proc. ACM Program. Lang.5, OOPSLA (2021), 1–30. doi:10.1145/3485517 , Vol. 1, No. 1, Article . Publication date: September 2025

    work page doi:10.1145/3485517 2021

  47. [2024]

    ACM Program

    PyDex: Repairing Bugs in Introductory Python Assignments using LLMs.Proc. ACM Program. Lang.8, OOPSLA1 (2024), 1100–1124. doi:10.1145/3649850

    doi:10.1145/3649850 2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.