Pith. sign in

REVIEW 3 major objections 3 minor 84 references

Code embeddings trained on public Scratch projects transfer to classroom tasks like correctness prediction and progress tracking without task-specific fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 12:50 UTC pith:J6TEHBSD

load-bearing objection A competent first comparison of code embeddings for Scratch with a clean sprite-naming result, but the correctness-prediction claim is weaker than the abstract suggests and needs trivial baselines plus a de-contaminated evaluation. the 3 major comments →

arxiv 2607.19291 v1 pith:J6TEHBSD submitted 2026-07-21 cs.SE

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

Benedikt Fein , Gordon Fraser This is my paper
classification cs.SE
keywords Scratchcode embeddingslearning analyticsprogram correctnessGGNNsprite namingblock-based programmingtransfer learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Code embeddings — vector representations that capture what a program does — can be transferred to Scratch, the block-based language used by millions of young learners. The paper trains five embedding models on roughly 1.8 million Scratch sprites using a new 'sprite naming' task, and compares them to four pre-trained large language models. The graph-based model that encodes control and data flow (GGNN) names sprites far better than token-sequence or AST-only models, and the same pre-trained embeddings estimate whether a student's program passes its tests and how far a student has progressed toward a solution, without per-task retraining. This makes fine-tuning-free learning analytics feasible in small classrooms where labelled data is scarce.

Core claim

The paper's central discovery is that the 'sprite naming' task — predicting the name of a sprite from the code inside it — both benchmarks how well an embedding captures Scratch semantics and serves as a pre-training task. Across five models and four LLMs, the GGNN, a gated graph neural network whose input graph augments the abstract syntax tree with control-flow, data-flow, and Scratch-specific event/message edges, achieves the best sprite naming scores (F1=0.55). Using this pre-trained model, the authors then show that the Euclidean distance between a student program's embedding and a model solution's embedding correlates with the proportion of passed test cases (r² up to 0.49 on one class

What carries the argument

The central object is the sprite-naming pre-training task combined with the GGNN's program graph. The graph is built from the abstract syntax tree plus semantic edges (LastRead, LastWrite, GuardedBy, ComputedFrom) and Scratch-specific edges connecting broadcast events to receivers and custom-procedure calls to definitions; multiple rounds of message passing along these edges let the model capture the concurrent, event-driven structure that flat token sequences miss. The second load-bearing mechanism is the embedding-fitness measure: the normalised Euclidean distance between a student project and a single model solution in embedding space, used as a surrogate for functional correctness.

Load-bearing premise

The single load-bearing premise is that Euclidean distance in pre-trained embedding space between a student program and one model solution is a valid proxy for functional correctness, and that the evaluation projects were not already part of the pre-training data.

What would settle it

A concrete test: collect a previously unpublished classroom exercise with its own reference solution and test suite, pre-train GGNN on the public sprite-naming corpus, and measure the correlation between embedding fitness and test-passing rate. If r² falls to near zero on several such exercises (as it nearly does for FruitCatching), the transfer claim fails. A complementary probe is to deliberately remove all projects similar to the evaluation project from the pre-training set and check whether the correlation collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Teachers could receive automatic, per-student estimates of whether a Scratch program works, computed from embeddings alone without running test suites or fine-tuning models to the task.
  • The sprite-naming benchmark gives the community a scalable pre-training task for future Scratch code embeddings, filling the gap left by missing documentation or method-naming corpora.
  • An embedding-based large language model can stand in for a dedicated graph model on whole-program tasks such as category classification and progress estimation, at the cost of GPU or API resources.
  • The observed performance drop on non-Latin writing systems implies that multilingual subword tokenization is a prerequisite for equitable classroom deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Euclidean-distance proxy likely underestimates correctness when students solve a task differently from the reference solution (FruitCatching r²=0.13); combining distance with syntax-aware structural similarity (e.g., AST edit distance) is a testable way to strengthen the surrogate.
  • Because BoatRace was probably present in the pre-training corpus, the reported r²=0.45 likely overstates transfer; a fair evaluation would require a fresh classroom dataset with a contemporaneous private base project.
  • The same recipe — pre-train on sprite naming, then use progress-variance projection — could apply to other event-driven visual languages or even event-driven textual systems, since the graph explicitly encodes message passing and concurrency.
  • A live classroom dashboard could use these embeddings at minute-level granularity (as in RQ5) to flag students who deviate from the expected path, before test suites are even run.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper presents a systematic evaluation of code embedding approaches for Scratch programs. It introduces a 'sprite naming' task as a proxy code-summarisation benchmark, trains five dedicated models (NeuralCodeSum, code2vec, code2seq, ASTNN, GGNN) on 656k mined Scratch projects, and compares them against four pre-trained LLMs under zero-shot prompting. It further evaluates multilingual robustness (RQ2), fine-tuning strategies for whole-program classification (RQ3), and the use of pre-trained embeddings—without fine-tuning—as surrogates for functional correctness (RQ4) and student progress (RQ5). The headline results are that graph-based GGNN embeddings outperform AST-path and flat-token models on sprite naming, and that embeddings transfer to progress prediction with strong correlation (RQ5, GGNN r²=0.735). The central claim that embeddings transfer to correctness prediction is, however, only weakly supported for unseen projects and is confounded by training-set overlap on one project.

Significance. If the transfer claims hold, this would be a valuable first comparative study of code embeddings for a block-based visual language, with practical implications for learning analytics in K-12 settings. The paper's strengths include the very large training corpus (1.8M sprites), the use of established model implementations, the multi-language evaluation, and a public replication package with preprocessing tooling. The GGNN result on sprite naming is clean and internally consistent, and the progress-projection result (RQ5) is a genuine positive finding. However, the correctness-prediction evidence in RQ4 is currently the weakest link: the strongest correlation (BoatRace, r²=0.45) is undermined by the authors' own admission of training-data overlap, the unseen-project correlations are modest at best (r²=0.13–0.33), and no trivial baselines are reported. Because the abstract explicitly claims transfer to 'predicting functional correctness', this issue is load-bearing and needs to be addressed before the central claim can be accepted.

major comments (3)
  1. [§5.4.1, Fig. 10, Eq. (2)] The BoatRace result is explicitly caveated: the project 'is based on a publicly available project... our sprite naming training dataset likely contains similar programs.' This makes r²=0.45 unsuitable as evidence for transfer to unseen tasks. Please re-run the analysis either excluding BoatRace or reporting it separately as a leakage check. The remaining unseen-project correlations (FruitCatching r²=0.13, Spaceship r²=0.33 for GGNN) are weak, so the correctness-transfer claim needs to be either softened or supported by additional evidence.
  2. [§4.6.2, Eq. (2)] No trivial baselines are reported. Since f_embed is a normalized distance to the model solution, a simple baseline such as program size (block count, AST node count), number of sprites, or edit distance to the solution could plausibly correlate with test fitness—e.g., because more complete programs are larger. Without comparing against such baselines, the observed correlations cannot be attributed to the structural or semantic content of the embeddings. Please add these controls and report whether the embedding distance explains variance beyond them.
  3. [Fig. 10 (b,c)] The abstract claims that embeddings enable 'predicting functional correctness', but the evidence includes non-significant results: FruitCatching with Qwen-per-sprite has r²=0.08 and p=0.069. Across the nine reported project/model combinations no multiple-comparison correction is applied. Please report adjusted p-values or use a suitable correction, and explicitly characterize which combinations actually support the claim. This is important because the current 'statistically significant correlation for most combinations' statement is too strong given the borderline and non-significant cases.
minor comments (3)
  1. [Fig. 10] The axis labels render 'r/uni00B2' in the provided PDF; this should be r². Please check the final rendering.
  2. [§5.2] Typo: 'the number of operators and operators used in the code' should presumably be 'operators and operands'.
  3. [§4.3.3] The METEOR metric is computed with an English WordNet; the paper acknowledges this, but it might be worth stating explicitly that the multilingual comparison in RQ2 excludes METEOR for this reason, which is already done. No action needed beyond the existing statement.

Circularity Check

0 steps flagged

No circular derivation chain; the only circularity-adjacent issue is acknowledged BoatRace training overlap, which is a leakage/confound rather than a fit-renamed-as-prediction.

full rationale

The paper's load-bearing transfer claims (RQ4/RQ5) do not reduce to their inputs by construction. The embedding fitness is defined as normalized Euclidean distance to a model solution (Eq. 2), and the target fitness is defined independently as the Whisker test pass rate (Eq. 1); the models are pre-trained only on sprite names, and no correctness/progress label is used for training or tuning. RQ4 and RQ5 therefore test a hypothesis against an external oracle rather than predicting a fitted quantity. The self-citations ([24], [25], [28], [66]) supply shared datasets, test suites, and model implementations; they do not encode the claimed result that pre-trained Scratch embeddings correlate with test fitness. The most circularity-adjacent passage is the paper's own admission: 'Since the BoatRace project is based on a publicly available project published by the CodeClubRikScratchuser (cf. Section 4.5.1), our sprite naming training dataset likely contains similar programs.' This is a training-data overlap that may inflate the BoatRace r²=0.45, but it is data leakage/selection bias, not a definitional or fitted-input circularity. Similarly, the absence of trivial baselines (e.g., block count) in RQ4 weakens the causal attribution to embeddings but is an experimental-control gap, not a circular derivation. No uniqueness theorem from the authors is invoked, and no ansatz is smuggled in via self-citation in a load-bearing way. The central empirical comparisons against independent Whisker oracles are self-contained, so circularity is minimal (0-2 band); I assign 1 due to the repeated reliance on the authors' own prior tools/datasets, none of which is load-bearing for the transfer claim.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The paper introduces a new benchmark task (sprite naming) and new preprocessing for Scratch, but no new physical or theoretical entities. The central claim rests on the reliability of LitterBox parsing, Whisker test suites, language detection, and the transferability of an existing projection method, plus an imperfect leakage assumption for BoatRace.

free parameters (5)
  • GGNN embedding size = 128
    Selected from {128, 256, 512} by validation loss; used for all GGNN downstream embeddings in RQ3-RQ5.
  • GGNN graph convolution layers = 12
    Selected from {4, 8, 12, 16} during hyperparameter search; controls how much structural information propagates.
  • code2vec/code2seq AST path length = 8
    Chosen from values up to 12; longer paths did not improve results and increased preprocessing cost.
  • Number of sampled AST paths = 200
    Random per-sprite path samples; halving or doubling did not improve validation performance.
  • NeuralCodeSum transformer layers and embedding size = 6 layers, size 512
    Selected from search spaces 4-12 layers and 128-512 embedding size.
axioms (6)
  • domain assumption LitterBox's AST and control/data-flow extraction faithfully represent Scratch program semantics.
    All model input formats are derived from LitterBox ASTs (§3.1, §3.6). If this conversion is wrong, the comparative results are invalid.
  • domain assumption The ScratchBlocks textual format is present in LLM pre-training corpora and is therefore a fair prompt input.
    Used to justify prompting LLMs with ScratchBlocks rather than raw JSON (§3.3).
  • domain assumption Whisker test suites are valid measures of functional correctness for the student projects.
    RQ4 and RQ5 define test fitness f_test from Whisker test-suite results (§4.6.1).
  • domain assumption lingua-rs language detection with >=90% confidence correctly identifies the programmer's natural language.
    Used to construct the ten per-language datasets in RQ2 (§4.4.1).
  • domain assumption The progress-variance-projection by Paassen et al. remains valid for Scratch embeddings and for models other than the original Python encoder.
    RQ5 reuses this projection without modification (§4.7.2).
  • domain assumption Training/evaluation splits prevent leakage from the RQ1 sprite-naming corpus into downstream tasks.
    For RQ4, the authors note the BoatRace project is likely similar to RQ1 training data (§5.4.1), so this assumption is only partially met.

pith-pipeline@v1.3.0-alltime-deepseek · 34619 in / 12472 out tokens · 119412 ms · 2026-08-01T12:50:24.718621+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of EmbeddedKittens: An Evaluation of Code Embeddings for Scratch." pith.science (2026). https://pith.science/paper/J6TEHBSD

@misc{pith2026260719291,
  author       = {Pith},
  title        = {Pith review of: EmbeddedKittens: An Evaluation of Code Embeddings for Scratch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J6TEHBSD}},
  note         = {Machine review of arXiv:2607.19291}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The trend of embedding source code for machine learning applications also enables new opportunities in learning analytics in programming education, but which code embedding approach is most suitable for learning analytics remains an open question. A common approach to embedding source code lies in treating the code as a token sequence similar to natural language when training large language models~(LLMs). However, in case of visual block-based programming languages like Scratch, this approach cannot be applied directly. While text-based representations of block-based code can be created to apply LLMs to this problem, other dedicated embedding models could potentially exhibit improved performance by capturing additional structural information. In this paper, we therefore instantiate four LLMs and five different popular embedding approaches for Scratch programs, create a token-prediction and two different classification tasks with corresponding datasets, and empirically evaluate the models on them. Our experiments demonstrate that a transfer of code embeddings to the educational environment of Scratch is feasible. The embedding models trained on large open Scratch datasets capture relevant structural and semantic information about the code to enable learning analytics like predicting functional correctness of student programs, in the typically small classroom setting without requiring further task-specific model fine-tuning.

Figures

Figures reproduced from arXiv: 2607.19291 by Benedikt Fein, Gordon Fraser.

Figure 1
Figure 1. Figure 1: Example of the Scratch code as it appears inside a sprite and its abstract syntax tree (ast). Each full Scratch program can contain multiple such sprites that each contain a separate independent set of scripts. or be used to track students’ exercise progress [59]. Again, these approaches mainly focus on applications to textual programming languages. However, especially in early software engineering educati… view at source ↗
Figure 2
Figure 2. Figure 2: Processing steps for the program from Fig. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example code and its representation for GGNN as graph based on the ast with additional edges for control and data flow information. LastLexicalUse edges are not shown since the relevant connections are already made by LastWrite and LastRead edges. Edges are shown directionally to highlight their construction. an abstract BLOCK node. Inputs to statements, i.e. expressions, have a distinctly different shape,… view at source ↗
Figure 4
Figure 4. Figure 4: Sprite naming dataset structure. The code below is written in the Scratch programming language. It is in the ScratchBlocks format which you know for example from the Scratch community forums. The code belongs to a single sprite in the program. I want to find a suitable name for this sprite. Suggest 10 names for the sprite. Only return the list of names with one name per line. ``` {{ source_code }} ``` [PI… view at source ↗
Figure 6
Figure 6. Figure 6: Project classification dataset label distributions. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Whisker test results of the student projects. bug-free model solution and multiple incomplete or faulty student solution attempts. Each base project is accompanied by a Whisker [17] test suite that verifies whether the required functionality is implemented correctly in the student projects. Most students implemented a meaningful proportion of the functionality (cf [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Project Category: Visualisation of the embedding space after t-SNE dimensionality reduction. [PITH_FULL_IMAGE:figures/full_fig_p026_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Project remixes: Visualisation of the embedding space after t-SNE dimensionality reduction. Colouring based on the base [PITH_FULL_IMAGE:figures/full_fig_p027_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Linear least-squares regression between test and embedding space distances between student projects and example solution. [PITH_FULL_IMAGE:figures/full_fig_p028_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Progress-variance-projection of student program snapshots for the whole class captured every minute. The example student [PITH_FULL_IMAGE:figures/full_fig_p031_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

84 extracted references · 11 canonical work pages

  1. [1]

    Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A transformer-based approach for source code summarization. In Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.449

    doi:10.18653/v1/2020.acl-main.449 2020

  2. [2]

    Efthimia Aivaloglou and Felienne Hermans. 2016. How kids code and how we know: an exploratory study on the Scratch repository. InConference on International Computing Education Research (ICER). ACM. doi:10.1145/2960310.2960325

    arXiv 2016

  3. [3]

    Efthimia Aivaloglou, Felienne Hermans, Jesus Moreno-Leon, and Gregorio Robles. 2017. A dataset of Scratch programs: scraped, shaped and scored. InInternational Conference on Mining Software Repositories (MSR). IEEE. doi:10.1109/msr.2017.45

    work page doi:10.1109/msr.2017.45 2017

  4. [4]

    Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to represent programs with graphs. InInternational Conference on Learning Representations (ICLR). arXiv. arXiv: 1711.00740[cs.LG]

    Pith/arXiv arXiv 2018

  5. [5]

    Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. Code2seq: generating sequences from structured representations of code. InInternational Conference on Learning Representations (ICLR). arXiv. arXiv: 1808.01400[cs.LG]

    Pith/arXiv arXiv 2018

  6. [6]

    Uri Alon and Eran Yahav. 2021. On the bottleneck of graph neural networks and its practical implications. InInternational Conference on Learning Representations (ICLR). arXiv. arXiv: 2006.05205[cs.LG]. Manuscript submitted to ACM 34 Benedikt Fein and Gordon Fraser

    Pith/arXiv arXiv 2021

  7. [7]

    Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019. Code2vec: learning distributed representations of code.Proceedings of the ACM on Programming Languages, 3. doi:10.1145/3290353

    doi:10.1145/3290353 2019

  8. [9]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. InACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Association for Computational Linguistics. https://aclanthology.org/W05-0909/

    2005

  9. [10]

    Robert Bazzocchi, Micah Flemming, and Lisa Zhang. 2020. Analyzing CS1 student code using code embeddings. InACM Technical Symposium on Computer Science Education (SIGCSE). ACM. doi:10.1145/3328778.3372601

    arXiv 2020

  10. [11]

    Tadeusz Caliński and Jerzy Harabasz. 1974. A dendrite method for cluster analysis.Communications in Statistics, 3, 1, (Jan. 1974). doi:10.1080/03610 927408827101

    work page doi:10.1080/03610 1974

  11. [12]

    Laura Caspari, Luisa Greifenstein, Ute Heuer, and Gordon Fraser. 2023. Scratchlog: live learning analytics for scratch. InConference on Innovation and Technology in Computer Science Education (ITiCSE). ACM, (June 2023). doi:10.1145/3587102.3588836

    arXiv 2023

  12. [13]

    Xiaolin Chai, Yan Sun, and Yan Gao. 2023. Towards data-driving multi-view evaluation framework for Scratch.Tsinghua Science and Technology, 29, 2, (Sept. 2023). doi:10.26599/tst.2023.9010016

    arXiv 2023

  13. [14]

    Jinyin Chen, Keke Hu, Yue Yu, Zhuangzhi Chen, Qi Xuan, Yi Liu, and Vladimir Filkov. 2020. Software visualization and deep transfer learning for effective software defect prediction. InInternational Conference on Software Engineering (ICSE). ACM. doi:10.1145/3377811.3380389

    arXiv 2020

  14. [15]

    Liuqing Chen, Shuhong Xiao, Yunnong Chen, Yaxuan Song, Ruoyu Wu, and Lingyun Sun. 2024. ChatScratch: an AI-augmented system toward autonomous visual programming learning for children aged 6-12. InConference on Human Factors in Computing Systems (CHI). ACM. doi:10.1145 /3613904.3642229

    arXiv 2024

  15. [16]

    DeepSeek-Coder-V2: breaking the barrier of closed-source models in code intelligence

    DeepSeek-AI et al. DeepSeek-Coder-V2: breaking the barrier of closed-source models in code intelligence. (June 2024). arXiv: 2406.11931 [cs.SE]

    Pith/arXiv arXiv 2024

  16. [17]

    Adina Deiner, Patric Feldmeier, Gordon Fraser, Sebastian Schweikl, and Wengran Wang. 2023. Automated test generation for Scratch programs. Empirical Software Engineering, 28, 3, (May 2023). doi:10.1007/s10664-022-10255-x

    doi:10.1007/s10664-022-10255-x 2023

  17. [18]

    Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets.Journal of Machine Learning Research, 7, 1. http://jmlr.org/papers /v7/demsar06a.html

    2006

  18. [19]

    Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: exploring prompt engineering for solving CS1 problems using natural language. InACM Technical Symposium on Computer Science Education (SIGCSE). ACM. doi:10.1145/3545945.3569823

    arXiv 2023

  19. [20]

    Stefania Druga and Amy J Ko. 2025. Scratch Copilot: supporting youth creative coding with AI. InInteraction Design and Children (IDC). ACM. doi:10.1145/3713043.3727051

    arXiv 2025

  20. [21]

    Scratch Copilot evaluation: assessing AI-assisted creative coding for families

    Stefania Druga and Nancy Otero. Scratch Copilot evaluation: assessing AI-assisted creative coding for families. (May 2023). arXiv: 2305.10417 [cs.HC]

    Pith/arXiv arXiv 2023

  21. [22]

    MMTEB: massive multilingual text embedding benchmark

    Kenneth Enevoldsen et al. MMTEB: massive multilingual text embedding benchmark. (Feb. 2025). arXiv: 2502.13595[cs.CL]

    arXiv 2025

  22. [23]

    Benedikt Fein, Patric Feldmeier, Gordon Fraser, and Florian Obermüller. 2026. Reasoning about bugs in learners’ Scratch programs using large language models. InInternational Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). ACM. doi:10.1145/3786580.3786949

    arXiv 2026

  23. [24]

    Benedikt Fein, Isabella Graßl, Florian Beck, and Gordon Fraser. 2022. An evaluation of code2vec embeddings for Scratch. InInternational Conference on Educational Data Mining (EDM). International Educational Data Mining Society. doi:10.5281/zenodo.6853103

    work page doi:10.5281/zenodo.6853103 2022

  24. [25]

    Benedikt Fein, Maximilian Jungwirth, Gordon Fraser, and Florian Kandlinger. 2026. Challenges of deploying code embeddings: an industrial case study on method name generation.Automated Software Engineering, 33, 2, (Feb. 2026). doi:10.1007/s10515-026-00592-2

    work page doi:10.1007/s10515-026-00592-2 2026

  25. [26]

    Benedikt Fein, Florian Obermüller, and Gordon Fraser. 2022. Catnip: an automated hint generation tool for Scratch. InConference on Innovation and Technology in Computer Science Education (ITiCSE). ACM. doi:10.1145/3502718.3524820

    arXiv 2022

  26. [27]

    Benedikt Fein, Florian Obermüller, and Gordon Fraser. 2025. LitterBox+: an extensible framework for LLM-enhanced Scratch static code analysis. InInternational Conference on Automated Software Engineering (ASE): Tool Demonstration Track. IEEE. doi:10.1109/ASE63991.2025.00357

    arXiv 2025

  27. [28]

    Gordon Fraser, Ute Heuer, Nina Körber, Florian Obermüller, and Ewald Wasmeier. 2021. LitterBox: a linter for Scratch programs. InInternational Conference on Software Engineering: Joint Track on Software Engineering Education and Training (ICSE-JSEET). IEEE. doi:10.1109/icse-seet52601.202 1.00028

    work page doi:10.1109/icse-seet52601.202 2021

  28. [29]

    Milton Friedman. 1937. The use of ranks to avoid the assumption of normality implicit in the analysis of variance.Journal of the American Statistical Association, 32, 200, (Dec. 1937). doi:10.1080/01621459.1937.10503522

    arXiv 1937

  29. [30]

    Rao Fu, Ziyang Luo, Hongzhan Lin, Zhen Ye, and Jing Ma. 2025. ScratchEval: are GPT-4o smarter than my child? evaluating large multimodal models with visual programming challenges. InConference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. doi...

    work page doi:10.18653/v1/2025.naacl-short.57 2025

  30. [31]

    Gemma 3 technical report

    Gemma Team et al. Gemma 3 technical report. (Mar. 2025). arXiv: 2503.19786[cs.CL]

    Pith/arXiv arXiv 2025

  31. [32]

    Víctor González-Calatayud, Paz Prendes-Espinosa, and Rosabel Roig-Vila. 2021. Artificial intelligence for student assessment: a systematic review. Applied Sciences, 11, 12, (June 2021). doi:10.3390/app11125467. Manuscript submitted to ACM EmbeddedKittens: An Evaluation of Code Embeddings for Scratch 35

    work page doi:10.3390/app11125467 2021

  32. [33]

    Isabella Graßl, Benedikt Fein, and Gordon Fraser. 2025. Detecting gender stereotypes in Scratch programming tutorials. InKoli Calling. ACM. doi:10.1145/3769994.3770019

    arXiv 2025

  33. [34]

    Isabella Graßl and Gordon Fraser. 2022. Scratch as social network: topic modeling and sentiment analysis in Scratch projects. InInternational Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). ACM. doi:10.1145/3510458.3513021

    arXiv 2022

  34. [35]

    Elisabeth Griebl, Benedikt Fein, Florian Obermüller, Gordon Fraser, and René Just. 2023. On the applicability of language models to block-based programs. InInternational Conference on Software Engineering (ICSE). IEEE. doi:10.1109/icse48619.2023.00199

    arXiv 2023

  35. [36]

    Rajarshi Haldar and Julia Hockenmaier. 2024. Analyzing the performance of large language models on code summarization. InJoint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING). ELRA and ICCL. https://aclanthology.org/2024.lrec- main.89/

    2024

  36. [37]

    1979.Elements of software science

    Maurice Howard Halstead. 1979.Elements of software science. Elsevier.isbn: 0444002154

    1979

  37. [38]

    no ceiling

    Brian Harvey and Jens Mönig. 2010. Bringing “no ceiling” to Scratch: can one language serve kids and computer scientists? InConstructionism

    2010

  38. [39]

    Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu

    Abram Hindle, Earl T. Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016. On the naturalness of software.Communications of the ACM, 59, 5, (Apr. 2016). doi:10.1145/2902362

    work page doi:10.1145/2902362 2016

  39. [40]

    Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.Neural Computation, 9, 8, (Nov. 1997). doi:10.1162/neco.1997.9.8.1735

    doi:10.1162/neco.1997.9.8.1735 1997

  40. [41]

    Xinyi Hou et al. 2024. Large language models for software engineering: a systematic literature review.ACM Transactions on Software Engineering and Methodology, 33, 8, (Nov. 2024). doi:10.1145/3695988

    doi:10.1145/3695988 2024

  41. [42]

    Sathvik Joel, Jie Wu, and Fatemeh Fard. 2025. A survey on LLM-based code generation for low-resource and domain-specific programming languages.ACM Transactions on Software Engineering and Methodology, (Oct. 2025). doi:10.1145/3770084

    doi:10.1145/3770084 2025

  42. [43]

    Barr, Jacques Klein, and Tegawendé F

    Abdoul Kader Kaboré, Earl T. Barr, Jacques Klein, and Tegawendé F. Bissyandé. 2023. Codegrid: a grid representation of code. InInternational Symposium on Software Testing and Analysis (ISSTA). ACM. doi:10.1145/3597926.3598141

    arXiv 2023

  43. [44]

    Bissyandé

    Patrick Keller, Abdoul Kader Kaboré, Laura Plein, Jacques Klein, Yves Le Traon, and Tegawendé F. Bissyandé. 2021. What you see is what it means! semantic representation learning of code based on visualization and transfer learning.ACM Transactions on Software Engineering and Methodology, 31, 2, (Dec. 2021). doi:10.1145/3485135

    work page doi:10.1145/3485135 2021

  44. [45]

    The Stack: 3 TB of permissively licensed source code

    Denis Kocetkov et al. The Stack: 3 TB of permissively licensed source code. (Nov. 2022). arXiv: 2211.15533[cs.CL]

    Pith/arXiv arXiv 2022

  45. [46]

    Sentencepiece: a simple and language independent subword tokenizer and detokenizer for neural text processing

    Taku Kudo and John Richardson. Sentencepiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. (Aug. 2018). arXiv: 1808.06226[cs.CL]

    Pith/arXiv arXiv 2018

  46. [47]

    Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. 2022. SySeVR: a framework for using deep learning to detect software vulnerabilities.IEEE Transactions on Dependable and Secure Computing, 19, 4, (July 2022). doi:10.1109/tdsc.2021.3051525

    arXiv 2022

  47. [48]

    Chin-Yew Lin. 2004. ROUGE: a package for automatic evaluation of summaries. InText Summarization Branches Out. Association for Computational Linguistics, (July 2004). https://aclanthology.org/W04-1013/

    2004

  48. [49]

    StarCoder 2 and The Stack v2: The Next Generation

    Anton Lozhkov et al. StarCoder 2 and The Stack v2: The Next Generation. (Feb. 2024). arXiv: 2402.19173[cs.SE]

    Pith/arXiv arXiv 2024

  49. [50]

    Dipeeka Luitel, Shiva Nejati, and Mehrdad Sabetzadeh. 2024. Requirements-driven slicing of Simulink models using LLMs. InInternational Requirements Engineering Conference Workshops (REW). IEEE. doi:10.1109/rew61692.2024.00014

    arXiv 2024

  50. [51]

    Zhang, Mike Papadakis, Maxime Cordy, Xiaofei Xie, and Yves Le Traon

    Wei Ma, Mengjie Zhao, Ezekiel Soremekun, Qiang Hu, Jie M. Zhang, Mike Papadakis, Maxime Cordy, Xiaofei Xie, and Yves Le Traon. 2022. Graph- Code2Vec: generic code embedding via lexical and program dependence analyses. InInternational Conference on Mining Software Repositories (MSR). ACM. doi:10.1145/3524842.3528456

    arXiv 2022

  51. [52]

    Xiangyue Ma, Xiaoting Du, Chenglong Li, Jiangtao Meng, Xiaoke Fang, Wenjie Ding, and Zheng Zheng. 2026. How do large language models perform in deep learning code generation? an empirical study.ACM Transactions on Software Engineering and Methodology, (May 2026). doi:10.114 5/3816024

    2026

  52. [53]

    John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn Eastmond. 2010. The Scratch programming language and environment. ACM Transactions on Computing Education, 10, 4, (Nov. 2010). doi:10.1145/1868358.1868363

    arXiv 2010

  53. [54]

    Thomas J. McCabe. 1976. A complexity measure.IEEE Transactions on Software Engineering, SE-2, 4, (Dec. 1976). doi:10.1109/tse.1976.233837

    arXiv 1976

  54. [55]

    McGill and Adrienne Decker

    Monica M. McGill and Adrienne Decker. 2020. Tools, languages, and environments used in primary and secondary computing education. In Conference on Innovation and Technology in Computer Science Education (ITiCSE). ACM. doi:10.1145/3341525.3387365

    arXiv 2020

  55. [56]

    Qing Mi, Yiqun Hao, Liwei Ou, and Wei Ma. 2022. Towards using visual, semantic and structural features to improve code readability classification. Journal of Systems and Software, 193, (Nov. 2022). doi:10.1016/j.jss.2022.111454

    arXiv 2022

  56. [57]

    George A. Miller. 1995. WordNet: a lexical database for English.Communications of the ACM, 38, 11, (Nov. 1995). doi:10.1145/219717.219748

    arXiv 1995

  57. [58]

    1963.Distribution-free multiple comparisons

    Peter Bjorn Nemenyi. 1963.Distribution-free multiple comparisons. Ph.D. Dissertation. Princeton University

    1963

  58. [59]

    Benjamin Paassen, Jessica McBroom, Bryn Jeffries, Irena Koprinska, and Kalina Yacef. 2021. Mapping Python programs to vectors using recursive neural encodings.Journal of Educational Data Mining, 13, 3, (Oct. 2021). doi:10.5281/zenodo.5634224

    doi:10.5281/zenodo.5634224 2021

  59. [60]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: a method for automatic evaluation of machine translation. InAnnual Meeting on Association for Computational Linguistics (ACL). Association for Computational Linguistics. doi:10.3115/1073083.1073135

    arXiv 2001

  60. [61]

    Chris Piech, Jonathan Huang, Andy Nguyen, Mike Phulsuksombati, Mehran Sahami, and Leonidas Guibas. 2015. Learning program embeddings to propagate feedback on student code. InInternational Conference on Machine Learning (ICML). PMLR. https://proceedings.mlr.press/v37/piech15.ht ml. Manuscript submitted to ACM 36 Benedikt Fein and Gordon Fraser

    2015

  61. [62]

    Peng Qi, Yan Sun, Hong Luo, and Mohsen Guizani. 2022. Scratch-Rec: a novel Scratch recommendation approach adapting user preference and programming skill for enhancing learning to program.Applied Intelligence, 52, 8, (Jan. 2022). doi:10.1007/s10489-021-02970-7

    work page doi:10.1007/s10489-021-02970-7 2022

  62. [63]

    Nishat Raihan, Mohammed Latif Siddiq, Joanna C. S. Santos, and Marcos Zampieri. 2025. Large language models in computer science education: a systematic literature review. InACM Technical Symposium on Computer Science Education (SIGCSE). ACM. doi:10.1145/3641554.3701863

    arXiv 2025

  63. [64]

    Lianne Roest, Hieke Keuning, and Johan Jeuring. 2024. Next-step hint generation for introductory programming using large language models. In Australasian Computing Education Conference (ACE). ACM. doi:10.1145/3636243.3636259

    arXiv 2024

  64. [65]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An empirical evaluation of using large language models for automated unit test generation.IEEE Transactions on Software Engineering, 50, 1, (Jan. 2024). doi:10.1109/tse.2023.3334955

    arXiv 2024

  65. [66]

    Sebastian Schweikl and Gordon Fraser. 2025. RePurr: automated repair of block-based learners’ programs. InInternational Conference on the Foundations of Software Engineering (FSE). ACM. doi:10.1145/3715786

    doi:10.1145/3715786 2025

  66. [67]

    Neural machine translation of rare words with subword units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. (Aug. 2015). arXiv: 1508.07909 [cs.CL]

    Pith/arXiv arXiv 2015

  67. [68]

    Samuel Sanford Shapiro and Martin Wilk. 1965. An analysis of variance test for normality (complete samples).Biometrika, 52, 3-4, (Dec. 1965). doi:10.1093/biomet/52.3-4.591

    doi:10.1093/biomet/52.3-4.591 1965

  68. [69]

    Yuan Si, Daming Li, Hanyuan Shi, and Jialu Zhang. 2026. VisionScratch: LLM-based automated feedback generation using code-produced videos for Scratch programs. InInternational Conference on the Foundations of Software Engineering (FSE). ACM. doi:10.1145/3808163

    work page doi:10.1145/3808163 2026

  69. [70]

    Stitch: step-by-step LLM guided tutoring for Scratch

    Yuan Si, Kyle Qi, Daming Li, Hanyuan Shi, and Jialu Zhang. Stitch: step-by-step LLM guided tutoring for Scratch. (Oct. 2025). arXiv: 2510.26634 [cs.SE]

    arXiv 2025

  70. [71]

    OpenAI GPT-5 system card

    Aaditya Singh et al. OpenAI GPT-5 system card. (Dec. 2025). arXiv: 2601.03267[cs.CL]

    Pith/arXiv arXiv 2025

  71. [72]

    Sandro Speth, Niklas Meißner, and Steffen Becker. 2023. Investigating the use of AI-generated exercises for beginner and intermediate programming courses: a chatgpt case study. InConference on Software Engineering Education & Training (CSEE&T). IEEE. doi:10.1109/cseet58097.2023.00030

    arXiv 2023

  72. [73]

    [SW] Peter M. Stahl, lingua-rs version 1.7.2, 2025.url: https://github.com/pemistahl/lingua-rs,swhid: ⟨swh:1:rel:1a486dd6ad25a610d9f181855c5 92234adf7c31f;origin=https://github.com/pemistahl/lingua-rs;visit=swh:1:snp:f055241154533f746f30d7a9fd85fb8f7371234d⟩

    2025

  73. [74]

    Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. 2025. Source code summarization in the era of large language models. InInternational Conference on Software Engineering (ICSE). IEEE. doi:10.1109/icse55347.2025.00 034

    arXiv 2025

  74. [75]

    Alaaeddin Swidan and Felienne Hermans. 2023. A framework for the localization of programming languages. InInternational Conference on Systems, Programming, Languages and Applications: Software for Humanity (SPLASH). ACM. doi:10.1145/3622780.3623645

    arXiv 2023

  75. [76]

    Zhao Tian and Junjie Chen. 2026. Aligning requirement for large language model’s codegeneration. InInternational Conference on Software Engineering (ICSE). ACM. doi:10.1145/3744916.3764572

    arXiv 2026

  76. [77]

    Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.Journal of Machine Learning Research, 9, 86. http://jmlr.org/pape rs/v9/vandermaaten08a.html

    2008

  77. [78]

    András Vargha and Harold D. Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25, 2, (June 2000). doi:10.3102/10769986025002101

    doi:10.3102/10769986025002101 2000

  78. [79]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InAdvances in Neural Information Processing Systems (NIPS). https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee2435 47dee91fbd053c1c4a845aa-Paper.pdf

    2017

  79. [80]

    EmbeddingGemma: powerful and lightweight text representations

    Henrique Schechter Vera et al. EmbeddingGemma: powerful and lightweight text representations. (Sept. 2025). arXiv: 2509.20354[cs.CL]

    Pith/arXiv arXiv 2025

  80. [81]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. InConference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. doi:10.18653/v1/2021.emnlp-main.685

    doi:10.18653/v1/2021.emnlp-main.685 2021

Showing first 80 references.