{"id":"4bc56be1-b857-450f-94c3-f0666cf1baff","arxiv_id":"2607.19291","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Structure-aware embeddings such as GGNN transfer to Scratch and can support sprite naming and, with weaker evidence, correctness and progress prediction.","lead":"Researchers tested whether code embeddings, vector representations of programs used in machine learning, work for Scratch, a visual block-based language for children. They found that structure-aware models beat text-based LLMs on a new sprite-naming task, and pre-trained embeddings can track student progress and, more weakly, predict program correctness without fine-tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ4 correctness-prediction evidence may be confounded by training-set overlap and lacks trivial baselines; unseen-project correlations are weak (r²=0.13–0.33), so transfer claim is not yet established.","rationale":"The reader's verdict (CONDITIONAL) correctly identifies that RQ4/RQ5 rest on the assumption that Euclidean distance in embedding space is a valid correctness proxy and that the evaluation projects were not effectively seen in pre-training. I agree that this is the central weakness. My partial disagreement is that the most load-bearing issue is not just the validity of the distance proxy or the BoatRace contamination, but the absence of any trivial baseline that would rule out the null hypothesis that simpler program-size features explain the correlation. The paper even acknowledges FruitCatching's weaker correlation is likely due to alternative solutions (Section 5.4.1), which is itself an admission that distance-to-a-single-solution is not a robust correctness proxy. The strongest correlation is on the contaminated project, so the headline 'feasible without fine-tuning' rests on weak or confounded evidence. The proposed test—de-contamination plus a size-based baseline—would directly reveal whether the embedding correlation survives on truly unseen projects and whether it outperforms a trivial heuristic. Since the paper provides a replication package, this test is executable in principle. I therefore keep the verdict CONDITIONAL: the direction is credible, but the correctness-prediction claim needs de-contaminated evaluation and baseline comparisons before it can be accepted.","tokens_in":34943,"tokens_out":3575,"duration_ms":39315,"concrete_test":"Use the provided replication package (Zenodo DOI 10.5281/zenodo.21273159) to: (1) Detect near-duplicates between the three RQ4 classroom projects (BoatRace, FruitCatching, Spaceship) and the 656k sprite-naming training projects, using project-ID lookups or LitterBox AST similarity; exclude any overlapping projects from the evaluation. (2) Recompute the GGNN and Qwen embedding-fitness correlations (Eq. 2 vs. Eq. 1) on this de-contaminated set and report r² and p per project. (3) Add a trivial baseline: for each project p, compute f_size(p) = 1 - |size(s)-size(p)| / max_q |size(s)-size(q)| using total block count (and, separately, AST node count); correlate this with f_test. If a trivial baseline yields r² comparable to the embedding-based r² on the unseen projects, then the claim that embeddings add semantic value for correctness prediction is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that pre-trained Scratch embeddings capture semantic information enabling correctness prediction without fine-tuning. RQ4 tests this by correlating embedding distance to a single model solution with Whisker test fitness. The strongest correlation (BoatRace, GGNN r²=0.45) is undermined by the paper's own admission (Section 5.4.1) that BoatRace is based on a publicly available CodeClubRik project and that the sprite-naming training set likely contains similar programs. On genuinely unseen projects, the correlations are much weaker: FruitCatching r²=0.13 (p=0.018) and Spaceship r²=0.33 (p=0.010). These explain only 13–33% of variance, which is weak support for a claim of 'predicting functional correctness.' Furthermore, no trivial baselines are reported. Since f_embed is defined as normalized distance to the model solution, a simple baseline based on code size (e.g., block count, AST node count) could plausibly correlate with test fitness and might match or exceed the embedding-based correlations. Without such a control, the observed correlation cannot be attributed to the embeddings' structural/semantic encoding; it may be an artifact of program completeness or size. Thus, the strongest evidence for transfer is likely inflated, and the remaining evidence is weak and uncontrolled.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a systematic evaluation of code embedding approaches for Scratch programs. It introduces a 'sprite naming' task as a proxy code-summarisation benchmark, trains five dedicated models (NeuralCodeSum, code2vec, code2seq, ASTNN, GGNN) on 656k mined Scratch projects, and compares them against four pre-trained LLMs under zero-shot prompting. It further evaluates multilingual robustness (RQ2), fine-tuning strategies for whole-program classification (RQ3), and the use of pre-trained embeddings—without fine-tuning—as surrogates for functional correctness (RQ4) and student progress (RQ5). The headline results are that graph-based GGNN embeddings outperform AST-path and flat-token models on sprite naming, and that embeddings transfer to progress prediction with strong correlation (RQ5, GGNN r²=0.735). The central claim that embeddings transfer to correctness prediction is, however, only weakly supported for unseen projects and is confounded by training-set overlap on one project.","tokens_in":35309,"tokens_out":2411,"duration_ms":27620,"significance":"If the transfer claims hold, this would be a valuable first comparative study of code embeddings for a block-based visual language, with practical implications for learning analytics in K-12 settings. The paper's strengths include the very large training corpus (1.8M sprites), the use of established model implementations, the multi-language evaluation, and a public replication package with preprocessing tooling. The GGNN result on sprite naming is clean and internally consistent, and the progress-projection result (RQ5) is a genuine positive finding. However, the correctness-prediction evidence in RQ4 is currently the weakest link: the strongest correlation (BoatRace, r²=0.45) is undermined by the authors' own admission of training-data overlap, the unseen-project correlations are modest at best (r²=0.13–0.33), and no trivial baselines are reported. Because the abstract explicitly claims transfer to 'predicting functional correctness', this issue is load-bearing and needs to be addressed before the central claim can be accepted.","major_comments":[{"comment":"The BoatRace result is explicitly caveated: the project 'is based on a publicly available project... our sprite naming training dataset likely contains similar programs.' This makes r²=0.45 unsuitable as evidence for transfer to unseen tasks. Please re-run the analysis either excluding BoatRace or reporting it separately as a leakage check. The remaining unseen-project correlations (FruitCatching r²=0.13, Spaceship r²=0.33 for GGNN) are weak, so the correctness-transfer claim needs to be either softened or supported by additional evidence.","section":"§5.4.1, Fig. 10, Eq. (2)"},{"comment":"No trivial baselines are reported. Since f_embed is a normalized distance to the model solution, a simple baseline such as program size (block count, AST node count), number of sprites, or edit distance to the solution could plausibly correlate with test fitness—e.g., because more complete programs are larger. Without comparing against such baselines, the observed correlations cannot be attributed to the structural or semantic content of the embeddings. Please add these controls and report whether the embedding distance explains variance beyond them.","section":"§4.6.2, Eq. (2)"},{"comment":"The abstract claims that embeddings enable 'predicting functional correctness', but the evidence includes non-significant results: FruitCatching with Qwen-per-sprite has r²=0.08 and p=0.069. Across the nine reported project/model combinations no multiple-comparison correction is applied. Please report adjusted p-values or use a suitable correction, and explicitly characterize which combinations actually support the claim. This is important because the current 'statistically significant correlation for most combinations' statement is too strong given the borderline and non-significant cases.","section":"Fig. 10 (b,c)"}],"minor_comments":[{"comment":"The axis labels render 'r/uni00B2' in the provided PDF; this should be r². Please check the final rendering.","section":"Fig. 10"},{"comment":"Typo: 'the number of operators and operators used in the code' should presumably be 'operators and operands'.","section":"§5.2"},{"comment":"The METEOR metric is computed with an English WordNet; the paper acknowledges this, but it might be worth stating explicitly that the multilingual comparison in RQ2 excludes METEOR for this reason, which is already done. No action needed beyond the existing statement.","section":"§4.3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about the BoatRace overlap, which is good, but the central transfer claim in the abstract is stronger than the current RQ4 results justify. I would be willing to accept after the authors add trivial baselines, remove or reclassify BoatRace, and temper the correctness claim. The RQ5 result is a strong positive and should be emphasized as the main evidence of transfer."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a useful empirical comparison, and the headlining sprite-naming result holds up. The authors train five embeddings and prompt four LLMs on a new sprite-naming task over ~1.8M sprites, and the ordering is clean: GGNN (graph with control/data flow) > AST-based > flat-token > LLMs. They also evaluate on ten languages, which is a nice extra. The replication package is real (Zenodo, open-source preprocessing). This is the first structured comparison of its kind for Scratch, and it is worth engaging with.\n\nThe weak spot is exactly where the stress-test note points. RQ4 claims the embeddings predict functional correctness without fine-tuning, but the evidence is thinner than the abstract suggests. The strongest correlation is BoatRace (GGNN r²=0.45), and the authors admit in §5.4.1 that BoatRace is based on a public CodeClubRik project likely present in the sprite-naming training set. On genuinely unseen projects the correlations are r²=0.13–0.33. Those are statistically significant in some cases (FruitCatching p=0.018 for GGNN), but they explain little variance. Worse, there is no trivial baseline: f_embed is a normalized L2 distance to the model solution, and something as simple as program size or block count could plausibly correlate with test fitness. Without that control, the transfer claim is not established. The authors are honest about the BoatRace overlap, but they still lean on it in the summary.\n\nRQ3 and RQ5 are less problematic. The fine-tuning comparison and the progress-variance projection show the embeddings carry usable signal, and RQ5's correlations (r²≈0.71–0.74) are respectable. The multilingual analysis is also solid, though the subtokenization caveat is real.\n\nMinor notes: the LLM comparison is fair enough given the prompt design, but the LLMs' poor sprite-naming scores are not a surprise—generating exact strings is not their strength. The paper could be trimmed, but that is not a substantive flaw.\n\nBottom line: the sprite-naming evaluation is a solid contribution and deserves to be published after revision. RQ4 needs either a de-contaminated evaluation (exclude BoatRace or re-train without similar projects) and at least one trivial baseline, or a toned-down claim. I'd send it to review. I'd cite it for the sprite-naming result. Reading group: maybe.","headline":"A competent first comparison of code embeddings for Scratch with a clean sprite-naming result, but the correctness-prediction claim is weaker than the abstract suggests and needs trivial baselines plus a de-contaminated evaluation.","tokens_in":35743,"tokens_out":2087,"would_cite":true,"duration_ms":21781,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Code embeddings trained on public Scratch projects transfer to classroom tasks like correctness prediction and progress tracking without task-specific fine-tuning.","keywords":["Scratch","code embeddings","learning analytics","program correctness","GGNN","sprite naming","block-based programming","transfer learning"],"falsifier":"A concrete test: collect a previously unpublished classroom exercise with its own reference solution and test suite, pre-train GGNN on the public sprite-naming corpus, and measure the correlation between embedding fitness and test-passing rate. If r² falls to near zero on several such exercises (as it nearly does for FruitCatching), the transfer claim fails. A complementary probe is to deliberately remove all projects similar to the evaluation project from the pre-training set and check whether the correlation collapses.","tokens_in":34838,"feed_emoji":"🧩","tokens_out":4626,"duration_ms":40043,"temperature":0.7,"pith_summary":"Code embeddings — vector representations that capture what a program does — can be transferred to Scratch, the block-based language used by millions of young learners. The paper trains five embedding models on roughly 1.8 million Scratch sprites using a new 'sprite naming' task, and compares them to four pre-trained large language models. The graph-based model that encodes control and data flow (GGNN) names sprites far better than token-sequence or AST-only models, and the same pre-trained embeddings estimate whether a student's program passes its tests and how far a student has progressed toward a solution, without per-task retraining. This makes fine-tuning-free learning analytics feasible in small classrooms where labelled data is scarce.","feed_headline":"Scratch code embeddings predict student progress without fine-tuning","feed_subtitle":"Trained on 1.8 million sprites, the graph-based GGNN transfers to correctness and progress tasks.","key_machinery":"The central object is the sprite-naming pre-training task combined with the GGNN's program graph. The graph is built from the abstract syntax tree plus semantic edges (LastRead, LastWrite, GuardedBy, ComputedFrom) and Scratch-specific edges connecting broadcast events to receivers and custom-procedure calls to definitions; multiple rounds of message passing along these edges let the model capture the concurrent, event-driven structure that flat token sequences miss. The second load-bearing mechanism is the embedding-fitness measure: the normalised Euclidean distance between a student project and a single model solution in embedding space, used as a surrogate for functional correctness.","core_discovery":"The paper's central discovery is that the 'sprite naming' task — predicting the name of a sprite from the code inside it — both benchmarks how well an embedding captures Scratch semantics and serves as a pre-training task. Across five models and four LLMs, the GGNN, a gated graph neural network whose input graph augments the abstract syntax tree with control-flow, data-flow, and Scratch-specific event/message edges, achieves the best sprite naming scores (F1=0.55). Using this pre-trained model, the authors then show that the Euclidean distance between a student program's embedding and a model solution's embedding correlates with the proportion of passed test cases (r² up to 0.49 on one class","pith_inferences":["The Euclidean-distance proxy likely underestimates correctness when students solve a task differently from the reference solution (FruitCatching r²=0.13); combining distance with syntax-aware structural similarity (e.g., AST edit distance) is a testable way to strengthen the surrogate.","Because BoatRace was probably present in the pre-training corpus, the reported r²=0.45 likely overstates transfer; a fair evaluation would require a fresh classroom dataset with a contemporaneous private base project.","The same recipe — pre-train on sprite naming, then use progress-variance projection — could apply to other event-driven visual languages or even event-driven textual systems, since the graph explicitly encodes message passing and concurrency.","A live classroom dashboard could use these embeddings at minute-level granularity (as in RQ5) to flag students who deviate from the expected path, before test suites are even run."],"forward_implications":["Teachers could receive automatic, per-student estimates of whether a Scratch program works, computed from embeddings alone without running test suites or fine-tuning models to the task.","The sprite-naming benchmark gives the community a scalable pre-training task for future Scratch code embeddings, filling the gap left by missing documentation or method-naming corpora.","An embedding-based large language model can stand in for a dedicated graph model on whole-program tasks such as category classification and progress estimation, at the cost of GPU or API resources.","The observed performance drop on non-Latin writing systems implies that multilingual subword tokenization is a prerequisite for equitable classroom deployment."],"fun_headline_variants":["Sprite naming picks GGNN as best Scratch code embedder","Graph-based embeddings capture Scratch semantics best","No fine-tuning: GGNN embeddings predict Scratch correctness","GGNN embeddings transfer to Scratch without fine-tuning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The single load-bearing premise is that Euclidean distance in pre-trained embedding space between a student program and one model solution is a valid proxy for functional correctness, and that the evaluation projects were not already part of the pre-training data.","fun_headline_variants_meta":{"raw":{"variants":["Sprite naming picks GGNN as best Scratch code embedder","Graph-based embeddings capture Scratch semantics best","No fine-tuning: GGNN embeddings predict Scratch correctness","GGNN embeddings transfer to Scratch without fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001092,"raw_usage":{"total_tokens":4391,"prompt_tokens":732,"completion_tokens":3659,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":3599}},"tokens_in":476,"tokens_out":3659,"duration_ms":20572,"temperature":1.0,"reasoning_tokens":3599,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T12:50:24.718621+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: collect a previously unpublished classroom exercise with its own reference solution and test suite, pre-train GGNN on the public sprite-naming corpus, and measure the correlation between embedding fitness and test-passing rate. If r² falls to near zero on several such exercises (as it nearly does for FruitCatching), the transfer claim fails. A complementary probe is to deliberately remove all projects similar to the evaluation project from the pre-training set and check whether the correlation collapses.","supporting_citations":[],"review_version":1}