{"id":"39f23a83-7e1d-4c97-931e-fd3b1c2e7168","arxiv_id":"2505.14425","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning LLMs on synthetic instructions transfers well to simple spatial tasks but degrades on regular, repetitive layouts when instructions are human-authored.","lead":"This paper tests how well LLMs that were trained only on templated, rule-based instructions can follow free-form human instructions for building colored block arrangements on a grid. It finds that the models succeed on simple layouts but lose much of their skill on repetitive layouts that need abstract reading between the lines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The simple-vs-regular comparison in Table 4 is confounded by prompt-template differences (Appendix A.1, Figs. 11–12), so the claim that human regular-board instructions are intrinsically harder to ground is not yet established; a matched-template control is required.","rationale":"The paper's intended contribution is to show where synthetic-only fine-tuning fails: high scores on synthetic instructions of both board types, lower scores on human instructions, with a larger absolute gap on regular boards (Table 4). I read the claim as an empirical generalization about instruction complexity, not a formal theorem. What must be true for it to hold is that the simple-vs-regular contrast isolates structural/compositional complexity. Appendix A.1 and Figures 11 and 12 show it does not: simple boards require generating a function definition plus a usage line using only put(), while regular boards provide a combo function signature and require only a usage snippet. This is an internally acknowledged difference, and the paper's own Table 2 shows prompt-style manipulations change simple-board scores by more than 0.2 for Qwen2.5-Coder-32B. That makes the SB/RB comparison the weakest load-bearing step. I am not claiming the results are fabricated or that the ST→HA gap is an artifact; within regular boards, the template is held constant between synthetic and human instructions, so the drop from 1.00 to 0.54 stands. The issue is specifically the attribution to 'more abstract or relational understanding' via the regular-board condition. A matched-template control would settle whether the regular-board deficit is a property of the instructions or of the prompt/API design. The reader's weakest-assumption analysis identified this same confound, and the conditional verdict remains appropriate; I would not escalate to REJECT because the evidence is recoverable and the core direction of the finding is plausible.","tokens_in":17599,"tokens_out":5607,"duration_ms":55110,"concrete_test":"Take the 130 regular-board human instructions from the HA/RB test set and re-run the fine-tuned Qwen2.5-Coder-32B (and, ideally, Llama3.3-70B) under the simple-board prompt template of Fig. 11, expanding repeated placements into explicit put() calls in the gold code, so only instruction content differs from the simple-board condition. Symmetrically, run simple-board human instructions under the regular-board template with functions pre-defined. If HA/RB accuracy rises substantially toward HA/SB (or HA/SB falls under the regular template), the central claim is a prompt-format artifact; if the gap persists under matched templates, the structural-complexity interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that fine-tuning solely on synthetic data is insufficient for transfer to 'more abstract or relational' instructions rests on Table 4's simple-vs-regular gap, e.g., Qwen2.5-Coder-32B at 0.98→0.68 on simple boards but 1.00→0.54 on regular boards. However, the two board types use different prompt templates: simple boards (Fig. 11) ask for a Function definition plus a Usage call using only put(), while regular boards (Fig. 12) pre-define a $COMBO_NAME function and ask only for an Output snippet. Therefore the SB/RB contrast varies output format, available API, and output length alongside structural complexity. The paper concedes in §5.1 that small models' reversal 'may be due to differences in the expected output structure,' and Table 2 shows prompt-style changes (FSG/FSC) can move Qwen2.5-Coder-32B's simple-board accuracy from 0.68 to 0.45 or 0.32, demonstrating that format/API changes alone can create large accuracy swings. Thus the observed regular-board drop cannot be uniquely attributed to intrinsic complexity or relational abstraction in human instructions. The ST→HA gap within regular boards is not confounded by template, but the paper's headline 'more complex tasks' and the simple-vs-regular contrast used to support it are.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether instruction-tuned LLMs fine-tuned only on synthetic instructions generalize to human-authored instructions in a grounded spatial-reasoning task. Using the SARTCo dataset, the authors fine-tune several open-weight models (Qwen2.5-Coder, Qwen3, Llama3) and compare execution success on synthetic versus human-authored instructions, on simple and regular boards. They also evaluate transfer to the HEXAGONS and TidyBot tasks and provide an error analysis and an instruction-similarity analysis. The central claim is that synthetic-only fine-tuning yields strong performance on simple synthetic boards but degrades on human-authored instructions, especially on regular boards that require more abstract or relational understanding.","tokens_in":17919,"tokens_out":4131,"duration_ms":40497,"significance":"If the main claim is established, the paper would be a useful, controlled demonstration that synthetic instruction tuning does not automatically transfer to naturalistic human instructions in grounded tasks. The design has notable strengths: aligned synthetic/human instruction pairs for the same target boards, an executable-code evaluation metric with exact board matching, a human baseline, cross-domain probes on HEXAGONS and TidyBot, and a detailed error taxonomy. These assets make the study potentially valuable for the instruction-following and embodied-language communities. However, the central simple-versus-regular comparison is currently confounded by prompt-template differences, and the main quantitative conclusions lack statistical support. The contribution can be salvaged with additional experiments and analysis.","major_comments":[{"comment":"The simple-board vs. regular-board comparison is confounded by prompt-template differences. The simple-board prompt (Fig. 11) requires a Function definition plus a Usage call and exposes only the put() API, while the regular-board prompt (Fig. 12) pre-defines a $COMBO_NAME function and asks only for an Output snippet. The paper itself states in Appendix A.1 that the prompt structure differs between the two board types. Therefore the SB/RB accuracy gaps in Table 4 (e.g., Qwen2.5-Coder-32B at 0.68 vs. 0.54 on human-authored instructions) cannot be uniquely attributed to board complexity, repetition, or the need for more abstract relational understanding. Table 2 reinforces the concern: merely changing the prompt style for the same board type (FSG/FSC) moves Qwen2.5-Coder-32B's simple-board accuracy from 0.68 to 0.45 or 0.32, which is the same magnitude as many SB/RB differences. A matched-template control is required, for example evaluating regular-board human instructions under the full Function+Usage prompt, or evaluating simple-board tasks under the regular-board Output-only prompt, before the 'more complex tasks' conclusion can be drawn.","section":"§5.1, Table 4; Appendix A.1, Figs. 11–12"},{"comment":"The main quantitative claims rest on point estimates without confidence intervals or significance tests. Each condition uses only 130 test boards, and the outcome is binary exact-match execution success. For example, Qwen2.5-Coder-32B's regular-board improvement from 0.23 to 0.54 after fine-tuning, or the SB/RB difference of 0.68 vs. 0.54, could be within binomial sampling noise; a bootstrap interval or a McNemar test on paired boards would establish the reliability of these differences. Without this, the statements that 'performance degrades significantly on more complex tasks' and that regular-board gains are 'low' are not statistically grounded.","section":"Tables 1, 2, 4"},{"comment":"The fine-tuning configuration is reported inconsistently between the main text and the appendix. Section 4.1 lists lora_dropout=0.10 and '20 steps', while Appendix A.3 lists lora_dropout=0.0 and four gradient accumulation steps, and Appendix A.3.1 mentions early stopping with patience 2. Since the paper's central comparisons are between fine-tuned and base models, a single reproducible configuration is essential. Please reconcile these numbers and state exactly which configuration produced Tables 1, 2, and 4.","section":"§4.1 vs. Appendix A.3"},{"comment":"The instruction-similarity analysis is presented as explaining the execution-success gap ('These results suggest that lower semantic and lexical alignment ... limits the generalization ability'), but the reported BLEU and embedding-similarity values are only correlations with execution success, computed over the same boards, and BLEU is near zero for regular boards regardless of model behavior. The causal direction is not established; for instance, both the low BLEU and the low execution success could follow from regular-board instructions being more abstract, without similarity itself being the bottleneck. Please either soften the causal language or add a regression/partial-correlation analysis that controls for the number of shapes and board type.","section":"§5.2, Table 5"}],"minor_comments":[{"comment":"Figure 5's caption contains an apparent typo: '41.5% and 1 “Element. Mismatch” 58.5%' should presumably read 'and Element Mismatch 58.5%.'","section":"§5.2, Fig. 6"},{"comment":"The caption contains a spelling error: 'all the models performance was detoriated' should be 'deteriorated.'","section":"Table 7 caption"},{"comment":"The word 'nodel' appears in the caption: 'Error Categorization across nodel responses'; it should be 'model responses.'","section":"Table 10 caption"},{"comment":"The model is referred to as 'Llama3-70B' in the qualitative analysis but as 'Llama3.3-70B' everywhere else; please use one consistent name.","section":"§5.2"},{"comment":"The Human-Baseline row places 0.98 and 0.76 in the SB/RB columns, but the 'Before/After' structure of the table makes it look like a model score; please add a footnote or restructure the row so it is clear this is an independent human upper bound, not a before/after fine-tuning value.","section":"Table 1"},{"comment":"Tables 2 and 3 use the columns SB/RB for both FSG and FSC, but these prompt styles are introduced as 'regular board prompt styles'; clarify in the text whether the same styles were applied to simple boards and why this is a meaningful comparison.","section":"Tables 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question, and the authors have assembled a valuable testbed with aligned instruction pairs, executable evaluation, and human baselines. The main weakness is the confounded simple-vs-regular comparison; without a matched-template control, the headline claim about regular boards requiring more abstract or relational understanding is not established. Given the availability of the dataset and the relatively straightforward nature of the required control experiments, major revision is the appropriate verdict. I would also ask the editor to ensure the authors address the missing statistical grounding and the hyperparameter inconsistencies, since these affect reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this about the paper: it is a genuinely useful empirical study of synthetic-to-human instruction transfer for grounded spatial reasoning. The core phenomenon, that fine-tuning on synthetic instructions transfers poorly to human-authored instructions, is real and clearly shown. But the paper's more specific story, that regular boards are harder because of structural complexity, is confounded by the fact that simple and regular boards use different prompt templates.\n\nWhat is new and good: they fine-tune a range of LLMs on synthetic instructions only and evaluate on both synthetic and human instructions for the same boards, with an execution-based metric, a detailed error taxonomy, embedding-similarity analysis, and cross-domain probes on HEXAGONS and TidyBot. The human baseline is a nice addition. The strongest result is within regular boards: for example, Qwen2.5-Coder-32B drops from 1.00 on synthetic to 0.54 on human instructions. That gap is not confounded by template and supports the paper's central negative finding.\n\nThe stress-test concern lands. Simple boards (Appendix A.1, Figure 11) ask the model to generate a Function definition and a Usage call using only put(). Regular boards (Figure 12) pre-define the $COMBO_NAME function and ask only for an Output snippet. So the simple-vs-regular contrast varies output format, API surface, and output length alongside structural complexity. The paper's own prompt-variation results (Table 2) show that changing the prompt style alone can swing Qwen2.5-Coder-32B's simple-board accuracy from 0.68 to 0.45 or 0.32. That does not sink the paper, but it means the 'more complex tasks' conclusion needs a matched-template control, or at least a reanalysis that separates instruction style from board structure.\n\nOther soft spots are more minor: there are no confidence intervals or significance tests; test sets are 130 boards per type; hyperparameters are inconsistent between Section 4.1 (lora_dropout=0.10, batch size=8, lr=1e-4, '3 epochs, 20 steps') and Appendix A.3 (dropout=0.0, 'three epochs, and four gradient accumulation steps'); and no code or data links are provided, so exact replication is not currently possible. These are all fixable in revision.\n\nWho is this for? People working on instruction tuning, synthetic data, and grounded spatial language. The error taxonomy and similarity analysis are worth borrowing. It deserves serious peer review; with the template confound addressed and the training details cleaned up, it would be a solid benchmark contribution. I would send it to review rather than desk reject.","headline":"Useful benchmark study of synthetic-to-human transfer in spatial grounding, but the simple-vs-regular 'complexity' claim is confounded by prompt-template differences.","tokens_in":18399,"tokens_out":2625,"would_cite":true,"duration_ms":23893,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on synthetic instructions alone does not transfer to human-authored spatial language, especially for repetitive arrangements.","keywords":["spatial grounding","instruction tuning","synthetic data","generalization gap","code generation","human-authored instructions","large language models","execution-based evaluation"],"falsifier":"Run the regular-board evaluation with the simple-board prompt template, full function definition plus Function and Usage labels, and watch the 1.00-to-0.54 drop on human-authored instructions. If matching the template shrinks or erases the gap, the claim that human regular instructions are intrinsically harder to ground is false.","tokens_in":17398,"feed_emoji":"🧩","tokens_out":6671,"duration_ms":62683,"temperature":0.7,"pith_summary":"This paper studies whether language models that are tuned to turn instructions into code for building block arrangements can generalize from templated synthetic instructions to free-form human instructions. The authors fine-tune several instruction-tuned models on synthetic instruction–code pairs only, then evaluate the generated code by executing it in a virtual 8x8 grid and comparing the resulting board to the target. The central finding is that transfer is strong for simple arrangements but breaks down on repetitive \"regular\" boards: the best fine-tuned model executes synthetic instructions almost perfectly but drops to about half accuracy on human-authored regular instructions. The paper argues that synthetic-only supervision cannot supply the abstract, relational understanding that human phrasing of repeated structures requires.","feed_headline":"Template-trained LLMs fail on human spatial instructions","feed_subtitle":"Even the best fine-tuned model drops from near-perfect on templated grids to 54% on repetitive human-authored instructions.","key_machinery":"The load-bearing setup is a paired instruction benchmark in which every target board comes with two instruction styles, a template-generated synthetic instruction and a free-form human instruction, plus a ground-truth Python function. The model must emit a function definition or usage call that is executed in a virtual simulator, and success means the simulated 8x8 board matches the target exactly. This design lets the authors vary instruction style and board type, simple non-repetitive versus regular repetitive, while holding the target board fixed. Supporting analyses use embedding similarity between synthetic and human instructions and error categorization of the executed code.","core_discovery":"On its own terms, the paper establishes that the generalization gap from synthetic to human instructions is controlled by structural complexity rather than by model size alone. Fine-tuning on synthetic data yields near-perfect execution on synthetic instructions for several models, for example 0.98 to 1.00 for Qwen2.5-Coder-32B on simple and regular boards, but human-authored instructions drop to 0.68 on simple boards and 0.54 on regular boards. The gap is larger for regular boards, where human instructions express repetition and symmetry implicitly, and it persists under prompt-style variations and few-shot prompting. Error analysis shows that fine-tuning mostly fixes low-level placement errors, leaving element mismatches and referential ambiguity as the residual bottleneck, and embedding similarity between human and synthetic instructions for the same board predicts how much transfer occurs.","pith_inferences":["Because the simple and regular board conditions use different prompt templates and output formats, part of the regular-board gap could reflect prompt format rather than human instruction complexity, and a template-matched comparison would separate the two.","Embedding distance to selected synthetic templates could serve as a cheap predictor of where fine-tuning will fail, allowing targeted paraphrase collection for low-similarity boards.","The null transfer on the hexagonal-grid task suggests that grid geometry and coordinate mapping dominate transfer, so testing the same instruction style on the original grid size would isolate instruction style from environment shift.","For robotics pipelines, a practical takeaway is to collect human paraphrases only for repetitive and relational structures rather than for all tasks."],"forward_implications":["Synthetic-only fine-tuning is a reliable route only for non-repetitive spatial instructions; deployments with repetitive patterns will need human or human-like training data.","Prompt variations, such as adding a function signature or schematic description, and few-shot examples do not close the regular-board gap, so the limitation is not simple prompt underspecification.","The residual errors after fine-tuning are mainly element mismatches rather than board-placement crashes, meaning models can place objects but misassign shape, color, or order.","Cross-domain transfer is uneven: the same fine-tuning helps on an object-arrangement task for strong models but gives no boost on a hexagonal-grid drawing task, so gains do not automatically generalize to new spatial formats.","Instruction similarity predicts transfer: boards whose human instructions are semantically close to their synthetic templates benefit more from fine-tuning."],"supporting_citations":[{"why":"This citation supplies the dataset of paired synthetic and human instructions with ground-truth code and target boards used for fine-tuning and evaluation.","marker":"Kranti et al., 2024b"},{"why":"This citation provides the game-play evaluation harness that executes the model-generated code and measures abort and success rates.","marker":"Chalamalasetti et al., 2023"},{"why":"This citation gives the sentence-embedding model used to compute semantic similarity between synthetic and human instructions.","marker":"Reimers and Gurevych, 2019"},{"why":"This citation provides the HEXAGONS drawing task used to test cross-domain transfer after fine-tuning.","marker":"Lachmy et al., 2022"},{"why":"This citation provides the TidyBot arrangement task used to test cross-domain transfer after fine-tuning.","marker":"Wu et al., 2023"}],"fun_headline_variants":["Synthetic training, human failure: LLM spatial gap","Complexity, not size, predicts LLM instruction gap","Template-trained LLMs stumble on human spatial language","Human instructions break spatial LLMs trained on templates","Why synthetic-only training fails for spatial reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on treating simple-versus-regular as a pure difference in structural complexity, even though the two board types are presented with different prompt templates and output formats in the appendix.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic training, human failure: LLM spatial gap","Complexity, not size, predicts LLM instruction gap","Template-trained LLMs stumble on human spatial language","Human instructions break spatial LLMs trained on templates","Why synthetic-only training fails for spatial reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000991,"raw_usage":{"total_tokens":4138,"prompt_tokens":818,"completion_tokens":3320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":3246}},"tokens_in":434,"tokens_out":3320,"duration_ms":22079,"temperature":1.0,"reasoning_tokens":3246,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:34:32.769908+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the regular-board evaluation with the simple-board prompt template, full function definition plus Function and Usage labels, and watch the 1.00-to-0.54 drop on human-authored instructions. If matching the template shrinks or erases the gap, the claim that human regular instructions are intrinsically harder to ground is false.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This citation provides the HEXAGONS drawing task used to test cross-domain transfer after fine-tuning."}],"review_version":1}