{"id":"a5fc20a6-c1af-4f4f-8dec-8e68f2a08412","arxiv_id":"2506.00845","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Post-training alignment on synthetic graph data yields statistically significant gains on some real-world implicit-graph tasks, but gains are inconsistent and multi-step reasoning remains fragile.","lead":"This paper tests whether reinforcement learning (GRPO and DPO) on synthetic graph problems, such as connectivity and shortest path, helps large language models solve real-world tasks with hidden graph structure. The results are mixed: some tasks improve significantly, but gains are task-dependent, and the authors find models still struggle with multi-step composition and hallucinated intermediate steps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that post-training alignment, not SFT alone, drives real-world gains is unsupported: Table 2's significance is measured against zero-shot, and the only large alignment-vs-SFT gains come from post-hoc checkpoint-selected LLAMA runs.","rationale":"Read in good faith, the paper's central claim is empirical: post-training alignment on synthetic graph data should improve real-world implicit graph reasoning beyond SFT. The synthetic reward design is rule-based and verifiable, and the paper's negative findings on compositionality and hallucination are honest and useful. The weakest link is not the implicit-graph analogy, which the authors already treat cautiously, but the inferential validity of the headline improvement claim. The reader flagged checkpoint selection and abstract overstatement, and I agree with those concerns, but the more load-bearing issue is that the reported significance is computed against zero-shot, whereas the contribution requires alignment to beat the SFT row. For QWEN, no alignment setting beats SFT on average, so the only evidence for the central claim comes from LLAMA settings that were explicitly stopped early and selected from checkpoints. This makes the z-test p-values conditional on post hoc selection, and the '5 task settings' claim becomes a maximum over many correlated comparisons without multiplicity control. A corrected protocol could plausibly leave some settings significant, for example Blocksworld, so outright rejection is not warranted; the appropriate outcome is a conditional verdict tied to a re-analysis with the correct baseline and without post hoc selection.","tokens_in":25349,"tokens_out":9816,"duration_ms":99792,"concrete_test":"Recompute Table 2 with the SYNTHETIC SFT row as the reference baseline for every alignment setting, using pre-registered fixed epochs or early stopping based only on synthetic validation, and apply a multiple-comparison correction across all alignment settings per dataset. Report separately for QWEN and LLAMA how many of the 8 datasets retain at least one alignment setting significantly better than SFT; if QWEN has zero and LLAMA's count drops below 5, the central claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's stated contribution is that post-training alignment on synthetic data generalizes where SFT does not, so the relevant baseline is the SYNTHETIC SFT row of Table 2. However, the table footnote says statistical significance is tested against zero-shot, not against SFT. Under the relevant comparison the evidence is inconsistent: for QWEN2.5-7B, every alignment row has a lower average increase than SFT (SFT +10.6%; DPO-P +9.7%, DPO-S +7.9%, GRPO-P W/SFT +2.0%, GRPO-S W/SFT +6.2%), so alignment does not beat SFT on average. For LLAMA-3.1-8B, the large alignment-vs-SFT gains (+25.4%, +36.5%, +36.7%) come from settings that Appendix C says were trained for about 3.2 epochs using saved checkpoints 'for better representation of model's performance,' i.e., selected with reference to results, which invalidates the z-test p-values. Section 4.2 further defines the headline '5 task settings' as 'at least one alignment setting' significant, a maximum over 12+ configurations per dataset with no multiple-comparison correction. Thus the central 'not SFT alone' claim rests on comparisons that are either not against the stated baseline, not consistent across models, or not statistically valid as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage recipe: optional supervised fine-tuning on synthetic connectivity and shortest-path tasks, followed by DPO or GRPO post-training alignment with solution-based or process-based rule-based rewards, and evaluation on held-out synthetic tasks and eight real-world tasks with implicit graph structure (multi-hop QA, structured commonsense reasoning, and Blocksworld planning/verification). The headline claim is that post-training alignment on purely synthetic graph data yields statistically significant real-world improvements on five task settings, with an average gain of about 13%, and that process-based rewards outperform solution-based rewards on synthetic tasks while the real-world picture is mixed. The paper also analyzes the compositionality gap between single-step and multi-step reasoning and the prevalence of hallucinated intermediate steps.","tokens_in":25607,"tokens_out":8421,"duration_ms":72886,"significance":"If the central comparison were valid, the paper would make a useful empirical contribution: it addresses a timely question, publicly releases code and results, uses verifiable rule-based rewards, tests two LLM families on eight real-world benchmarks, and includes a careful decomposition of single-step versus multi-step failures. The synthetic reward design is transparent and easy to audit, and the correlation between reward score and synthetic accuracy (Pearson 0.951) supports the internal validity of the reward functions. However, the statistical framing—significance measured against zero-shot, post-hoc checkpoint selection for several LLAMA runs, and a '5 out of 8' claim that is a maximum over alignment configurations—currently substantially weakens the evidence for the paper's stated 'alignment, not SFT alone' claim. These issues are fixable with reanalysis and re-reporting, but they are load-bearing for the main contribution.","major_comments":[{"comment":"Table 2's footnote says statistical significance is measured against zero-shot performance, but the paper's contribution claim is that post-training alignment, not SFT alone, drives real-world gains. Against the relevant baseline (the SYNTHETIC SFT row), the QWEN2.5-7B results show no average advantage for alignment: SFT achieves +10.6% average increase, while DPO-P W/SFT, DPO-S W/SFT, GRPO-P W/SFT, and GRPO-S W/SFT achieve +9.7%, +7.9%, +2.0%, and +6.2%, respectively. The LLAMA3.1-8B comparisons that favor alignment (+25.4%, +36.5%, +36.7%) are affected by the checkpoint-selection issue in Appendix C. The paper should either test significance against synthetic SFT and report those numbers, or explicitly reframe the claim as 'improvement over zero-shot' and remove the 'not SFT alone' wording.","section":"Section 4.2 / Table 2"},{"comment":"Appendix C reports that for LLAMA3.1-8B settings GRPO-P W/O SFT, GRPO-S W/O SFT, and GRPO-S W/SFT, the authors 'train around 3.2 epochs (rather than standard 8 epochs) using saved checkpoints for better representation of model's performance.' Choosing checkpoints with reference to the evaluation invalidates the z-test p-values for those rows and inflates the LLAMA gains reported in Table 2. These runs should be excluded from the headline significance claims or reported as exploratory with no p-values. In addition, no confidence intervals or repeated-seed variability are reported anywhere, so the reader cannot assess the stability of the 13% average gain.","section":"Appendix C"},{"comment":"The headline '5 task settings' (abstract: '5 datasets') is a maximum over the up to 12 alignment configurations per dataset, not a consistent result: the same paragraph reports only 3 of 8 settings with consistent significant improvements. No multiple-comparison correction is applied to the per-configuration z-tests. The 'consistent' and 'at least one configuration' counts should be reported separately, and either the maximum-based claim or the average gain should be qualified accordingly.","section":"Section 4.2 / abstract"},{"comment":"The exact test-set sizes for the real-world benchmarks are inconsistent. Section 3.1 states 'For all real-world datasets, we randomly sampled 1000 instances from each dataset,' but Appendix D says 'Blocksworld dataset has only 500 instances per setting.' Since the z-tests in Table 2 depend directly on these sample sizes, the paper must state the per-dataset test size for every benchmark, including Blocksworld planning and verification.","section":"Sections 3.1 and Appendix D"},{"comment":"The negative Pearson correlation of -0.336 between synthetic-task performance and real-world performance is presented as a side observation, but it bears directly on the transfer mechanism. If training on synthetic connectivity/shortest-path problems were teaching a general graph-reasoning skill, one would not expect negative correlation across training settings. Please reconcile this correlation with the transfer claim, or restrict the claim to the structurally closest tasks (e.g., Blocksworld) and provide a concrete mechanism-based test, such as per-setting correlations or an ablation that isolates the reward signal.","section":"Appendix E.3"}],"minor_comments":[{"comment":"Please fix typos, including 'Methdology' (Section 2 heading), 'incooperates commensense' (Section 3.1), 'alightnment' (Section 4.2), and 'huallicination' (Limitations).","section":"Throughout"},{"comment":"The reported average gain is inconsistent: 12.9% in the abstract, 13% in Section 1, and 13.6% in Section 4.2. Please reconcile the numbers and define the baseline over which the average is taken.","section":"Abstract / Sections 1 and 4.2"},{"comment":"The reward definitions list 'Incorrect answer penalty,' 'Incorrect reasoning penalty,' and 'Format error penalty' all set to 0; these are not penalties. Either rename them as zero rewards or explain why zero values are intended.","section":"Appendix B"},{"comment":"The text refers to 'Appendix B Table 3,' but the mixing-real-world-data table appears in Appendix A. Please fix the cross-reference.","section":"Section 5.1"},{"comment":"Several model responses in Table 5 are followed by a '%' symbol with no legend; please explain what it denotes, and ensure Figure 2's two-panel layout has clear labels for 'final wrong/final correct' in both panels.","section":"Table 5 / Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The anonymous code link is a plus. In revision, I would ask the authors to (i) report significance tests against the synthetic-SFT baseline, (ii) remove or clearly flag the checkpoint-selected LLAMA runs, (iii) apply a multiple-comparison correction or downgrade the '5 of 8' claim to the consistent-improvement count, and (iv) state per-dataset sample sizes. The paper's central idea is publishable after this reanalysis, and the empirical scope is appropriate for cs.LG."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real value here is the negative evidence, not the positive spin. The paper documents a compositionality gap and hallucinated intermediate steps in LLM graph reasoning after synthetic alignment, with honest analysis in Section 5 and the appendices. That is a useful contribution. The new application of GRPO/DPO with rule-based process and solution rewards to synthetic connectivity/shortest-path data is legitimate, and the finding that process rewards help on synthetic tasks but not reliably on real ones is worth reporting.\n\nBut the central claim in the abstract and intro—that post-training alignment gives statistically significant real-world improvements and beats SFT alone—does not hold up to the paper's own tables. Significance in Table 2 is tested against zero-shot, not against the synthetic SFT baseline. For QWEN, every alignment row has a lower average increase than SFT alone (+10.6% for SFT; best alignment +9.7%, most below +7%). For LLAMA, the two largest GRPO-S gains (+36.7% and +5.6% versus +6.5% for SFT) include the +36.7% setting that Appendix C admits was selected via saved checkpoints after around 3.2 epochs, and the +36.5% GRPO-P W/SFT row is not among the checkpoint-selected runs but still appears without confidence intervals. The '5 task settings' in the abstract is really '5 settings where at least one of 12+ alignment configurations was significant,' with no multiple-comparison correction; only 3 settings show consistent gains.\n\nSo the paper's headline is overstated, and the stress-test is essentially correct. The effect of alignment versus SFT is inconsistent across models, and the authors do not provide a valid statistical comparison against the relevant baseline. That said, the weaker claim—synthetic alignment sometimes helps, sometimes hurts, and the real bottlenecks are compositionality and hallucination—is defensible and well supported. The appendix material on single-step versus multi-step and the correlation analysis (reward accuracy 0.951, synthetic-real correlation -0.336) is thoughtful.\n\nWho should read this: anyone working on LLM graph reasoning or synthetic-data generalization. The negative results are worth knowing even if the headline numbers are not reliable. I would not cite the paper as evidence that alignment beats SFT; I would cite it as evidence of transfer limits and hallucination. The paper deserves a serious referee after major revision: the authors must re-run significance tests against SFT, report confidence intervals, disclose checkpoint selection for all runs, and either correct the abstract or reframe the contribution honestly.\n\nRecommendation: engage with it, but on the revised frame.","headline":"Useful negative results about synthetic graph alignment, but the headline claim that alignment beats SFT on real-world tasks doesn't survive the paper's own tables.","tokens_in":26172,"tokens_out":3388,"would_cite":true,"duration_ms":31567,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Post-training alignment on synthetic graph puzzles transfers to real-world reasoning, lifting accuracy by about 13 percent on average.","keywords":["graph reasoning","post-training alignment","reinforcement learning","synthetic data","GRPO","DPO","process reward","generalization"],"falsifier":"A concrete check: drop the two Blocksworld settings, the tasks most similar in format to the synthetic ones, and recompute the average real-world gain; if the 13 percent improvement and the five significant settings largely vanish, the claimed transfer to implicit graph reasoning is really transfer to planning-style tasks. A complementary experiment aligns a model on synthetic arithmetic or sorting data with the same GRPO recipe and tests the same eight real-world tasks, since comparable gains would show the active ingredient is generic reward-driven reasoning rather than graph structure.","tokens_in":25134,"feed_emoji":"🧠","tokens_out":7227,"duration_ms":62993,"temperature":0.7,"pith_summary":"The paper tries to establish that the right way to use synthetic graph data, whose answers are already known exactly, is not to fine-tune a model to imitate those answers but to reward it for producing them during post-training alignment. It claims that alignment with rule-based rewards, especially the on-policy method GRPO, helps a language model internalize the essentials of graph reasoning, and that this skill transfers: on eight real-world tasks with implicit graph structure, the recipe yields statistically significant gains on five settings, averaging about 13 percent over baselines. If true, this matters because it points to a cheap route to improving LLM reasoning on problems where no graph is written down, using only synthetic training data whose correctness a program can verify for free. It also directly contradicts the prior conclusion that synthetic graph training only helps on synthetic graphs.","feed_headline":"Synthetic graph training lifts real-world reasoning by 13%","feed_subtitle":"Rewarding right answers on made-up connectivity puzzles beats plain fine-tuning on five of eight real tasks.","key_machinery":"The carrying mechanism is a rule-based reward scheme keyed to a forced response format. Each response must contain three tagged sections, a think section, a response section of numbered reasoning steps, and an answer section with the final verdict, and a Python program with NetworkX verifies the content. The solution-based reward compares only the answer to ground truth; the process-based reward rebuilds the graph the model's steps refer to, scores each step as a correct fact or a hallucination, and applies a penalty of -2 for invented nodes, edges, or weights. These rewards feed two alignment algorithms: GRPO, an on-policy reinforcement learning method that estimates advantages from groups of rollouts, and DPO, an off-policy preference method trained on high-temperature rollouts. The claim is that scoring the reasoning process rather than just the product forces the policy to learn graph-valid reasoning instead of memorized output patterns, and that this is what makes the learned behavior transfer to tasks where no graph is written down.","core_discovery":"The paper's central claim is that post-training alignment on synthetic connectivity and shortest-path problems generalizes to real-world tasks with implicit graph structure, where plain supervised fine-tuning fails to transfer. The authors design two rule-based rewards for the synthetic problems: a solution-based reward that scores only the final answer, and a process-based reward that reconstructs the graph implied by the model's stated reasoning steps, rewarding correct statements and penalizing hallucinated nodes, edges, and weights. They use these rewards to align off-the-shelf and SFT-then-aligned 7B and 8B instruction-tuned models with GRPO and DPO. They report that GRPO consistently outperforms DPO on synthetic tasks; that process-based rewards beat solution-based rewards by about 24 percent on synthetic tasks under GRPO but show no consistent edge on real-world tasks; that the aligned models achieve statistically significant improvements on five of eight real-world task settings with an average gain of about 13 percent; and that remaining failures concentrate in two places, the compositionality gap between correct single-step answers and correct multi-step answers, and multi-step answers reached through hallucinated or unexplained intermediate steps.","pith_inferences":["The paper's own negative correlation between synthetic and real-world performance (-0.336 Pearson) suggests its headline gains may be carried mainly by tasks structurally closest to the synthetic ones, such as plan verification; re-weighting the eight task settings to exclude Blocksworld might shrink the average gain toward zero, a re-analysis the paper does not run.","A natural extension is to replace the synthetic reward's graph reconstruction with a verifier that checks single-hop facts in a knowledge base, turning the compositionality gap itself into part of the reward signal; this would test whether step-level verification, rather than synthetic graph exposure, is the active ingredient.","If the mechanism is genuinely graph-generic, aligning on synthetic problems from other well-verified structures such as constraint satisfaction or sorting networks should produce analogous real-world gains; if it does not, the improvements are specific to graph-shaped reasoning and the analogy to real-world tasks is weaker than claimed."],"forward_implications":["If the recipe is right, synthetic-data alignment becomes a low-cost route to improving real-world reasoning, since the training data and reward signals need no human annotation.","The GRPO-over-DPO result implies that on-policy reward optimization is the transferable part of alignment, with off-policy preference learning adding little after SFT.","The process-reward advantage on synthetic but not real tasks implies fine-grained step verification helps most where the target solution space is exactly the training space.","The observed compositionality gap means even successful alignment does not fix multi-step reasoning, so gains should shrink as required step counts grow.","SFT before GRPO can hurt performance, evidence that pattern memorization can interfere with reward-driven learning and that the two stages are not automatically complementary."],"supporting_citations":[{"why":"Defines the synthetic connectivity and shortest-path tasks in natural language that the paper's training data is built from.","marker":"(Wang et al., 2023)"},{"why":"Supplies the NLGift synthetic dataset and the baseline finding that synthetic SFT alone does not transfer to real-world tasks, which this work sets out to beat.","marker":"(Zhang et al., 2024b)"},{"why":"Contributes GRPO, the on-policy RL algorithm that carries most of the reported gains.","marker":"(Shao et al., 2024)"},{"why":"Contributes DPO, the off-policy preference method used for the comparison alignment setting.","marker":"(Rafailov et al., 2023)"},{"why":"NetworkX is the tool that reconstructs and checks the graph implied by each response's reasoning steps in the process-based reward.","marker":"(Hagberg et al., 2008)"},{"why":"StrategyQA is a primary real-world multi-hop QA evaluation target for the transfer claim.","marker":"(Geva et al., 2021)"},{"why":"Knowledge Crosswords supplies the multi-hop QA dataset with explicit constraint structure used for evaluation and single-step probing.","marker":"(Ding et al., 2024)"},{"why":"Blocksworld planning and verification tasks are the real-world benchmarks showing the largest alignment gains.","marker":"(Valmeekam et al., 2023)"}],"fun_headline_variants":["Post-training alignment on synthetic graphs improves real-world reasoning","Alignment on synthetic graph data yields 13% real-world gain","Synthetic graph training with RL alignment beats SFT on real tasks","Process rewards help synthetic graphs, not real-world tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that real-world tasks such as strategy questions, commonsense reasoning, and block-stacking planning engage the same graph reasoning machinery as synthetic connectivity and shortest-path problems, so rewards earned on synthetic puzzles teach transferable skills rather than task-specific response formats.","fun_headline_variants_meta":{"raw":{"variants":["Post-training alignment on synthetic graphs improves real-world reasoning","Alignment on synthetic graph data yields 13% real-world gain","Synthetic graph training with RL alignment beats SFT on real tasks","Process rewards help synthetic graphs, not real-world tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001622,"raw_usage":{"total_tokens":6489,"prompt_tokens":1018,"completion_tokens":5471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":5403}},"tokens_in":634,"tokens_out":5471,"duration_ms":38555,"temperature":1.0,"reasoning_tokens":5403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:56:12.399001+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: drop the two Blocksworld settings, the tasks most similar in format to the synthetic ones, and recompute the average real-world gain; if the 13 percent improvement and the five significant settings largely vanish, the claimed transfer to implicit graph reasoning is really transfer to planning-style tasks. A complementary experiment aligns a model on synthetic arithmetic or sorting data with the same GRPO recipe and tests the same eight real-world tasks, since comparable gains would show the active ingredient is generic reward-driven reasoning rather than graph structure.","supporting_citations":[{"cited_title":"Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change","cited_arxiv_id":null,"evidence_quote":"Blocksworld planning and verification tasks are the real-world benchmarks showing the largest alignment gains."}],"review_version":1}