{"id":"15efd248-3dd3-455b-ad25-c3c745e89c86","arxiv_id":"2504.20752","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding synthetic knowledge-graph facts to raise the ratio of inferred to atomic facts produces a grokking-like jump in OOD accuracy on comparison-style 2WikiMultiHopQA questions, but not on composition questions.","lead":"This paper tests whether a small language model can learn multi-hop reasoning the way networks learn during grokking, by adding synthetic facts to its training data. The result is a near-perfect score on simple comparison questions from a Wikipedia-derived benchmark, but the effect vanishes on harder compositional questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Composition experiment at phi=6.25 shows no OOD jump, directly contradicting the claimed threshold-sufficiency of phi_G.","rationale":"The reader's weakest_assumption focuses on the missing same-size low-phi control for the comparison task, which is a valid confound. However, the composition result is even more load-bearing because it is an in-paper falsification of the threshold-sufficiency claim: phi=6.25 was explicitly claimed to trigger grokking, yet the model never grokked. This is not a hypothetical confound but an observed failure of the stated mechanism on the same model, training setup, and augmentation pipeline. I therefore center the critique on the composition contradiction, while agreeing with the reader that the comparison result is also confounded by data quantity and yes/no format. The reader's rationale mentions the composition failure as a contradiction to sufficiency, so there is partial agreement. The recommended verdict remains REJECT: the central claim as stated is not supported, and the abstract's benchmark-level overstatement further weakens the paper. The concrete test (raising phi to 16 for composition) would settle whether the mechanism can be salvaged by a higher threshold or fails entirely.","tokens_in":14555,"tokens_out":4224,"duration_ms":40194,"concrete_test":"Hold atomic facts fixed at 800 and train the composition model with inferred facts increased to 12,800 (phi=16), keeping all other hyperparameters identical. If OOD accuracy remains near 7% with no late-phase jump, the phi-threshold mechanism is falsified for composition. If OOD jumps, the paper's cited threshold (phi>3.6 or phi=6.25) is wrong and must be revised, undermining the generality of the central claim.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that raising the relation-specific ratio phi_r above an empirical threshold triggers a grokking transition (Section 3.2, Conclusion). The paper's own composition experiment (Section 4.3, Figure 4(c), Table 3) sets phi=6.25, explicitly described as enough to trigger grokking dynamics in practice, yet OOD accuracy stays at 7% with no late-phase improvement. This is a direct internal counterexample: either the threshold is far higher for composition than for comparison, or phi is not the causal driver. The paper does not revise phi_G or offer a principled explanation; it only says complex multi-hop relations are harder to internalize, which is post hoc. Separately, the comparison success (phi=8, OOD 96%) is confounded because augmentation increased the number of training facts roughly 80x (120 atomic and 60 inferred to 1,000 atomic and 8,000 inferred) and the task is a yes/no format where a majority-class baseline would score 50%; without a same-size low-phi control or a format-matched baseline, the comparison jump cannot be attributed to phi. The abstract's claim of 'matching or exceeding current state-of-the-art' also overstates Table 3, where average OOD is 0.52, below GPT-4o (0.56) and o1-mini (0.60). These issues together leave the central mechanism unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data augmentation method for real-world multi-hop reasoning, augmenting 2WikiMultiHopQA with LLM-generated synthetic atomic and inferred facts to raise the relation-specific ratio phi_r of inferred to atomic facts above an empirically adopted threshold. The authors train an 8-layer GPT-2-style transformer from scratch and report a late-phase OOD accuracy jump on comparison queries after augmentation to phi=8, while composition queries at phi=6.25 show no such jump. The abstract claims up to 95-100% accuracy on 2WikiMultiHopQA and state-of-the-art performance, but Table 3 reports average OOD accuracy of 0.52, below the GPT-4o and o1-mini baselines.","tokens_in":14791,"tokens_out":2939,"duration_ms":29896,"significance":"If the central claim were supported—that raising phi_r alone triggers grokking on real-world data and exceeds strong baselines—the result would be notable for bridging synthetic grokking studies and practical NLP. The paper is also honest in reporting the composition null result and in listing factuality and feasibility limitations. However, the central causal claim is not isolated from data-volume and format confounds, and the composition experiment directly contradicts the asserted threshold sufficiency, so the significance currently rests on an unsupported mechanism.","major_comments":[{"comment":"The composition experiment at phi=6.25, described in §3.2.2 as \"enough to trigger grokking dynamics in practice,\" shows OOD accuracy stuck at 7% with no late-phase jump. This is a direct internal counterexample to the paper's central claim that crossing the phi threshold induces grokking; the explanation given (\"complex multi-hop relations appear harder to internalize\") is post hoc and does not reconcile the contradiction with the proposed threshold mechanism.","section":"§4.3, Figure 4(c), Table 3"},{"comment":"The augmentation procedure changes both the ratio phi_r and the total number of training examples: comparison data goes from 120 atomic and 60 inferred facts to 1,000 atomic and 8,000 inferred facts, and composition data from 200/100 to 800/5,000. There is no same-size low-phi control, so the observed OOD improvement on comparison cannot be attributed to phi_r rather than to increased data volume, LLM-generated query genre uniformity, or other dataset artifacts.","section":"§4.2 vs. §4.3"},{"comment":"The abstract claims the approach \"achieves up to 95-100% accuracy on 2WikiMultiHopQA - substantially improving over strong baselines and matching or exceeding current state-of-the-art results,\" but the reported average OOD accuracy in Table 3 is 0.52, below GPT-4o (0.56) and o1-mini (0.60). The \"up to\" phrasing cherry-picks the comparison column while the composition column is 0.07, and the comparison average is dominated by a yes/no format with a 50% majority baseline that is not reported.","section":"Table 3 and Abstract"},{"comment":"The experimental protocol reports \"the best run\" out of three random seeds for the central quantitative results, with only a vague \"variability ±1-2%\" note. For a claim about a qualitative phenomenon like a late-phase jump, per-seed curves or at least error bars are needed; the best-run selection undermines confidence that the reported 96% OOD comparison accuracy is a stable, reproducible outcome rather than a favorable seed.","section":"§4.1"},{"comment":"The phi_G thresholds (3.6, 18, and the chosen 8 and 6.25) are imported from prior synthetic-task work (Wang et al., 2024) and neither fitted nor validated on this real-world setup. The composition result at phi=6.25 suggests the thresholds do not transfer, so the paper does not establish that any threshold was crossed in a causally meaningful sense; this is a correctness risk tied to the paper's central claim.","section":"§2.2, §3.2"}],"minor_comments":[{"comment":"Algorithm 2 uses the token \"graph.augment atomic()\" with an inconsistent separator; either \"graph.augment_atomic()\" or a hyphen-free style should be used throughout.","section":"Algorithms 1 and 2"},{"comment":"The caption for panel (a) mentions \"original and grokked GPT2-small\" but the text in §4.2 says the original data produces no late-phase jump; the caption does not clearly distinguish which curve is which and should be revised for readability.","section":"Figure 4 caption"},{"comment":"The last row of Table 3 is typeset as \"0.52GPT2-Small\", merging the average OOD cell with the model name; this formatting error should be corrected.","section":"Table 3"},{"comment":"The proof of Lemma 1 contains typos such as \"asymtotic\" and \"boudned\" (also in the text of the lemma statement); these should be fixed.","section":"Appendix A.1"},{"comment":"The in-text citation \"Loshchilov et al., 2017\" refers to a paper whose reference entry lists only two authors plus \"et al.\"; the reference list should be completed consistently per the venue's style.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript header claims appearance in ICML 2025 proceedings, but the content appears to be an arXiv preprint; this provenance mismatch should be verified by the editor. The main finding is a familiar grokking-type claim without the controlled experiments needed to support it, and the composition null result appears to falsify the paper's own threshold criterion. I see no currently feasible revision within the manuscript's stated scope that would address the confounds without redoing the experimental design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. The paper reports a real empirical phenomenon: on comparison-style multi-hop questions, a GPT-2-small trained on heavily LLM-augmented data shows a late-phase out-of-distribution accuracy jump, reaching 96% OOD—better than GPT-4o's 25% on that subset. That is new and worth taking seriously. But the load-bearing claim that this jump is caused by crossing a relation-specific ratio phi_r is not supported by their own data, and the abstract overstates the benchmark results.\n\nWhat the paper does well: it takes grokking out of modular arithmetic and onto Wikipedia-derived QA, defines an explicit ID/OOD distinction, and shows a clear training curve where OOD accuracy rises late on comparison questions. The augmentation pipeline—generate atomic facts, then pair them into yes/no comparison questions—is simple and easy to follow. The paper also honestly shows the composition failure: Figure 4(c) and Table 3 give OOD accuracy of 7%, with no late-phase jump.\n\nNow the soft spots, in proportion. The central mechanism fails on its own terms. Section 3.2 says phi=6.25 for composition is \"enough to trigger grokking dynamics in practice,\" but Section 4.3 shows no grokking. The dismissal—\"complex multi-hop relations appear harder to internalize\"—is post hoc. Either the threshold is much higher for composition, or phi is not the causal driver; the paper offers no revised threshold or principled account. Second, the comparison result is confounded: augmentation increased the number of training facts roughly 80-fold and switched the task to a yes/no format, where majority class gives 50%. There is no same-size low-phi control, so the jump cannot be uniquely attributed to phi. Third, the abstract's \"95-100% accuracy ... matching or exceeding state-of-the-art\" ignores that average OOD is 52%, below GPT-4o (56%) and o1-mini (60%). That is selective reporting. No code or data is released, though the setup is standard enough to reproduce.\n\nThis is a paper with a real empirical nugget buried under an overclaimed mechanism. A reader working on grokking or data augmentation for multi-hop QA will find useful observations; a reader looking for a rigorous test of phi-threshold theories will be frustrated.\n\nRecommendation: send it to peer review—the phenomenon deserves scrutiny and the authors are asking the right question. But it needs major revision: add same-size controls, explain or bound the composition counterexample, and rewrite the abstract to match Table 3.","headline":"A genuine late-phase OOD jump on comparison-style QA is buried under an overclaimed phi-threshold mechanism that the paper's own composition experiment contradicts.","tokens_in":15380,"tokens_out":3284,"would_cite":false,"duration_ms":33150,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By raising a relation-level ratio of inferred to atomic facts past a threshold, this paper induces grokking in a GPT-2-style transformer on real-world multi-hop QA, reaching 96% out-of-distribution accuracy on comparison questions.","keywords":["grokking","data augmentation","multi-hop question answering","knowledge graphs","out-of-distribution generalization","synthetic data","transformer interpretability","emergent reasoning circuits"],"falsifier":"Train the same model on an augmented comparison dataset with the same total number of examples as the high-$\\phi_r$ setting but with $\\phi_r$ kept near the original value of about 0.5 (equal atomic and inferred counts), and measure out-of-distribution accuracy across training; if the jump still appears, the threshold claim is wrong. A high-$\\phi_r$ dataset with open-ended answers would further test whether the yes/no format rather than $\\phi_r$ drives the result.","tokens_in":14299,"feed_emoji":"🧠","tokens_out":9890,"duration_ms":84457,"temperature":0.7,"pith_summary":"The paper tries to show that grokking—the sudden late-training jump from memorization to generalization—can be induced on real-world factual question answering, not just on toy arithmetic tasks. It defines, for each relation, the ratio $\\phi_r$ of inferred (multi-hop) facts to atomic (single-hop) facts and argues that raising $\\phi_r$ past an empirical threshold makes a GPT-2-style transformer form internal reasoning circuits. On the 2WikiMultiHopQA benchmark, augmented data push the model to 100% in-distribution and 96% out-of-distribution accuracy on comparison questions, matching or exceeding much larger pretrained models. A sympathetic reader would care because this suggests a distribution-level recipe for unlocking implicit multi-hop reasoning in modest-sized transformers.","feed_headline":"Synthetic facts trigger grokking in real-world multi-hop QA","feed_subtitle":"Raising a relation-level fact ratio past a threshold lifts a 124M-parameter model to 96% out-of-distribution accuracy.","key_machinery":"The load-bearing quantity is the relation-specific generalization ratio $\\phi_r = |\\mathcal{F}_{I,r}|/|\\mathcal{F}_{A,r}|$, the number of multi-hop inferred facts per atomic fact involving relation $r$. The mechanism it controls is the formation of a 'generalizing circuit,' an internal transformer subnetwork that chains relation steps rather than retrieving memorized answers. Two analytic lemmas give the ceiling on how many $n$-hop paths a sparse knowledge graph can generate: the relation-specific branching factor $b_r$ bounds $\\phi_{n,r}$ by roughly $b_r^{n-1}$ as the entity set grows, so if $b_r$ is too low, full generalization is impossible without added edges or nodes. The method's work is to push $\\phi_r$ past an empirical threshold (about 8 for comparison and 6.25 for composition here) through language-model-generated synthetic atomic and inferred facts.","core_discovery":"The central claim is that real-world multi-hop reasoning can be grokked once the data distribution supplies enough multi-step facts relative to single-step facts. For each relation $r$, the paper tracks $\\phi_r = |\\mathcal{F}_{I,r}| / |\\mathcal{F}_{A,r}|$, the ratio of inferred facts to atomic facts involving $r$; below a threshold the model memorizes, and above it the model's out-of-distribution accuracy jumps late in training, the signature of grokking. Experimentally, augmenting 2WikiMultiHopQA comparison data from 120 atomic and 60 inferred facts to 1,000 atomic and 8,000 inferred facts ($\\phi_r = 8$) produces 100% in-distribution and 96% out-of-distribution accuracy with an 8-layer GPT-2-style transformer. Compositional queries, augmented to $\\phi_r = 6.25$, reach 93% in-distribution accuracy but only 7% out-of-distribution, showing that the threshold is necessary but not sufficient for every relation shape. The paper also reports that even factually wrong synthetic data can strengthen generalization, because the model is pushed toward relational structure instead of memorized content.","pith_inferences":["Holding dataset size fixed while varying only $\\phi_r$—a same-size low-$\\phi$ control—would be the cleanest causal test; the paper does not include it, so the threshold interpretation remains an open extension.","The yes/no format of the comparison questions may itself ease generalization; applying the same augmentation to free-form or multi-choice answers would test whether the effect survives format changes.","Because incorrect synthetic facts can help, augmentation can be read as shaping distribution geometry rather than preserving factuality, suggesting deliberate injection of structured distractors as a design tool with separate factuality safeguards.","The threshold $\\phi_G$ likely shifts with model scale and pretraining, so mapping it across widths, depths, and pretrained initializations would connect this recipe to scaling laws for grokking."],"forward_implications":["On structured comparison questions, a 124-million-parameter GPT-2-style model reaches 100% in-distribution and 96% out-of-distribution accuracy, roughly matching or surpassing much larger pretrained systems on the same benchmark.","Raising $\\phi_r$ is a distribution-shaping recipe: the original 2WikiMultiHopQA data, with $\\phi \\approx 0.5$, show no late-training out-of-distribution jump, while the augmented version does.","Relation types differ in how easily they grok: composition reaches 93% in-distribution accuracy but only 7% out-of-distribution despite augmentation, so the threshold is necessary but not sufficient.","Factually incorrect synthetic data can still improve generalization, implying that relational structure, not fact accuracy, drives circuit formation.","Grokking extends from small synthetic tasks to real-world Wikipedia-based factual reasoning, making the phenomenon relevant to practical question answering."],"supporting_citations":[{"why":"Introduces the grokking phenomenon and the delayed memorization-to-generalization transition that this paper aims to reproduce on real-world data.","marker":"Power et al., 2022"},{"why":"Supplies the relation-specific ratio $\\phi_r$ and its empirical threshold values that the augmentation strategy targets.","marker":"Wang et al., 2024"},{"why":"Constructs the 2WikiMultiHopQA benchmark and its structured and unstructured subsets used in all experiments.","marker":"Ho et al., 2020"},{"why":"Provides the mechanistic-interpretability framing of grokking as circuit formation that motivates the analysis.","marker":"Nanda et al., 2023"},{"why":"Establishes the link between knowledge graph completion and multi-hop QA that the augmentation procedure builds on.","marker":"Liu et al., 2022"},{"why":"Explains grokking through circuit efficiency, supporting the claim that increasing inferred facts encourages generalizing circuits.","marker":"Varma et al., 2023"}],"fun_headline_variants":["Grokking goes real: synthetic data unlocks multi-hop reasoning","Raise fact ratio, watch transformers grok real-world QA","Even wrong synthetic facts trigger grokking in multi-hop QA","From memorization to generalization via synthetic fact injection","95% on multi-hop QA: synthetic data flips on grokking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the late-training out-of-distribution jump is caused by the ratio $\\phi_r$ crossing a threshold, not by the larger number of training examples or the simple yes/no format of the augmented comparison questions.","fun_headline_variants_meta":{"raw":{"variants":["Grokking goes real: synthetic data unlocks multi-hop reasoning","Raise fact ratio, watch transformers grok real-world QA","Even wrong synthetic facts trigger grokking in multi-hop QA","From memorization to generalization via synthetic fact injection","95% on multi-hop QA: synthetic data flips on grokking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1518,"prompt_tokens":1046,"completion_tokens":472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":386}},"tokens_in":662,"tokens_out":472,"duration_ms":4704,"temperature":1.0,"reasoning_tokens":386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:21:10.866326+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model on an augmented comparison dataset with the same total number of examples as the high-$\\phi_r$ setting but with $\\phi_r$ kept near the original value of about 0.5 (equal atomic and inferred counts), and measure out-of-distribution accuracy across training; if the jump still appears, the threshold claim is wrong. A high-$\\phi_r$ dataset with open-ended answers would further test whether the yes/no format rather than $\\phi_r$ drives the result.","supporting_citations":[],"review_version":1}