Pith. sign in

REVIEW 5 major objections 5 minor 19 references

Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By raising a relation-level ratio of inferred to atomic facts past a threshold, this paper induces grokking in a GPT-2-style transformer on real-world multi-hop QA, reaching 96% out-of-distribution accuracy on comparison questions.

desk verdict A genuine late-phase OOD jump on comparison-style QA is buried under an overclaimed phi-threshold mechanism that the paper's own composition experiment contradicts. read the letter →

arxiv 2504.20752 v2 pith:U46NP3PI submitted 2025-04-29 cs.CL cs.AIcs.LG

Roman Abramov , Felix Steinbauer , Gjergji Kasneci This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords grokkingdataaugmentationmulti-hopquestionansweringknowledgegraphsout-of-distributiongeneralizationsynthetictransformerinterpretabilityemergentreasoningcircuits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that grokking—the sudden late-training jump from memorization to generalization—can be induced on real-world factual question answering, not just on toy arithmetic tasks. It defines, for each relation, the ratio $\phi_r$ of inferred (multi-hop) facts to atomic (single-hop) facts and argues that raising $\phi_r$ past an empirical threshold makes a GPT-2-style transformer form internal reasoning circuits. On the 2WikiMultiHopQA benchmark, augmented data push the model to 100% in-distribution and 96% out-of-distribution accuracy on comparison questions, matching or exceeding much larger pretrained models. A sympathetic reader would care because this suggests a distribution-level recipe for unlocking implicit multi-hop reasoning in modest-sized transformers.

What carries the argument

The load-bearing quantity is the relation-specific generalization ratio $\phi_r = |\mathcal{F}_{I,r}|/|\mathcal{F}_{A,r}|$, the number of multi-hop inferred facts per atomic fact involving relation $r$. The mechanism it controls is the formation of a 'generalizing circuit,' an internal transformer subnetwork that chains relation steps rather than retrieving memorized answers. Two analytic lemmas give the ceiling on how many $n$-hop paths a sparse knowledge graph can generate: the relation-specific branching factor $b_r$ bounds $\phi_{n,r}$ by roughly $b_r^{n-1}$ as the entity set grows, so if $b_r$ is too low, full generalization is impossible without added edges or nodes. The method's work is to push $\phi_r$ past an empirical threshold (about 8 for comparison and 6.25 for composition here) through language-model-generated synthetic atomic and inferred facts.

What would settle it

Train the same model on an augmented comparison dataset with the same total number of examples as the high-$\phi_r$ setting but with $\phi_r$ kept near the original value of about 0.5 (equal atomic and inferred counts), and measure out-of-distribution accuracy across training; if the jump still appears, the threshold claim is wrong. A high-$\phi_r$ dataset with open-ended answers would further test whether the yes/no format rather than $\phi_r$ drives the result.

Watch

Extended reading notes

Core claim

The central claim is that real-world multi-hop reasoning can be grokked once the data distribution supplies enough multi-step facts relative to single-step facts. For each relation $r$, the paper tracks $\phi_r = |\mathcal{F}_{I,r}| / |\mathcal{F}_{A,r}|$, the ratio of inferred facts to atomic facts involving $r$; below a threshold the model memorizes, and above it the model's out-of-distribution accuracy jumps late in training, the signature of grokking. Experimentally, augmenting 2WikiMultiHopQA comparison data from 120 atomic and 60 inferred facts to 1,000 atomic and 8,000 inferred facts ($\phi_r = 8$) produces 100% in-distribution and 96% out-of-distribution accuracy with an 8-layer GPT-2-style transformer. Compositional queries, augmented to $\phi_r = 6.25$, reach 93% in-distribution accuracy but only 7% out-of-distribution, showing that the threshold is necessary but not sufficient for every relation shape. The paper also reports that even factually wrong synthetic data can strengthen generalization, because the model is pushed toward relational structure instead of memorized content.

Load-bearing premise

The load-bearing premise is that the late-training out-of-distribution jump is caused by the ratio $\phi_r$ crossing a threshold, not by the larger number of training examples or the simple yes/no format of the augmented comparison questions.

Editorial extensions

If this is right

  • On structured comparison questions, a 124-million-parameter GPT-2-style model reaches 100% in-distribution and 96% out-of-distribution accuracy, roughly matching or surpassing much larger pretrained systems on the same benchmark.
  • Raising $\phi_r$ is a distribution-shaping recipe: the original 2WikiMultiHopQA data, with $\phi \approx 0.5$, show no late-training out-of-distribution jump, while the augmented version does.
  • Relation types differ in how easily they grok: composition reaches 93% in-distribution accuracy but only 7% out-of-distribution despite augmentation, so the threshold is necessary but not sufficient.
  • Factually incorrect synthetic data can still improve generalization, implying that relational structure, not fact accuracy, drives circuit formation.
  • Grokking extends from small synthetic tasks to real-world Wikipedia-based factual reasoning, making the phenomenon relevant to practical question answering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Holding dataset size fixed while varying only $\phi_r$—a same-size low-$\phi$ control—would be the cleanest causal test; the paper does not include it, so the threshold interpretation remains an open extension.
  • The yes/no format of the comparison questions may itself ease generalization; applying the same augmentation to free-form or multi-choice answers would test whether the effect survives format changes.
  • Because incorrect synthetic facts can help, augmentation can be read as shaping distribution geometry rather than preserving factuality, suggesting deliberate injection of structured distractors as a design tool with separate factuality safeguards.
  • The threshold $\phi_G$ likely shifts with model scale and pretraining, so mapping it across widths, depths, and pretrained initializations would connect this recipe to scaling laws for grokking.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a data augmentation method for real-world multi-hop reasoning, augmenting 2WikiMultiHopQA with LLM-generated synthetic atomic and inferred facts to raise the relation-specific ratio phi_r of inferred to atomic facts above an empirically adopted threshold. The authors train an 8-layer GPT-2-style transformer from scratch and report a late-phase OOD accuracy jump on comparison queries after augmentation to phi=8, while composition queries at phi=6.25 show no such jump. The abstract claims up to 95-100% accuracy on 2WikiMultiHopQA and state-of-the-art performance, but Table 3 reports average OOD accuracy of 0.52, below the GPT-4o and o1-mini baselines.

Significance. If the central claim were supported—that raising phi_r alone triggers grokking on real-world data and exceeds strong baselines—the result would be notable for bridging synthetic grokking studies and practical NLP. The paper is also honest in reporting the composition null result and in listing factuality and feasibility limitations. However, the central causal claim is not isolated from data-volume and format confounds, and the composition experiment directly contradicts the asserted threshold sufficiency, so the significance currently rests on an unsupported mechanism.

major comments (5)
  1. [§4.3, Figure 4(c), Table 3] The composition experiment at phi=6.25, described in §3.2.2 as "enough to trigger grokking dynamics in practice," shows OOD accuracy stuck at 7% with no late-phase jump. This is a direct internal counterexample to the paper's central claim that crossing the phi threshold induces grokking; the explanation given ("complex multi-hop relations appear harder to internalize") is post hoc and does not reconcile the contradiction with the proposed threshold mechanism.
  2. [§4.2 vs. §4.3] The augmentation procedure changes both the ratio phi_r and the total number of training examples: comparison data goes from 120 atomic and 60 inferred facts to 1,000 atomic and 8,000 inferred facts, and composition data from 200/100 to 800/5,000. There is no same-size low-phi control, so the observed OOD improvement on comparison cannot be attributed to phi_r rather than to increased data volume, LLM-generated query genre uniformity, or other dataset artifacts.
  3. [Table 3 and Abstract] The abstract claims the approach "achieves up to 95-100% accuracy on 2WikiMultiHopQA - substantially improving over strong baselines and matching or exceeding current state-of-the-art results," but the reported average OOD accuracy in Table 3 is 0.52, below GPT-4o (0.56) and o1-mini (0.60). The "up to" phrasing cherry-picks the comparison column while the composition column is 0.07, and the comparison average is dominated by a yes/no format with a 50% majority baseline that is not reported.
  4. [§4.1] The experimental protocol reports "the best run" out of three random seeds for the central quantitative results, with only a vague "variability ±1-2%" note. For a claim about a qualitative phenomenon like a late-phase jump, per-seed curves or at least error bars are needed; the best-run selection undermines confidence that the reported 96% OOD comparison accuracy is a stable, reproducible outcome rather than a favorable seed.
  5. [§2.2, §3.2] The phi_G thresholds (3.6, 18, and the chosen 8 and 6.25) are imported from prior synthetic-task work (Wang et al., 2024) and neither fitted nor validated on this real-world setup. The composition result at phi=6.25 suggests the thresholds do not transfer, so the paper does not establish that any threshold was crossed in a causally meaningful sense; this is a correctness risk tied to the paper's central claim.
minor comments (5)
  1. [Algorithms 1 and 2] Algorithm 2 uses the token "graph.augment atomic()" with an inconsistent separator; either "graph.augment_atomic()" or a hyphen-free style should be used throughout.
  2. [Figure 4 caption] The caption for panel (a) mentions "original and grokked GPT2-small" but the text in §4.2 says the original data produces no late-phase jump; the caption does not clearly distinguish which curve is which and should be revised for readability.
  3. [Table 3] The last row of Table 3 is typeset as "0.52GPT2-Small", merging the average OOD cell with the model name; this formatting error should be corrected.
  4. [Appendix A.1] The proof of Lemma 1 contains typos such as "asymtotic" and "boudned" (also in the text of the lemma statement); these should be fixed.
  5. [References] The in-text citation "Loshchilov et al., 2017" refers to a paper whose reference entry lists only two authors plus "et al."; the reference list should be completed consistently per the venue's style.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the phi ratio is counted from the constructed corpus, the threshold is imported from external prior work, and the OOD jump is an empirical observation.

full rationale

I find no step in which a 'prediction' or 'first-principles result' reduces to its own inputs by construction. The generalization ratio phi_r is directly counted from the augmented corpus (Definition 2.10 and Section 3.2: 8,000 inferred facts over 1,000 atomic facts for comparison; 5,000 over 800 for composition), rather than fitted to the OOD outcome. The threshold phi_G is adopted from Wang et al. (2024), an external prior study whose authors do not overlap with the present paper, so it is independent support rather than a self-citation chain. The comparison experiment reports an observed late-phase OOD accuracy jump in Section 4.3 / Figure 4(b), and the composition experiment reports no such jump in Section 4.3 / Figure 4(c). Neither of these results is derived from the definition of phi_r; both are empirical training curves. The paper itself discloses the composition null result ('OOD Accuracy remains low, showing no late-phase improvement'), which is an internal evidence problem for the claimed threshold-sufficiency of phi, but it is not circularity. Likewise, the abstract's 'matching or exceeding current state-of-the-art' wording is weakened by Table 3's average OOD 0.52 versus GPT-4o's 0.56 and o1-mini's 0.60, but overstatement is a correctness issue, not a circular reduction. There are no self-citations, no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors, no ansatz smuggled in via citation, and no known result merely renamed. The central positive finding is a designed demonstration built on an externally sourced threshold, which creates confounds and generalization risks but does not make the derivation circular.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new particles or physical entities are introduced. The 'generalizing circuits' are explanatory constructs inherited from prior mechanistic interpretability work and are not directly measured in this paper; the Limitations section admits their mechanics remain only partially understood.

free parameters (3)
  • phi_G (minimal generalization ratio threshold) = approx. 3.6 (slow), up to 18 (fast), cited from Wang et al. (2024)
    The paper's design targets (phi=8 for comparison, phi=6.25 for composition) are chosen to exceed this prior empirical threshold; it is not re-estimated here.
  • Augmentation dataset sizes = 1,000 atomic + 8,000 inferred (comparison); 800 atomic + 5,000 inferred (composition)
    Hand-selected sizes to raise phi above phi_G; no sensitivity analysis is reported.
  • Model hyperparameters (learning rate, batch size, weight decay) = 5e-5, 512, 1.0
    Standard choices; no tuning is reported relative to grokking behavior.
assumptions (4)
  • domain assumption The empirical threshold phi_G derived from small synthetic algorithmic tasks (Wang et al., 2024) transfers unchanged to real-world Wikipedia text and this 8-layer GPT-2 architecture.
    Central to the claim; Sections 2.2 and 3.2.1 rely on it without independent validation. The composition experiment contradicts it.
  • domain assumption Under the random-graph model, each potential directed edge exists independently with probability b/(|V|-1) (Section 2.5, Lemma 1).
    Used to derive the asymptotic bounds on phi_r and to justify the need for augmentation; the real Wikipedia KG is not a random graph.Empirically, Figure 5 shows only shape-level agreement.
  • domain assumption LLM-generated synthetic facts do not introduce systematic leakage between training and OOD test queries, and OOD test queries are 'entirely new combinations' (Section 4.1).
    The construction of the OOD test set and any deduplication against training synthetic data are not described, so this assumption is unverified.
  • domain assumption The answer-format consistency in the augmented data (e.g., the yes/no same-country format) cannot be exploited to answer the queries, so OOD accuracy reflects latent relational reasoning.
    No negative control with format-only answers is run; Section 4.6 success cases are nearly all of the same phrasal shape.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers." pith.science (2026). https://pith.science/paper/U46NP3PI

@misc{pith2026250420752,
  author       = {Pith},
  title        = {Pith review of: Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U46NP3PI}},
  note         = {Machine review of arXiv:2504.20752}
}
abstract

Transformers have achieved great success in numerous NLP tasks but continue to exhibit notable gaps in multi-step factual reasoning, especially when real-world knowledge is sparse. Recent advances in grokking have demonstrated that neural networks can transition from memorizing to perfectly generalizing once they detect underlying logical patterns - yet these studies have primarily used small, synthetic tasks. In this paper, for the first time, we extend grokking to real-world factual data and address the challenge of dataset sparsity by augmenting existing knowledge graphs with carefully designed synthetic data to raise the ratio $\phi_r$ of inferred facts to atomic facts above the threshold required for grokking. Surprisingly, we find that even factually incorrect synthetic data can strengthen emergent reasoning circuits rather than degrade accuracy, as it forces the model to rely on relational structure rather than memorization. When evaluated on multi-hop reasoning benchmarks, our approach achieves up to 95-100% accuracy on 2WikiMultiHopQA - substantially improving over strong baselines and matching or exceeding current state-of-the-art results. We further provide an in-depth analysis of how increasing $\phi_r$ drives the formation of generalizing circuits inside Transformers. Our findings suggest that grokking-based data augmentation can unlock implicit multi-hop reasoning capabilities, opening the door to more robust and interpretable factual reasoning in large-scale language models.

Figures

Figures reproduced from arXiv: 2504.20752 by the authors.

Figure 1
Figure 1. Average accuracy on 2WikiMultiHopQA for comparison task. Despite GPT2-small being a model with 124 million parame￾ters, grokked version achieves almost 100% accuracy, beating the most recent gpt-4o and o1-mini models. From Toy Grokking to Real-World Data. Recent work on grokking (Power et al., 2022) has shown that, under certain conditions, overparameterized neural networks sud￾denly transition from pure memorizatio… view at source ↗
Figure 2
Figure 2. Exemplary Knowledge Graph with Synthesized Data. Four original nodes (black) and three relations (blue) result in two inferred facts. Two additional synthetic nodes and relations (red) extend the amount of inferred facts by four. Consequently, ϕI = ϕ2 increases from 2 3 ≈ 0.66 to 6 5 = 1.2. successor node (no branching ambiguity): ∀ri ∈ pn : ∃vi−1, vi ∈ V such that I(vi−1, ri) = vi ∧ ∀vj ∈ V [PITH_FULL_IMAGE:figure… view at source ↗
Figure 3
Figure 3. Conceptual difference between ID and OOD: In￾distribution (ID, orange) and Out-of-distribution (OOD, red) in￾ferred facts are shown. All green components are seen during training, including all atomic facts (AF) and some inferred facts (IF). Definition 2.12 (In-distribution vs. Out-of-distribution). In-distribution (ID): An inferred fact derived from com￾binations of atomic facts that were present in the training da… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a) Accuracy on the comparison task for original and grokked GPT2-small on OOD-test set. We can observe that the OOD accuracy remains almost the same when it reaches 100%, but the difference becomes evident afterward. The grokked transformer continues to improve in acc…
Figure 5
Figure 5. Figure 5: Growth of ϕ3,r (y-axis) with |V| (x-axis) for br = 2. The red line is ϕ3,r = [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 4 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Reconciling modern machine-learning practice and the classical bias–variance trade-off

    Belkin, M., Hsu, D., Ma, S., and Mandal, S. Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116 0 (32): 0 15849--15854, 2019. doi:10.1073/pnas.1903070116. URL https://www.pnas.org/doi/abs/10.1073/pnas.1903070116

    doi:10.1073/pnas.1903070116 2019

  3. [3]

    Unifying grokking and double descent

    Davies, X., Langosco, L., and Krueger, D. Unifying grokking and double descent. CoRR, abs/2303.06173, 2023. doi:10.48550/ARXIV.2303.06173. URL https://doi.org/10.48550/arXiv.2303.06173

    arXiv doi:10.48550/arxiv.2303.06173 2023

  4. [4]

    Constructing A Multi -hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

    Ho, X., Duong Nguyen, A.-K., Sugawara, S., and Aizawa, A. Constructing A Multi -hop QA Dataset for Comprehensive Evaluation of Reasoning Steps . In Proceedings of the 28th International Conference on Computational Linguistics , pp.\ 6609--6625, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics. doi:10.18653/v1/2020.coli...

    doi:10.18653/v1/2020.coling-main.580 2020

  5. [5]

    Training Dynamics of Deep Network Linear Regions

    Humayun, A. I., Balestriero, R., and Baraniuk, R. G. Training dynamics of deep network linear regions. CoRR, abs/2310.12977, 2023. doi:10.48550/ARXIV.2310.12977. URL https://doi.org/10.48550/arXiv.2310.12977

    work page Pith review arXiv doi:10.48550/arxiv.2310.12977 2023

  6. [6]

    G., Kim, K., and Lee, K

    Lee, J., Kang, B. G., Kim, K., and Lee, K. M. Grokfast: Accelerated Grokking by Amplifying Slow Gradients , June 2024. URL http://arxiv.org/abs/2405.20233. arXiv:2405.20233 [cs]

    arXiv 2024

  7. [7]

    Joint Knowledge Graph Completion and Question Answering

    Liu, L., Du, B., Xu, J., Xia, Y., and Tong, H. Joint Knowledge Graph Completion and Question Answering . In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '22, pp.\ 1098--1108, New York, NY, USA, August 2022. Association for Computing Machinery. ISBN 978-1-4503-9385-0. doi:10.1145/3534678.3539289. URL https://dl...

    arXiv 2022

  8. [8]

    J., and Tegmark, M

    Liu, Z., Michaud, E. J., and Tegmark, M. Omnigrok: Grokking beyond algorithmic data. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/pdf?id=zDiHoIWa0q1

    work page 2023

Show all 19 references
  1. [9]

    Fixing weight decay regularization in adam

    Loshchilov, I., Hutter, F., et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5, 2017

    2017 arXiv

  2. [10]

    Deep double descent: Where bigger models and more data hurt

    Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. Deep double descent: Where bigger models and more data hurt. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. UR...

    2020

  3. [11]

    Progress measures for grokking via mechanistic interpretability

    Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://...

    2023

  4. [12]

    Multi-scale feature learning dynamics: Insights for double descent

    Pezeshki, M., Mitra, A., Bengio, Y., and Lajoie, G. Multi-scale feature learning dynamics: Insights for double descent. In Chaudhuri, K., Jegelka, S., Song, L., Szepesv \' a ri, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning, ICML 2022, 17-23 ...

    2022

  5. [13]

    v., and Back, T

    Plaat, A., Wong, A., Verberne, S., Broekens, J., Stein, N. v., and Back, T. Reasoning with Large Language Models , a Survey , July 2024. URL http://arxiv.org/abs/2407.11511. arXiv:2407.11511 [cs]

    2024

  6. [14]

    Grokking: Generalization beyond overfitting on small algorithmic datasets

    Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. Grokking: Generalization beyond overfitting on small algorithmic datasets. CoRR, abs/2201.02177, 2022. URL https://arxiv.org/abs/2201.02177

    2022 arXiv

  7. [15]

    Thilak, V., Littwin, E., Zhai, S., Saremi, O., Paiss, R., and Susskind, J. M. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. CoRR, abs/2206.04817, 2022. doi:10.48550/ARXIV.2206.04817. URL https://doi.org/10.48550/arXiv.2206.04817

    2022 doi arXiv

  8. [16]

    MuSiQue : Multihop Questions via Single -hop Question Composition

    Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. MuSiQue : Multihop Questions via Single -hop Question Composition . Transactions of the Association for Computational Linguistics, 10: 0 539--554, May 2022. ISSN 2307-387X. doi:10.1162/tacl_a_00475. URL https://doi....

    2022 doi

  9. [17]

    Explaining grokking through circuit efficiency, September 2023

    Varma, V., Shah, R., Kenton, Z., Kramár, J., and Kumar, R. Explaining grokking through circuit efficiency, September 2023. URL http://arxiv.org/abs/2309.02390. arXiv:2309.02390 [cs]

    2023 arXiv

  10. [18]

    Grokked Transformers are Implicit Reasoners : A Mechanistic Journey to the Edge of Generalization , May 2024

    Wang, B., Yue, X., Su, Y., and Sun, H. Grokked Transformers are Implicit Reasoners : A Mechanistic Journey to the Edge of Generalization , May 2024. URL http://arxiv.org/abs/2405.15071. arXiv:2405.15071 [cs]

    2024 arXiv

  11. [19]

    W., Salakhutdinov, R., and Manning, C

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. HotpotQA : A Dataset for Diverse , Explainable Multi -hop Question Answering , September 2018. URL http://arxiv.org/abs/1809.09600. arXiv:1809.09600 [cs]

    2018 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.