Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers

Devin White; Joshua Barron

REVIEW 3 major objections 5 minor 14 references

Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pre-training capacity alone decides whether a transformer learns rules or memorizes facts, and mixing tasks suppresses extrapolation at every size tested.

desk verdict The isolated capacity sweep is a clean, useful result, but the joint-training conclusion is confounded and the 'intrinsic' trade-off is overstated. read the letter →

arxiv 2506.09099 v2 pith:QSOT6O3W submitted 2025-06-10 cs.LG cs.AIcs.CL

Joshua Barron , Devin White This is my paper

classification cs.LGcs.AIcs.CL

keywords memorizationgeneralizationmodelcapacityarithmeticextrapolationfactualrecallsyntheticpre-trainingTransformergrokking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that model capacity alone, independent of regularization, decides whether a pre-trained Transformer learns general rules or memorizes training examples. Training character-level models from scratch on two synthetic tasks, it finds that the smallest model extrapolates to withheld arithmetic cases like 5+7 but cannot recall capital-city facts, while larger models recall every fact but fail every withheld arithmetic case. When the two task types are mixed in a joint pre-training run, no tested model, from 1.46k parameters to 10.63M, extrapolates to the withheld cases. The authors interpret this as an intrinsic capacity-driven trade-off with consequences for small language models and hallucination mitigation, while cautioning that the small model's success is narrow extrapolation within the 0–9 digit range, not general arithmetic.

What carries the argument

The central object is a capacity sweep of Transformer models indexed by embedding size, from a minimal single-layer, single-head network with n=14 to a 10.63M-parameter GPT-style multi-layer transformer. The capacity axis is the independent variable: all models share the same optimizer settings, and a controlled regularization condition (dropout 0.0, weight decay 0.1) is used to rule out regularization differences. The two synthetic character-level tasks supply the dependent variables: arithmetic expressions of the form <a+b=c> and <a-b=c> with all pairs involving 5 and 7 withheld from training, and 50 capital-city statements that are identical in training and validation. The proposed mechanism is an implicit simplicity prior: a model too small to store all input-output mappings is forced toward compressible rules, while a model large enough to store direct mappings converges to memorization. The paper's Section 5.1 check, feeding n14 digits 0–19, is what turns generalization into the narrower claim of extrapolation within the training domain.

What would settle it

Retrain the n14 and n28 models on arithmetic only but with sequences padded to block size 24 and evaluated in the same format used in the Combined Test; if n14 still extrapolates to all 40 withheld (5,7) cases at that block size, the joint-task result depends on the task mix, while if n14 loses extrapolation with padding alone, the paper's joint-training interpretation collapses.

Watch

Extended reading notes

Core claim

In the paper's own terms, pre-training exhibits a capacity-dependent trade-off: smaller models are able to learn and extrapolate the underlying arithmetic rules but fail to memorize factual content, while larger models excel at factual recall but struggle with extrapolation, and under joint training no model achieves both. The evidence is a sweep of Transformer sizes, a 1.46k-parameter single-layer net (n14), 5.26k (n28), 19.94k (n56), and a 10.63M six-layer GPT-style model, trained on character-level addition and subtraction with the (5,7) families withheld and on 50 capital-city statements. n14 scores 40/40 on the withheld arithmetic cases but only 8.2% on facts; n28 and above score 100% on facts but 0/40 on withheld arithmetic. In the combined setting, n14 collapses on both tasks and every larger model memorizes the in-distribution tasks while scoring 0/40 on (5,7). The authors label n14's arithmetic success narrow extrapolation because the model scores 0% on addition with digits 0–19, and they report the same pattern under a controlled regularization condition (dropout 0.0, weight decay 0.1).

Load-bearing premise

The combined test pads arithmetic sequences to block size 24 and mixes them with capital-city sequences in the same batch, and the paper assumes this setup is a fair test of joint learning; if the padding or the heterogeneous mixing disrupts arithmetic optimization by itself, the joint-training conclusion does not follow.

Editorial extensions

If this is right

If capacity alone drives the trade-off, shrinking a model is not just a compute saving; it is a substantive inductive-bias choice that favors rule extraction over fact storage.
Mixed-task pre-training that combines factual and reasoning data may systematically suppress extrapolation at every scale, so curricula or architectural separation would be needed to get both behaviors.
The n14 result, being narrow extrapolation confined to digits 0–9, does not extend to multi-digit arithmetic; the paper itself says so.
For small language models, the trade-off implies they may be better suited to reasoning-constrained deployment, such as robotics or embedded systems, and worse at knowledge-dense recall tasks.
Hallucination mitigation that pushes harder on factual recall could, on this account, actively reduce a model's ability to generalize.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next test is to separate the two confounds in the Combined Test: train arithmetic-only at block size 24 and train the two tasks in alternating batches, to check whether padding or sequence mixing alone erases extrapolation.
The capacity threshold for memorization, sitting between n14 and n28, suggests an external-memory or retrieval-augmented design could let a small rule-learning model defer fact storage to a separate store, a modular split the paper floats only as a future direction.
The trade-off may depend on the optimizer's preference for easy low-loss solutions; scheduling or curriculum interventions that let the arithmetic solution emerge before facts are introduced might change which mode wins, which the paper's fixed-schedule experiments do not test.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: there is a real, reproducible-looking result here in the isolated arithmetic and capital-cities tasks, and the authors deserve credit for running a controlled-regularization check and for including a limitations section that quietly walks back the strength of the 'generalization' claim. But the paper's most cited line—'no model achieves both'—is built on a comparison that changes three things at once, so it doesn't hold up as stated.

What's new: the capacity sweep from n14 to n56 with withheld (5,7) arithmetic is a clean demonstration that tiny models can extrapolate to unseen combinations while larger models memorize but fail on those same cases. The joint-training result (all models drop to 0/40 on held-out arithmetic) is a fresh data point, and the grokking check (1.5M iterations for MLT, still 0/40) is a useful negative result. The controlled regularization appendix is a good-faith check that the capacity effect isn't just dropout/weight-decay differences. The citation pattern is fine—grokking, memorization-without-overfitting, and the SFT/RL contrast are all there.

Where it goes soft. The Combined Test changes block size (9 to 24), batch size (882 to 576), and adds padding and task mixing in one step. There is no control with arithmetic-only at block size 24/batch size 576, so the loss of extrapolation cannot be attributed to the addition of the capitals task. The stress-test note is exactly right. The abstract says the trade-off is 'intrinsic'; the paper's own Section 5.1 shows n14 gets 0% on 0-19 arithmetic, so the extrapolation is shallow, not robust generalization. That limitation is honestly stated, but it undercuts the 'intrinsic' phrasing. Also, all numbers are single-seed deterministic runs with best-of-training/best-of-10 selection and no error bars. For a workshop paper that's a minor problem; for a claim stated as a consistent law, it needs multiple seeds.

Bottom line: the isolated capacity effect is worth engaging with and probably real. The joint-training conclusion needs a matched control before it supports the paper's central claim. The paper is honest enough to include the limitations that weaken its own strongest statements. I'd send it to review, but the referee should require the matched control and multiple seeds.

Referee Report

3 major / 5 minor

Summary. The paper pre-trains small character-level Transformers on two synthetic tasks: arithmetic (addition/subtraction with (5,7) combinations held out) and capital-city factual recall. The authors report that the smallest model (n14) extrapolates to the held-out arithmetic cases but fails at factual recall, while larger models (n28, n56, an MLT) memorize facts but score 0/40 on held-out arithmetic. In a joint-training condition with arithmetic and capital-city data mixed, the paper reports that no model generalizes to the held-out arithmetic cases, leading to the claim that pre-training exhibits an intrinsic capacity-dependent trade-off between memorization and generalization. The paper also includes a controlled-regularization replication and an extended-training appendix on grokking.

Significance. If the results are correct, the paper offers a clean, controlled demonstration that model capacity alone (independent of explicit regularization) can bias Transformers toward either rule extrapolation or memorization, and that joint training on both types of tasks suppresses extrapolation across all tested sizes. The use of genuinely held-out arithmetic combinations and the separate controlled-regularization condition are strengths, and the stark 40/40 versus 0/40 contrast is memorable. However, the central joint-training claim rests on a comparison that changes multiple experimental variables at once, and the single-seed, best-of-training evaluation protocol limits the reliability of the quantitative claims. The capacity-dependent trend in isolated arithmetic and capital-city tasks is more credible than the joint-training 'incompatibility' conclusion, but both would be substantially strengthened by multi-seed runs and properly matched controls.

major comments (3)

[§3.1, Tables 4 and 7] The Combined Test condition changes three variables at once relative to the arithmetic-only baseline: block size (9 to 24), batch size (882 to 576), and the addition of capital-city sequences, while also padding each 9-character arithmetic line to block size 24. No control is reported with arithmetic-only at block size 24 and batch size 576, nor with capital-city-only under the combined data layout. Consequently, the collapse of held-out (5,7) extrapolation in Tables 4 and 7 and Figure 3 cannot be attributed to task mixing; it could equally be caused by longer context, padding tokens, altered sequence-to-token ratio, or the new batch size. This is a load-bearing gap because the abstract's 'no model achieves both' claim and the Discussion's 'incompatibility' conclusion depend directly on this comparison. The authors should run arithmetic-only at block size 24/batch size 576 (with padding) and report the (5,7) accuracy, and ideally also a capitals-only condition under the same layout, before claiming that joint training suppresses extrapolation.
[§3.3, Tables 2–7] The evaluation protocol reports the 'best combined evaluation score across training' and evaluates each test input 10 times, taking the best result. This is a maximum over checkpoints and over samples, not a representative performance statistic. For the success claims (n14 40/40, capital-city percentages such as 8.2% and 12.4%), this selection can only inflate the reported accuracy; for the failure claims (0/40 for larger models), the selection is conservative, so those particular numbers are robust. The paper should report mean/standard deviation over checkpoints or, preferably, multiple seeds, and should state the checkpoint or sampling rule used for each reported number. Without this, the quantitative contrast between n14 and larger models is difficult to interpret.
[§3.2] The smallest model n14 was not chosen independently of the outcome: the authors state that they 'incrementally increased n until the model successfully learned a basic arithmetic task.' This selects on the dependent variable. With only one model at the 'small' end of the capacity range and a deterministic single seed, the claim that limited capacity intrinsically biases toward rule extraction rests on a single run of a model selected for that behavior. Multiple random restarts or a sweep over nearby sizes (e.g., n12–n20) would be needed to establish that the extrapolation capability is a stable property of small capacity rather than a lucky initialization or a specific architectural coincidence.

minor comments (5)

[§4.2] There is a typo: 'a increase in parameter count' should be 'an increase in parameter count.'
[§4.3 and Appendix A.3] The text says 'Controlled Regularization did not alter this trend' but the controlled-regularization Tables 5–7 change both dropout and weight decay relative to the original n-models (which had dropout 0.0 and weight decay 0). This is a legitimate control, but the paper should state explicitly that the 'original' and 'controlled' conditions differ in these hyperparameters rather than implying they are identical except for regularization.
[Table 4] The column 'Combined' is not defined in the table caption or in the text; please specify whether it is an average of the three preceding accuracy columns or a separate metric.
[§5.1 and Limitations] The paper uses 'generalization' throughout the abstract and introduction, but §5.1 and the Limitations section correctly clarify that the n14 behavior is 'narrow extrapolation,' not robust generalization. Please align the terminology consistently so that the claims match the evidence, especially in the Discussion and Conclusion.
[References and figures] Minor formatting issues: some references have inconsistent spacing or punctuation (e.g., 'V y .' and '1st ed. 2024 Edition'), and Figure 7's log-scaled x-axis is not mentioned in the caption. Please proofread for typos such as 'experiemnt' in §4.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the trade-off is measured on independent held-out arithmetic cases and a fixed factual recall set, and no fitted parameter or self-citation is reused to define the result.

full rationale

The paper's central empirical claims are derived from two synthetic tasks whose target quantities are external to the training objective. Arithmetic extrapolation is measured on expressions involving both 5 and 7 that are explicitly withheld from training and validation, so the reported 40/40 success of n14 and 0/40 failures of larger models are genuine held-out measurements rather than quantities reconstructed from the training data. Capital-city memorization is measured on a fixed set of 50 factual statements, and the paper openly states that the same examples are used for training and validation, which is appropriate for a pure recall test. No model parameter or evaluation score is fitted to the held-out (5,7) answers and then renamed as a prediction; the reported accuracies are direct evaluation results. The paper contains no load-bearing self-citation: its references to nanoGPT, simplicity priors, and prior memorization studies are background support, not premises that already contain the capacity-dependent conclusion. The most serious methodological weakness is not circularity but experimental confounding in the Combined Test, where block size and batch size change simultaneously with the addition of the second task, weakening attribution of the joint-training effect. That is a validity threat, not a circular reduction. Consequently, the derivation chain is self-contained with respect to its measurements, and no circular step can be exhibited with a specific equation or definitional equivalence.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities or fitted constants; its central result rests on the domain assumption that the synthetic tasks isolate generalization and memorization, on the faithfulness of the nanoGPT implementation, and on the paper-specific choice to report best-of-training/best-of-10 accuracy.

free parameters (1)

Embedding size n=14 (smallest viable model) = 14
Chosen by incrementing n until the model learned basic arithmetic; defines the 'small capacity' endpoint of the trade-off sweep.

assumptions (3)

domain assumption Synthetic character-level tasks cleanly isolate generalization (arithmetic extrapolation) from memorization (factual recall)
The entire interpretation depends on the two tasks being valid and separable probes; the paper does not validate this assumption against other tasks.
standard math The nanoGPT implementation and standard cross-entropy training faithfully represent Transformer pre-training dynamics
The paper builds directly on Karpathy's nanoGPT with minor modifications; no formal verification is provided.
ad hoc to paper Logging the best evaluation score across training and best of 10 samples per input is a representative performance statistic
Section 3.3 states this selection rule without justification; it can systematically inflate reported accuracies.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers." pith.science (2026). https://pith.science/paper/QSOT6O3W

@misc{pith2026250609099,
  author       = {Pith},
  title        = {Pith review of: Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QSOT6O3W}},
  note         = {Machine review of arXiv:2506.09099}
}

read the original abstract

The relationship between memorization and generalization in large language models (LLMs) remains an open area of research, with growing evidence that the two are deeply intertwined. In this work, we investigate this relationship by pre-training a series of capacity-limited Transformer models from scratch on two synthetic character-level tasks designed to separately probe generalization (via arithmetic extrapolation) and memorization (via factual recall). We observe a consistent trade-off: small models extrapolate to unseen arithmetic cases but fail to memorize facts, while larger models memorize but fail to extrapolate. An intermediate-capacity model exhibits a similar shift toward memorization. When trained on both tasks jointly, no model (regardless of size) succeeds at extrapolation. These findings suggest that pre-training may intrinsically favor one learning mode over the other. By isolating these dynamics in a controlled setting, our study offers insight into how model capacity shapes learning behavior and offers broader implications for the design and deployment of small language models.

Figures

Figures reproduced from arXiv: 2506.09099 by the authors.

**Figure 1.** Addition and subtraction accuracy over the math evaluation, which includes the four withheld (5, 7)/(7, 5) cases. Red dots indicate where both curves hit 100%, signifying genuine extrapolation to the unseen pairs. Only n14 ever earns these red markers; larger models plateau below perfect overall accuracy due to being unable to extrapolate to the withheld combinations. age generalization over memorization. 4.2. Fac… view at source ↗

**Figure 3.** Training Dynamics on Joint Task (Arithmetic + Factual): Accuracy over 30k iterations for models trained on both tasks. No model generalizes to held-out (5,7) arithmetic cases. The n14 model regresses relative to its arithmetic-only performance, and all larger models prioritize memorization. This reveals the incompatibility of generalization and memorization under simultaneous training conditions. Plotting the accur… view at source ↗

**Figure 4.** Arithmetic extrapolation with controlled regularization: only n14 generalizes to unseen (5,7); larger models plateau at 0%. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: Factual recall under controlled regularization: n28 and above achieve perfect scores; n14 remains below. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Joint performance with controlled regularization: no model extrapolates arithmetic; larger models still memorize facts. B. Grokking Beyond the core training stages, we explored whether extended training might promote generalization, a phenomenon known as grokking (Powe…

**Figure 7.** Figure 7: illustrates this: while in-distribution accuracy rapidly saturates and remains high, extrapolation performance stays flat at zero [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 2 canonical work pages

[1]

L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

arXiv
[4]

9, 2024); accessed May 19,

GitHub repository; commit 93a43d9(Dec. 9, 2024); accessed May 19,

work page 2024
[6]

doi: 10.1016/j.neunet.2024.106550

ISSN 0893-6080. doi: 10.1016/j.neunet.2024.106550. URL http://dx. doi.org/10.1016/j.neunet.2024.106550. Nezhurina, M., Cipolina-Kun, L., Cherti, M., and Jitsev, J. Alice in wonderland: Simple tasks showing complete rea- soning breakdown in state-of-the-art large language mod- els,

arXiv 2024
[7]

Pareja, A., Nayak, N

URL https: //arxiv.org/abs/2203.02155. Pareja, A., Nayak, N. S., Wang, H., Killamsetty, K., Su- dalairaj, S., Zhao, W., Han, S., Bhandwaldar, A., Xu, G., Xu, K., Han, L., Inglis, L., and Srivastava, A. Unveil- ing the secret recipe: A guide for supervised fine-tuning small llms,

arXiv
[8]

Power, A., Burda, Y ., Edwards, H., Babuschkin, I., and Misra, V

URL https://arxiv.org/abs/ 2412.13337. Power, A., Burda, Y ., Edwards, H., Babuschkin, I., and Misra, V . Grokking: Generalization beyond overfitting on small algorithmic datasets,

arXiv
[9]

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al

URL https:// arxiv.org/abs/2201.02177. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9,

arXiv
[11]

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A

URLhttps://arxiv.org/abs/2205.10770. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. At- tention is all you need.Advances in neural information processing systems, 30,

arXiv
[12]

Wang, X., Antoniades, A., Elazar, Y ., Amayuelas, A., Al- balak, A., Zhang, K., and Wang, W

URL https: //arxiv.org/abs/2405.15071. Wang, X., Antoniades, A., Elazar, Y ., Amayuelas, A., Al- balak, A., Zhang, K., and Wang, W. Y . Generalization v.s. memorization: Tracing language models’ capabil- ities back to pretraining data, 2025a. URL https: //arxiv.org/abs/2407.14985. 7 Too Big to Think Wang, Y ., Yang, Q., Zeng, Z., Ren, L., Liu, L., Peng, B...

arXiv

Show all 14 references

[13]

Controlled Regularization

URL https:// arxiv.org/abs/2310.06714. 8 Too Big to Think A. Controlled Regularization Experiments To verify that the observed results were not artifacts of differing regularization settings, we re-ran all experiments under “Controlled Regularization” (dropout = 0.0,weight dec...

arXiv
[14]

Goldilocks zone

Extended training of the MLT model (10.63M parameters) on the arithmetic-only task for 1.5M iterations. X-axis is log-scaled. In-distribution addition and subtraction accuracies plateau near 98%, while held-out (5,7) generalization remains at 0%, indicating persistent memoriza...
[2019]

M., Hauth, A., Millican, K., et al

Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

arXiv
[2022]

Maltoni, D

URL https://arxiv.org/abs/2205.10343. Maltoni, D. and Ferrara, M. Arithmetic with language models: From memorization to computation.Neural Net- works, 179:106550, November

arXiv
[2024]

2024 Edition

1st ed. 2024 Edition. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

2024 arXiv
[2025]

org/abs/2501.17161

URL https://arxiv. org/abs/2501.17161. Friedland, G.Information-Driven Machine Learning: Data Science as an Engineering Discipline. Springer, 1st edi- tion,

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

arXiv

[2] [4]

9, 2024); accessed May 19,

GitHub repository; commit 93a43d9(Dec. 9, 2024); accessed May 19,

work page 2024

[3] [6]

doi: 10.1016/j.neunet.2024.106550

ISSN 0893-6080. doi: 10.1016/j.neunet.2024.106550. URL http://dx. doi.org/10.1016/j.neunet.2024.106550. Nezhurina, M., Cipolina-Kun, L., Cherti, M., and Jitsev, J. Alice in wonderland: Simple tasks showing complete rea- soning breakdown in state-of-the-art large language mod- els,

arXiv 2024

[4] [7]

Pareja, A., Nayak, N

URL https: //arxiv.org/abs/2203.02155. Pareja, A., Nayak, N. S., Wang, H., Killamsetty, K., Su- dalairaj, S., Zhao, W., Han, S., Bhandwaldar, A., Xu, G., Xu, K., Han, L., Inglis, L., and Srivastava, A. Unveil- ing the secret recipe: A guide for supervised fine-tuning small llms,

arXiv

[5] [8]

Power, A., Burda, Y ., Edwards, H., Babuschkin, I., and Misra, V

URL https://arxiv.org/abs/ 2412.13337. Power, A., Burda, Y ., Edwards, H., Babuschkin, I., and Misra, V . Grokking: Generalization beyond overfitting on small algorithmic datasets,

arXiv

[6] [9]

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al

URL https:// arxiv.org/abs/2201.02177. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9,

arXiv

[7] [11]

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A

URLhttps://arxiv.org/abs/2205.10770. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. At- tention is all you need.Advances in neural information processing systems, 30,

arXiv

[8] [12]

Wang, X., Antoniades, A., Elazar, Y ., Amayuelas, A., Al- balak, A., Zhang, K., and Wang, W

URL https: //arxiv.org/abs/2405.15071. Wang, X., Antoniades, A., Elazar, Y ., Amayuelas, A., Al- balak, A., Zhang, K., and Wang, W. Y . Generalization v.s. memorization: Tracing language models’ capabil- ities back to pretraining data, 2025a. URL https: //arxiv.org/abs/2407.14985. 7 Too Big to Think Wang, Y ., Yang, Q., Zeng, Z., Ren, L., Liu, L., Peng, B...

arXiv