REVIEW 4 cited by
Emergent properties with repeated examples
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We study the performance of transformers as a function of the number of repetitions of training examples with algorithmically generated datasets. On three problems of mathematics: the greatest common divisor, modular multiplication, and matrix eigenvalues, we show that for a fixed number of training steps, models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples. We also demonstrate that two-set training - repeated use of a small random subset of examples, along normal sampling on the rest of the training set - provides for faster learning and better performance. This highlights that the benefits of repetition can outweigh those of data diversity. These datasets and problems provide a controlled setting to shed light on the still poorly understood interplay between generalization and memorization in deep learning.
Forward citations
Cited by 4 Pith papers
-
Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability
Transformers can in-context predict PCG outputs on unseen parameters; required context length scales as sqrt(m), and curriculum training with smaller moduli is necessary for large moduli.
-
Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
Different procedural pretraining tasks create complementary, transferable structures in a transformer's attention and MLP weights, and structures from different tasks can be combined into one initialization.
-
Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks
Trained MLPs and transformers solving modular addition can be unified under an approximate Chinese Remainder Theorem, and deep or embedding-based networks learn only O(log n) frequency features.
-
Procedural Pretraining: Warming Up Language Models with Abstract Data
A short warm-up on procedural data (brackets, sorting, sets) makes language models more accurate and more data-efficient on language, code, and informal math.
Discussion (0). Continue with ORCID to comment.