REVIEW 8 cited by
Parameterized Synthetic Text Generation with SimpleStories
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present SimpleStories, a large synthetic story dataset in simple language, consisting of 2 million samples each in English and Japanese. Through parameterizing prompts at multiple levels of abstraction, we achieve control over story characteristics at scale, inducing syntactic and semantic diversity. Ablations on a newly trained model suite show improved sample efficiency and model interpretability compared to the TinyStories dataset. We open-source all constituent parts of model creation, hoping to enable novel ways to study the end-to-end training process. As a byproduct, we move the frontier regarding the fewest-parameter language model that outputs grammatical natural language.
Forward citations
Cited by 8 Pith papers
-
Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail
Neural scaling occurs because larger models maintain learning on weaker eigenmodes of the eNTK that smaller models cannot access.
-
Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail
Larger neural networks keep learning from ever-weaker spectral directions of the empirical neural tangent kernel, a capacity the authors call spectral reach.
-
Towards Human-Level Book-Writing Capability
A prompt-to-book training framework that derives hierarchical summaries from public-domain novels and inverts them to supervise long-context models toward human literary prose instead of assistant-style output.
-
Towards Human-Level Book-Writing Capability
A supervised fine-tuning approach using inverted multi-resolution planning scaffolds from public-domain novels trains models to generate book-length stories with more human-like literary qualities than standard instru...
-
Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space
LLMs perform in-context learning as trajectories through a structured low-dimensional conceptual belief space, with the structure visible in both behavior and internal representations and causally manipulable via inte...
-
Annotations Mitigate Post-Training Mode Collapse
Annotation-anchored training reduces semantic diversity collapse in post-trained language models by a factor of six compared to standard supervised fine-tuning while preserving instruction-following and improving with scale.
-
Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matching
Mixture-of-experts flow matching enables non-autoregressive language models to achieve autoregressive-level quality in three sampling steps, delivering up to 1000x faster inference than diffusion models.
-
Sessa: Selective State Space Attention
Sessa integrates attention within recurrent paths to achieve power-law memory tails and flexible non-decaying selective retrieval, outperforming baselines on long-context tasks.
Discussion (0). Sign in to comment.