{"id":"e5840dd8-6fcf-4bb4-8d51-692dd26e6cd6","arxiv_id":"2501.00522","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training tiny language models on GPT-4-simplified, low-complexity text improves their scores on simplified GLUE tasks and instruction-following, compared with training on the original larger corpus.","lead":"The authors built a pipeline that uses GPT-4 to rewrite ordinary language training data into a simpler form, using a 2,000-word vocabulary and generic names, then trained tiny language models on these 'leaner' datasets. Their experiments suggest these small models learn faster, but the gains are measured only on benchmarks simplified by the same pipeline, not on original tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central efficiency claim is measured only on LEANER-GLUE, a benchmark generated with the same GPT-4 simplification style as LEANER-Pretrain; without an original-GLUE check, the Table 4 advantage may just reflect training/test distribution matching rather than learning efficiency.","rationale":"The reader's verdict is CONDITIONAL with the same weakest assumption, and I agree. The load-bearing condition for the central claim is that LEANER-GLUE is a valid proxy for original GLUE, because it is the only downstream measure in EXP1 and EXP3. If simplification merely makes the benchmark easier or changes task semantics, then 'LEANER pretraining outperforms Original pretraining' could be true while saying nothing about learning efficiency on real-world language tasks. The label-verification filter in Algorithm 2 adds a specific reason for concern: it does not guard against construct shift; it only enforces GPT-4 label agreement on the simplified text, which may bias the retained set toward easy or style-matched instances. The tokenizer trained only on LEANER data is a second confound, but the decisive missing control is an original-GLUE evaluation. The arithmetic error (29% not 41%) is noted but does not change the qualitative verdict. Since the reader already conditioned acceptance on this missing validation, my stress-test does not move the verdict.","tokens_in":37290,"tokens_out":6461,"duration_ms":66585,"concrete_test":"Primary check: re-run EXP1 with original GLUE as the downstream benchmark. Using the same eight 14M checkpoints from Table 4, fine-tune each on original GLUE training splits with the Section 4.1.2 hyperparameter grid and 5-seed protocol, then evaluate on original GLUE dev sets; train the tokenizer on each model's own pretraining corpus rather than LEANER-only to avoid tokenizer mismatch. If LEANER-pretrained models no longer beat Original-pretrained models on average, the efficiency claim is an artifact of distribution alignment. Secondary: compute the Spearman correlation between each model's LEANER-GLUE and original-GLUE average scores; a low correlation (<0.7) would indicate that LEANER-GLUE is not measuring the same construct.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All central evidence for 'leaner pre-training boosts learning efficiency' comes from Table 4, evaluated on LEANER-GLUE (Section 4.1.3). LEANER-GLUE is constructed with the same GPT-4 rewriting family as LEANER-Pretrain: the same 'preschool children, ≤2,000 words' prompt and world-setting simplification (Sections 3.1.2 and G.2), with Algorithm 2 retaining a revised sample only if GPT-4 predicts the original label. That filter does not validate construct preservation; it selects instances GPT-4 can still label after simplification, potentially removing hard or semantically shifted cases. Consequently, LEANER-Pretrain may simply be closer in distribution to the evaluation benchmark than Original-Pretrain is, so the measured advantage may be style matching rather than general learning efficiency. No experiment evaluates on original GLUE, and no error bars are reported for the normalized averages. Additionally, the headline '41% smaller' is arithmetically wrong: 71M vs 100M is a 29% reduction, weakening the quantitative efficiency claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LEANER, a suite of simplified language datasets for tiny language models: a 71M-token pre-training corpus, a 7M-token instruction-tuning corpus, a simplified version of GLUE (LEANER-GLUE), and a generated instruction-following benchmark (LEANER-Eval). The datasets are produced by prompting GPT-4 to rewrite source texts with a preschool-level vocabulary (≤2,000 words), simplified grammar, and a constrained world setting. The authors train 14M-parameter BERT, LLAMA, XLNET, and MAMBA models on either LEANER-Pretrain or Original-Pretrain, and report that LEANER pre-training yields higher average scores on LEANER-GLUE despite the smaller corpus size. They also compare instruction-following quality under four data recipes using a 14M LLAMA and explore curriculum-learning strategies for pre-training a 1M LLAMA on LEANER-Pretrain. The central claim is that a smaller, simplified pre-training corpus improves learning efficiency and that simplified environments can serve as resource-efficient proxies for larger-scale LM research.","tokens_in":37518,"tokens_out":3764,"duration_ms":38876,"significance":"If the central claim holds, the paper would contribute a useful, publicly released resource for low-resource LM research: the dataset suite, code, and detailed training configurations are all made available, and the architecture-comparison and curriculum-learning experiments are the kind of reproducible study that the community needs. The paper also makes an explicit falsifiable prediction, namely that leaner pre-training outperforms larger original pre-training on downstream linguistic tasks. However, the significance is conditional on the validity of LEANER-GLUE as a benchmark that preserves the construct of the original GLUE tasks. The paper does not report any evaluation on original GLUE, and the benchmark is constructed with the same GPT-4 simplification pipeline as the training data, so the reported efficiency gain may reflect train/test distribution matching rather than general learning efficiency. The extensive experimental documentation and the release of datasets are strengths, but the central evaluation strategy currently prevents the main claim from being accepted as established.","major_comments":[{"comment":"The central claim that leaner pre-training improves learning efficiency is supported only by scores on LEANER-GLUE, a benchmark built with the same GPT-4 simplification pipeline as LEANER-Pretrain: both use the same preschool-level, ≤2,000-word vocabulary prompt and the same world-setting simplification guidelines (Section 3.1.2 and Appendix G). Algorithm 2 retains a revised sample only if GPT-4 can still reproduce the original label after simplification; this filters out samples that GPT-4 cannot label, but it does not verify that the simplified text preserves the original task construct, difficulty, or label semantics. A model pre-trained on simplified text may therefore score higher on LEANER-GLUE simply because it is closer in distribution to the simplified benchmark than Original-Pretrain is. The paper reports no results on original GLUE and no human or external validation of label preservation. I would like to see at least a subset evaluation on the original GLUE tasks using the same fine-tuning protocol, or a human-validated construct-preservation study on a sample of each of the eight LEANER-GLUE tasks; without one of these, the advantage shown in Table 4 is not evidence of general learning efficiency.","section":"Section 3.2 (Algorithm 2) and Section 4.1.3 (Table 4)"},{"comment":"All models, including those pre-trained on Original-Pretrain, use a tokenizer retrained on the LEANER-Training corpus with a vocabulary limited to 2K tokens. This creates a potential confound in the architecture-comparison experiment: the original web, book, textbook, and conversation text is tokenized with a vocabulary built from simplified, preschool-level data, which may inflate tokenization length, increase out-of-vocabulary replacements, and degrade models trained on Original-Pretrain. The paper does not report out-of-vocabulary rates or tokenization statistics on the original corpus. A fair comparison should train two matched-vocabulary tokenizers (one on each corpus), or demonstrate that the LEANER-trained tokenizer does not systematically disadvantage the Original-Pretrain condition.","section":"Section 4.1.1 (Tokenizer section)"},{"comment":"The instruction-following experiment in EXP2 is evaluated by GPT-4 using a grammar/coherence/specificity rubric, on questions that were themselves generated by GPT-4 from LEANER-Instruct examples. The evaluation prompt explicitly asks the grader to be lenient and to reflect relative differences between models, which invites evaluator bias. The paper provides no evidence that the rubric correlates with human judgments, and the reported differences between recipes are small (e.g., +0.6 grammar for LEANER-Ori over Ori-Ori out of 5). A human-rated subset of at least 100 responses with inter-annotator agreement would make these claims testable; without it, the instruction-following results are not robust evidence for the paper's second research question.","section":"Section 4.2 (Table 5) and Appendix G.3"}],"minor_comments":[{"comment":"The claim that LEANER-Pretrain is '41% smaller' is arithmetically incorrect: 71M tokens versus 100M tokens is a 29% reduction, not 41%. This quantitative description should be corrected wherever it appears.","section":"Section 4.1.3 and Abstract"},{"comment":"Table 4 reports averages over five random seeds but gives no standard deviations or confidence intervals; given the small model sizes and the variance-prone WNLI task mentioned in Section 4.1.2, the reader cannot assess whether the observed differences between LEANER and Ori pre-training are significant.","section":"Table 4 and Appendix H.2"},{"comment":"The Limitations section acknowledges that 71M tokens are insufficient for instruction-following and that LLM rewriting introduces bias, but it does not address the more fundamental issue that LEANER-GLUE is built with the same simplification pipeline as LEANER-Pretrain; a brief statement of this limitation and its implications for interpreting Table 4 would be appropriate.","section":"Section 7 (Limitations)"},{"comment":"There are several typographical and formatting errors, including 'preforms' for 'performs' in the contribution list of Section 1, the inconsistent capitalization 'LEANER -glue' in Appendix H, and the unusual spacing in 'L EANER' throughout the tables and text; these should be cleaned up.","section":"Throughout the text"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a strong and potentially useful empirical claim, but the central evidence is currently circular: the evaluation benchmark is constructed with the same GPT-4 simplification procedure as the training corpus, and no original-GLUE check is provided. I recommend requiring either an original-GLUE evaluation (even on a subset) or a human-validated construct-preservation analysis for LEANER-GLUE before the efficiency claim can be accepted. The tokenizer confound in Section 4.1.1 should also be addressed, as it directly affects the fairness of the LEANER-versus-Original comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, the dataset suite is a real contribution: LEANER-Pretrain, LEANER-Instruct, LEANER-GLUE, and LEANER-Eval, with a pipeline that cuts noise, caps vocabulary at 2,000 words, simplifies world references, and preserves genre patterns across diverse sources. Prior tiny datasets like TinyStories and TinyDialogues are single-genre; this one covers web, books, wiki, textbooks, conversation, code, and math, and it is released with code. That is a reusable resource for low-resource LM research. Second, the central claim that leaner pre-training boosts learning efficiency is only shown on LEANER-GLUE, which is generated with the same GPT-4 simplification prompt family as the training data. The stress-test concern lands: the label-verification filter in Algorithm 2 checks that GPT-4 can still predict the original label, but that does not establish construct preservation, and no experiment on original GLUE is reported. So the advantage in Table 4 may be distribution matching rather than general learning efficiency.\n\nThe paper does other things well. The architecture comparison (BERT, LLAMA, XLNET, MAMBA) is controlled for parameter count and tokenizer, and the curriculum experiments with a 1M model are a sensible use of the simplified testbed. The n-gram entropy numbers give a concrete measure of simplification. The limitations section is honest about the instruction-following weakness and LLM bias.\n\nSoft spots are real but not fatal to the dataset value. The arithmetic error is sloppy: 71M is 29% smaller than 100M, not 41%. Table 4 reports five-seed averages without standard deviations, and a few differences are small enough that error bars matter. EXP2's LEANER-Ori beating LEANER-LEANER complicates the story; the paper's explanation is plausible but underdeveloped. The largest issue is the missing original-GLUE check, which the authors could add with modest effort, or they could reframe the claim as efficiency within simplified environments rather than for general LM learning.\n\nThis paper deserves peer review. A serious referee should not desk-reject it; the dataset artifact alone is worth community scrutiny. The revision request should center on external validation or a carefully narrowed claim, plus fixing the arithmetic and reporting variance.\n\nFor my own work, I would cite the dataset suite if I needed a simplified-pretraining testbed, and I would bring this to a reading group to discuss the evaluation circularity. The authors are clearly thinking about the right problems, and the mistakes are fixable.","headline":"Useful simplified-language dataset testbed, but the headline efficiency claim is only demonstrated on a benchmark built with the same GPT-4 simplification pipeline, so the general claim needs validation on original tasks.","tokens_in":38072,"tokens_out":2342,"would_cite":true,"duration_ms":25003,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training on simplified, lower-noise text lets tiny language models beat models trained on 41% more data.","keywords":["simplified language environment","tiny language models","dataset simplification","learning efficiency","curriculum learning","LEANER dataset suite","instruction following","GLUE benchmark simplification"],"falsifier":"Evaluate the same 14M models on the original, unsimplified GLUE benchmark instead of LEANER-GLUE: if the leaner-pretrained model no longer beats the original-pretrained model, the learning-efficiency gain is an artifact of the simplified benchmark rather than of the training data. A cheaper check is to build a version of LEANER-GLUE without the LLM label-verification step (Algorithm 2) and see whether the performance advantage persists.","tokens_in":37078,"feed_emoji":"🧸","tokens_out":4751,"duration_ms":44171,"temperature":0.7,"pith_summary":"The paper claims that language models learn more efficiently when trained on deliberately simplified text: datasets with noise removed, vocabulary capped at roughly 2,000 common words, and complex real-world references replaced with plain equivalents. It introduces a pipeline that rewrites standard pre-training, instruction, and benchmark data with an LLM, producing the LEANER suite: a 71M-token pre-training set, a 7M instruction set, a simplified GLUE benchmark, and an instruction-following evaluation set. The central empirical claim is that tiny models pre-trained on this smaller, simpler corpus outperform the same models pre-trained on the larger original corpus on the simplified GLUE tasks, and also show better grammar, coherence, and answer specificity when following instructions. If true, this would give researchers a cheap testbed where small models and small datasets reproduce the behavior of large-scale training, making architecture and curriculum comparisons affordable.","feed_headline":"Simpler training text beats 41% more data for tiny language models","feed_subtitle":"A 2,000-word-vocabulary corpus lifts tiny models' grammar and task scores while cutting data volume by roughly 40%.","key_machinery":"The load-bearing mechanism is the text-revision pipeline: GPT-4 rewrites each sample to use a 2,000-word vocabulary, simplify grammar and world references (names, places, times), strip ads and formatting noise, and preserve genre-specific structure. A companion label-verification step (Algorithm 2) keeps a simplified GLUE sample only when an LLM predicting the label matches the original label, targeting 70% coverage of each task. The paper frames dataset complexity as information entropy: simplification lowers n-gram entropy, and an appendix argument bounds dataset complexity below by entropy, making entropy reduction the principled target. Curriculum experiments use a 1M-parameter LLAMA proxy and an iterative sample-introduction schedule built from three difficulty metrics (sentence length, self-loss, and a reference LM's loss).","core_discovery":"The paper's central discovery is that 'leaner' pre-training improves the learning efficiency of tiny language models. Every architecture tested at 14M parameters (BERT, XLNet, LLAMA, and Mamba) scores higher on the LEANER-GLUE benchmark when pre-trained on the 71M-token LEANER-Pretrain corpus than on the 100M-token original corpus, even though the leaner corpus is 41% smaller. The same pattern appears for instruction-following: a 14M LLAMA pre-trained and instruction-tuned on leaner data beats the original-data recipe on grammar, coherence, and specificity. The paper also reports that an iterative curriculum driven by an external LM's loss reaches peak downstream performance roughly 20% earlier in training steps and with roughly 20% less data than training on all data in random order.","pith_inferences":["The paper's central comparison is evaluated only on the simplified benchmark, which was built with the same GPT-4 style of simplification as the training data; a test on the original, unsimplified GLUE tasks would separate genuine learning-efficiency gains from artifacts of train-test distribution match.","The improvement could come from dataset distribution shift rather than simplification per se; comparing against a randomly sampled 71M-token subset of the original corpus would distinguish 'less data' from 'simpler data'.","The label-verification step in LEANER-GLUE curation may select systematically easier samples; ablating that step (retaining simplified samples regardless of LLM label agreement) would show whether the benchmark's difficulty or its construction drives the reported gains."],"forward_implications":["If the claim holds, simpler and cleaner data is a viable route to smaller models: effective data volume for basic linguistic competence can be cut by roughly 40% without hurting downstream scores.","The LEANER suite provides a low-cost proxy environment for comparing model architectures and training techniques at the 1M-14M parameter scale, and the authors argue the resulting rankings may transfer to larger models.","The curriculum result implies pre-training schedules can save about 20% of steps and 20% of data by introducing samples in an order derived from a reference model's loss.","A simplified benchmark like LEANER-GLUE opens the door to zero-shot evaluation of tiny models within the same vocabulary and knowledge scope used for training."],"supporting_citations":[{"why":"Defines the GLUE benchmark that LEANER-GLUE simplifies, supplying the task construct and labels the label-verification step must preserve.","marker":"(Wang et al., 2019)"},{"why":"GPT-4 is the revision engine that rewrites all LEANER training, instruction, and evaluation texts.","marker":"(Achiam et al., 2023)"},{"why":"A pretrainer's guide that determines the source composition of the pre-training corpus (web, books, wiki, textbook, conversation, code, math).","marker":"(Longpre et al., 2024)"},{"why":"TinyStories is the prior tiny-dataset baseline whose single-theme limitation the paper positions LEANER-Pretrain against.","marker":"(Eldan & Li, 2023)"},{"why":"BERT is one of the four 14M architectures compared on LEANER-GLUE.","marker":"(Devlin et al., 2019)"},{"why":"LLAMA is the transformer-decoder architecture used in EXP2 and EXP3, including the 1M proxy models.","marker":"(Touvron et al., 2023)"},{"why":"Mamba is the state-space-model architecture whose learning efficiency is compared against transformers.","marker":"(Gu & Dao, 2024a)"},{"why":"SimCSE provides the sentence-embedding similarity used to deduplicate LEANER-Eval questions and to partition them by similarity to training data.","marker":"(Gao et al., 2022)"}],"fun_headline_variants":["71M cleaner tokens beat 100M for tiny LMs","Simplify text, boost tiny LM: 41% less data wins","Tiny LMs learn faster on lean, 2000-word corpora","Cleaner data cuts tiny LM training steps by 20%","Small corpus, big results: tiny LMs outperform with less"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4-simplified GLUE benchmark preserves the original tasks' construct and labels well enough that higher scores on the simplified version reflect genuine gains in linguistic ability, rather than an artifact of training and testing on similarly simplified data.","fun_headline_variants_meta":{"raw":{"variants":["71M cleaner tokens beat 100M for tiny LMs","Simplify text, boost tiny LM: 41% less data wins","Tiny LMs learn faster on lean, 2000-word corpora","Cleaner data cuts tiny LM training steps by 20%","Small corpus, big results: tiny LMs outperform with less"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1461,"prompt_tokens":1045,"completion_tokens":416,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":326}},"tokens_in":661,"tokens_out":416,"duration_ms":5194,"temperature":1.0,"reasoning_tokens":326,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:47:51.716222+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the same 14M models on the original, unsimplified GLUE benchmark instead of LEANER-GLUE: if the leaner-pretrained model no longer beats the original-pretrained model, the learning-efficiency gain is an artifact of the simplified benchmark rather than of the training data. A cheaper check is to build a version of LEANER-GLUE without the LLM label-verification step (Algorithm 2) and see whether the performance advantage persists.","supporting_citations":[],"review_version":1}