{"id":"e006e628-bead-49df-93f9-0f0e2ae235c5","arxiv_id":"2508.19873","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Training a tiny BERT first on Simple Wikipedia and then on everyday Wikipedia improves perplexity, while difficulty heuristics such as sentence length, word rarity, and Flesch score give no consistent gain over random order.","lead":"Researchers trained a small language model on simple English first, then everyday English, and found the model predicted text better than with random order. Classic difficulty measures such as sentence length or word rarity gave no consistent benefit, suggesting human-curated simplicity is a more useful curriculum signal.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Update-count confound: Sequential trains ~17% longer than Baseline SL+EL; no update-matched control is reported.","rationale":"The reader identified the label-to-difficulty assumption as the weakest point and also mentioned the update-count confound in the rationale. I agree both are real concerns, but the update-count confound is more load-bearing because it directly challenges the existence of the reported curriculum benefit, not just its interpretation. If an update-matched control eliminates the advantage, the central claim fails regardless of label validity. The proposed test is cheap and decisive. The reader's verdict is already CONDITIONAL, and my analysis does not move it to ACCEPT or REJECT: the paper's evidence is suggestive but incomplete, so CONDITIONAL remains appropriate. Hence UNCHANGED.","tokens_in":6712,"tokens_out":3746,"duration_ms":44978,"concrete_test":"Train Baseline SL+EL (random ordering) with an update budget matched to Sequential's 781,333 updates (e.g., cap training at that step count and evaluate), and conversely train Sequential with early stopping at Baseline SL+EL's 665,333 updates. If the perplexity gaps (overall and SL) collapse to non-significance under either matching, the curriculum ordering effect is confounded with training duration. Report the resulting p-values and effect sizes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that SL-first curriculum ordering improves perplexity—rests on comparing Sequential (65.31 overall, 57.83 SL) to Baseline SL+EL (69.61 overall, 64.15 SL), with p=.019 and p=.001. But Table 3 shows Sequential uses 781,333±122,292 updates versus Baseline SL+EL's 665,333±102,111, a ~17% increase. Training to convergence with early stopping means more updates typically lowers final loss/perplexity; the reported significance tests compare only final perplexities, not perplexity at matched update counts. The paper acknowledges 'Both incremental and sequential strategies require more updates than Baseline SL+EL to reach these improvements' but does not control for this. Incremental also uses 781k updates yet fails to improve overall perplexity (p=.598), which weakens a pure 'more compute' explanation, but the schedule-update interaction remains unexamined. Without an update-matched random baseline or an analysis that isolates ordering from training duration, the claim that 'structuring it via a curriculum...consistently improves perplexity' is not yet established. This is more load-bearing than the label-noise assumption because it threatens the empirical effect itself, not just the interpretive label-to-difficulty step.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether editorially curated simple language (Simple Wikipedia, SL) provides a useful signal for curriculum learning (CL) in masked language model pre-training. Using BERT-tiny and the Simple Wikipedia dataset, the authors compare label-based curricula (sequential SL→EL, incremental SL→SL+EL, anti-sequential EL→SL) with competence-based curricula driven by sentence length, word rarity, and Flesch Reading Ease, plus random baselines. Each condition is run over 15 seeds with one-sided significance tests and Holm–Bonferroni adjustment. The main reported result is that the sequential curriculum significantly improves overall and SL perplexity over a random SL+EL baseline, while the anti-sequential order does not, and none of the competence-based curricula yields significant gains. The paper concludes that human intuition about linguistic difficulty can guide CL, whereas shallow heuristics cannot.","tokens_in":7053,"tokens_out":6290,"duration_ms":74747,"significance":"Strengths: the study is comparatively clean: public dataset, 15 seeds, explicit significance testing, an anti-sequential control, and a negative-result arm for heuristics. If the main comparison were fully controlled, the paper would be a useful empirical datapoint for the curriculum-learning debate and for the role of simplified language in pretraining. The significance is limited by two issues: the sequential condition trains for substantially more updates than its baseline, and the SL/EL labels are article-level proxies for sentence-level difficulty. Both concerns are addressable, but they currently affect the central claim rather than a peripheral detail.","major_comments":[{"comment":"The central comparison—Sequential vs. Baseline SL+EL—does not isolate ordering from training duration. Sequential uses 781,333±122,292 updates vs. 665,333±102,111 for Baseline SL+EL, about 17% more. The reported p-values compare final perplexities after different numbers of update steps; longer training generally moves final perplexity in the same direction as the claimed effect. The fact that Incremental trains for the same 781k updates yet fails to improve overall perplexity weakens a pure 'more compute' account, but it does not resolve the schedule–update interaction. Please provide an update-matched control (e.g., a random SL+EL model trained for the same number of updates as Sequential) or report perplexity at matched update counts, and also report the early-stopping protocol.","section":"Sec. 4, Table 3"},{"comment":"The paper's interpretive conclusion—that 'human intuition about linguistic difficulty' drives the gain—rests on the assumption that article-level SL/EL labels reflect sentence-level difficulty. The authors acknowledge that inherited labels introduce noise, but the issue is more than noise: SL vs. EL may encode register, topic, or article identity rather than perceived simplicity. The empirical ordering effect (SL first helps) remains of interest, but it does not by itself establish the difficulty-based mechanism. I recommend either validating the proxy (e.g., human or readability annotation of a sample) or narrowing the conclusion to Simple-Wikipedia-first training.","section":"Sec. 3, Dataset"}],"minor_comments":[{"comment":"'no improvement in neither overall nor SL perplexity' should read 'no improvement in either overall or SL perplexity'.","section":"Sec. 4"},{"comment":"The competence-function hyperparameters T=50,000 and c0=0.05 were chosen after observing that shorter competence phases yield better results. Please describe the selection procedure (validation set? grid search?) and report sensitivity; this tuning is not part of the central comparison, but it affects the negative arm.","section":"Appendix A"},{"comment":"The negative statements about competence-based curricula are based on non-significant differences with n=15 seeds. This supports 'no consistent gain observed', but phrases like 'no benefit' or 'fail to effectively separate' should be accompanied by effect sizes or an equivalence/power analysis, or restricted to 'we did not detect an improvement'.","section":"Sec. 4 / Table 5"},{"comment":"Please specify the train/validation/test split and whether the reported perplexities are on a held-out test set. This also helps rule out test contamination in the SL-first condition.","section":"Sec. 3, Training Setup"},{"comment":"The histograms use raw counts, but SL and EL have different numbers of sentences. Density plots would make the overlap of the distributions easier to assess.","section":"Figure 1"},{"comment":"Consider reporting effect sizes (e.g., Cohen's d or matched difference) for the significant comparisons; with 15 seeds, p-values alone do not convey the magnitude of the improvement.","section":"Sec. 4 / Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially publishable contribution. The update-count confound in the headline result is the main technical obstacle; if the authors can provide an update-matched control or an analysis that separates ordering from duration, the central claim would be substantially stronger. The label-proxy issue is also important because it bears on the paper's title and interpretation. The manuscript is within the journal's scope and the experimental design otherwise shows good statistical hygiene."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know one thing up front: this is a careful, small-scale empirical paper, and the core result is plausible. The authors show that training a BERT-tiny MLM on Simple Wikipedia first and then Everyday Wikipedia (sequential) significantly beats a random SL+EL mix on overall and SL perplexity, while competence-based heuristics (length, word rarity, FRE) give nothing. The new bit is using article-level SL/EL labels as a global curriculum signal and directly comparing them against those heuristics. That comparison is the paper's contribution, and it is a new application rather than a new framework.\n\nWhat the paper does well: 15 seeds, proper multiple-comparison correction, an anti-sequential control that makes the ordering effect look real, and an honest negative arm for competence-based curricula. The distribution plots showing that the heuristics don't separate SL from EL are a nice diagnostic. The incremental strategy (SL then SL+EL) uses the same number of updates as sequential but does not improve overall perplexity, which weakens a pure \"more compute\" explanation.\n\nWhere the soft spots are. First, the update-count confound is real but not fatal. Sequential runs ~781k updates versus ~665k for Baseline SL+EL, a 17% gap, and the significance tests only compare final perplexities. An update-matched random control would make the ordering claim airtight. The incremental result helps, but the schedule-update interaction remains unexamined. Second, the paper's interpretive step — that human intuition about linguistic difficulty is what drives the gains — relies on article-level labels being a valid proxy for sentence-level difficulty. The paper acknowledges the label noise, but if the labels mostly encode topic or register rather than perceived simplicity, the \"human intuition\" framing doesn't follow, even if the ordering effect itself persists. That is a conceptual soft spot, not a fatal one.\n\nOverall, I think the central empirical claim mostly holds for this specific setup: BERT-tiny, MLM pretraining, this corpus. The paper is honest about scope and about what it cannot show. It is not a framework paper and not a large-model demonstration, but it is a legitimate data point for curriculum learning and pretraining. I'd send it to a serious referee, with the expectation that the authors address the update-matched baseline and dig into label noise.\n\nFor me: I'd bring it to a reading group if we were discussing curriculum learning or data selection. I don't work directly in this space, so I probably wouldn't cite it in the next year, but I'd point people to it as a fair, reproducible negative-positive result pairing.","headline":"A clean, small-scale empirical result — SL-first ordering helps BERT-tiny MLM perplexity — but the update-count confound and the label-to-difficulty assumption keep it from being conclusive.","tokens_in":7437,"tokens_out":1785,"would_cite":false,"duration_ms":21817,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a language model on Simple Wikipedia before everyday text consistently lowers perplexity, especially on simple-language text.","keywords":["curriculum learning","language model pre-training","Simple Wikipedia","perplexity","difficulty heuristics","BERT-tiny","data ordering","readability"],"falsifier":"Re-run the Sequential and Baseline SL+EL conditions with SL/EL labels randomly permuted across sentences while keeping all text identical. If the scrambled labels still yield significantly better perplexity for the 'sequential' order, then the gain is caused by something other than human-curated difficulty. A complementary check is to compare article labels against human sentence-level simplicity ratings: if the labels do not predict those ratings, the paper's difficulty interpretation is unsupported.","tokens_in":6705,"feed_emoji":"📉","tokens_out":9723,"duration_ms":99797,"temperature":0.7,"pith_summary":"This paper asks whether human-curated simple language can supply the 'easy' half of a curriculum for pretraining a small masked language model. Using the article-level Simple/Everyday labels of the Simple Wikipedia corpus, it shows that training a BERT-tiny model first on Simple Wikipedia and then on everyday text consistently lowers perplexity—overall and on simple-language text—compared with the same data in random order (best overall 65.31 vs 69.61; simple-language 57.83 vs 64.15). The gain depends on ordering: reversing the curriculum (everyday first) loses it, and simply mixing the two registers does not help. By contrast, competence-based curricula built on sentence length, word rarity, or Flesch Reading Ease show no consistent gains, because those heuristics barely separate the two classes. The paper's conclusion is that human intuition about linguistic difficulty, encoded in curated labels, carries structure that shallow heuristics miss.","feed_headline":"Train on simple text first, perplexity drops","feed_subtitle":"A BERT-tiny model improves when Simple Wikipedia comes before everyday text; reversed order loses the gain.","key_machinery":"The central object is the article-level SL/EL label from the Simple Wikipedia corpus, treated as a human judgment about whether a sentence is simple. The sequential curriculum starts on SL and then trains on EL; the incremental curriculum trains on SL and then on the combined set; an anti-sequential control starts on EL. Against these, the competence-based curriculum sorts sentences by sentence length, normalized word rarity, or Flesch Reading Ease and expands the training set according to a square-root competence function. Evaluation uses perplexity (lower is better) of a BERT-tiny model—a two-layer, 128-hidden-unit transformer—over the full set and the SL/EL subsets, with 15 seeds and Holm","core_discovery":"The paper's central claim is that human-curated simple-language labels, not surface-level text statistics, carry the signal that makes curriculum learning work in masked language-model pretraining. Using the Simple Wikipedia corpus, where each sentence inherits an article-level tag for simple (SL) or everyday (EL) language, the authors train a BERT-tiny model under several orderings. The sequential curriculum—SL first, EL second—achieves the best overall perplexity (65.31 ± 4.19) and best simple-language perplexity (57.83 ± 4.52), significantly better than the randomly shuffled SL+EL baseline on both (p = .019 and p = .001, Holm–Bonferroni adjusted). Reversing the order to EL-then-SL elimina","pith_inferences":["Testable extension: randomly permute SL/EL labels across the same sentences and rerun the sequential experiment; if the gain survives, the active ingredient is not human-curated difficulty but some artifact of the split.","The same ordering hypothesis could be tested on other register pairs, such as child-directed versus adult text, to see whether the SL-first benefit generalizes beyond Simple Wikipedia.","Because the effect is demonstrated at BERT-tiny scale, a natural next question is whether larger models or causal objectives still benefit; the scaffold might be needed only when the model's capacity is small.","The asymmetry the paper observes—everyday language seems to cover much of simple language but not vice versa—suggests curriculum ordering may act as a way to avoid gradient conflicts; a curriculum that interleaves with decreasing EL weight might outperform the discrete two-phase switch."],"forward_implications":["Human-curated simple-language labels, ordered SL-first, improve BERT-tiny's overall and SL perplexity relative to the same data in random order (p = .019 and p = .001).","The ordering is doing the work: anti-sequential (EL first) does not beat the EL-only baseline and significantly worsens EL perplexity.","Simply adding simple data without ordering does not help SL perplexity and can worsen it, consistent with asymmetric negative interference between the two registers.","Shallow heuristics—sentence length, word rarity, Flesch Reading Ease—fail to separate SL from EL, explaining why competence-based curricula show no consistent gain over random order.","The gains come with more training updates (about 781k vs 665k for the random mixed baseline), so the practical value depends on whether the perplexity improvement is worth the extra compute."],"supporting_citations":[{"why":"Formalizes curriculum learning as training from easy to hard, the paradigm the paper tests.","marker":"Bengio et al., 2009"},{"why":"Supplies the 'starting small' principle that motivates beginning with simplified text.","marker":"Elman, 1993"},{"why":"Provides the Simple Wikipedia corpus and its SL/EL labels, the paper's difficulty signal.","marker":"Coster and Kauchak, 2011"},{"why":"Provides the competence-based curriculum algorithm and word-rarity heuristic used as the comparison baseline.","marker":"Platanios et al., 2019"},{"why":"Defines the Flesch Reading Ease score used as one shallow difficulty heuristic.","marker":"Flesch, 1948"},{"why":"Reported that data-based curricula give little gain for masked language modeling, the negative result this paper reproduces for competence-based curricula.","marker":"Surkov et al., 2022"},{"why":"Supplies the 'modeling easiness' heuristic approach the paper also draws on for difficulty measures.","marker":"Ranaldi et al., 2023"},{"why":"Gives the negative-interference framing used to interpret why random SL+EL mixing does not help.","marker":"Wang et al., 2020"},{"why":"Supports the explanation that compositional structure in simple language benefits deep models.","marker":"Galke et al., 2024"}],"fun_headline_variants":["Simple text first: a curriculum that works","Human intuition beats shallow heuristics in curriculum","Curriculum: start with simple language, improve perplexity","Simple Wikipedia first: better BERT pretraining","Order matters: simple before complex boosts LM"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the Simple Wikipedia article-level labels are a valid human-curated proxy for sentence-level difficulty; if the labels mostly encode article topic or identity rather than simplicity, the conclusion that human intuition about difficulty drives curriculum gains does not follow, even though the ordering effect itself might persist.","fun_headline_variants_meta":{"raw":{"variants":["Simple text first: a curriculum that works","Human intuition beats shallow heuristics in curriculum","Curriculum: start with simple language, improve perplexity","Simple Wikipedia first: better BERT pretraining","Order matters: simple before complex boosts LM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000133,"raw_usage":{"total_tokens":941,"prompt_tokens":679,"completion_tokens":262,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":423,"completion_tokens_details":{"reasoning_tokens":193}},"tokens_in":423,"tokens_out":262,"duration_ms":3152,"temperature":1.0,"reasoning_tokens":193,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:22:12.104120+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Sequential and Baseline SL+EL conditions with SL/EL labels randomly permuted across sentences while keeping all text identical. If the scrambled labels still yield significantly better perplexity for the 'sequential' order, then the gain is caused by something other than human-curated difficulty. A complementary check is to compare article labels against human sentence-level simplicity ratings: if the labels do not predict those ratings, the paper's difficulty interpretation is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Simple Wikipedia corpus and its SL/EL labels, the paper's difficulty signal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reported that data-based curricula give little gain for masked language modeling, the negative result this paper reproduces for competence-based curricula."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 'modeling easiness' heuristic approach the paper also draws on for difficulty measures."}],"review_version":1}