Pith. sign in

REVIEW 2 major objections 6 minor 21 references

Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning

T0 review · 2 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training a language model on Simple Wikipedia before everyday text consistently lowers perplexity, especially on simple-language text.

desk verdict A clean, small-scale empirical result — SL-first ordering helps BERT-tiny MLM perplexity — but the update-count confound and the label-to-difficulty assumption keep it from being conclusive. read the letter →

arxiv 2508.19873 v1 pith:7MAU6ZMB submitted 2025-08-27 cs.CL

Vanessa Toborek , Sebastian Müller , Tim Selbach , Tamás Horváth , Christian Bauckhage This is my paper · ORCID
classification cs.CL
keywords curriculumlearninglanguagemodelpre-trainingSimpleWikipediaperplexitydifficultyheuristicsBERT-tinydataorderingreadability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether human-curated simple language can supply the 'easy' half of a curriculum for pretraining a small masked language model. Using the article-level Simple/Everyday labels of the Simple Wikipedia corpus, it shows that training a BERT-tiny model first on Simple Wikipedia and then on everyday text consistently lowers perplexity—overall and on simple-language text—compared with the same data in random order (best overall 65.31 vs 69.61; simple-language 57.83 vs 64.15). The gain depends on ordering: reversing the curriculum (everyday first) loses it, and simply mixing the two registers does not help. By contrast, competence-based curricula built on sentence length, word rarity, or Flesch Reading Ease show no consistent gains, because those heuristics barely separate the two classes. The paper's conclusion is that human intuition about linguistic difficulty, encoded in curated labels, carries structure that shallow heuristics miss.

What carries the argument

The central object is the article-level SL/EL label from the Simple Wikipedia corpus, treated as a human judgment about whether a sentence is simple. The sequential curriculum starts on SL and then trains on EL; the incremental curriculum trains on SL and then on the combined set; an anti-sequential control starts on EL. Against these, the competence-based curriculum sorts sentences by sentence length, normalized word rarity, or Flesch Reading Ease and expands the training set according to a square-root competence function. Evaluation uses perplexity (lower is better) of a BERT-tiny model—a two-layer, 128-hidden-unit transformer—over the full set and the SL/EL subsets, with 15 seeds and Holm

What would settle it

Re-run the Sequential and Baseline SL+EL conditions with SL/EL labels randomly permuted across sentences while keeping all text identical. If the scrambled labels still yield significantly better perplexity for the 'sequential' order, then the gain is caused by something other than human-curated difficulty. A complementary check is to compare article labels against human sentence-level simplicity ratings: if the labels do not predict those ratings, the paper's difficulty interpretation is unsupported.

Watch

Extended reading notes

Core claim

The paper's central claim is that human-curated simple-language labels, not surface-level text statistics, carry the signal that makes curriculum learning work in masked language-model pretraining. Using the Simple Wikipedia corpus, where each sentence inherits an article-level tag for simple (SL) or everyday (EL) language, the authors train a BERT-tiny model under several orderings. The sequential curriculum—SL first, EL second—achieves the best overall perplexity (65.31 ± 4.19) and best simple-language perplexity (57.83 ± 4.52), significantly better than the randomly shuffled SL+EL baseline on both (p = .019 and p = .001, Holm–Bonferroni adjusted). Reversing the order to EL-then-SL elimina

Load-bearing premise

The load-bearing premise is that the Simple Wikipedia article-level labels are a valid human-curated proxy for sentence-level difficulty; if the labels mostly encode article topic or identity rather than simplicity, the conclusion that human intuition about difficulty drives curriculum gains does not follow, even though the ordering effect itself might persist.

Editorial extensions

If this is right

  • Human-curated simple-language labels, ordered SL-first, improve BERT-tiny's overall and SL perplexity relative to the same data in random order (p = .019 and p = .001).
  • The ordering is doing the work: anti-sequential (EL first) does not beat the EL-only baseline and significantly worsens EL perplexity.
  • Simply adding simple data without ordering does not help SL perplexity and can worsen it, consistent with asymmetric negative interference between the two registers.
  • Shallow heuristics—sentence length, word rarity, Flesch Reading Ease—fail to separate SL from EL, explaining why competence-based curricula show no consistent gain over random order.
  • The gains come with more training updates (about 781k vs 665k for the random mixed baseline), so the practical value depends on whether the perplexity improvement is worth the extra compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Testable extension: randomly permute SL/EL labels across the same sentences and rerun the sequential experiment; if the gain survives, the active ingredient is not human-curated difficulty but some artifact of the split.
  • The same ordering hypothesis could be tested on other register pairs, such as child-directed versus adult text, to see whether the SL-first benefit generalizes beyond Simple Wikipedia.
  • Because the effect is demonstrated at BERT-tiny scale, a natural next question is whether larger models or causal objectives still benefit; the scaffold might be needed only when the model's capacity is small.
  • The asymmetry the paper observes—everyday language seems to cover much of simple language but not vice versa—suggests curriculum ordering may act as a way to avoid gradient conflicts; a curriculum that interleaves with decreasing EL weight might outperform the discrete two-phase switch.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper investigates whether editorially curated simple language (Simple Wikipedia, SL) provides a useful signal for curriculum learning (CL) in masked language model pre-training. Using BERT-tiny and the Simple Wikipedia dataset, the authors compare label-based curricula (sequential SL→EL, incremental SL→SL+EL, anti-sequential EL→SL) with competence-based curricula driven by sentence length, word rarity, and Flesch Reading Ease, plus random baselines. Each condition is run over 15 seeds with one-sided significance tests and Holm–Bonferroni adjustment. The main reported result is that the sequential curriculum significantly improves overall and SL perplexity over a random SL+EL baseline, while the anti-sequential order does not, and none of the competence-based curricula yields significant gains. The paper concludes that human intuition about linguistic difficulty can guide CL, whereas shallow heuristics cannot.

Significance. Strengths: the study is comparatively clean: public dataset, 15 seeds, explicit significance testing, an anti-sequential control, and a negative-result arm for heuristics. If the main comparison were fully controlled, the paper would be a useful empirical datapoint for the curriculum-learning debate and for the role of simplified language in pretraining. The significance is limited by two issues: the sequential condition trains for substantially more updates than its baseline, and the SL/EL labels are article-level proxies for sentence-level difficulty. Both concerns are addressable, but they currently affect the central claim rather than a peripheral detail.

major comments (2)
  1. [Sec. 4, Table 3] The central comparison—Sequential vs. Baseline SL+EL—does not isolate ordering from training duration. Sequential uses 781,333±122,292 updates vs. 665,333±102,111 for Baseline SL+EL, about 17% more. The reported p-values compare final perplexities after different numbers of update steps; longer training generally moves final perplexity in the same direction as the claimed effect. The fact that Incremental trains for the same 781k updates yet fails to improve overall perplexity weakens a pure 'more compute' account, but it does not resolve the schedule–update interaction. Please provide an update-matched control (e.g., a random SL+EL model trained for the same number of updates as Sequential) or report perplexity at matched update counts, and also report the early-stopping protocol.
  2. [Sec. 3, Dataset] The paper's interpretive conclusion—that 'human intuition about linguistic difficulty' drives the gain—rests on the assumption that article-level SL/EL labels reflect sentence-level difficulty. The authors acknowledge that inherited labels introduce noise, but the issue is more than noise: SL vs. EL may encode register, topic, or article identity rather than perceived simplicity. The empirical ordering effect (SL first helps) remains of interest, but it does not by itself establish the difficulty-based mechanism. I recommend either validating the proxy (e.g., human or readability annotation of a sample) or narrowing the conclusion to Simple-Wikipedia-first training.
minor comments (6)
  1. [Sec. 4] 'no improvement in neither overall nor SL perplexity' should read 'no improvement in either overall or SL perplexity'.
  2. [Appendix A] The competence-function hyperparameters T=50,000 and c0=0.05 were chosen after observing that shorter competence phases yield better results. Please describe the selection procedure (validation set? grid search?) and report sensitivity; this tuning is not part of the central comparison, but it affects the negative arm.
  3. [Sec. 4 / Table 5] The negative statements about competence-based curricula are based on non-significant differences with n=15 seeds. This supports 'no consistent gain observed', but phrases like 'no benefit' or 'fail to effectively separate' should be accompanied by effect sizes or an equivalence/power analysis, or restricted to 'we did not detect an improvement'.
  4. [Sec. 3, Training Setup] Please specify the train/validation/test split and whether the reported perplexities are on a held-out test set. This also helps rule out test contamination in the SL-first condition.
  5. [Figure 1] The histograms use raw counts, but SL and EL have different numbers of sentences. Density plots would make the overlap of the distributions easier to assess.
  6. [Sec. 4 / Table 3] Consider reporting effect sizes (e.g., Cohen's d or matched difference) for the significant comparisons; with 15 seeds, p-values alone do not convey the magnitude of the improvement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central claim rests on direct empirical comparisons against random baselines.

full rationale

The paper's central result—that a sequential SL-first curriculum improves perplexity relative to a random SL+EL baseline—is an empirical measurement, not a quantity derived from its own inputs. The label-based curricula are defined by the externally produced Simple Wikipedia article-level labels (Coster and Kauchak, 2011), and the reported improvements (overall perplexity 65.31 vs. 69.61, SL perplexity 57.83 vs. 64.15, p=.019 and p=.001) are direct comparisons against a random baseline with the same data. No parameter is fitted to the target perplexity values; the only tuned hyperparameters (competence function T=50,000, c0=0.05) are used in the competence-based negative-result arm and do not enter the label-based claim. The acknowledged update-count difference (~781k vs. ~665k updates) is a potential confound for the empirical effect, but it is not a circularity: the comparison is still an external, falsifiable experiment, and the paper does not define the outcome in terms of the training schedule. The article-level label-noise concern is likewise a validity threat to the interpretation that human intuition drives the gains, but it does not make the derivation circular, because the labels are independently curated rather than constructed from the model's own predictions. There are no load-bearing self-citations: the paper does not cite its own authors' prior work to justify its core premise, and all external references (Platanios et al., Surkov et al., etc.) are standard prior work used for context or comparison, not for uniqueness claims. Thus no step in the paper's chain reduces by definition or by construction to its inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central label-based result introduces no fitted parameters; the free parameters listed belong to the competence-based (negative-result) arm. The main conceptual assumption is the SL/EL label-to-difficulty mapping, together with the evaluation choice.

free parameters (3)
  • T (competence phase duration) = 50,000
    Chosen in Appendix A after observing that shorter competence phases gave better results; not fixed a priori.
  • c0 (initial competence) = 0.05
    Set with T as competence-function parameters for competence-based CL; selected based on experiments.
  • competence update interval = 5,000 steps
    Decides how often the active training set is expanded; hand-chosen in Appendix A.
assumptions (3)
  • domain assumption Article-level Simple Wikipedia SL/EL labels are a valid proxy for human-curated sentence-level linguistic difficulty.
    Section 3, Dataset acknowledges label noise from article-level labels; the paper's central interpretation depends on SL encoding human-perceived simplicity.
  • domain assumption Perplexity on held-out text is a sufficient evaluation signal for MLM pre-training quality.
    Section 3, Evaluation; no downstream task or generation evaluation is reported.
  • domain assumption BERT-tiny trained to convergence with early stopping is representative enough to draw conclusions about CL for LM pre-training.
    Section 3, Training Setup; the paper explicitly limits to BERT-tiny and 15 seeds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning." pith.science (2026). https://pith.science/paper/7MAU6ZMB

@misc{pith2026250819873,
  author       = {Pith},
  title        = {Pith review of: Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7MAU6ZMB}},
  note         = {Machine review of arXiv:2508.19873}
}
read the original abstract

Curriculum learning (CL) aims to improve training by presenting data from "easy" to "hard", yet defining and measuring linguistic difficulty remains an open challenge. We investigate whether human-curated simple language can serve as an effective signal for CL. Using the article-level labels from the Simple Wikipedia corpus, we compare label-based curricula to competence-based strategies relying on shallow heuristics. Our experiments with a BERT-tiny model show that adding simple data alone yields no clear benefit. However, structuring it via a curriculum -- especially when introduced first -- consistently improves perplexity, particularly on simple language. In contrast, competence-based curricula lead to no consistent gains over random ordering, probably because they fail to effectively separate the two classes. Our results suggest that human intuition about linguistic difficulty can guide CL for language model pre-training.

Figures

Figures reproduced from arXiv: 2508.19873 by the authors.

Figure 1
Figure 1. Distribution of sentence-level difficulty heuristics for SL and EL. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 9 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    utze, Andreas S\

    Alessia Battisti, Dominik Pf\"utze, Andreas S\"auberli, Marek Kostrzewa, and Sarah Ebling. 2020. https://aclanthology.org/2020.lrec-1.404/ A Corpus for Automatic Readability Assessment and Text Simplification of German . In Proceedings of the 12th Language Resources and Evaluation Conference, LREC 2020 , pages 3302--3311. European Language Resources Association

    work page 2020

  4. [4]

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. https://dl.acm.org/doi/10.1145/1553374.1553380 Curriculum Learning . In Proceedings of the 26th International Conference On Machine Learning, ICML 2009, pages 41--48

    arXiv 2009

  5. [5]

    William Coster and David Kauchak. 2011. https://aclanthology.org/P11-2117 Simple English Wikipedia : A New Text Simplification Task . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies , ACL 2011 , volume 3, pages 665--669. Association for Computational Linguistics

    work page 2011

  6. [6]

    Jeffrey L. Elman. 1993. https://doi.org/10.1016/S0010-0277(02)00106-3 Learning and development in neural networks: the importance of starting small . Cognition, 48(1):71--99

    work page doi:10.1016/s0010-0277(02)00106-3 1993

  7. [7]

    Rudolph Flesch. 1948. https://doi.org/10.1037/h0057532 A New Readability Yardstick . Journal of Applied Psychology, 32:221--233

    doi:10.1037/h0057532 1948

  8. [8]

    Lukas Galke, Yoav Ram, and Limor Raviv. 2024. https://doi.org/10.1038/s41467-024-55158-1 What Makes a Language Easy to Deep-Learn? Deep Neural Networks and Humans Similarly Benefit from Compositional Structure . Nature Communications, 15(1):10816

    doi:10.1038/s41467-024-55158-1 2024

Show all 21 references
  1. [9]

    Huebner, Elior Sulem, Cynthia Fisher, and Dan Roth

    Philip A. Huebner, Elior Sulem, Cynthia Fisher, and Dan Roth. 2021. https://doi.org/10.18653/v1/2021.conll-1.49 BabyBERTa : Learning More Grammar With Small-Scale Child-Directed Language . In Proceedings of the 25th Conference on Computational Natural Language Learning , CoNLL...

    2021 doi

  2. [10]

    David Kauchak. 2013. https://aclanthology.org/P13-1151/ Improving Text Simplification Language Modeling Using Unsimplified Text Data . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2013 , volume 1, pages...

    2013

  3. [11]

    Cao Liu, Shizhu He, Kang Liu, and Jun Zhao. 2018. https://doi.org/10.24963/IJCAI.2018/587 Curriculum Learning for Natural Answer Generation . In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018 , pages 4223--4229. ijcai.org

    2018 doi

  4. [12]

    Evan Lucas, Dylan Gaines, Tagore Rao Kosireddy, Kevin Li, and Timothy C. Havens. 2024. https://aclanthology.org/2024.conll-babylm.19/ Using Curriculum Masking Based on Child Language Development to Train a Large Language Model with Limited Training Data . In The 2nd BabyLM Cha...

    2024

  5. [13]

    Tasnim Mohiuddin, Philipp Koehn, Vishrav Chaudhary, James Cross, Shruti Bhosale, and Shafiq Joty. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.113 Data Selection Curriculum for Neural Machine Translation . In Findings of the Association for Computational Linguistics :...

    2022 doi

  6. [14]

    Aaron Mueller and Tal Linzen. 2023. https://doi.org/10.18653/V1/2023.ACL-LONG.629 How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases . In Proceedings of the 61st Annual Meeting of the Association for Computation...

    2023 doi

  7. [15]

    Koichi Nagatsuka, Clifford Broni-Bediako, and Masayasu Atsumi. 2021. https://doi.org/10.26615/978-954-452-072-4_112 Pre-Training a BERT with Curriculum Learning by Increasing Block-Size of Input Text . In Proceedings of the 12th International Conference on Recent Advances in N...

    2021 doi

  8. [16]

    Mitchell

    Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom M. Mitchell. 2019. https://doi.org/10.18653/v1/n19-1119 Competence-Based Curriculum Learning for Neural Machine Translation . 1:1162--1172

    2019 doi

  9. [17]

    Leonardo Ranaldi, Giulia Pucci, and Fabio Massimo Zanzotto. 2023. https://doi.org/10.26615/978-954-452-092-2_101 Modeling Easiness for Training Transformers with Curriculum Learning . In Proceedings of the 14th International Conference on Recent Advances in Natural Language Pr...

    2023 doi

  10. [18]

    Maxim Surkov, Vladislav Mosin, and Ivan Yamshchikov. 2022. https://doi.org/10.18653/v1/2022.insights-1.16 Do Data-based Curricula Work ? In Proceedings of the Third Workshop on Insights from Negative Results in NLP , pages 119--128. Association for Computational Linguistics

    2022 doi

  11. [19]

    Xin Wang, Yudong Chen, and Wenwu Zhu. 2021. https://ieeexplore.ieee.org/document/9392296 A Survey on Curriculum Learning . IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9)

    2021

  12. [20]

    Lipton, and Yulia Tsvetkov

    Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.359 On Negative Interference in Multilingual Models : Findings and A Meta-Learning Treatment . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pr...

    2020 doi

  13. [21]

    Wojciech Zaremba and Ilya Sutskever. 2015. https://arxiv.org/abs/1410.4615 Learning to Execute . Preprint, arXiv:1410.4615

    2015 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.