Pith. sign in

REVIEW 3 major objections 6 minor 39 references

Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Using a development set to stop training early can misreport low-resource NLP accuracy by up to 18 percentage points.

desk verdict An honest empirical caution about dev-set early stopping in low-resource NLP, but the headline 18-point gap is largely an artifact of a weak mean-epoch baseline. read the letter →

arxiv 1909.01522 v2 pith:2QMFJ7HI submitted 2019-09-04 cs.CL

Katharina Kann , Kyunghyun Cho , Samuel R. Bowman This is my paper · ORCID
classification cs.CL
keywords low-resourceNLPdevelopmentsetearlystoppingrealisticevaluationcross-lingualtuningmorphologicalinflectionhistoricaltextnormalizationtransliteration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Low-resource NLP papers often tune neural models by early stopping on a development set, even though in a true data-poor deployment those examples would more plausibly be used for training. This paper replicates three recent low-resource sequence-to-sequence studies—historical text normalization, morphological inflection, and transliteration—and compares the standard DevSet protocol with a realistic alternative in which the number of training epochs is chosen from other languages for the same task (DevLang). Across languages, average absolute accuracy differs by up to 1.4 percentage points, while individual languages differ by as much as 18.0 points, as in Azeri morphological inflection. The authors conclude that development-set early stopping gives a poor estimate of real-world performance and that low-resource research should move away from large development sets when real-world settings are the goal.

What carries the argument

The central mechanism is a two-phase training protocol that isolates the stopping signal. In the stopping-point selection phase, models for the development languages are trained with the original early-stopping rule, and the DevLang epoch for each target language is the rounded average of the best epochs across those other languages. In the main training phase, each target model is trained once, preserving the DevSet early-stopped checkpoint and the additional DevLang checkpoint at the precomputed epoch. Because both models come from the same training run, any accuracy difference is attributable to the stopping point rather than to random initialization or data variation.

What would settle it

Rerun the Azeri MORPH setup with several different pools of development languages; if the DevLang epoch estimate shifts enough that the 18-point DevSet–DevLang gap does not reproduce, the claim that dev-set stopping reliably misestimates real-world performance would be undercut.

Watch

Extended reading notes

Core claim

The central discovery is a measurement of what the stopping signal is worth in low-resource neural NLP, across morphological inflection (MORPH), historical text normalization (NORM), and transliteration (TRANSL). Training each model once and keeping two checkpoints—the early-stopped model chosen by the target language's development set (DevSet) and the model at the epoch averaged from development languages (DevLang)—the paper finds DevSet accuracy is higher in 72 of 103 MORPH languages and 8 of 10 NORM languages, while DevLang is higher in 2 of 5 TRANSL languages. Mean test accuracy is 51.3 vs 50.0 for MORPH, 74.9 vs 74.2 for NORM, and 21.8 vs 22.3 for TRANSL. The largest gap is 18.0 absolute accuracy points on Azeri, where DevSet stops at epoch 217 with 64% accuracy and DevLang runs to epoch 324 with 46%. The paper reads this as evidence that dev-set early stopping can either overestimate or underestimate real-world performance and should not be treated as a neutral evaluation choice.

Load-bearing premise

The comparison assumes that the rounded average of the best stopping epochs across development languages is a realistic stand-in for the stopping decision a practitioner without a target-language development set would actually make.

Editorial extensions

If this is right

  • Published absolute accuracies from low-resource experiments that early-stop on a relatively large development set can differ from a no-dev-set setup by up to 18 percentage points.
  • Tuning the number of training epochs on development languages is a workable realistic baseline and is the alternative the paper's conclusion points to.
  • For morphological inflection and historical text normalization, development-set early stopping usually overestimates performance, whereas for transliteration it can underestimate it.
  • Comparisons between proposed methods in low-resource NLP should report or control for the stopping regime, since the stopping rule alone can move results.
  • Development examples may be better used as training data when the goal is to estimate real-world performance rather than to compare methods in a fixed benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the DevLang rule is an unweighted cross-language average, so languages whose learning dynamics differ strongly could make the no-dev-set gap larger or smaller than the maxima reported.
  • Editorial inference: weighting development languages by proximity to the target language, or adapting the stopping epoch during training, could yield a stronger realistic baseline and shrink the measured gap.
  • Editorial inference: the same DevSet-versus-DevLang comparison could be run for tagging, parsing, and classification tasks to see whether the finding extends beyond sequence transduction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper empirically compares two training protocols for low-resource NLP: early stopping on a target-language development set (DevSet) versus fixing the number of training epochs by averaging the best epochs over other 'development languages' (DevLang). Using public code and data from three prior studies (historical text normalization, morphological inflection, transliteration), the authors find that average test-accuracy differences between the two protocols are small (up to 1.4% absolute), but that individual languages can differ by as much as 18.0% (MORPH, Azeri). They conclude that using large development sets for early stopping can give a poor estimate of real-world performance and recommend that low-resource NLP research move away from this practice. The paper includes per-language results in appendices and a brief limitations section.

Significance. The paper addresses an important methodological question: whether the common practice of using a development set for early stopping in low-resource NLP research distorts reported performance relative to realistic deployment settings. Its strengths are the transparent reuse of prior public code and data, a clear experimental design that isolates the stopping rule, and detailed per-language transparency in the appendices. If the empirical claims hold, the paper could influence how low-resource NLP results are reported. However, the significance is currently tempered by the sensitivity of the conclusions to the specific DevLang heuristic and by the absence of any statistical validation, both of which leave the central claim conditional rather than established.

major comments (3)
  1. [Section 3, 'Stopping point selection phase'; Tables 3, 6, 7] The DevLang protocol's mean-epoch heuristic is not justified as a representative 'realistic' alternative. In the MORPH experiments (Appendix A), the DevLang epoch count is approximately 324 for nearly all languages, while DevSet best epochs range from 28 (Tibetan) to 881 (French). For Azeri, DevSet reaches 64% accuracy at epoch 217, but the DevLang model is trained to epoch 324 and achieves only 46%, yielding the headline 18.0% gap in Table 3. This gap may be an artifact of selecting the mean rather than a consequence of early stopping on a development set. The paper should either justify the mean as the natural real-world choice or test robustness to alternative summary statistics (e.g., median, a high quantile, or a learning-curve-based rule). Without such a sensitivity analysis, the central conclusion that dev sets give a poor estimate of real-world performance is conditional on this specific baseline.
  2. [Tables 2 and 3] The paper reports no measures of uncertainty or significance. Table 2's counts (e.g., 72 languages where DevSet > DevLang for MORPH) and Table 3's average differences are presented as evidence of systematic overestimation, but no confidence intervals, significance tests, or multiple random seeds are provided. Given that the paper's recommendation rests on these systematic differences being real, the authors should report a paired significance test across languages (e.g., a signed-rank test) or bootstrap intervals. As it stands, the reader cannot determine whether the 72/23/8 split is likely to arise by chance.
  3. [Section 3, equal-size requirement] The DevLang protocol requires equal training set sizes across languages, and the study is restricted to datasets satisfying this condition (Section 3 and Table 1). In many real-world low-resource settings, language datasets vary substantially in size, and a simple epoch average would not transfer directly. The paper does not discuss whether its conclusions generalize to such settings, which weakens the broad practical take-away in Section 6 and the abstract.
minor comments (6)
  1. [Section 3 and Tables 4-7] The text states 'We round this number to an integer,' but Tables 4–7 report non-integer DevLang epoch values such as 43.44 and 323.11; please clarify whether the main training phase used rounded integers or the exact averages.
  2. [Section 4.2] The sentence 'We randomly choose ten development languages' for MORPH does not report a random seed or a sensitivity analysis; please provide the seed or show that the results do not depend on this particular choice.
  3. [Figure 1] The axes labeled 'Δ epochs' and 'Δ accuracy' should be defined explicitly as DevLang minus DevSet, and a regression line or correlation coefficient should be included if the claim about 'longer training seems better for MORPH' is to be supported visually.
  4. [Section 6, 'Practical take-aways'] The final recommendation is phrased unconditionally ('should move away') despite the paper's own limitations section noting that dev sets may be sufficient for comparing methods in equal settings; please add conditions or soften the wording to align with the evidence.
  5. [Abstract] The statement 'differences are as big as 18.0% accuracy' highlights a single outlier among over a hundred language-task pairs; consider also reporting the median and interquartile range of the differences to give a balanced picture.
  6. [References and Table 1] There is a typo in the reference for Schulz et al.: 'NAAACL' should be 'NAACL'; the same typo appears in Table 1 caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical protocol comparison, not a derivation, and its central claim is anchored to external datasets and prior implementations.

full rationale

The paper compares two training protocols, DevSet (early stopping on a target-language development set) and DevLang (epoch count tuned on other languages), and reports the resulting accuracy differences. No parameter is fitted to the target data and then renamed as a prediction; the DevLang epoch is computed as the average of best epochs over development languages, explicitly excluding the target language when the target is itself a development language, and the comparison is made on test sets from the original papers. The central claim that development sets can give a poor estimate of real-world performance is an empirical observation about the magnitude of the differences, not a quantity that is forced by construction. The authors also state a limitation: 'we do not claim to show that using a validation set always over- or underestimates real-world performance,' which further shows the claim is not definitionally guaranteed. The skeptic's concern that the mean-epoch heuristic may be a weak or unrepresentative baseline is a robustness/correctness critique about whether DevLang reflects realistic practice, not a circularity: the paper does not define DevLang in terms of the outcome it predicts, nor does it rely on self-citation to establish its conclusion. Thus no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities; the paper's claims rest on domain assumptions about the transferability of early stopping behavior across languages and the faithfulness of the original experimental setups. The only constructed quantity, the DevLang epoch count, is a simple statistic, not a fitted parameter.

assumptions (3)
  • domain assumption The training sets for all languages in a given task are of equal size, which is required to average epoch counts across languages.
    Section 3 states 'It is important for our experiments that the training sets for all languages are of the same size', because otherwise the number of examples per epoch would differ and the average best epoch would not be directly comparable.
  • domain assumption The original hyperparameters and model architectures from the prior papers are appropriate for all languages in the study.
    The paper reuses the implementations and hyperparameters from Bollmann et al. (2018), Sharma et al. (2018), and Upadhyay et al. (2018) without per-language tuning, assuming these are reasonable across the language set.
  • domain assumption Development languages are representative of target languages with respect to training dynamics.
    The DevLang protocol computes the average best epoch over development languages and applies it to target languages. This assumes a transferable relationship between language learning curves, which is the weakest loaded assumption of the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set." pith.science (2026). https://pith.science/paper/2QMFJ7HI

@misc{pith2026190901522,
  author       = {Pith},
  title        = {Pith review of: Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2QMFJ7HI}},
  note         = {Machine review of arXiv:1909.01522}
}
read the original abstract

Development sets are impractical to obtain for real low-resource languages, since using all available data for training is often more effective. However, development sets are widely used in research papers that purport to deal with low-resource natural language processing (NLP). Here, we aim to answer the following questions: Does using a development set for early stopping in the low-resource setting influence results as compared to a more realistic alternative, where the number of training epochs is tuned on development languages? And does it lead to overestimation or underestimation of performance? We repeat multiple experiments from recent work on neural models for low-resource NLP and compare results for models obtained by training with and without development sets. On average over languages, absolute accuracy differs by up to 1.4%. However, for some languages and tasks, differences are as big as 18.0% accuracy. Our results highlight the importance of realistic experimental setups in the publication of low-resource NLP research results.

Figures

Figures reproduced from arXiv: 1909.01522 by the authors.

Figure 1
Figure 1. Difference in accuracy (DevLang-DevSet) depending on the difference in training epochs (DevLang [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 32 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Roee Aharoni and Yoav Goldberg. 2017. Morphological inflection generation with hard monotonic attention. In ACL

    work page 2017

  4. [4]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR

    2015

  5. [5]

    Marcel Bollmann, Joachim Bingel, and Anders S gaard. 2017. Learning attention for historical text normalization by learning to pronounce. In ACL

    work page 2017

  6. [6]

    Marcel Bollmann, Anders S gaard, and Joachim Bingel. 2018. Multi-task learning for historical text normalization: S ize matters. In DeepLo

    work page 2018

  7. [7]

    Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41--75

    1997

  8. [8]

    Ryan Cotterell, Christo Kirov, John Sylak-Glassman, G \'e raldine Walther, Ekaterina Vylomova, Arya D McCarthy, Katharina Kann, Sebastian Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. The C o NLL -- SIGMORPHON 2018 shared task: U niversal morphological reinflection. In C o NLL -- SIGMORPHON

    work page 2018

Show all 39 references
  1. [9]

    Hal Daume III. 2007. Frustratingly easy domain adaptation. In ACL

    2007

  2. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT : P re-training of deep bidirectional transformers for language understanding. In NAACL

    2019

  3. [11]

    Banchs, Min Zhang, Haizhou Li, and A Kumaran

    Xiangyu Duan, Rafael E. Banchs, Min Zhang, Haizhou Li, and A Kumaran. 2015. Proceedings of the fifth named entity workshop. In Fifth Named Entity Workshop

    2015

  4. [12]

    Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why does unsupervised pre-training help deep learning? JMLR, 11:625--660

    2010

  5. [13]

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML

    2017

  6. [14]

    Dan Garrette, Jason Mielens, and Jason Baldridge. 2013. Real-world semi-supervised learning of POS -taggers for low-resource languages. In ACL

    2013

  7. [15]

    Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In ACL

    2016

  8. [16]

    Reza Haffari, Colin Cherry, George Foster, Shahram Khadivi, and Bahar Salehi. 2018. Proceedings of the workshop on deep learning approaches for low-resource NLP

    2018

  9. [17]

    Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

    1997

  10. [18]

    Ann Irvine and Chris Callison-Burch. 2013. Combining bilingual and comparable corpora for low resource machine translation. In WMT

    2013

  11. [19]

    Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi \'e gas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. TACL, 5:339--351

    2017

  12. [20]

    Katharina Kann, Ryan Cotterell, and Hinrich Sch \"u tze. 2017. One-shot neural cross-lingual transfer for paradigm completion. In ACL

    2017

  13. [21]

    Katharina Kann, Jesus Manuel Mager Hois, Ivan Vladimir Meza Ruiz, and Hinrich Sch \"u tze. 2018. Fortification of neural morphological segmentation models for polysynthetic minimal-resource languages. In NAACL

    2018

  14. [22]

    Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. O pen NMT : O pen-source toolkit for neural machine translation. In ACL

    2017

  15. [23]

    Nikola Ljube s ic, Katja Zupan, Darja Fi s er, and Tomaz Erjavec. 2016. Normalising slovene data: historical texts vs. user-generated content. In KONVENS

    2016

  16. [24]

    Maren Mahsereci, Lukas Balles, Christoph Lassner, and Philipp Hennig. 2017. Early stopping without a validation set. arXiv:1703.09580

    2017 arXiv

  17. [25]

    Peter Makarov and Simon Clematide. 2018. Imitation learning for neural morphological string transduction. In EMNLP

    2018

  18. [26]

    Carolin Odebrecht, Malte Belz, Amir Zeldes, Anke L \"u deling, and Thomas Krause. 2017. RIDGES herbology: designing a diachronic multi-layer corpus. Language Resources and Evaluation, 51(3):695--725

    2017

  19. [27]

    Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. 2018. Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS

    2018

  20. [28]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL

    2018

  21. [29]

    Eva Pettersson. 2016. Spelling normalisation and linguistic analysis of historical text for information extraction. Ph.D. thesis, Acta Universitatis Upsaliensis

    2016

  22. [30]

    Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv:1706.05098

    2017 arXiv

  23. [31]

    Claudia Schulz, Steffen Eger, Johannes Daxenberger, Tobias Kahse, and Iryna Gurevych. 2018. Multi-task learning for argumentation mining in low-resource settings. In NAAACL

    2018

  24. [32]

    Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: S ummarization with pointer-generator networks. In ACL

    2017

  25. [33]

    Abhishek Sharma, Ganesh Katrapati, and Dipti Misra Sharma. 2018. IIT(BHU)--IIITH at CoNLL--SIGMORPHON 2018 shared task on universal morphological reinflection. In CoNLL--SIGMORPHON

    2018

  26. [34]

    Shyam Upadhyay, Jordan Kodner, and Dan Roth. 2018. Bootstrapping transliteration with constrained discovery for low-resource languages. In EMNLP

    2018

  27. [35]

    Gael Vaamonde. 2015. Userguide for digital edition of texts in ps post scriptum

    2015

  28. [36]

    Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR

    2019

  29. [37]

    David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In HLT

    2001

  30. [38]

    Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. arXiv:1805.07513

    2018 arXiv

  31. [39]

    Xiaojin Jerry Zhu. 2005. Semi-supervised learning literature survey. Technical report, University of Wisconsin-Madison, Department of Computer Sciences

    2005

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.