{"id":"280ef69f-7724-4794-acdc-31ca80593650","arxiv_id":"1909.01522","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Using a development set for early stopping can change reported accuracy for low-resource NLP models, with per-language differences up to 18 percentage points compared to tuning training length on other languages.","lead":"This paper tests whether using a development set to stop training language models inflates reported accuracy for low-resource NLP tasks. Across three tasks, average accuracy differences are small, but individual languages can differ by up to 18 percentage points, so evaluation choices matter.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DevLang protocol's mean-epoch heuristic may be a strawman; the central 'dev sets mislead' claim is not tested against a robust no-dev-set alternative.","rationale":"The reader's weakest_assumption identifies exactly this issue: the DevLang protocol's averaged stopping epoch may be a poor proxy for the optimal per-language epoch. I agree with that assessment. This is load-bearing because the entire quantitative comparison, and especially the 18.0% maximum difference that drives the abstract, is measured against this single heuristic. The paper's own Limitations section acknowledges that the effect depends on how representative the validation set is, but it does not address the representativeness of the DevLang epoch rule. A meaningful stress test is to vary that rule. Since the needed experiments are straightforward with the available code, the appropriate verdict remains CONDITIONAL: the paper is an honest and useful study, but its central quantitative claims require a robustness check before the recommendation to move away from development sets can be considered established.","tokens_in":9104,"tokens_out":6202,"duration_ms":66635,"concrete_test":"Re-run the MORPH experiments with DevLang epochs chosen by (i) the median of best epochs across the ten development languages, and (ii) leave-one-out cross-validation on development languages that selects the epoch maximizing held-out development accuracy. Compare the resulting DevLang accuracies to DevSet using the same evaluation as Table 3 and Appendix A. If the maximum absolute difference drops below 5% or the sign of the average difference changes, the reported overestimation/underestimation conclusions are artifacts of the averaging heuristic. The public code bases (SIGMORPHON 2018 baselines) make this a direct re-training with a modified epoch rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest link is Section 3's stopping-point selection: DevLang uses the average of best epochs over development languages as the target epoch count. In the MORPH experiments (Appendix A), this average collapses to a single value around 324 for almost all languages, while DevSet best epochs range from 28 to 881. Many languages are therefore either under-trained (e.g., Finnish best at 799 vs 324) or over-trained (e.g., Azeri best at 217 vs 324). The large accuracy gaps in Table 3, including the headline 18.0% difference for Azeri, may reflect the poor fit of the mean rather than an inherent overestimation from early stopping. Figure 1 shows substantial scatter between epoch difference and accuracy difference, confirming that a single average epoch is not a good predictor for individual languages. The paper's recommendation to abandon large dev sets would be stronger if the alternative were more representative of what a practitioner would do without a target dev set, such as using the median, a high quantile, or a stopping rule based on development-language learning curves. Without such a sensitivity check, the conclusion that dev sets give a poor estimate of real-world performance is conditional on this specific, and arguably weak, baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically compares two training protocols for low-resource NLP: early stopping on a target-language development set (DevSet) versus fixing the number of training epochs by averaging the best epochs over other 'development languages' (DevLang). Using public code and data from three prior studies (historical text normalization, morphological inflection, transliteration), the authors find that average test-accuracy differences between the two protocols are small (up to 1.4% absolute), but that individual languages can differ by as much as 18.0% (MORPH, Azeri). They conclude that using large development sets for early stopping can give a poor estimate of real-world performance and recommend that low-resource NLP research move away from this practice. The paper includes per-language results in appendices and a brief limitations section.","tokens_in":9312,"tokens_out":8146,"duration_ms":78824,"significance":"The paper addresses an important methodological question: whether the common practice of using a development set for early stopping in low-resource NLP research distorts reported performance relative to realistic deployment settings. Its strengths are the transparent reuse of prior public code and data, a clear experimental design that isolates the stopping rule, and detailed per-language transparency in the appendices. If the empirical claims hold, the paper could influence how low-resource NLP results are reported. However, the significance is currently tempered by the sensitivity of the conclusions to the specific DevLang heuristic and by the absence of any statistical validation, both of which leave the central claim conditional rather than established.","major_comments":[{"comment":"The DevLang protocol's mean-epoch heuristic is not justified as a representative 'realistic' alternative. In the MORPH experiments (Appendix A), the DevLang epoch count is approximately 324 for nearly all languages, while DevSet best epochs range from 28 (Tibetan) to 881 (French). For Azeri, DevSet reaches 64% accuracy at epoch 217, but the DevLang model is trained to epoch 324 and achieves only 46%, yielding the headline 18.0% gap in Table 3. This gap may be an artifact of selecting the mean rather than a consequence of early stopping on a development set. The paper should either justify the mean as the natural real-world choice or test robustness to alternative summary statistics (e.g., median, a high quantile, or a learning-curve-based rule). Without such a sensitivity analysis, the central conclusion that dev sets give a poor estimate of real-world performance is conditional on this specific baseline.","section":"Section 3, 'Stopping point selection phase'; Tables 3, 6, 7"},{"comment":"The paper reports no measures of uncertainty or significance. Table 2's counts (e.g., 72 languages where DevSet > DevLang for MORPH) and Table 3's average differences are presented as evidence of systematic overestimation, but no confidence intervals, significance tests, or multiple random seeds are provided. Given that the paper's recommendation rests on these systematic differences being real, the authors should report a paired significance test across languages (e.g., a signed-rank test) or bootstrap intervals. As it stands, the reader cannot determine whether the 72/23/8 split is likely to arise by chance.","section":"Tables 2 and 3"},{"comment":"The DevLang protocol requires equal training set sizes across languages, and the study is restricted to datasets satisfying this condition (Section 3 and Table 1). In many real-world low-resource settings, language datasets vary substantially in size, and a simple epoch average would not transfer directly. The paper does not discuss whether its conclusions generalize to such settings, which weakens the broad practical take-away in Section 6 and the abstract.","section":"Section 3, equal-size requirement"}],"minor_comments":[{"comment":"The text states 'We round this number to an integer,' but Tables 4–7 report non-integer DevLang epoch values such as 43.44 and 323.11; please clarify whether the main training phase used rounded integers or the exact averages.","section":"Section 3 and Tables 4-7"},{"comment":"The sentence 'We randomly choose ten development languages' for MORPH does not report a random seed or a sensitivity analysis; please provide the seed or show that the results do not depend on this particular choice.","section":"Section 4.2"},{"comment":"The axes labeled 'Δ epochs' and 'Δ accuracy' should be defined explicitly as DevLang minus DevSet, and a regression line or correlation coefficient should be included if the claim about 'longer training seems better for MORPH' is to be supported visually.","section":"Figure 1"},{"comment":"The final recommendation is phrased unconditionally ('should move away') despite the paper's own limitations section noting that dev sets may be sufficient for comparing methods in equal settings; please add conditions or soften the wording to align with the evidence.","section":"Section 6, 'Practical take-aways'"},{"comment":"The statement 'differences are as big as 18.0% accuracy' highlights a single outlier among over a hundred language-task pairs; consider also reporting the median and interquartile range of the differences to give a balanced picture.","section":"Abstract"},{"comment":"There is a typo in the reference for Schulz et al.: 'NAAACL' should be 'NAACL'; the same typo appears in Table 1 caption.","section":"References and Table 1"}],"recommendation":"major_revision","confidential_remarks":"This is a useful empirical study that raises a legitimate concern about evaluation practices in low-resource NLP. The main issue is that the DevLang baseline, used to represent the 'realistic' alternative, is a single heuristic that may be unrepresentative of what practitioners would actually do; the headline 18% difference could be an artifact of the mean-epoch choice. The revision should add sensitivity analyses (e.g., other ways of selecting the DevLang epoch count) and basic statistical assessments. With those additions, the paper's central claim could become convincing; without them, the conclusion is too strong for the evidence provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper does something genuinely useful: it re-runs three published low-resource NLP experiments under two training protocols—early stopping on a development set (DevSet) and a more realistic no-target-dev-set alternative (DevLang)—and measures how much the reported accuracy changes. That is a new, concrete measurement, and the paper's directional warning, that a large dev set can distort real-world performance estimates, survives its own caveats. Second, the quantitative headline, especially the 18.0% MORPH gap, is not as clean as it looks because the DevLang baseline is a single heuristic that fits many languages poorly.\n\nThe good parts are real. The experimental design is transparent and externally anchored: same code, same data, same hyperparameters as the original papers, with one modification. The per-language details are in the appendix, and the limitations paragraph explicitly says dev sets may be fine for comparing methods in equal settings and that over- or underestimation depends on how representative the dev set is. That measured framing is a credit to the authors.\n\nThe soft spots are concentrated in Section 3. DevLang averages the best epochs over development languages; in MORPH that average collapses to roughly 324 for almost all languages, while DevSet best epochs range from 28 to 881. The headline Azeri case has a DevSet best at epoch 217, so DevLang trains 107 epochs past its optimum and loses 18 points mostly because of a bad epoch choice, not because dev-set early stopping inherently misleads. Figure 1 shows the same scatter: the mean epoch is not a reliable predictor for individual languages. The paper would be substantially stronger if it compared against a few alternative no-dev-set strategies—median, a high quantile, or a stopping rule based on development-language learning curves. Without that, the quantitative claim is conditional on this specific baseline. The absence of error bars or significance tests is a minor additional weakness, not a fatal one.\n\nNet assessment: this is a legitimate empirical contribution and the practical warning is directionally correct—large dev sets in genuinely low-resource settings are unrealistic and can give misleading absolute numbers. But the current evidence supports a cautious recommendation to report both settings or to use more realistic baselines, not a blanket call to abandon dev sets. The paper deserves a serious referee and a revision that strengthens the alternative protocol before publication.","headline":"An honest empirical caution about dev-set early stopping in low-resource NLP, but the headline 18-point gap is largely an artifact of a weak mean-epoch baseline.","tokens_in":9803,"tokens_out":2411,"would_cite":true,"duration_ms":24353,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using a development set to stop training early can misreport low-resource NLP accuracy by up to 18 percentage points.","keywords":["low-resource NLP","development set","early stopping","realistic evaluation","cross-lingual tuning","morphological inflection","historical text normalization","transliteration"],"falsifier":"Rerun the Azeri MORPH setup with several different pools of development languages; if the DevLang epoch estimate shifts enough that the 18-point DevSet–DevLang gap does not reproduce, the claim that dev-set stopping reliably misestimates real-world performance would be undercut.","tokens_in":8914,"feed_emoji":"📉","tokens_out":11271,"duration_ms":91118,"temperature":0.7,"pith_summary":"Low-resource NLP papers often tune neural models by early stopping on a development set, even though in a true data-poor deployment those examples would more plausibly be used for training. This paper replicates three recent low-resource sequence-to-sequence studies—historical text normalization, morphological inflection, and transliteration—and compares the standard DevSet protocol with a realistic alternative in which the number of training epochs is chosen from other languages for the same task (DevLang). Across languages, average absolute accuracy differs by up to 1.4 percentage points, while individual languages differ by as much as 18.0 points, as in Azeri morphological inflection. The authors conclude that development-set early stopping gives a poor estimate of real-world performance and that low-resource research should move away from large development sets when real-world settings are the goal.","feed_headline":"Dev-set early stopping skews low-resource NLP scores by up to 18%","feed_subtitle":"No target-language dev data means published accuracies may not reflect real-world low-resource performance.","key_machinery":"The central mechanism is a two-phase training protocol that isolates the stopping signal. In the stopping-point selection phase, models for the development languages are trained with the original early-stopping rule, and the DevLang epoch for each target language is the rounded average of the best epochs across those other languages. In the main training phase, each target model is trained once, preserving the DevSet early-stopped checkpoint and the additional DevLang checkpoint at the precomputed epoch. Because both models come from the same training run, any accuracy difference is attributable to the stopping point rather than to random initialization or data variation.","core_discovery":"The central discovery is a measurement of what the stopping signal is worth in low-resource neural NLP, across morphological inflection (MORPH), historical text normalization (NORM), and transliteration (TRANSL). Training each model once and keeping two checkpoints—the early-stopped model chosen by the target language's development set (DevSet) and the model at the epoch averaged from development languages (DevLang)—the paper finds DevSet accuracy is higher in 72 of 103 MORPH languages and 8 of 10 NORM languages, while DevLang is higher in 2 of 5 TRANSL languages. Mean test accuracy is 51.3 vs 50.0 for MORPH, 74.9 vs 74.2 for NORM, and 21.8 vs 22.3 for TRANSL. The largest gap is 18.0 absolute accuracy points on Azeri, where DevSet stops at epoch 217 with 64% accuracy and DevLang runs to epoch 324 with 46%. The paper reads this as evidence that dev-set early stopping can either overestimate or underestimate real-world performance and should not be treated as a neutral evaluation choice.","pith_inferences":["Editorial inference: the DevLang rule is an unweighted cross-language average, so languages whose learning dynamics differ strongly could make the no-dev-set gap larger or smaller than the maxima reported.","Editorial inference: weighting development languages by proximity to the target language, or adapting the stopping epoch during training, could yield a stronger realistic baseline and shrink the measured gap.","Editorial inference: the same DevSet-versus-DevLang comparison could be run for tagging, parsing, and classification tasks to see whether the finding extends beyond sequence transduction."],"forward_implications":["Published absolute accuracies from low-resource experiments that early-stop on a relatively large development set can differ from a no-dev-set setup by up to 18 percentage points.","Tuning the number of training epochs on development languages is a workable realistic baseline and is the alternative the paper's conclusion points to.","For morphological inflection and historical text normalization, development-set early stopping usually overestimates performance, whereas for transliteration it can underestimate it.","Comparisons between proposed methods in low-resource NLP should report or control for the stopping regime, since the stopping rule alone can move results.","Development examples may be better used as training data when the goal is to estimate real-world performance rather than to compare methods in a fixed benchmark."],"supporting_citations":[{"why":"Supplies the historical text normalization datasets and hyperparameters replicated for the NORM experiments.","marker":"Bollmann et al. (2018)"},{"why":"Supplies the 103-language dataset released for the 2018 shared task on universal morphological reinflection, which defines the MORPH low-resource setting.","marker":"Cotterell et al. (2018)"},{"why":"Supplies the pointer-generator model, hyperparameters, and early-stopping schedule used for MORPH.","marker":"Sharma et al. (2018)"},{"why":"Supplies the transliteration datasets, model, hyperparameters, and early-stopping schedule.","marker":"Upadhyay et al. (2018)"},{"why":"Provides the sequence-to-sequence implementation and hyperparameter defaults used for the normalization experiments.","marker":"Klein et al. (2017)"},{"why":"Provides the attention-based encoder-decoder architecture underlying the NORM and TRANSL models.","marker":"Bahdanau et al. (2015)"},{"why":"Provides the pointer-generator network with copy mechanism used for morphological inflection.","marker":"See et al. (2017)"},{"why":"Provides the hard monotonic attention mechanism used in the transliteration model.","marker":"Aharoni and Goldberg (2017)"}],"fun_headline_variants":["Dev-set early stopping can flip low-resource NLP scores by 18%","Low-resource NLP dev-set bias: up to 18% accuracy shift","Realistic low-resource NLP: dev-set early stopping misleads","Using dev sets in low-resource NLP skews results by 18%","Low-resource NLP: dev-set stop point changes accuracy by 18%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the rounded average of the best stopping epochs across development languages is a realistic stand-in for the stopping decision a practitioner without a target-language development set would actually make.","fun_headline_variants_meta":{"raw":{"variants":["Dev-set early stopping can flip low-resource NLP scores by 18%","Low-resource NLP dev-set bias: up to 18% accuracy shift","Realistic low-resource NLP: dev-set early stopping misleads","Using dev sets in low-resource NLP skews results by 18%","Low-resource NLP: dev-set stop point changes accuracy by 18%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1331,"prompt_tokens":950,"completion_tokens":381,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":281}},"tokens_in":566,"tokens_out":381,"duration_ms":3672,"temperature":1.0,"reasoning_tokens":281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:14:37.088120+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Azeri MORPH setup with several different pools of development languages; if the DevLang epoch estimate shifts enough that the 18-point DevSet–DevLang gap does not reproduce, the claim that dev-set stopping reliably misestimates real-world performance would be undercut.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the historical text normalization datasets and hyperparameters replicated for the NORM experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 103-language dataset released for the 2018 shared task on universal morphological reinflection, which defines the MORPH low-resource setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pointer-generator model, hyperparameters, and early-stopping schedule used for MORPH."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the transliteration datasets, model, hyperparameters, and early-stopping schedule."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sequence-to-sequence implementation and hyperparameter defaults used for the normalization experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pointer-generator network with copy mechanism used for morphological inflection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the hard monotonic attention mechanism used in the transliteration model."}],"review_version":1}