{"id":"9f58e058-25a0-488f-84b7-8ceeb5e1a212","arxiv_id":"2506.05227","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Autoencoding is the best auxiliary objective for morphological inflection when unlabeled data is tiny; character-level masked language modeling wins with more unlabeled data, and segment oracle masking helps further.","lead":"The paper compares 13 self-supervised objectives for training a small transformer to inflect words in 19 very low-resource languages. The results give practical guidance for language documentation: copy the word when unlabeled data is tiny, switch to masked-character training as unlabeled data grows, and mask known morpheme pieces when possible.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AE-vs-MLM 'data availability' trend in Table 3 is confounded: rows differ in supervised-set size, sampling with/without replacement, filtering, and source corpora, so the central conclusion is not uniquely supported.","rationale":"The paper is a focused empirical study with a clear experimental apparatus, and the segment-masking result on 5 languages is useful but secondary. The main load-bearing issue is the inference from Table 3 to a monotone 'data availability' trend. The reader's concern is on target: the row labels change several properties at once. In addition to the factors the reader lists, ud-1k differs from ud-200 in supervised-set size (1k vs 200), while the unlabeled UD sample is essentially the same, so even the first step of the trend is not an unlabeled-data manipulation. A controlled scaling sweep that varies only unique unlabeled types, or a matched with/without-replacement comparison on the same unique-type pool, would settle it. The absence of variance estimates makes the small margins (AE vs cmlm-iid in ud-1k: 1.16; cmlm-iid vs AE in ud-wl-NR: 1.19) additionally hard to interpret, but the confound is the more fundamental problem. Given the otherwise reasonable methodology and practical takeaways, the correct response is to keep the conditional verdict: the paper should be accepted only if the conclusion is reframed or the controlled experiment is run. No change from the reader's verdict.","tokens_in":20156,"tokens_out":7974,"duration_ms":80509,"concrete_test":"Run a matched scaling experiment on the 19 languages: fix the supervised set at the 200-per-POS ud-200 samples; from a single UD pool with fixed filters and sources (e.g., the ud-wl-NR pools), sample unlabeled words without replacement at 1k, 2k, 3k, and 5k unique types; train AE and cmlm-iid with identical hyperparameters and at least 3 seeds. If cmlm-iid overtakes AE as unique-type count increases with all else fixed, the trend is real. As a second arm, hold the 5k unique-type pool fixed and compare it against 5k draws with replacement from the same pool; if the rank order flips between the two arms, the 'growth' effect is actually a duplicate-removal/unique-type effect, and the paper's wording must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that AE dominates with very limited unlabeled data and MLM/CMLM dominates as unlabeled data grows (Conclusion; Section 6.2). The support is the row-wise ordering in Table 3, but the rows are not a controlled size sweep. (i) ud-1k vs ud-200 changes the supervised set from 1k to 200 examples while keeping the same 5k unlabeled UD sample, so the first two rows speak to supervised-data size, not unlabeled-data size. (ii) Moving from ud-200 to ud-wl / ud-vnadj changes word-length and POS filters; moving to ud-wl-NR / ud-vnadj-NR additionally changes sampling with replacement to without replacement, raises Types from about 3.2k to about 5.0k, and for some languages adds web-scraped English EWT and Turkish BOUN tokens (Appendix A). Thus 'data availability' is not the only variable that changes where the CMLM overtake of AE appears (Tables 1 and 3). The same ordering could be driven by duplicate removal, n-gram diversity (N-grams jump from about 4.7k to 6.3-7.6k), or noise from web data. Without a test that manipulates unlabeled unique-type count while holding supervision, filtering, source, and sampling fixed, the headline trend is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies self-supervised auxiliary objectives for low-resource morphological inflection. Using encoder-decoder transformers on 19 languages, the authors compare autoencoding (AE) with character-level masked language modeling (CMLM) and T5-style span denoising under iid, suffix-biased, prefix-biased, and oracle morpheme-segment masking, and with deletion variants. The headline claims are that AE is best when the unlabeled pool is very small, that CMLM-based denoising overtakes AE as unlabeled data grows, that segment-based masking reliably improves CMLM, and that stronger inductive biases affect predictions in interpretable ways but rarely beat uniform CMLM. The paper also presents a copying analysis that links AE to lemma copying and denoising to training-output n-gram generation.","tokens_in":20490,"tokens_out":7741,"duration_ms":85049,"significance":"The paper asks a well-motivated question and provides a broad, systematic comparison: 19 languages, 13 objective/dataset variants, per-language tables, and an explicit analysis of copying behavior. These empirical resources are valuable and largely reproducible in structure. If the central trend were established, the paper would offer a practical data-dependent recipe for choosing AE vs MLM in low-resource settings. However, the headline trend is currently confounded by simultaneous changes in supervised set size, filtering, sampling scheme, and source corpora across the rows of Table 3, and one 'always' claim is contradicted by the paper's own Table 4. The paper's contribution is therefore promising but needs a controlled unlabeled-size experiment and more cautious claims before the main conclusions can be accepted.","major_comments":[{"comment":"The headline trend in the Conclusion ('for small sets of unlabeled words, autoencoding is a strong objective... tasks based on MLM perform better as the unlabeled dataset grows') is not established, because the supporting rows are not a controlled sweep of unlabeled data availability. The ud-1k to ud-200 comparison changes the supervised set from 1k to 200-600 examples while keeping the same 5k unlabeled sample; the ud-200 to ud-wl/ud-vnadj comparisons change word-length and part-of-speech filters; and the ud-wl-NR/ud-vnadj-NR rows additionally change sampling from with-replacement to without-replacement, roughly doubling the number of unique types (Table 1: Types 2659 vs 4903/5060), increasing n-gram counts, and adding English-EWT and Turkish-BOUN web-derived tokens (Appendix A). The performance ordering in Table 3 could therefore be driven by deduplication, lexical diversity, or web-data noise rather than by unlabeled data size. Please add an experiment that varies the number of unique unlabeled types while holding supervision, filtering, source corpus, and sampling scheme fixed.","section":"5.1; Tables 1 and 3"},{"comment":"The claim that 'When using the CMLM objective, the segmentation variant ... always outperforms the corresponding setup without segmentation' is contradicted by Table 4: Russian cmlm-pref is 69.50 while cmlm-seg-pref is 68.60; Italian cmlm-suff is 51.90 while cmlm-seg-suff is 47.90; and English cmlm-iid is 90.60 while cmlm-seg-iid is 90.50. The Abstract's 'sampling masks based on known morpheme boundaries consistently improves performance' is likewise too strong given these exceptions. The text should report the average improvement and explicitly discuss the per-language cases where the segment oracle hurts.","section":"Section 6; Table 4"},{"comment":"The experiments appear to be single runs with no variance or significance information, and several load-bearing differences are small. On ud-wl-NR the best denoiser (cmlm-iid, 51.68) beats AE (50.49) by 1.19 points; on ud-vnadj-NR t5-iid (51.48) beats AE (50.06) by 1.42 points. Without standard errors, paired significance tests, or multiple seeds, the statement that 'denoising clearly outperforms autoencoding' is not supported. The same concern applies to the aggregate CMLM-vs-T5 comparison in Figure 3a, where the reported averages may be within run-to-run noise.","section":"Section 6; Tables 3 and 5-10"}],"minor_comments":[{"comment":"The Hungarian row reports cmlm-prefix = 3.40, an extreme outlier relative to the same condition in Tables 6-10 (e.g., 55.30 in ud-200 and 55.80 in ud-wl). The column mean is consistent with the printed value, so this appears to be an anomalous run rather than a simple typo; please verify the run or report multiple seeds.","section":"Table 5"},{"comment":"The sentence 'autoencoding is a strong objective due to its inductive bias towards copying form the source string' contains a typo: 'form' should be 'from'.","section":"Conclusion"},{"comment":"The heading 'Exact hyperparamters' contains a typo; it should read 'hyperparameters'.","section":"Appendix C"},{"comment":"The caption says results are 'averaged over all datasets,' but the right panel uses only the 5-language segmentation dataset; the caption should state this distinction explicitly.","section":"Figure 2"},{"comment":"In Example (5), the deletion example 'bake -> baked' is clear, but the text should state explicitly that the self-supervised deletion task uses no inflection tag, mirroring the [TASK] notation in Eq. (1), to avoid confusion with supervised inflection pairs.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper's central empirical claim depends partly on two prior papers by the same group for the CMLM objective and the ud-1k baseline; I do not see this as grounds for rejection, and the per-language tables mitigate the concern. I would, however, ask the authors to state explicitly whether experimental configurations and per-run seeds will be released, since the number of conditions (19 languages x multiple objectives x 6 datasets) makes reproducibility a practical issue. The fit with a computational linguistics journal is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this is a solid, systematic empirical study of self-supervised auxiliary objectives for morphological inflection, and the practical takeaway—use autoencoding when you have very few unlabeled words, switch to character MLM as the pool grows—is plausible and mostly supported. But the headline trend is not as clean as the text makes it, and one bold claim in the abstract is contradicted by the paper's own per-language results.\n\nWhat's genuinely new: the sweep itself—13 objectives, 19 languages, per-language tables in the appendix—and a nice copy-bias analysis showing AE models copy more from the lemma while CMLM generates more from training-data trigrams. That mechanistic evidence is the best part. The UD dataset variants, word-length/POS filtering, and without-replacement sampling are useful probes, even if they don't isolate a single variable.\n\nThe soft spots: the central claim—that autoencoding dominates with small unlabeled sets and MLM takes over as unlabeled data grows—rests on comparing datasets that differ in several ways at once. Moving from ud-200 to ud-wl changes word-length filtering; ud-vnadj changes POS filtering; ud-wl-NR and ud-vnadj-NR also switch from sampling with replacement to without, which raises unique types from about 3.2k to 5k, and for English and Turkish adds web-scraped corpora with known noise. The accuracy ordering in Table 3 could be driven by duplicate removal, lexical diversity, or noise, not by 'data availability' per se. A controlled sweep that varies only the number of unique unlabeled words while holding everything else fixed would establish the claim.\n\nAlso, the abstract says segment masking 'consistently improves performance,' but Table 4 shows rus/prefix, ita/suffix, and eng/iid where the segment variant is worse than its non-segment counterpart. The average is better, so 'consistently' is too strong. And there are no error bars or released code/data; for a comparative study, that's a real limitation.\n\nWho benefits: people working on low-resource morphology or character-level transduction will find this a useful reference for choosing auxiliary tasks. It deserves a serious referee, but the revision should fix the overclaims, add a controlled size sweep, and report variance. I wouldn't desk-reject it.","headline":"Solid empirical sweep with a useful AE-to-MLM recipe, but the data-availability trend is confounded by simultaneous dataset changes and the 'consistently' segment-masking claim fails on its own Table 4.","tokens_in":21006,"tokens_out":3615,"would_cite":true,"duration_ms":38361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When unlabeled word lists are tiny, training a morphological-inflection model to copy words beats character masked language modeling; as the unlabeled lexicon grows, masked modeling takes over, and oracle morpheme-mask sampling helps most.","keywords":["morphological inflection","self-supervised learning","low-resource NLP","autoencoding","character masked language modeling","multitask training","language documentation","encoder-decoder transformer"],"falsifier":"Re-train the same 19-language models on unlabeled sets that hold every property fixed except the number of unique word types, for instance by subsampling the ud-wl-NR sets to 2,000, 3,000, and 4,000 unique types. If autoencoding no longer wins at the smallest unique-type counts, or if denoising does not improve monotonically with unique types, the data-availability explanation is wrong.","tokens_in":19974,"feed_emoji":"🔤","tokens_out":5420,"duration_ms":56982,"temperature":0.7,"pith_summary":"The paper asks which self-supervised auxiliary objective best helps a character-level encoder-decoder transformer learn morphological inflection when labeled data is extremely scarce. Training on 19 languages with 13 objectives and unlabeled sets of around 5,000 words, it finds that pure autoencoding—training the model to copy an unlabeled word unchanged—gives the highest inflection accuracy when the unlabeled vocabulary is very small. As the unlabeled dataset grows in unique word types, character masked language modeling (masking characters and reconstructing them) overtakes autoencoding. The paper also reports that masking entire morphemes instead of individual characters consistently improves accuracy on the five languages where gold morpheme boundaries are available.","feed_headline":"Autoencoding beats masked language modeling on tiny word lists","feed_subtitle":"Choosing the right auxiliary task for low-resource inflection depends on how many unique words you have.","key_machinery":"The load-bearing object is the multitask training setup: a small encoder-decoder transformer is trained simultaneously on supervised inflection pairs (lemma plus inflection tags to inflected form) and on a self-supervised task over unlabeled words, with the inflection tags replaced by a special task symbol. The decisive mechanism is the difference between autoencoding, which presents an unchanged word as both source and target and thereby teaches copying, and character masked language modeling (CMLM), which masks 25% of characters and teaches the model to regenerate sequences from the training distribution. The third component is the mask sampling strategy: iid uniform character masking, suffix- or prefix-skewed masking, deletion, and oracle morpheme-segment masking.","core_discovery":"On the paper's own terms, the central discovery is a data-availability ranking of auxiliary objectives for low-resource morphological inflection. With very limited unlabeled data (a few thousand tokens with many duplicates), autoencoding is the strongest auxiliary task, and the paper attributes this to an inductive bias toward copying the source string. With more diverse unlabeled data (roughly 5,000 unique word types), character masked language modeling becomes more effective, because it pushes the model to generate character sequences beyond those it copied. A secondary discovery is that sampling masks according to known morpheme boundaries beats character-level masking for the CMLM objective on concatenative languages, while almost all other hand-designed inductive biases—suffix/prefix mask skewing, deletion, and span merging—fail to beat uniform character masking on average.","pith_inferences":["Because the AE-to-MLM crossover is established across datasets that change lexical diversity, sampling without replacement, and part-of-speech filtering at the same time, the cleanest extension is to vary unique-type count alone while holding all other dataset properties fixed.","The morpheme-mask result suggests a practical research direction: approximate morpheme boundaries from raw text rather than using a gold oracle, and use those segments to guide mask sampling; the paper does not test this unsupervised approximation.","The paper's trigram analysis implies a testable prediction: autoencoding models' errors should be dominated by over-copying, while CMLM errors should come from generating plausible but wrong sequences; this can be checked on error sets beyond the languages analyzed."],"forward_implications":["Practitioners with fewer than a few thousand unlabeled word tokens and little supervision should choose autoencoding as the auxiliary task.","When an unlabeled lexicon of several thousand unique word types is available, character masked language modeling is the safer default, with gains up to roughly 9 absolute accuracy points over autoencoding.","If morpheme boundaries can be obtained or approximated, sampling masks over whole morphemes under CMLM is the best-performing configuration tested.","Uniform masking beats linguistically skewed suffix and prefix masks on average, so for unknown morphology, high mask variance is more valuable than a typological prior."],"supporting_citations":[{"why":"Supplies the exact ud-1k benchmark, the multitask setup, and the autoencoding baseline that beats denoising at 1k supervised samples; this paper reproduces and extends that result.","marker":"Purushothama et al. (2024)"},{"why":"Defines masked language modeling, the base objective the paper adapts to the character level as CMLM.","marker":"Devlin et al. (2019)"},{"why":"Supplies the dynamic masking procedure (80% mask, 10% random, 10% unchanged) that CMLM inherits.","marker":"Liu et al. (2019)"},{"why":"Provides the span-masked T5 denoising objective that the paper compares against CMLM and autoencoding.","marker":"Raffel et al. (2020)"},{"why":"Proposes tuning copying bias in inflection models, the hypothesis the paper invokes to explain autoencoding's success.","marker":"Liu and Hulden (2022)"},{"why":"Confirms that training inflection models to copy synthetic words improves generalization, supporting the copy-bias interpretation.","marker":"Yang et al. (2022)"},{"why":"Introduces the specific CMLM formulation the paper uses as its character-level masking objective.","marker":"Wiemerslage et al. (2023)"},{"why":"Supplies the SIGMORPHON 2023 shared task data used for all supervised inflection training and evaluation.","marker":"Goldman et al. (2023)"},{"why":"Supplies the gold morpheme segmentation data used to build the oracle segment-masking datasets.","marker":"Batsuren et al. (2022)"},{"why":"Supplies the Universal Dependencies treebanks from which the unlabeled word sets are sampled.","marker":"Zeman et al. (2023)"}],"fun_headline_variants":["Data size decides: autoencode vs mask for low-resource inflection","Morpheme-aware masking lifts low-resource inflection","Autoencoding wins when word lists are tiny","For scarce data, copy beats fill-in-the-blank"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The strongest trend—that autoencoding is better when unlabeled data is tiny while masked language modeling wins when it is larger—is read from datasets that change sampling with and without replacement, word-length filtering, part-of-speech filtering, and unique vocabulary size at the same time; if those other changes, not unlabeled-data size, drive the ordering, the crossover claim is not established.","fun_headline_variants_meta":{"raw":{"variants":["Data size decides: autoencode vs mask for low-resource inflection","Morpheme-aware masking lifts low-resource inflection","Autoencoding wins when word lists are tiny","For scarce data, copy beats fill-in-the-blank"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000727,"raw_usage":{"total_tokens":3210,"prompt_tokens":850,"completion_tokens":2360,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":2294}},"tokens_in":466,"tokens_out":2360,"duration_ms":19362,"temperature":1.0,"reasoning_tokens":2294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:22:35.812216+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train the same 19-language models on unlabeled sets that hold every property fixed except the number of unique word types, for instance by subsampling the ud-wl-NR sets to 2,000, 3,000, and 4,000 unique types. If autoencoding no longer wins at the smallest unique-type counts, or if denoising does not improve monotonically with unique types, the data-availability explanation is wrong.","supporting_citations":[],"review_version":1}