Pith. sign in

REVIEW 3 major objections 5 minor 14 references

Improving Low-Resource Morphological Inflection via Self-Supervised Objectives

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read When unlabeled word lists are tiny, training a morphological-inflection model to copy words beats character masked language modeling; as the unlabeled lexicon grows, masked modeling takes over, and oracle morpheme-mask sampling helps most.

desk verdict Solid empirical sweep with a useful AE-to-MLM recipe, but the data-availability trend is confounded by simultaneous dataset changes and the 'consistently' segment-masking claim fails on its own Table 4. read the letter →

arxiv 2506.05227 v1 pith:6I7BSLWB submitted 2025-06-05 cs.CL

Adam Wiemerslage , Katharina von der Wense This is my paper
classification cs.CL
keywords morphologicalinflectionself-supervisedlearninglow-resourceNLPautoencodingcharactermaskedlanguagemodelingmultitasktrainingdocumentationencoder-decodertransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks which self-supervised auxiliary objective best helps a character-level encoder-decoder transformer learn morphological inflection when labeled data is extremely scarce. Training on 19 languages with 13 objectives and unlabeled sets of around 5,000 words, it finds that pure autoencoding—training the model to copy an unlabeled word unchanged—gives the highest inflection accuracy when the unlabeled vocabulary is very small. As the unlabeled dataset grows in unique word types, character masked language modeling (masking characters and reconstructing them) overtakes autoencoding. The paper also reports that masking entire morphemes instead of individual characters consistently improves accuracy on the five languages where gold morpheme boundaries are available.

What carries the argument

The load-bearing object is the multitask training setup: a small encoder-decoder transformer is trained simultaneously on supervised inflection pairs (lemma plus inflection tags to inflected form) and on a self-supervised task over unlabeled words, with the inflection tags replaced by a special task symbol. The decisive mechanism is the difference between autoencoding, which presents an unchanged word as both source and target and thereby teaches copying, and character masked language modeling (CMLM), which masks 25% of characters and teaches the model to regenerate sequences from the training distribution. The third component is the mask sampling strategy: iid uniform character masking, suffix- or prefix-skewed masking, deletion, and oracle morpheme-segment masking.

What would settle it

Re-train the same 19-language models on unlabeled sets that hold every property fixed except the number of unique word types, for instance by subsampling the ud-wl-NR sets to 2,000, 3,000, and 4,000 unique types. If autoencoding no longer wins at the smallest unique-type counts, or if denoising does not improve monotonically with unique types, the data-availability explanation is wrong.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a data-availability ranking of auxiliary objectives for low-resource morphological inflection. With very limited unlabeled data (a few thousand tokens with many duplicates), autoencoding is the strongest auxiliary task, and the paper attributes this to an inductive bias toward copying the source string. With more diverse unlabeled data (roughly 5,000 unique word types), character masked language modeling becomes more effective, because it pushes the model to generate character sequences beyond those it copied. A secondary discovery is that sampling masks according to known morpheme boundaries beats character-level masking for the CMLM objective on concatenative languages, while almost all other hand-designed inductive biases—suffix/prefix mask skewing, deletion, and span merging—fail to beat uniform character masking on average.

Load-bearing premise

The strongest trend—that autoencoding is better when unlabeled data is tiny while masked language modeling wins when it is larger—is read from datasets that change sampling with and without replacement, word-length filtering, part-of-speech filtering, and unique vocabulary size at the same time; if those other changes, not unlabeled-data size, drive the ordering, the crossover claim is not established.

Editorial extensions

If this is right

  • Practitioners with fewer than a few thousand unlabeled word tokens and little supervision should choose autoencoding as the auxiliary task.
  • When an unlabeled lexicon of several thousand unique word types is available, character masked language modeling is the safer default, with gains up to roughly 9 absolute accuracy points over autoencoding.
  • If morpheme boundaries can be obtained or approximated, sampling masks over whole morphemes under CMLM is the best-performing configuration tested.
  • Uniform masking beats linguistically skewed suffix and prefix masks on average, so for unknown morphology, high mask variance is more valuable than a typological prior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the AE-to-MLM crossover is established across datasets that change lexical diversity, sampling without replacement, and part-of-speech filtering at the same time, the cleanest extension is to vary unique-type count alone while holding all other dataset properties fixed.
  • The morpheme-mask result suggests a practical research direction: approximate morpheme boundaries from raw text rather than using a gold oracle, and use those segments to guide mask sampling; the paper does not test this unsupervised approximation.
  • The paper's trigram analysis implies a testable prediction: autoencoding models' errors should be dominated by over-copying, while CMLM errors should come from generating plausible but wrong sequences; this can be checked on error sets beyond the languages analyzed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies self-supervised auxiliary objectives for low-resource morphological inflection. Using encoder-decoder transformers on 19 languages, the authors compare autoencoding (AE) with character-level masked language modeling (CMLM) and T5-style span denoising under iid, suffix-biased, prefix-biased, and oracle morpheme-segment masking, and with deletion variants. The headline claims are that AE is best when the unlabeled pool is very small, that CMLM-based denoising overtakes AE as unlabeled data grows, that segment-based masking reliably improves CMLM, and that stronger inductive biases affect predictions in interpretable ways but rarely beat uniform CMLM. The paper also presents a copying analysis that links AE to lemma copying and denoising to training-output n-gram generation.

Significance. The paper asks a well-motivated question and provides a broad, systematic comparison: 19 languages, 13 objective/dataset variants, per-language tables, and an explicit analysis of copying behavior. These empirical resources are valuable and largely reproducible in structure. If the central trend were established, the paper would offer a practical data-dependent recipe for choosing AE vs MLM in low-resource settings. However, the headline trend is currently confounded by simultaneous changes in supervised set size, filtering, sampling scheme, and source corpora across the rows of Table 3, and one 'always' claim is contradicted by the paper's own Table 4. The paper's contribution is therefore promising but needs a controlled unlabeled-size experiment and more cautious claims before the main conclusions can be accepted.

major comments (3)
  1. [5.1; Tables 1 and 3] The headline trend in the Conclusion ('for small sets of unlabeled words, autoencoding is a strong objective... tasks based on MLM perform better as the unlabeled dataset grows') is not established, because the supporting rows are not a controlled sweep of unlabeled data availability. The ud-1k to ud-200 comparison changes the supervised set from 1k to 200-600 examples while keeping the same 5k unlabeled sample; the ud-200 to ud-wl/ud-vnadj comparisons change word-length and part-of-speech filters; and the ud-wl-NR/ud-vnadj-NR rows additionally change sampling from with-replacement to without-replacement, roughly doubling the number of unique types (Table 1: Types 2659 vs 4903/5060), increasing n-gram counts, and adding English-EWT and Turkish-BOUN web-derived tokens (Appendix A). The performance ordering in Table 3 could therefore be driven by deduplication, lexical diversity, or web-data noise rather than by unlabeled data size. Please add an experiment that varies the number of unique unlabeled types while holding supervision, filtering, source corpus, and sampling scheme fixed.
  2. [Section 6; Table 4] The claim that 'When using the CMLM objective, the segmentation variant ... always outperforms the corresponding setup without segmentation' is contradicted by Table 4: Russian cmlm-pref is 69.50 while cmlm-seg-pref is 68.60; Italian cmlm-suff is 51.90 while cmlm-seg-suff is 47.90; and English cmlm-iid is 90.60 while cmlm-seg-iid is 90.50. The Abstract's 'sampling masks based on known morpheme boundaries consistently improves performance' is likewise too strong given these exceptions. The text should report the average improvement and explicitly discuss the per-language cases where the segment oracle hurts.
  3. [Section 6; Tables 3 and 5-10] The experiments appear to be single runs with no variance or significance information, and several load-bearing differences are small. On ud-wl-NR the best denoiser (cmlm-iid, 51.68) beats AE (50.49) by 1.19 points; on ud-vnadj-NR t5-iid (51.48) beats AE (50.06) by 1.42 points. Without standard errors, paired significance tests, or multiple seeds, the statement that 'denoising clearly outperforms autoencoding' is not supported. The same concern applies to the aggregate CMLM-vs-T5 comparison in Figure 3a, where the reported averages may be within run-to-run noise.
minor comments (5)
  1. [Table 5] The Hungarian row reports cmlm-prefix = 3.40, an extreme outlier relative to the same condition in Tables 6-10 (e.g., 55.30 in ud-200 and 55.80 in ud-wl). The column mean is consistent with the printed value, so this appears to be an anomalous run rather than a simple typo; please verify the run or report multiple seeds.
  2. [Conclusion] The sentence 'autoencoding is a strong objective due to its inductive bias towards copying form the source string' contains a typo: 'form' should be 'from'.
  3. [Appendix C] The heading 'Exact hyperparamters' contains a typo; it should read 'hyperparameters'.
  4. [Figure 2] The caption says results are 'averaged over all datasets,' but the right panel uses only the 5-language segmentation dataset; the caption should state this distinction explicitly.
  5. [Section 4] In Example (5), the deletion example 'bake -> baked' is clear, but the text should state explicitly that the self-supervised deletion task uses no inflection tag, mirroring the [TASK] notation in Eq. (1), to avoid confusion with supervised inflection pairs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical comparison whose central claims rest on new experiments; self-citations are used only as baselines or method definitions, not as proofs of the main result.

full rationale

This paper does not present a derivation chain; it is an empirical study comparing auxiliary self-supervised objectives for morphological inflection. The central claim—that autoencoding is stronger with very limited unlabeled data and that character masked language modeling becomes more effective as unlabeled data grows—is based on the measured accuracies in Table 3 and the dataset statistics in Table 1. There is no equation-level reduction: the auxiliary objectives (AE, CMLM, T5-style span masking, segment masking) are defined independently of the inflection task, and the downstream accuracy is not a fitted parameter of those definitions. The main self-citations are to Wiemerslage et al. (2023) for the CMLM objective and to Purushothama et al. (2024) for the ud-1k setup and a baseline number. These are not load-bearing in a circular sense: the CMLM implementation is described in the paper itself, and the ud-1k baseline is either reproduced or explicitly marked as taken from published results, with the paper's new claims about lower-resource settings coming from its own new runs. The 'copying' analysis in Section 7 independently measures trigram provenance and is not assumed into the conclusion. The methodological concern that the AE-versus-MLM trend is confounded with sampling, filtering, and corpus differences is a validity threat, not a circularity: it does not make the conclusion equal to its inputs by definition. No pattern from the circularity taxonomy applies, and the self-citations do not rise to load-bearing status.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

All free parameters are fixed design choices rather than fitted constants; no derived result is tuned to match a target. The domain assumptions are about data and benchmark validity, and the paper's own Limitations section concedes the benchmark-proxy assumption. No new physical or linguistic entities are introduced.

free parameters (4)
  • mask sampling rate = 25% of character positions
    Chosen for CMLM from prior work (Wiemerslage et al. 2023), not optimized here; main comparisons compare objectives at this fixed rate.
  • suffix/prefix skew probability = 95% over final/initial third, 5% over rest
    Hand-set heuristic for skewing masks toward word edges; the results for prefix/suffix strategies depend on this arbitrary split.
  • supervised sample size = 200 inflection pairs per part of speech
    This simulates the low-resource setting; performance differences across datasets are tied to this choice.
  • unlabeled pool size = 5000 words (fewer for amh and san)
    Arbitrary scale for the external lexicon; the AE-to-MLM crossover likely depends on this size.
assumptions (4)
  • domain assumption UD treebank word lists are a valid source of unlabeled data for morphological inflection.
    Section 5.1 samples 5k words from UD treebanks; if these words are not representative of inflected forms (for example short, repeated, or noisy tokens), the auxiliary tasks may not actually help inflection.
  • domain assumption Gold morpheme boundaries from SIGMORPHON 2022 segmentation data provide useful oracle supervision.
    Section 5.1 and Appendix B construct the segment-masking dataset from canonical and surface segmentation; the segment-masking improvement is measured only for 5 concatenative languages.
  • domain assumption Accuracy on the SIGMORPHON 2023 benchmark is a proxy for usefulness in language documentation.
    Stated in Limitations: the authors 'test all methods on a standard benchmark by simulating a low-resource scenario'; real documentation-language conditions may differ.
  • domain assumption Multitask training with summed losses transfers self-supervised objectives to the supervised inflection task.
    Section 5.2 sums supervised and self-supervised losses in the same batch; without this assumption the comparison of objectives would be hard to interpret.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Low-Resource Morphological Inflection via Self-Supervised Objectives." pith.science (2026). https://pith.science/paper/6I7BSLWB

@misc{pith2026250605227,
  author       = {Pith},
  title        = {Pith review of: Improving Low-Resource Morphological Inflection via Self-Supervised Objectives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6I7BSLWB}},
  note         = {Machine review of arXiv:2506.05227}
}
read the original abstract

Self-supervised objectives have driven major advances in NLP by leveraging large-scale unlabeled data, but such resources are scarce for many of the world's languages. Surprisingly, they have not been explored much for character-level tasks, where smaller amounts of data have the potential to be beneficial. We investigate the effectiveness of self-supervised auxiliary tasks for morphological inflection -- a character-level task highly relevant for language documentation -- in extremely low-resource settings, training encoder-decoder transformers for 19 languages and 13 auxiliary objectives. Autoencoding yields the best performance when unlabeled data is very limited, while character masked language modeling (CMLM) becomes more effective as data availability increases. Though objectives with stronger inductive biases influence model predictions intuitively, they rarely outperform standard CMLM. However, sampling masks based on known morpheme boundaries consistently improves performance, highlighting a promising direction for low-resource morphological modeling.

Figures

Figures reproduced from arXiv: 2506.05227 by the authors.

Figure 1
Figure 1. Example of the multitask training; left: in￾flection with the past tense tag; right: self-supervised masked language modeling. @ represents the mask to￾ken; [TASK] is a special tag for the self-supervised task. inherent inductive bias towards particular tasks like inflection (Aharoni and Goldberg, 2017; Makarov and Clematide, 2018; Wu and Cotterell, 2019). Instead, we take inspiration from high-resource NLP where pr… view at source ↗
Figure 2
Figure 2. Results for masking v.s. deleting for all models averaged over all datasets. On the right are the segmentation [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Average accuracy over all ud datasets when masking. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Language specific results for the ud-vnadj-NR dataset. Bars represent the max test accuracy of each [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The percent of trigams in predicted inflections that were copied from the lemma or from the training [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

  1. [6]

    In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 176–203, Seattle, Washing- ton

    SIGMORPHON– UniMorph 2022 shared task 0: Generalization and typologically diverse morphological inflection. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 176–203, Seattle, Washing- ton. Association for Computational Linguistics. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvinine...

    work page 2022

  2. [11]

    InFindings of the Association for Computational Linguistics: ACL 2023, pages 3351–3365

    An investigation of noise in morphological inflection. InFindings of the Association for Computational Linguistics: ACL 2023, pages 3351–3365. Shijie Wu and Ryan Cotterell

    work page 2023

  3. [12]

    InProceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main V olume, pages 1901–1907, Online

    Ap- plying the transformer to character-level transduction. InProceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main V olume, pages 1901–1907, Online. Association for Computational Linguistics. Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and ...

    work page 1901

  4. [13]

    @" and "www

    Universal dependencies 2.12. LINDAT/CLARIAH- CZ digital library at the Institute of Formal and Ap- plied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. Appendix A UD Data When we sample from UD data, we use exactly the corpora from Purushothama et al. (2024) except for two cases. For both English and Turkish, there is not suff...

    work page 2024

  5. [400]

    When masking unlabeled data, we always sample 25% of tokens for masking

    We train with the Warmup Inverse Square Root schedule with 4k warmup steps. When masking unlabeled data, we always sample 25% of tokens for masking. All experiments are trained on NVIDIA GPUs and implemented with yoyodyne1 (Wiemerslage et al., 2024). 1https://github.com/CUNY-CL/yoyodyne t5-prefix cmlm-prefix t5-suffix cmlm-suffix t5-iid cmlm-iid AE afb75....

    work page arXiv 2024

  6. [2014]

    Adam: A method for stochas- tic optimization.arXiv preprint arXiv:1412.6980. Jordan Kodner, Salam Khalifa, Khuyagbaatar Bat- suren, Hossep Dolatian, Ryan Cotterell, Faruk Akkus, Antonios Anastasopoulos, Taras Andrushko, Arya- man Arora, Nona Atanalov, Gábor Bella, Elena Budianskaya, Yustinus Ghanggo Ate, Omer Gold- man, David Guriel, Simon Guriel, Silvia ...

    arXiv

  7. [2017]

    InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (V ol- ume 1: Long Papers), pages 2004–2015

    Morphologi- cal inflection generation with hard monotonic atten- tion. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (V ol- ume 1: Long Papers), pages 2004–2015. Khuyagbaatar Batsuren, Gábor Bella, Aryaman Arora, Viktor Martinovic, Kyle Gorman, Zdenˇek Žabokrt- ský, Amarsanaa Ganbold, Šárka Dohnalová, Magda Ševˇ...

    work page 2004

  8. [2018]

    InProceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2877–2882, Brussels, Belgium

    Imitation learning for neural morphological string transduction. InProceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2877–2882, Brussels, Belgium. Association for Com- putational Linguistics. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi

    work page 2018

Show all 14 references
  1. [2019]

    BERT: Pre-training of deep bidirectional transformers for language under- standing. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long and Short Papers), pages 4171...

    2019

  2. [2020]

    InProceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5251–5262, Online

    IGT2P: From in- terlinear glossed texts to paradigms. InProceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5251–5262, Online. Association for Computational Linguistics. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carro...

    2020

  3. [2021]

    Adam Wiemerslage, Kyle Gorman, and Katharina von der Wense

    Finetuned lan- guage models are zero-shot learners.arXiv preprint arXiv:2109.01652. Adam Wiemerslage, Kyle Gorman, and Katharina von der Wense

    arXiv

  4. [2022]

    InProceedings of the 19th SIGMOR- PHON Workshop on Computational Research in Pho- netics, Phonology, and Morphology, pages 103–116, Seattle, Washington

    The SIGMORPHON 2022 shared task on morpheme seg- mentation. InProceedings of the 19th SIGMOR- PHON Workshop on Computational Research in Pho- netics, Phonology, and Morphology, pages 103–116, Seattle, Washington. Association for Computational Linguistics. Paul F Christiano, Ja...

    2022

  5. [2023]

    InProceedings of the 20th SIG- MORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 117–125

    Sigmorphon–unimorph 2023 shared task 0: Typologically diverse morpho- logical inflection. InProceedings of the 20th SIG- MORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 117–125. Kyle Gorman

    2023

  6. [2024]

    InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18955–18970, Miami, Florida, USA

    Getting the most out of your training data: Exploring unsupervised tasks for morphological inflection. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18955–18970, Miami, Florida, USA. Association for Computational Linguistics. C...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.