{"id":"c3c098a8-b2d2-4187-86a0-d7f33a977736","arxiv_id":"2505.16800","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Jointly training a transformer to segment and gloss words, then adding LLM-generated training examples, improves canonical morpheme segmentation for several low-resource languages.","lead":"Low-resource languages rarely have enough labeled data to train good morpheme segmentation systems. This paper shows that jointly predicting glosses and adding LLM-generated training words improves segmentation accuracy on several such languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No filtering of LLM-generated synthetic examples against dev/test is reported; if generated surface forms overlap test words, the reported synthetic-data gains would be an artifact of data leakage.","rationale":"The multitask-learning result is the stronger and more interesting contribution: a shared encoder with two decoders for segmentation and glossing improves over a single-task baseline across most languages, with gains in ACC, F1, and edit distance. The synthetic-data result, however, is the part most vulnerable to a concrete artifact. The pipeline in Section 2.2 uses GPT-4o to generate new words from stems and morphemes that occur in the training split; the paper does not say that generated examples were filtered against dev/test. For tiny languages, the space of plausible surface forms is small, so exact overlap with the held-out set is not merely possible but likely unless explicitly avoided. The authors' Limitations section admits the model cannot generate new stems and that no qualitative error analysis was performed, which leaves this leakage risk unresolved. The reader's weakest assumption focused on hallucination and on the independence of the 6:2:2 unique-word split; my concern is closely related but distinct: contamination of the training set by synthetic copies of test words. It is a stronger threat because it would mechanically inflate all three test metrics, not just fail to transfer. I nevertheless recommend keeping the reader's CONDITIONAL verdict rather than moving to REJECT: the concern is a testable hypothesis, the multitask claim is independent and appears solid, and the authors may already have filtered the data without reporting it. The check I propose would settle the matter, so the appropriate outcome is acceptance conditional on that verification. My agreement with the reader is partial because we both identify the synthetic-data stage as the weak point, but I prioritize leakage over hallucination as the decisive failure mode.","tokens_in":8599,"tokens_out":4663,"duration_ms":35355,"concrete_test":"Obtain the released splits and the exact LLM-generated synthetic files, then (1) compute the overlap between every generated surface form and the dev/test surface-form lists for each language; (2) also check whether any generated canonical segmentation or gloss matches a test entry even when the surface string differs; (3) retrain the M+LLM(0.5) configuration with all overlapping examples removed and compare average ACC, F1, and ED against Table 3. If the average gains over the multitask-only model largely disappear or reverse after decontamination, the synthetic-data improvement is an artifact of leakage. If the gains persist, the claim survives this test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The synthetic-data contribution (Section 3.3, Table 3) rests on GPT-4o generating new training triples from stems and grammatical morphemes extracted only from the training split (Section 2.2). The paper never states that generated surface forms, canonical segmentations, or glosses were checked against the dev or test splits and removed on overlap. This matters because the test sets are small (e.g., Gitksan has 109 test words) and the generation prompt (Appendix A.4) combines a real stem with 2-5 grammatical morphemes from the training data, so the LLM can easily reproduce words that actually occur in the held-out test portion of the same dataset. If any such exact surface-form match is present in the augmented training set, the model can memorize the correct segmentation and inflate word accuracy and morpheme F1 for those test instances; edit distance also drops. The paper's own Limitations section concedes that synthetic data is constrained to existing vocabulary and stems, which increases rather than reduces this risk. Since code, splits, and generated data are withheld behind a placeholder link and no qualitative error analysis is provided, the reported synthetic-data gains (e.g., Gitksan ACC 52.29 to 56.88 at 75% augmentation) cannot be distinguished from leakage. This is the most load-bearing concern because it attacks one of the paper's two central claims, and it is directly checkable once the data are released.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a transformer-based sequence-to-sequence model for canonical morpheme segmentation, augmented in two ways: a multitask objective that jointly predicts the gloss tier alongside the segmentation, and synthetic training triples generated by GPT-4o using in-context learning from training-set stems and grammatical morphemes. Experiments are reported on seven languages from the SIGMORPHON 2023 dataset, with comparisons to a single-task Fairseq baseline and to the previously published TAMS system on three shared languages. The multitask model outperforms the baseline on average across languages, and adding synthetic data at 25%, 50%, and 75% of the training size yields mixed but generally positive gains. The paper also reports learning-curve experiments and a single-task ablation for the synthetic-data component.","tokens_in":8800,"tokens_out":3443,"duration_ms":28205,"significance":"If the results hold, the multitask contribution is a clean and practically useful finding: using the gloss tier, which is already part of IGT documentation, as auxiliary supervision improves canonical segmentation without additional annotation, and the gains appear consistent across languages and data sizes. The synthetic-data contribution is potentially interesting but is currently under-supported; the generation procedure is novel in using LLMs for this task, yet the absence of a leakage check and the lack of statistical rigor leave the reported gains uncertain. The paper is commendably explicit about its limitations, particularly the risk of LLM hallucination and the restriction to existing vocabulary, which is a strength in transparency.","major_comments":[{"comment":"The manuscript never reports filtering LLM-generated synthetic examples against the dev or test splits. Because generation uses stems and grammatical morphemes extracted from the training split, the LLM can reproduce surface forms that occur in the held-out test portion of the same dataset; with small test sets (e.g., Gitksan has 109 test words), even a few exact matches could explain the observed gains (e.g., Gitksan ACC rising from 52.29 to 56.88 at 75% augmentation). The code and generated data are behind a placeholder link, so this cannot currently be checked. The authors should add an explicit overlap-removal step or provide an analysis showing that no generated surface form coincides with any dev/test form; without this, the synthetic-data claim is not distinguishable from leakage.","section":"Section 2.2 / Section 3.3, Table 3"},{"comment":"All reported results are single runs with no error bars, seeds, or significance tests. The abstract and conclusions state that the approach 'significantly improves' performance, but the numerical differences are often small (e.g., average ACC 66.59 for the multitask model vs. 66.63 at 25% augmentation in Table 3), and some language-metric cells degrade with synthetic data (e.g., Lezgi F1 at 75% augmentation, Nyangbo F1 at 50% augmentation). Multi-seed experiments with standard deviations, or paired significance tests across the same train/dev/test splits, are needed to establish which gains are reliable.","section":"Section 3, Tables 2, 3, and 5"},{"comment":"The text states that the segmentation loss weight λ is tuned within the range 0.8 to 15, but λ is defined as the weight on L_seg with (1−λ) on L_gloss, so values above 1 make the gloss weight negative, and the appendix only reports λ from 0.5 to 1.0. This appears to be a typo, but since the multitask results depend directly on this weighting, the reported range should be corrected and the λ values used for each language should be reported alongside the final results.","section":"Section 2.1, Eq. (1), and Appendix A.2"}],"minor_comments":[{"comment":"The text says 'We report the results in Table 5,' but the relevant table is labeled Table 3; the appendix contains a separate Table 5, so the cross-reference should be fixed.","section":"Section 3.3"},{"comment":"The word 'complemetary' should be spelled 'complementary'.","section":"Section 2.1"},{"comment":"Number formatting is inconsistent: '3,558' appears with a comma while other numbers do not use thousands separators; please unify.","section":"Table 1"},{"comment":"The row label 'TAMS-CLS' is not defined in the text; please clarify what this baseline is.","section":"Section 3.1"},{"comment":"The example prompt says 'You are a linguistics expert of Natügu' but the same template is presumably used for all languages; the paper should state explicitly how the prompt is adapted per language, including how the list of grammatical morphemes and stems is selected.","section":"Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The multitask result is the stronger contribution and appears to be a clear improvement over the single-task baseline, but the paper currently overclaims 'significant' improvements without any variance estimation. The synthetic-data claim is the riskier one: the leakage concern raised by the skeptical review is real and directly checkable, and the authors should be required to either filter generated examples against dev/test or provide a demonstration that no overlap occurs. The placeholder code link and withheld splits should also be resolved before the paper can be considered further. The scope fits the journal, and the limitations section is honest, but the experimental methodology needs strengthening."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The substantive contribution here is the multitask model: one encoder, two decoders, jointly predicting canonical segmentation and glosses. It beats the single-task Fairseq baseline on all seven languages on average, and it beats the reported TAMS numbers on the three languages TAMS covers. That is a clean, plausible result, and it does not depend on anything exotic. The learning-curve analysis is also a nice touch—shows the benefit grows with data, which is what you'd want if the mechanism is shared linguistic structure.\n\nThe synthetic-data half is where I part ways with the paper's own framing. The abstract says the approach 'significantly improves' segmentation, but Table 3 tells a messier story: several languages get worse at some augmentation ratios (Lezgi F1 drops at 0.5 and 0.75, Tsez accuracy drops, Nyangbo F1 drops at every ratio). The averages improve modestly, but the headline overstates it. More importantly, the stress-test note is correct: there is no reported filtering of LLM-generated surface forms against the dev or test splits. Given the prompt combines a real stem with 2–5 grammatical morphemes from the same dataset, and test sets are tiny (Gitksan has 109 words), exact overlap is not just possible but likely. If even a handful of test words appear in the augmented training set, the word-accuracy and edit-distance gains become artifacts. The Limitations section admits the generation process is limited to existing vocabulary but never mentions checking overlap. That is a load-bearing gap, and it is directly checkable once the code and data are released.\n\nThe other soft spots are minor by comparison: no error bars or multi-seed runs, the code link is a placeholder, and the splits are withheld. These are standard for a workshop-style submission but should be fixed in a journal version.\n\nThe multitask result alone is a solid empirical contribution for computational morphology and documentary linguistics. The synthetic-data contribution should be treated as unverified until the leakage check is done. I would send this to peer review, but I would ask the authors to release code, splits, and generated data, and to add a filtering step plus a leakage audit. If they can show the gains survive that, the paper becomes a useful reference. If not, the multitask half still stands on its own.","headline":"The multitask glossing result is real and worth refereeing; the LLM synthetic-data claim is compromised until leakage is ruled out.","tokens_in":641,"tokens_out":836,"would_cite":false,"duration_ms":23680,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Multitask learning with glosses and LLM-generated training examples improves low-resource canonical morpheme segmentation by about 3.6 average points and cuts edit distance by roughly 30 percent.","keywords":["canonical morpheme segmentation","low-resource languages","multitask learning","synthetic data","in-context learning","interlinear glossed text","transformer","morphological generalization"],"falsifier":"Have a linguist annotate a sample of the generated words for grammatical validity, or test the model on held-out words with unseen stems; if most generated forms are judged invalid or accuracy on unseen stems does not beat the baseline, the synthetic-data gains are pattern repetition rather than morphological learning.","tokens_in":8326,"feed_emoji":"🧩","tokens_out":7776,"duration_ms":58412,"temperature":0.7,"pith_summary":"This paper claims that two sources of already available information can improve canonical morpheme segmentation in low-resource languages: the gloss tier of interlinear glossed texts, used as an auxiliary training objective, and synthetic word examples generated by a large language model. Canonical segmentation restores underlying morphemes from surface forms, such as 'happiness' to 'happy-ness', and it is a costly step in language documentation when annotated data are scarce. Across seven languages from the SIGMORPHON 2023 shared task, the multitask model improves average word accuracy from 62.95 to 66.59 and morpheme F1 from 77.71 to 81.35, while reducing total edit distance by about 30 percent; adding LLM-generated examples at 75 percent of the gold data pushes the averages to 67.82 and 81.96. The result matters because it suggests documentation workflows can obtain better segmentation without additional human annotation.","feed_headline":"Multitask and LLM data lift low-resource segmentation 3.6 points","feed_subtitle":"Predicting glosses and adding LLM examples cut edit-distance errors by about 30 percent.","key_machinery":"The load-bearing mechanism is a shared encoder with two task decoders: the input is the orthographic surface word, one decoder emits the canonical segmentation, and the other emits the morpheme glosses, trained with the joint loss $L_{\\text{total}} = \\lambda L_{\\text{seg}} + (1-\\lambda)L_{\\text{gloss}}$, with $\\lambda$ tuned near 0.9. A second mechanism is prompt-based synthetic-data generation: a large language model receives a stem, its meaning, example segmentations and glosses from the training set, and the list of grammatical morphemes, then outputs new triples that approximate interlinear glossed text. Together the two mechanisms add information without new human annotation, since glosses are already part of the documentary format and the synthetic examples expand coverage of rare or unseen morpheme combinations.","core_discovery":"The paper's central claim is that a transformer sequence-to-sequence model with one shared encoder and two decoders, one emitting canonical segmentation and one emitting morpheme glosses, learns morphological structure better than a single-task segmenter, and that adding LLM-synthesized surface-form/segmentation/gloss triples as training data pushes accuracy further. The shared latent representation forces the model to encode information that predicts both the boundaries and the grammatical labels, which the authors observe reduces the unnecessary orthographic changes that a single-task baseline tends to make. On the low-resource languages, the multitask model reaches 66.59 average word accuracy and 81.35 morpheme F1 versus 62.95 and 77.71 for the baseline; adding synthetic examples at 75 percent of training size brings the averages to 67.82 and 81.96. The authors attribute the gains to better generalization from the glossing signal and to increased morphological coverage from the generated words, while noting that the synthetic gains may partly come from exposure to frequent patterns rather than genuine morphological generalization.","pith_inferences":["The paper's reported gains from synthetic data would be on firmer ground if a linguist's error analysis confirmed the generated forms; otherwise the improvements could come from exposure to frequent character patterns rather than true morphological generalization.","The same shared-encoder design could use other IGT tiers, such as the translation line or part-of-speech labels, as additional auxiliary tasks whenever those tiers are documented.","A sharper stress test would measure accuracy on held-out words whose stems never appear in the training set; if the gain disappears there, the synthetic data is not teaching new morphology.","Because the LLM prompt only recombines stems and affixes seen in training, this method will not create novel stems; pairing it with rule-based generation or human-in-the-loop correction would extend coverage to out-of-vocabulary forms."],"forward_implications":["A system that sees only the orthographic word can be trained to predict both the segmentation and the gloss, because both tiers already exist in interlinear glossed texts and require no extra curation.","Multitask learning alone raises average word accuracy from 62.95 to 66.59 and morpheme F1 from 77.71 to 81.35, cutting total edit distance by about 30 percent.","Adding LLM-generated examples at 75 percent of the gold data brings average accuracy to 67.82 and F1 to 81.96, with the largest relative gains in the sparsest languages such as Gitksan.","The learning-curve result implies the benefit of multitask learning grows as more annotated data accumulates, so adopting the model early in a documentation project compounds with later data collection."],"supporting_citations":[{"why":"Supplies the translation-assisted segmentation system and the 6:2:2 unique-word splits used for Arapaho, Lezgi, and Tsez.","marker":"Rice et al., 2024"},{"why":"Defines the SIGMORPHON 2023 shared-task dataset and languages on which all experiments are run.","marker":"Ginn et al., 2023"},{"why":"Establishes multitask learning as the mechanism that the joint segmentation-and-glossing decoder relies on.","marker":"Caruana, 1997"},{"why":"Provides the transformer sequence-to-sequence architecture that the shared encoder and dual decoders are built from.","marker":"Vaswani et al., 2017"},{"why":"Supplies the sequence-modeling toolkit whose transformer implementation the authors modify for the multitask objective.","marker":"Ott et al., 2019"},{"why":"Provides the character-level transformer hyperparameter settings that the model adopts.","marker":"Wu et al., 2021"},{"why":"The multitask transformer codebase the implementation is modified from, used to build the shared-encoder dual-decoder model.","marker":"Zhou et al., 2019"}],"fun_headline_variants":["Low-resource morphology: LLM data and glosses add 3.6 points","Synthetic data and multitask glosses boost segmentation by 3.6 points","Gloss multitasking and LLM examples lift low-resource segmentation accuracy","Synthetic LLM examples and gloss multitasking cut edit-distance errors by 30%","Multitask glosses plus synthetic data: 3.6 point segmentation gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM-generated surface forms, segmentations, and glosses are linguistically faithful; if they are hallucinated or only repeat frequent patterns, the synthetic-data improvements will not transfer to real morphology.","fun_headline_variants_meta":{"raw":{"variants":["Low-resource morphology: LLM data and glosses add 3.6 points","Synthetic data and multitask glosses boost segmentation by 3.6 points","Gloss multitasking and LLM examples lift low-resource segmentation accuracy","Synthetic LLM examples and gloss multitasking cut edit-distance errors by 30%","Multitask glosses plus synthetic data: 3.6 point segmentation gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001601,"raw_usage":{"total_tokens":6339,"prompt_tokens":864,"completion_tokens":5475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":5369}},"tokens_in":480,"tokens_out":5475,"duration_ms":29265,"temperature":1.0,"reasoning_tokens":5369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:54:35.885333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a linguist annotate a sample of the generated words for grammatical validity, or test the model on held-out words with unseen stems; if most generated forms are judged invalid or accuracy on unseen stems does not beat the baseline, the synthetic-data gains are pattern repetition rather than morphological learning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the translation-assisted segmentation system and the 6:2:2 unique-word splits used for Arapaho, Lezgi, and Tsez."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SIGMORPHON 2023 shared-task dataset and languages on which all experiments are run."}],"review_version":1}