{"id":"53643851-a5eb-4ed8-9b76-9fb615f465e4","arxiv_id":"2607.13372","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"For Irabu Ryukyuan, a POS tier improves neural grammatical glossing by +4.4 points with oracle POS and can more than halve data needs, but current tagger errors cancel the gain in a fully automatic pipeline.","lead":"This paper tests whether adding a part-of-speech (POS) line to interlinear glossing helps automated annotation for Irabu, an endangered Ryukyuan language. It finds that with perfect POS tags, glossing accuracy improves, especially with very little training data, but the benefit disappears when the POS tags are predicted by the model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data-multiplier claim is built on gold POS at test; pipeline-mode learning curves are untested, so the headline 'halves the data' may not survive realistic tagger degradation on small budgets.","rationale":"The reader's weakest assumption already identifies essentially the same soft spot: all quantitative conclusions come from a single corpus, and the learning-curve/data-multiplier results use gold POS at test time, so they apply only to the assisted scenario. My stress-test converges on this: the most load-bearing gap between the paper's central claim and its evidence is that the 'POS as a data multiplier' result — the headline contribution — has not been demonstrated under the fully automatic pipeline that the paper ultimately motivates. The paper is honest about this in §3.7 and §5, and its assisted-workflow recommendation is well supported by the clean Gold-POS vs No-POS ablation. Therefore the reader's CONDITIONAL verdict is appropriate; I do not see grounds to move it to ACCEPT or REJECT. The proposed pipeline-mode learning-curve experiment is a concrete, feasible check that would settle whether the data-multiplier claim generalizes to the automatic setting or needs to be explicitly scoped to assisted annotation.","tokens_in":19257,"tokens_out":7764,"duration_ms":90847,"concrete_test":"Retrain the full pipeline at each training budget (78, 155, 310, 620 utterances): train the POS tagger on that subset, feed its predicted POS to the glossing model at test, and compare against the No-POS glossing model at the same budget (5 seeds each). If the predicted-POS curve lies at or below the No-POS curve at the smaller budgets, the 'more than halves the glossed data' claim must be restricted to the assisted (gold-POS-at-test) scenario and the abstract/conclusion adjusted accordingly. As a secondary check, compute bootstrap confidence intervals on the horizontal distance ratio between the Gold-POS and No-POS learning curves to test whether the '>2×' data-saving is statistically distinguishable from 1×.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim that 'a POS tier more than halves the amount of glossed data needed' (Abstract, Contribution iv, §3.7) is derived from Table 5, where the POS-informed model receives gold POS at test time. In the deployment scenario the paper is ultimately aimed at — a documentation project that starts with only 6–47 minutes of annotated data — the POS tier would have to be produced by the pipeline's tagger, whose accuracy at those small training budgets is never measured. The break-even analysis in §3.6 shows that a real tagger must reach roughly 88% accuracy before predicted POS beats no POS, and that the real tagger at 88.1% yields only +0.40 points. Since tagger accuracy will decline as training data shrink, the gain may shrink, vanish, or even reverse at exactly the low-data regime where the paper claims the largest benefit. The paper honestly flags this in §3.7 ('a pipeline analogue would retrain the tagger at each size — future work') and in the Limitations, but the abstract and conclusion still state the data-multiplier result without this caveat, and the recommendation that 'richness beats quantity' depends on it. In addition, the data-equivalence values (×2.8, ×2.5, 'not matched even with the full corpus') are reported without confidence intervals: the claim that a POS-informed model on 310 utterances beats an ablated model on all 620 rests on 0.890 vs 0.888, a difference within the reported seed noise. Thus the magnitude of the central benefit — and especially its growth as data shrink — is less secure than the headline suggests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a fully supervised BiLSTM–CRF pipeline for morpheme segmentation, POS tagging, and grammatical glossing of Irabu Ryukyuan, trained on 47 minutes/620 utterances of a 774-utterance corpus, and evaluates how annotation richness (whether a per-morpheme POS tier is provided) and quantity (6–47 min budgets) affect glossing accuracy. The central findings are: gold POS improves test grammatical-gloss accuracy by +4.35±0.73 points over an ablated no-POS model, significant in all 5 seeds; the gain is larger at smaller training budgets (+11.6 at 12 minutes); a fully automatic pipeline does not realize the gain because the tagger's 12% errors poison the glossing model (wrong POS worse than no POS), with break-even tagger accuracy near 88%. The authors recommend quadrilinear annotation and a two-stage assisted-annotation workflow, and they release the code, corpus, and per-sentence outputs.","tokens_in":19591,"tokens_out":8254,"duration_ms":101593,"significance":"If the results hold, this is one of the few controlled, budget-centred measurements of annotation design for low-resource documentary linguistics. The core ablation is well controlled: same architecture, data splits, and seeds; exact McNemar tests; seed-level consistency; and an exact per-token decomposition of the pipeline loss. The paper also gives a falsifiable target for tagger quality (break-even near 88% accuracy) and is unusually candid about limitations, releasing code, gold corpus, and complete test outputs. The main quantitative claims rest on a single 405-token test set from one language, and the data-multiplier claim is computed in the gold-POS assisted setting rather than the fully automatic pipeline; these are load-bearing caveats for the paper's central recommendation.","major_comments":[{"comment":"The headline data-multiplier claim — 'a POS tier more than halves the amount of glossed data needed' (Abstract; Contribution iv) — is measured only under gold POS at test time (§3.7 explicitly says the curves use gold POS and that a pipeline analogue is future work). The paper's motivating scenario is automatic annotation of new recordings, where POS is predicted by a tagger. §3.6 shows break-even at ~88% tagger accuracy and that the current 88.1% tagger delivers only +0.40±0.77 points; tagger accuracy will not be higher at the 6–23-minute budgets where Table 5 shows the largest gains. Since the Conclusion's 'richness beats quantity' and the Stage-1 '30 rich minutes' recommendation depend on this claim, the abstract/conclusion must restrict it to the assisted setting, or the paper must add pipeline-mode learning curves (retrained tagger per budget).","section":"Abstract; §3.7/Table 5; §4"},{"comment":"The data-equivalence values (×2.8, ×2.5, 'not matched even with full corpus') are point estimates without confidence intervals or paired significance tests. The crossover anchoring the recommendation — Gold-POS at 310 utterances (0.890) vs No-POS at 620 (0.888) — is 0.2 points, within the seed SDs reported in Table 2. Report per-seed values and a paired/bootstrap test at each crossover, or soften the 'more than halves'/'not matched' wording. This is load-bearing for RQ2 and for the 'richness beats quantity' conclusion.","section":"§3.7, Table 5"}],"minor_comments":[{"comment":"The text states a 32-tag POS inventory with 31 tags in training, but Table 6 appears to list only 30 rows. Please reconcile the count.","section":"Appendix A; §3.1"},{"comment":"The text says the clitic ga occurs with six distinct grammatical glosses, but Figure 4 enumerates only five (gen, nom, foc, q, pur) from the four POS contexts shown. The remaining contexts are described in prose; please clarify the count and the sixth reading.","section":"§3.4, Figure 4"},{"comment":"The term 'position span-F1' is used without definition. Please define it and distinguish it from boundary-level F1 and token-level span F1.","section":"§3.2"},{"comment":"The log-odds analysis that supports the 'near-constant multiplier ≈1.8' claim is only described verbally. Since it is used to reconcile the two readings of Figure 7, consider including the regression details or per-budget log-odds values in the appendix.","section":"§3.7"}],"recommendation":"major_revision","confidential_remarks":"This is a single-author field-linguistics paper based on a self-annotated corpus, which is appropriate for the setting but means inter-annotator agreement is not addressed. The strengths are the reproducible code/data release, carefully controlled ablation, and honest caveats. The main revision I would require is aligning the abstract/conclusion data-multiplier and 'richness beats quantity' claims with the assisted gold-POS setting, plus confidence intervals for the Table 5 crossovers. With those changes, the paper would be a strong contribution to the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The core ablation — gold POS vs no POS under a fixed architecture — is clean and carefully analyzed, and the paper is refreshingly honest that the fully automatic pipeline gains nothing yet. The headline that a POS tier 'more than halves' the supervised data needed is only supported for the assisted scenario, because the learning curves in Table 5 use gold POS at test time. The paper says this in §3.7 and Limitations, but the abstract and conclusion still state the data-multiplier result without the caveat.\n\nWhat is new: treating the presence of a POS tier as an annotation-design variable, with five seeds, exact McNemar tests, an exact error-propagation decomposition, and a break-even analysis that gives a concrete target for tagger accuracy. The per-label results showing the gain concentrated on case and focus clitics are convincing. The worked example of how the model disambiguates 'ga' is a nice transparency exhibit. The null pipeline result is reported rather than buried; that is to the paper's credit.\n\nSoft spots, in proportion. Everything rests on one language and a test set of 405 grammatical-gloss tokens; the numbers should be read as one measurement, not constants. The data-equivalence values (×2.8, ×2.5, 'not matched even with the full corpus') are presented without confidence intervals, and the flagship comparison — POS-informed on 310 utterances vs ablated on all 620 — is 0.890 vs 0.888, which is inside the seed noise you would expect from the reported SDs. The 'wrong POS is worse than no POS' finding is based on 34 mistagged tokens; the direction is plausible and the decomposition is elegant, but the magnitude is fragile. The pipeline-mode learning curves, where the tagger is retrained at each budget, are never run; so the 'richness beats quantity' recommendation for the fully automatic setting is a projection, not a measurement. The claim that adding a POS tier is nearly free is asserted, not measured — no annotation time data are given.\n\nWho benefits: field linguists planning documentation workflows and anyone working on automated glossing. The practical recommendation — write down the POS tier — is reasonable and may save real time in the assisted setup. This is a serious, well-scoped case study; I would send it out rather than desk-reject. Revisions I'd ask for: either add a second language or narrow the headline to the assisted case, add intervals to the data-equivalence numbers, and preferably measure tagger accuracy at the small training budgets so the multiplier claim has a pipeline analogue.","headline":"Clean ablation of the POS tier, honestly reported null for the automatic pipeline, but the headline data-multiplier claim rests on gold POS and a 405-token test set — worth a serious referee, not a desk reject.","tokens_in":20140,"tokens_out":2682,"would_cite":true,"duration_ms":26746,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A per-morpheme POS tier materially improves automatic grammatical glossing when POS input is gold-quality and training data are scarce, but predicted-POS errors cancel the gain in a fully automatic pipeline.","keywords":["low-resource language documentation","interlinear glossing","POS tagging","quadrilinear annotation","Irabu Ryukyuan","neural sequence labeling","error propagation","annotation budget"],"falsifier":"Run the same controlled ablation (gold vs. no POS) on another agglutinative language with a comparable one-hour gold corpus: if the gold-POS gain is not positive and significant across five seeds, or if the gain does not grow as training data shrink, the central claim is contradicted. A cheaper check is to measure gloss accuracy on the 34 mistagged tokens in this corpus under a model trained with POS dropout or predicted-POS augmentation; if the 'wrong POS worse than no POS' effect disappears, the paper's mechanistic explanation is confirmed.","tokens_in":19103,"feed_emoji":"🏷️","tokens_out":4768,"duration_ms":47302,"temperature":0.7,"pith_summary":"This paper asks a practical question for endangered-language documentation: given a fixed, small annotation budget, does adding a per-morpheme part-of-speech (POS) tier to the standard three-line interlinear format make automated glossing better? Using a deliberately small neural pipeline trained on roughly one hour of fully annotated Irabu Ryukyuan discourse, it shows that gold-quality POS raises grammatical-gloss accuracy by 4.4 points on average, and the gain grows to 11.6 points when the training corpus is cut to a quarter. The gain is not yet realized automatically: when POS tags are predicted by the pipeline's own tagger, the few errors actively mislead the glossing model, so a wrong POS is worse than no POS. The paper frames the value as latent rather than lost, shows that it returns once the tagger clears roughly 90% accuracy, and recommends that documentation projects annotate quadrilinearly—text, POS, gloss, translation—so the tier is available for both assisted and future automatic workflows.","feed_headline":"Add a POS line, gain 4.4 to 11.6 points in automatic glossing","feed_subtitle":"A per-morpheme POS tier more than halves the glossed data needed, but only once the tagger passes 90%.","key_machinery":"The central object is the per-morpheme POS tier added to standard interlinear glossing, making the annotation quadrilinear (text + POS + gloss + translation). The argument is carried by a controlled ablation: a BiLSTM-CRF glossing model is trained with and without a POS embedding, holding all other factors fixed, under three conditions—gold POS, predicted POS, and no POS. The machinery includes the POS-to-gloss disambiguation rules the model learns (e.g., Irabu ga is genitive before a noun but nominative before a verb; as a focus particle it forces the focus gloss), and an error-propagation decomposition that attributes the lost gain to a train/test mismatch: the model, trained exclusively o","core_discovery":"The paper claims that an explicitly recorded per-morpheme POS tier—the missing fourth line in conventional trilinear interlinear glossing—is the key to making automatic glossing useful in low-resource documentation. Under gold POS, a BiLSTM-CRF glossing model improves grammatical-gloss accuracy by +4.35±0.73 points over a POS-ablated model, with the effect significant in all five seeds and concentrated on ambiguous case and focus clitics like Irabu ga, whose gloss is determined by its own POS and the category of the adjacent word. The gain grows as the training corpus shrinks, reaching +11.6 points at a quarter of the data, so a POS tier more than halves the amount of glossed data needed to","pith_inferences":["Because the POS tier is a closed, cheap inventory, the benefit may be larger for typologically similar languages where case and focus morphology are ambiguous from surface form alone; replicating the ablation on another agglutinative language would directly test the generality.","The poisoning result suggests model-side fixes the paper lists as future work—training with predicted or noised POS, joint POS-gloss modeling, or soft POS inputs—and one can hypothesize that scheduled sampling would recover most of the +4.4 point ceiling even with the current tagger.","The learning-curve analysis used gold POS at test time, so the 'POS as a data multiplier' conclusion applies to the assisted scenario; extending it to the fully automatic pipeline would require retraining the tagger at each data budget, which is likely to reduce but not eliminate the advantage.","For a project that cannot reach ~90% POS accuracy, spending annotation effort on the POS tier may not pay off in automatic glossing, though it still helps the assisted workflow; this is a caveat beyond the paper's own hedge."],"forward_implications":["Adding a POS tier gives an immediate accuracy boost in computer-assisted annotation, where a human supplies POS before gloss review, with roughly 40% fewer gloss errors to fix at full data.","A POS tier more than halves the amount of glossed data needed to reach a given gloss accuracy in the low-data regime, so documentation projects can invest in rich annotation of less discourse rather than thin annotation of more.","The learning-curve interaction implies that annotation richness (POS) is a more cost-effective investment than annotation quantity alone for agglutinative languages with comparable morphological transparency.","The break-even analysis sets a concrete target: once the tagger exceeds roughly 90% accuracy, the POS gain starts to pay off automatically; at 92–96% it recovers +1.6 to +3.2 points.","The two-stage design implies that projects should front-load about 30 minutes of quadrilinear annotation, then use the pipeline to draft annotations for review, folding corrections back into training."],"fun_headline_variants":["Add a POS tier to interlinear glossing: +4.4 to +11.6 points in accuracy","POS tier: the four-line trick that halves glossing data requirements","For Irabu, a POS tier improves glossing most when data is scarce","A POS line saves hours: +11.6 points with a quarter of the data","The missing fourth line: POS tier boosts automatic glossing by 4.4–11.6 pts"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The quantitative conclusions rest on a single corpus of 774 utterances (6,412 morpheme tokens) of one language, Irabu Ryukyuan, with a test set of only 405 grammatical-gloss tokens, and the learning-curve and data-multiplier analyses use gold POS at test time, so those results describe the assisted-annotation scenario rather than the fully automatic pipeline.","fun_headline_variants_meta":{"raw":{"variants":["Add a POS tier to interlinear glossing: +4.4 to +11.6 points in accuracy","POS tier: the four-line trick that halves glossing data requirements","For Irabu, a POS tier improves glossing most when data is scarce","A POS line saves hours: +11.6 points with a quarter of the data","The missing fourth line: POS tier boosts automatic glossing by 4.4–11.6 pts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1538,"prompt_tokens":899,"completion_tokens":639,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":524}},"tokens_in":643,"tokens_out":639,"duration_ms":7009,"temperature":1.0,"reasoning_tokens":524,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T05:19:12.495679+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same controlled ablation (gold vs. no POS) on another agglutinative language with a comparable one-hour gold corpus: if the gold-POS gain is not positive and significant across five seeds, or if the gain does not grow as training data shrink, the central claim is contradicted. A cheaper check is to measure gloss accuracy on the 34 mistagged tokens in this corpus under a model trained with POS dropout or predicted-POS augmentation; if the 'wrong POS worse than no POS' effect disappears, the paper's mechanistic explanation is confirmed.","supporting_citations":[],"review_version":1}