REVIEW 2 major objections
CMU-01 at the SIGMORPHON 2019 Shared Task on Crosslinguality and Context in Morphology
T0 review · 2 major / 0 minor · reviewed 2026-05-24 · grok-4.3
Pith's one-line read A hierarchical neural CRF predicts each morphological feature independently and transfers training from multiple typologically similar languages.
desk verdict Standard shared-task system paper using hierarchical neural CRFs and multilingual transfer, but the abstract supplies no ablations or numbers to show the transfer step actually helps. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Hierarchical neural conditional random field (CRF) with independent per-feature prediction, plus multi-lingual transfer training from typologically related languages.
What would settle it
An ablation that trains the same model on only the target language versus the proposed multi-language transfer set and measures whether the transfer step produces higher accuracy on held-out low-resource treebanks.
Extended reading notes
Core claim
The submission uses a hierarchical neural CRF that predicts each coarse-grained morphological feature independently for every token, paired with a multi-lingual transfer training regime that draws from multiple related languages of similar typology to compensate for under-resourced treebanks.
Load-bearing premise
Training on multiple typologically similar languages will reliably improve performance on under-resourced target treebanks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript describes the CMU-01 submission to SIGMORPHON 2019 task 2 on morphological analysis and lemmatization in context across 107 treebanks. It employs a hierarchical neural CRF that predicts coarse-grained features (POS, Case, etc.) independently and proposes a multi-lingual transfer regime from typologically similar languages to address under-resourced treebanks.
Significance. If the multi-lingual transfer regime can be shown to improve performance on low-resource targets, the work would offer a practical approach to morphological tagging in under-resourced settings. The hierarchical CRF design for independent feature prediction is a reasonable modeling choice, but without reported results the significance cannot be assessed.
major comments (2)
- [Abstract] Abstract: the claim that the multi-lingual transfer regime addresses the under-resourced setting is unsupported; the text supplies no quantitative results, ablation studies, monolingual baselines, or error analysis, so the load-bearing assumption that transfer from related languages reliably improves target performance remains unevaluated.
- [Abstract] Abstract: no implementation details are given for the transfer regime (language selection criteria, transfer mechanism, or how the hierarchical CRF enforces feature independence), preventing assessment of whether the proposed solution is reproducible or correctly specified.
Simulated Author's Rebuttal
We thank the referee for their comments on our description of the CMU-01 system for the SIGMORPHON 2019 shared task. We address the two major comments below and will revise the manuscript to strengthen the presentation.
read point-by-point responses
-
Referee: [Abstract] Abstract: the claim that the multi-lingual transfer regime addresses the under-resourced setting is unsupported; the text supplies no quantitative results, ablation studies, monolingual baselines, or error analysis, so the load-bearing assumption that transfer from related languages reliably improves target performance remains unevaluated.
Authors: The abstract summarizes the approach; the body of the manuscript reports the official shared-task scores across all 107 treebanks. We agree, however, that the manuscript lacks explicit monolingual baselines, ablation studies on the transfer component, and error analysis. These will be added in the revised version to provide direct quantitative support for the transfer regime. revision: yes
-
Referee: [Abstract] Abstract: no implementation details are given for the transfer regime (language selection criteria, transfer mechanism, or how the hierarchical CRF enforces feature independence), preventing assessment of whether the proposed solution is reproducible or correctly specified.
Authors: We agree that the current description is high-level. The revised manuscript will include concrete details on (i) how typologically similar languages are selected for transfer, (ii) the exact transfer training procedure, and (iii) the hierarchical CRF architecture that factors the prediction of coarse-grained morphological features. revision: yes
Circularity Check
No circularity: empirical system description with no derivations or self-referential reductions
full rationale
The paper is a shared-task system submission describing a hierarchical neural CRF model for morphological analysis plus a multi-lingual transfer regime motivated by under-resourced treebanks. No equations, fitted parameters, or predictions appear; the central claims are architectural choices and a training regime presented without any reduction to self-defined quantities, self-citation chains, or ansatzes. The absence of ablations noted by the reader is a limitation of evidence, not a circularity in the derivation chain. The work is therefore self-contained against external benchmarks with score 0.
Assumptions & free parameters
Cite this review
Pith. "Pith review of CMU-01 at the SIGMORPHON 2019 Shared Task on Crosslinguality and Context in Morphology." pith.science (2026). https://pith.science/paper/KNQRRQ73
@misc{pith2026190710129,
author = {Pith},
title = {Pith review of: CMU-01 at the SIGMORPHON 2019 Shared Task on Crosslinguality and Context in Morphology},
year = {2026},
howpublished = {\url{https://pith.science/paper/KNQRRQ73}},
note = {Machine review of arXiv:1907.10129}
}
read the original abstract
This paper presents the submission by the CMU-01 team to the SIGMORPHON 2019 task 2 of Morphological Analysis and Lemmatization in Context. This task requires us to produce the lemma and morpho-syntactic description of each token in a sequence, for 107 treebanks. We approach this task with a hierarchical neural conditional random field (CRF) model which predicts each coarse-grained feature (eg. POS, Case, etc.) independently. However, most treebanks are under-resourced, thus making it challenging to train deep neural models for them. Hence, we propose a multi-lingual transfer training regime where we transfer from multiple related languages that share similar typology.
Reviewed May 24, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.