Pith. sign in

REVIEW 3 major objections 5 minor 29 references

UD-KSL Treebank v1.3: A semi-automated framework for aligning XPOS-extracted units with UPOS tags

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A semi-automated alignment of Korean-specific tags with universal POS tags improves L2-Korean tagging and parsing, especially when data are scarce.

desk verdict A useful L2-Korean treebank extension and alignment pipeline, but the headline accuracy gains are confounded by changing test labels and single-run results. read the letter →

arxiv 2506.09009 v2 pith:6G5ESEMV submitted 2025-06-10 cs.CL

Hakyung Sung , Gyu-Ho Shin , Chanyoung Lee , You Kyung Sung , Boo Kyung Jung This is my paper
classification cs.CL
keywords UniversalDependenciesL2KoreanlearnercorpusXPOS-to-UPOSalignmentmorphosyntactictaggingdependencyparsinglow-resourceNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a semi-automated alignment between Korean-specific morphosyntactic tags (Sejong XPOS) and Universal Dependencies part-of-speech tags (UPOS) makes L2-Korean treebank annotations more consistent and improves downstream NLP performance. The authors build the alignment from eojeols, the whitespace-delimited morphosyntactic units of Korean, by extracting frequent XPOS sequences, mapping each sequence to a UPOS tag based on its syntactic head, and manually validating the mappings. They also add 2,998 manually annotated sentences from argumentative essays to the existing treebank. Fine-tuning tagging and parsing models on the aligned data improves UPOS and XPOS tagging accuracy, and in a low-resource setting with a transformer-based toolkit it also improves dependency parsing and lemmatization. The paper matters because label consistency is a cheap lever for improving learner-language resources, where expert annotation is expensive.

What carries the argument

The central machinery is the eojeol construction and its XPOS sequence. An eojeol is a whitespace-delimited morphosyntactic unit in Korean, often combining a lexical root with multiple grammatical morphemes. The framework extracts all eojeol-level XPOS sequences (2,080 unique constructions), keeps those occurring more than five times (96.41% of eojeols) for manual XPOS-to-UPOS alignment, and maps rare constructions with default heuristics. The mapping is guided by the syntactic head morpheme, so NNG+JKO maps to NOUN while VV+EC maps to VERB, and the whole process is realized as a lookup-table script followed by three rounds of human verification. This machinery is what lets fine-grained Sejong tags be converted into consistent universal tags without re-annotating the corpus from scratch.

What would settle it

Evaluate aligned and unaligned models on a single fixed held-out test set whose labels are independently adjudicated and never updated by the alignment script; if the accuracy gap largely disappears, the reported improvements come from changed test labels rather than from more consistent training data.

Watch

Extended reading notes

Core claim

The central claim is that XPOS-to-UPOS alignment improves morphosyntactic analysis of L2 Korean even though it deliberately discards some language-specific distinctions. The paper reports that the alignment corrected 3,063 UPOS tags in the newly annotated sentences and 11,691 tags in the earlier treebank, and that the corrected labels produce consistent gains in UPOS tagging across two toolkits. On the smaller working set, the transformer-based toolkit gains +4.04 UPOS F1, +3.51 XPOS F1, +5.22 lemma accuracy, and +4.67 UAS, while the dictionary-based toolkit shows smaller or slightly negative parsing changes. The authors interpret this asymmetry as evidence that label consistency matters most when data are limited and that transformer architectures are better positioned to exploit it. They also flag a UD constraint that forced an auxiliary verb appearing as clause root to be tagged ADJ, suggesting future guideline revision.

Load-bearing premise

The comparison assumes that training and testing on aligned versus unaligned labels is fair even though the test answer keys themselves differ, so part of the reported gain could be relabeling the test set rather than better models.

Editorial extensions

If this is right

  • Aligned UPOS labels improve morphosyntactic tagging for both toolkits tested, so label consistency is a viable accuracy lever without collecting new sentences.
  • In low-resource settings the alignment gains are largest, making the method a partial substitute for annotated data in under-resourced languages.
  • The dependency-parsing benefit depends on the model: transformer-based parsers gain substantially on small data, while the local-feature parser does not, so alignment should be paired with an appropriate architecture.
  • The 2,998 new argumentative-essay sentences diversify the treebank's genre coverage and enable future studies of L2 Korean argumentation with the same annotation layers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If the gains come from label consistency rather than from the added sentences, the same alignment recipe could be applied to the unaligned portions of other Korean UD treebanks, not just learner data.
  • Editorial inference: The frequency cutoff means rare but linguistically interesting constructions are mapped by default heuristics; auditing the constructions that occur five or fewer times would show whether the shortcut introduces systematic errors.
  • Editorial inference: The forced ADJ tagging for auxiliary verbs serving as clause roots suggests a testable revision to UD Korean guidelines; if the guidelines allowed AUX roots, parsing metrics might shift further.
  • Editorial inference: Because the two toolkits respond differently, the paper's reported 'aligned dataset improves parsing' claim is best read as 'alignment helps transformer-based parsing in low-resource settings,' and rerunning the experiment on a larger transformer model would clarify the architecture dependence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a semi-automated framework that aligns Sejong XPOS sequences with Universal Dependencies UPOS tags for an L2-Korean treebank, extends the corpus with 2,998 newly annotated argumentative-essay sentences, and evaluates the effect of alignment by fine-tuning spaCy and Trankit on aligned versus unaligned versions of the data. The authors report that alignment improves UPOS tagging, XPOS tagging, and, in some configurations, dependency-parsing accuracy, with the largest gains on the smaller working set.

Significance. The dataset extension and the public annotation guideline are concrete, reusable contributions to L2-Korean NLP, and the alignment framework addresses a real inconsistency between fine-grained XPOS annotations and automatically generated UPOS tags. If the accuracy claim were properly supported, the framework would be practically valuable for low-resource Korean dependency parsing. However, the central empirical claim is not currently well supported: the UPOS comparison changes the test gold labels between conditions, and the dependency-parsing evidence is mixed and lacks variance estimates. The resource contribution is solid, but the headline accuracy claim needs to be re-established with a shared held-out test set and appropriate uncertainty quantification.

major comments (3)
  1. [Section 4.1 and Table 3] The UPOS evaluation confounds training and test label changes. The alignment procedure changes the gold UPOS labels themselves, with Section 3.4 reporting 3,063 corrected UPOS tags in the working set and 11,691 in UD-KSL v1.2, and Section 4.1 states that both datasets were provided in fixed and unfixed versions. Since both the training and test splits switch between the unaligned and aligned label sets, the UPOS deltas (e.g., +6.31 for spaCy on v1.2 and +4.04 for Trankit on the working set) reflect a change in the evaluation target as much as a change in model quality. A model trained and scored on aligned labels can appear better simply because those labels are more consistent with XPOS morphology and therefore easier to predict. To support the abstract's accuracy claim, the authors need a shared held-out gold test set whose labels are fixed independently of the alignment process, or at minimum an analysis separating label-change effects from genuine tagging improvement. The same issue does not affect the XPOS, lemma, and dependency rows, whose gold labels are not changed by the alignment, so the objection is decisive for the UPOS component rather than every row of Table 3.
  2. [Abstract and Section 4.2] The claim that alignment 'enhances morphosyntactic tagging and dependency-parsing accuracy' is not supported by Table 3 for dependency parsing. spaCy is slightly negative on both datasets (UAS -0.24/-0.56, LAS -0.29/-0.19), and Trankit on UD-KSL v1.2 is negligible (UAS -0.23, LAS 0.00). The only positive dependency-parsing result is Trankit on the 301-sentence working-set test split (UAS +4.67, LAS +4.92). The conclusion should be qualified to that specific condition, or additional evidence is needed before the abstract can make an unqualified claim that alignment improves dependency-parsing accuracy.
  3. [Section 4.1] The experiments use a single 8:1:1 split with no repeated runs, bootstrap confidence intervals, or significance tests. Several reported differences are small (e.g., XPOS +0.16 for Trankit on v1.2, LEMMA +0.01 for Trankit on v1.2) and are likely within run-to-run variation, but the table reports them to two decimal places without any uncertainty estimate. The dependency-parsing gains on the small working-set test split in particular need error bars or significance testing before they can carry the low-resource claim in the abstract.
minor comments (5)
  1. [Section 4.2] In the paragraph on UPOS tagging, the working-set Trankit improvement is reported as +3.51, but Table 3 lists +4.04 for UPOS; the +3.51 value matches the XPOS row. This inconsistency should be corrected.
  2. [Section 4.1] The terms 'fixed/unfixed' and 'aligned/unaligned' are used interchangeably; Table 2 and Table 3 use 'unaligned/aligned' while Section 4.1 uses 'fixed/unfixed.' Pick one consistent terminology throughout.
  3. [Section 4.2] The statement that 'Trankit consistently showed greater improvements across tasks' is too strong, since Table 3 shows Trankit with negative or zero dependency-parsing deltas on UD-KSL v1.2. Qualify the claim to the settings where the improvement actually occurs.
  4. [Table 2] The disappearance of PROPN and SYM categories after alignment (shown as em-dashes in the aligned columns) is not explained; a footnote describing where those tokens were reassigned would improve clarity.
  5. [Section 3.4] The description of the frequency threshold of five would benefit from stating how many constructions fell below the threshold and how the default mapping heuristics for those constructions were validated, since they still cover 3.59% of eojeols.

Circularity Check

1 steps flagged · score 6.0 of 10

UPOS accuracy gains are computed against gold labels that the alignment process itself rewrote, so the headline tagging improvement is partially self-defined; parsing/XPOS rows are not affected and carry the rest of the claim.

  1. self definitional [Section 3.4 ('Semi-automatic alignment'), Section 4.1 ('Model training and evaluation'), Table 3]
    "This step corrected 3,063 UPOS tags in the annotated texts of the current work (Section 2.2) and 11,691 tags in the existing UD dataset (Section 2.1). ... Both datasets were provided in fixed and unfixed versions to evaluate the impact of data refinement on model performance."

    The 'fixed' condition's test UPOS labels are the product of the alignment step; the 'unfixed' condition's labels are the pre-alignment gold. The reported UPOS delta is therefore accuracy on label set A minus accuracy on label set B, where A is constructed from B by the framework under test. A model trained and evaluated on the aligned labels can score higher simply because the target labels are more consistent with the XPOS morphology the framework exploits. The gain is partly an artifact of changing the gold standard; the evaluation does not separate model improvement from label redefinition. Manual validation improves label quality but does not supply a common held-out gold, so the central UPOS claim is partially circular by construction.

full rationale

The corpus and alignment framework are real contributions, and the dependency-parsing and XPOS results are evaluated on unchanged gold labels, so the paper is not wholly circular. However, the abstract's broad claim that alignment 'enhances morphosyntactic tagging accuracy' rests partly on Table 3's UPOS rows, where the test labels in the aligned condition are the output of the alignment procedure and differ from the unaligned condition's test labels. That makes the UPOS improvement partly self-referential. No load-bearing self-citation chain or ansatz-smuggling pattern was found; the main issue is the changed evaluation target for UPOS. A shared held-out gold set, or reporting performance on identical labels annotated from both schemes, would resolve the ambiguity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the accuracy of the gold XPOS annotations, the head-morpheme convention for UPOS, the frequency-cutoff heuristic, and the validity of comparing models across different test gold labels. The paper provides no formal proof or external benchmark, and the last assumption is not justified.

free parameters (1)
  • frequency_threshold_for_manual_review = 5
    Hand-chosen cutoff in Section 3.2. Constructions appearing more than five times were manually reviewed, while rarer ones got default heuristics. The threshold affects which XPOS sequences receive human-checked UPOS alignment and therefore shapes the whole dataset.
assumptions (4)
  • domain assumption Gold-standard XPOS annotations in UD-KSL v1.2 are correct.
    Section 3.2 uses them as the input for alignment. If XPOS errors propagate, the alignment mapping is unreliable.
  • domain assumption The head morpheme of an eojeol determines its UPOS tag.
    Section 3.1 states this convention. It follows Korean UD practice but is an analytic choice.
  • ad hoc to paper A frequency threshold of five yields representative coverage of eojeol constructions.
    Section 3.2. The threshold covers 96.41% of eojeols but rare constructions are mapped by default heuristics, which may introduce errors.
  • ad hoc to paper Changing the test gold labels between conditions is a valid way to measure alignment benefit.
    Section 4.1 provides 'fixed and unfixed versions' of the datasets. The paper assumes this comparison is meaningful, but no common fixed test set is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UD-KSL Treebank v1.3: A semi-automated framework for aligning XPOS-extracted units with UPOS tags." pith.science (2026). https://pith.science/paper/6G5ESEMV

@misc{pith2026250609009,
  author       = {Pith},
  title        = {Pith review of: UD-KSL Treebank v1.3: A semi-automated framework for aligning XPOS-extracted units with UPOS tags},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6G5ESEMV}},
  note         = {Machine review of arXiv:2506.09009}
}
read the original abstract

The present study extends recent work on Universal Dependencies annotations for second-language (L2) Korean by introducing a semi-automated framework that identifies morphosyntactic constructions from XPOS sequences and aligns those constructions with corresponding UPOS categories. We also broaden the existing L2-Korean corpus by annotating 2,998 new sentences from argumentative essays. To evaluate the impact of XPOS-UPOS alignments, we fine-tune L2-Korean morphosyntactic analysis models on datasets both with and without these alignments, using two NLP toolkits. Our results indicate that the aligned dataset not only improves consistency across annotation layers but also enhances morphosyntactic tagging and dependency-parsing accuracy, particularly in cases of limited annotated data.

Figures

Figures reproduced from arXiv: 2506.09009 by the authors.

Figure 1
Figure 1. Example of the Korean C-test (Lee-Ellis, 2009) essays were submitted as image files and manually transcribed into machine-readable texts by native Korean speakers with advanced linguistic expertise, preserving all original errors (i.e., no a priori cor￾rections were made, nor was technical assistance applied, during manual transcription). All person￾ally identifying information was anonymized. Proficiency evaluation… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 25 canonical work pages

  1. [1]

    Yevgeni Berzak, Jessica Kenney, Carolyn Spadine, Jing Xian Wang, Lucia Lam, Keiko Sophie Mori, Sebastian Garza, and Boris Katz. 2016. https://aclanthology.org/P16-1070.pdf Universal dependencies for learner english . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 737--746

    work page 2016

  2. [2]

    Sungdai Cho and John Whitman. 2022. The Cambridge handbook of Korean linguistics. Cambridge University Press

    work page 2022

  3. [3]

    Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2020. https://arxiv.org/abs/2006.13979 Unsupervised cross-lingual representation learning for speech recognition . arXiv preprint arXiv:2006.13979

    arXiv 2020

  4. [4]

    Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. https://aclanthology.org/W13-1703/ Building a large annotated corpus of learner english: The nus corpus of learner english . In Proceedings of the eighth workshop on innovative use of NLP for building educational applications, pages 22--31

    work page 2013

  5. [5]

    Marie-Catherine de Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. https://direct.mit.edu/coli/article/47/2/255/98516/Universal-Dependencies Universal dependencies . Computational linguistics, 47(2):255--308

    work page 2021

  6. [6]

    Elisa Di Nuovo, Cristina Bosco, Alessandro Mazzei, Manuela Sanguinetti, and 1 others. 2019. https://iris.unito.it/bitstream/2318/1725048/1/paper28.pdf Towards an italian learner treebank in universal dependencies . In CEUR workshop proceedings, volume 2481, pages 1--6. CEUR-WS

    work page 2019

  7. [7]

    Masaki Eguchi and Kristopher Kyle. 2023. https://doi.org/10.18653/v1/2023.bea-1.35 Span identification of epistemic stance-taking in academic written english . In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 429--442. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.bea-1.35 2023

  8. [8]

    Stefan Th Gries and Andrea L Berez. 2017. https://link.springer.com/chapter/10.1007/978-94-024-0881-2_15 Linguistic annotation in/for corpus linguistics . Handbook of linguistic annotation, pages 379--409

    work page doi:10.1007/978-94-024-0881-2_15 2017

Show all 29 references
  1. [9]

    Ji Yoon Han, Tae Hwan Oh, Lee Jin, and Hansaem Kim. 2020. https://aclanthology.org/2020.udw-1.12/ Annotation issues in universal dependencies for korean and japanese . In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), pages 99--108

    2020

  2. [10]

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spacy: Industrial-strength natural language processing in python

    2020

  3. [11]

    Hiroshi Kanayama, Na-Rae Han, Masayuki Asahara, Jena D Hwang, Yusuke Miyao, Jinho D Choi, and Yuji Matsumoto. 2018. https://aclanthology.org/W18-6009/ Coordinate structures in universal dependencies for head-final languages . In Proceedings of the Second Workshop on Universal ...

    2018

  4. [12]

    Hiroshi Kanayama, Masayasu Muraoka, and Katsumasa Yoshikawa. 2017. https://aclanthology.org/K17-3028/ A semi-universal pipelined approach to the conll 2017 ud shared task . In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependenci...

    2017

  5. [13]

    Taku Kudo. 2005. Mecab: Yet another part-of-speech and morphological analyzer. http://mecab. sourceforge. net/

    2005

  6. [14]

    Kristopher Kyle, Masaki Eguchi, Aaron Miller, and Theodore Sither. 2022. https://aclanthology.org/2022.bea-1.7/ A dependency treebank of spoken second language english . In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 202...

    2022

  7. [15]

    Chanyoung Lee, Tae hwan Oh, and Hansam Kim. 2019. https://www.kci.go.kr/kciportal/landing/article.kci?arti_id=ART002472594 CJK UTF8 mj 한국어 보편 의존 구문 분석 CJK (universal dependencies) CJK UTF8 mj 방법론 연구 CJK [a study on universal dependency annotation for korean] . CJK UTF8 mj 언어사실...

    2019

  8. [16]

    John Lee, Herman Leung, and Keying Li. 2017. https://aclanthology.org/W17-0408/ Towards universal dependencies for learner chinese . In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), pages 67--71

    2017

  9. [17]

    Sunyoung Lee-Ellis. 2009. https://journals.sagepub.com/doi/pdf/10.1177/0265532208101007 The development and validation of a korean c-test using rasch analysis . Language Testing, 26(2):245--274

    2009 doi

  10. [18]

    o nstr \

    Johanna Mesch and Krister Sch \"o nstr \"o m. 2018. https://www.diva-portal.org/smash/record.jsf?dswid=-1630&pid=diva2 In 8th Workshop on the Representation and Processing of Sign Languages, Miyazaki, Japan, 12 May, 2018, pages 121--126. European Language Resources Association

    2018

  11. [19]

    Youngbin Noh, Jiyoon Han, Tae Hwan Oh, and Hansaem Kim. 2018. https://aclanthology.org/W18-6013.pdf Enhancing universal dependencies for korean . In Proceedings of the second Workshop on Universal Dependencies (UDW 2018), pages 108--116

    2018

  12. [20]

    Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. https://aclanthology.org/2020.acl-demos.14/ Stanza: A python natural language processing toolkit for many human languages . In Proceedings of the 58th Annual Meeting of the Association for Comput...

    2020

  13. [21]

    Alla Rozovskaya. 2024. https://aclanthology.org/2024.lrec-main.1486/ Universal dependencies for learner russian . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17112--17119

    2024

  14. [22]

    Ho-Min Sohn. 1999. The Korean language. New York, NY: Cambridge University Cambridge University Press

    1999

  15. [23]

    Hakyung Sung and Kristopher Kyle. 2024. https://aclanthology.org/2024.law-1.2/ Annotation scheme for english argument structure constructions treebank . In Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 12--18

    2024

  16. [24]

    Hakyung Sung and Gyu-Ho Shin. 2023. https://aclanthology.org/2023.bea-1.6/ Towards l2-friendly pipelines for learner corpora: A case of written production by l2-korean learners . In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications...

    2023

  17. [25]

    Hakyung Sung and Gyu-Ho Shin. 2024. https://aclanthology.org/2024.lrec-main.332/ Constructing a dependency treebank for second language learners of korean . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluatio...

    2024

  18. [26]

    Hakyung Sung and Gyu-Ho Shin. 2025. https://arxiv.org/abs/2503.14718 Second language korean universal dependency treebank v1.2: Focus on data augmentation and annotation scheme refinement . In Proceedings of the Workshop on Resources and Representations for Under-Resourced Lan...

    2025 arXiv

  19. [27]

    Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. 2021. https://aclanthology.org/2021.eacl-demos.pdf#page=94 Trankit: A light-weight transformer-based toolkit for multilingual natural language processing . In Proceedings of the 16th Conference of the...

    2021

  20. [28]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  21. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.