Pith. sign in

REVIEW 3 major objections 6 minor 16 references

Extending dependencies to the taggedPBC: Word order in transitive clauses

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Automatically transferred dependency labels, summarized as an $N_1$ ratio, separate expert classifications of transitive word order into verb-initial, verb-medial, verb-final, and free categories across 961 languages.

desk verdict Useful dataset release, but the headline N1-ratio result cannot separate verb-medial from verb-final and needs post-hoc stats or a redefinition. read the letter →

arxiv 2506.06785 v1 pith:JGVGR2XN submitted 2025-06-07 cs.CL

classification cs.CL
keywords computationaltypologywordorderdependencyprojectioncross-lingualannotationtransferparallelcorporataggedPBCCoNLL-Uunder-resourcedlanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper extends the taggedPBC, a part-of-speech-tagged parallel corpus covering more than 1,500 languages, by projecting English dependency annotations through word alignments into every language and converting the result to CoNLL-U format. Its central claim is that one number computed from these noisy annotations, the $N_1$ ratio (verses whose first word is an argument divided by verses whose first word is the predicate), separates languages classified by experts as verb-initial, verb-medial, verb-final, or free word order; across 961 overlapping languages the separation is significant at $p < 0.003$. The paper does not directly validate the accuracy of the transferred dependency relations, yet argues that at this scale the aggregate signal still lines up with expert typological judgments. If the claim is right, automatic transfer from a single high-resource language can give coarse but real word-order typology for languages with no hand-built treebanks.

What carries the argument

The load-bearing object is the $N_1$ ratio, defined as the number of transitive verses whose first word is labelled as an argument (subject or object) divided by the number whose first word is the predicate; it compresses the six-way word-order space into a single gradient that separates the four collapsed expert classes. The mechanism producing it is a transfer pipeline: English sentences are parsed with an automatic dependency parser, IBM Model 2 word alignments map the English words to words in each target language, and the English dependency and morphological labels are transferred across the alignment links into CoNLL-U format. The paper deliberately substitutes broad validation against typological databases for direct validation of the transferred dependency labels, treating the correlation as evidence that the labels carry usable signal despite their noise.

What would settle it

Hand-annotate the subject, object, and verb in a stratified sample of verses from 25 to 30 languages spanning several families and word-order types, recompute the $N_1$ ratio from those gold labels, and check whether it still separates verb-initial, verb-medial, verb-final, and free languages at the reported significance level; if the separation disappears or diverges from the transferred-data ratios, the correlation is an artifact of the English-based transfer.

Watch

Extended reading notes

Core claim

The discovery, stated on the paper's own terms, is that corpus measures built from dependency relations that were never hand-annotated can reproduce expert judgments about basic transitive word order. In the CoNLL-U version of the taggedPBC, English sentences are parsed automatically and their subject and object labels are copied to aligned words in each target language through word alignments. For every language the paper counts verses containing a verb plus subject and object relations, computes proportions for the six transitive orders, collapses these to verb-initial, verb-medial, verb-final, and free, and finds that the $N_1$ ratio differentiates the four classes with a one-way analysis of variance at $p < 0.003$. The paper also observes that every language corpus contains at least one verse for each of the six orders, that free-order languages cannot be separated from fixed ones by the six proportions alone, and that these gradient patterns support the reality of discrete categories: languages grammaticalize one order while still showing all orders to differing degrees.

Load-bearing premise

The result collapses if the transferred subject and object labels are systematically wrong, because the $N_1$ ratio would then be measuring English word order or alignment noise rather than the target language's clause structure; the paper does not test the labels directly, only their downstream correlation, a gap it acknowledges in the methodology.

Editorial extensions

If this is right

  • The $N_1$ ratio can be used to assign coarse transitive word-order classes to taggedPBC languages that are absent from the three expert typological databases, extending word-order typology to languages without hand-coded classifications.
  • Because all six transitive orders occur in every language's corpus, word order is better treated as gradient; the significant $N_1$-based separation suggests that discrete categories such as verb-initial, verb-medial, and verb-final emerge from the relative frequency of competing orders.
  • Free word order cannot be detected directly from the six order proportions, since free-order languages scatter across the verb-initial, verb-medial, and verb-final ranges; the $N_1$ ratio, not the proportions, is the measure that aligns with the free classification.
  • The English-centric annotation restricts the dataset's use: language-specific categories such as numeral classifiers are largely missing, no morphological annotations are present, and dependent clauses are not distinguished from main clauses, so the dataset supports coarse features like word order rather than fine-grained morphosyntactic analysis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A quick check the paper does not run is a permutation baseline: shuffle the dependency labels across aligned words and recompute the $N_1$ ratio; if the analysis-of-variance separation survives the shuffling, the correlation is alignment noise rather than target-language word order.
  • The paper's finding that free-order languages resist detection through order proportions suggests an untested corollary: within-corpus variance of verse-level $N_1$ ratios, or the entropy of the six-order distribution, may separate free from fixed languages even when the mean proportions do not.
  • Because subject and object labels are transferred under a nominative-accusative reading, the method may systematically mislabel the single argument of ergative languages; a bounded test is to recompute the $N_1$-ratio separation within a sample of ergative languages once hand-annotated dependencies exist.
  • The same transfer pipeline could be applied to a non-religious parallel corpus with comparable sentence counts; if the verb-initial, verb-medial, verb-final, and free separation persists there, the result generalizes beyond Bible-verses translation style rather than reflecting a genre-specific artifact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper extends the taggedPBC, a massively parallel Bible corpus with automatically transferred POS tags, by adding dependency annotations transferred from English SpaCy parses through IBM Model 2 word alignments, and releases the result in CoNLL-U format. It then tests whether corpus-derived measures—notably the 'N1 ratio', defined as the count of verses starting with an argument divided by those starting with a predicate—correlate with expert word-order classifications (VI, VM, VF, free) in WALS, Grambank, and Autotyp for 961 overlapping languages. The paper reports a significant one-way ANOVA (p < 0.003) and interprets this as showing that the N1 ratio clearly differentiates the four word-order groups, arguing that noisy large-scale parallel data can support coarse-grained typological investigation.

Significance. If the central correlation were robustly established, the paper would make a useful contribution to computational typology by showing that automatically transferred dependency annotations over a very large parallel corpus can recover a coarse typological distinction, and the released dependency-annotated corpora would be a valuable resource. The paper also deserves credit for making code and data publicly available and for positioning the work as a validation exercise rather than a claim of gold-standard annotation. However, the current statistical and conceptual evidence does not support the strong claim that the N1 ratio differentiates among all four word-order classes, and the unvalidated dependency transfer leaves the measure's reliability open to question.

major comments (3)
  1. [Section 3, N1 ratio definition and ANOVA] The N1 ratio, as defined in the text ('a count of verses that start with an argument divided by those that start with a predicate'), is structurally unable to distinguish verb-medial from verb-final orders. For canonical VM (SVO, OVS) and VF (SOV, OSV) orders, the clause-initial constituent is an argument, whereas for VI (VSO, VOS) orders it is a predicate. Thus the ratio can separate VI from non-VI languages, but it cannot by itself separate VM from VF. The one-way ANOVA with p < 0.003 across the four groups may therefore be driven entirely by the VI-versus-rest contrast. The paper reports no post-hoc pairwise tests, group means, standard deviations, or effect sizes, so the sentence 'the N1 ratio clearly differentiates between these groups' is not supported by the evidence presented. Please report the group-level statistics and post-hoc contrasts, and either reformulate the claim to 'differentiates VI from VM/VF' or provide a different measure that distinguishes VM from VF.
  2. [Section 2 and Section 3, dependency transfer validity] The N1 ratio depends on identifying which word at the start of a verse is an argument versus a predicate, and this identification comes from dependency relations transferred from English through automatic word alignments. The paper explicitly declines to validate the transferred dependencies (Section 2), yet the central correlation is based entirely on them. If the transferred subject/object labels are systematically biased—for example, if English SVO word order influences which token is labeled as the subject—the N1 ratio could reflect English or alignment artifacts rather than the target language's own word order. The paper should either provide direct validation of the transferred dependencies on a hand-annotated sample (as was done for POS tags in Ring 2025a) or conduct a robustness analysis showing that the reported ANOVA result is stable under plausible levels of dependency-label noise. Without this, the external correlation cannot be interpreted as evidence for the usefulness of the dependency annotations.
  3. [Section 3, inclusion of 'free' word order languages and subset representativeness] The paper states that 'free' word order languages are 'extremely difficult to distinguish' from other classes and that identifying free versus fixed order from corpus proportions 'does not seem to be possible.' Yet the ANOVA reported for the N1 ratio includes 'free' as one of the four groups. If free-order languages overlap heavily with all other groups, their inclusion can dilute or distort the test, and the claim of a four-way differentiation is misleading. The authors should run the analysis separately on fixed-order languages only, report pairwise comparisons between VI, VM, and VF, and state explicitly what the N1 ratio can and cannot distinguish. In addition, the manuscript does not discuss whether the 961-language subset is representative of the full taggedPBC or of the world's languages; this should be addressed, especially since the paper emphasizes the dataset's broad family and isolate coverage.
minor comments (6)
  1. [Throughout] There are several typographical errors and formatting issues: 'comparibility' should be 'comparability', 'classfications' should be 'classifications', 'ANOV A' and 'ANOV As' should be 'ANOVA' and 'ANOVAs', and the title page renders 'thetaggedPBC' without spacing.
  2. [Section 3, Figures 2–5] The figures are referenced but not fully described in the text; in particular, Figure 5 should include confidence intervals or error bars and a description of what the plotted values represent. As it stands, a reader cannot verify the reported ANOVA result from the figure alone.
  3. [Section 3, statistical reporting] The sentence 'see stats at the linked repository' is not sufficient for a refereed paper. The paper should include a table with group sizes, means, standard deviations, F-statistic, degrees of freedom, p-value, and effect size (e.g., eta-squared) for the ANOVA, as well as post-hoc test results.
  4. [Section 3, N1 ratio definition] The N1 ratio was introduced in Ring (2025a) for intransitive SV/VS word order. The paper should explain more explicitly how the ratio is adapted to transitive clauses and why clause-initial position, rather than the full distribution of six order patterns, is the chosen operationalization.
  5. [References] The reference to Baylor et al. (2024) contains a duplicated URL fragment ('https://arxiv.org/abs/2402.01513.2402.01513'); it should be corrected to a single URL.
  6. [Section 3, 'free' classification] The term 'free' word order is used without operationalizing how it is defined across WALS, Grambank, and Autotyp; these databases may use different criteria for 'free', which could affect the pooled analysis. Please clarify how the classifications were combined.

Circularity Check

1 steps flagged · score 4.0 of 10

N1 ratio's four-way word-order claim is partially definitional: it encodes verb-initial vs. not, but cannot distinguish verb-medial from verb-final.

  1. self definitional [Section 3, final paragraph of 'Correspondences with transitive word order']
    "Conducting an ANOV A using the N1 ratio as the dependent variable and the VI, VM, VF and free classifications as the fixed factors reveals that the N1 ratio clearly differentiates between these groups (p < 0.003; see Figure 5). Surprisingly, this measure, derived simply from a count of verses that start with an argument divided by those that start with a predicate, allow us to determine whether a language can be classified as having VI, VM, VF, or free word order"

    The N1 ratio is defined as argument-initial verses divided by predicate-initial verses. In the six transitive word orders, only VSO and VOS are predicate-initial; SVO, OVS, SOV, and OSV are all argument-initial. Hence the ratio cannot distinguish verb-medial (SVO/OVS) from verb-final (SOV/OSV) by construction. The reported ANOVA (p<0.003) can therefore only reflect a VI-versus-non-VI contrast, not a four-way differentiation. The claim that N1 'clearly differentiates' all four groups overstates what the measure can do: the VI/non-VI distinction is baked into the definition, while the VM/VF distinction is entirely absent. External databases provide independent labels, so the correlation is not wholly circular, but the four-way claim reduces to a binary contrast by construction.

full rationale

The primary derivation chain transfers dependencies from English via alignments, computes word-order proportions, and then tests the N1 ratio against external typological classifications. The external classifications are independent of the corpus measure, so the correlation is not fitted or derived from the same labels. However, the N1 ratio is definitionally tied to clause-initial position: it counts argument-initial versus predicate-initial verses. Since VI languages are predicate-initial and VM/VF languages are argument-initial, the ratio's correlation with VI versus the rest is expected by definition; it cannot separate VM from VF. The paper does not report post-hoc contrasts or group means, so the claimed four-way differentiation is not supported. This is a self-definitional overreach rather than a full circularity, because the expert databases still provide external grounding for the VI/non-VI contrast. The dependency-transfer accuracy is explicitly not directly validated, but that is a validity concern, not a circularity concern. Overall, the central claim is partially definitional but retains some independent empirical content, warranting a moderate score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of automatically transferred dependencies and POS tags. These are inherited from the author's prior work and from SpaCy's English parser; the paper contributes no new fitted parameters. The main domain assumptions concern transferability of dependencies and representativeness of Bible translations.

free parameters (2)
  • IBM Model 2 word alignment parameters (from Ring 2025a) = trained model parameters, not enumerated
    Used to identify translation equivalents for dependency transfer; quality of transferred dependencies and hence the N1 ratio depends on these alignments.
  • SpaCy English parser parameters = pretrained English pipeline, version unspecified
    English dependency trees transferred to target languages; the source of subject/object labels.
assumptions (4)
  • domain assumption Dependency relations transfer faithfully across translation equivalents in parallel Bible verses.
    The entire method in Section 2 assumes that if an English word is aligned to a target-language word, the dependency relation can be transferred to the target word. This is stated in Section 2 but never directly validated.
  • domain assumption UD 'subject' and 'object' labels are meaningful for all languages in the dataset.
    Section 3 acknowledges the subject/object debate for ergative languages but proceeds with the labels anyway.
  • domain assumption The Bible translations in the corpus reflect natural target-language word order rather than source-language translationese.
    The paper does not address the known issue of Bible translation style influencing word order; the entire typological inference depends on this.
  • domain assumption WALS, Grambank, and Autotyp classifications are accurate and comparable after reduction to VI/VM/VF/free.
    Section 3 reduces heterogeneous categories from three databases to four and assumes this is valid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Extending dependencies to the taggedPBC: Word order in transitive clauses." pith.science (2026). https://pith.science/paper/JGVGR2XN

@misc{pith2026250606785,
  author       = {Pith},
  title        = {Pith review of: Extending dependencies to the taggedPBC: Word order in transitive clauses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JGVGR2XN}},
  note         = {Machine review of arXiv:2506.06785}
}
read the original abstract

The taggedPBC (Ring 2025a) contains more than 1,800 sentences of pos-tagged parallel text data from over 1,500 languages, representing 133 language families and 111 isolates. While this dwarfs previously available resources, and the POS tags achieve decent accuracy, allowing for predictive crosslinguistic insights (Ring 2025b), the dataset was not initially annotated for dependencies. This paper reports on a CoNLLU-formatted version of the dataset which transfers dependency information along with POS tags to all languages in the taggedPBC. Although there are various concerns regarding the quality of the tags and the dependencies, word order information derived from this dataset regarding the position of arguments and predicates in transitive clauses correlates with expert determinations of word order in three typological databases (WALS, Grambank, Autotyp). This highlights the usefulness of corpus-based typological approaches (as per Baylor et al. 2023; Bjerva 2024) for extending comparisons of discrete linguistic categories, and suggests that important insights can be gained even from noisy data, given sufficient annotation. The dependency-annotated corpora are also made available for research and collaboration via GitHub.

Figures

Figures reproduced from arXiv: 2506.06785 by the authors.

Figure 1
Figure 1. Word order proportions for 3 languages Further, while Autotyp identifies basic transitive word order similarly to WALS (replacing “S” with “A”), some languages are identified by the position of the verb in relation to other elements (Tohono O’odham [ood] is listed as “Vxx”, for example). Accordingly, in the interest of compar￾ibility I reduced the 7 transitive word order possibilities to 4: Verb initial (“VI”, subsu… view at source ↗
Figure 2
Figure 2. Verb-initial proportions [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 5
Figure 5. N1 ratio correspondences [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 15 canonical work pages

  1. [1]

    Baylor, Emi, Esther Ploeger, & Johannes Bjerva. 2023. The past, present, and future of typological databases in NLP . In Bouamor, Houda, Juan Pino, & Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1163--1169. Singapore: Association for Computational Linguistics. ://aclanthology.org/2023.findings-emnlp.82/

  2. [2]

    Baylor, Emi, Esther Ploeger, & Johannes Bjerva. 2024. Multilingual gradient word-order typology from universal dependencies. ://arxiv.org/abs/2402.01513. 2402.01513

  3. [3]

    Bickel, Balthasar, Johanna Nichols, Taras Zakharko, Alena Witzlack-Makarevich, Kristine Hildebrandt, Michael Rie ler, Lennart Bierkandt, Fernando Z \'u \ n iga, & John B Lowe. 2023. The AUTOTYP database (v1.1.1)

  4. [4]

    Bjerva, Johannes. 2024. The role of typological feature prediction in nlp and linguistics. Computational Linguistics, 50(2):781--794. ISSN 0891-2017. ://doi.org/10.1162/coli\_a\_00498. https://direct.mit.edu/coli/article-pdf/50/2/781/2457439/coli\_a\_00498.pdf

  5. [5]

    Choudhary, Chinmay & Colm O ' riordan. 2023. Multilingual end-to-end dependency parsing with linguistic typology knowledge. In Beinborn, Lisa, Koustava Goswami, Saliha Murado g lu, Alexey Sorokin, Ritesh Kumar, Andreas Shcherbakov, Edoardo M. Ponti, Ryan Cotterell, & Ekaterina Vylomova (eds.), Proceedings of the 5th Workshop on Research in Computational L...

  6. [6]

    & Martin Haspelmath (eds.)

    Dryer, Matthew S. & Martin Haspelmath (eds.). 2013. WALS Online (v2020.4). Zenodo

  7. [7]

    Hammarstr \"o m, Harald, Robert Forkel, Martin Haspelmath, & Sebastian Bank. 2024. Glottolog 5.1. Leipzig: Max Planck Institute for Evolutionary Anthropology. ://glottolog.org. Accessed on 2025-04-03

  8. [8]

    Imani, Ayyoob, Silvia Severini, Masoud Jalili Sabet, Fran c ois Yvon, & Hinrich Sch \"u tze. 2022. Graph-based multilingual label propagation for low-resource part-of-speech tagging. In Goldberg, Yoav, Zornitsa Kozareva, & Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1577--1589. Abu Dhabi, U...

Show all 16 references
  1. [9]

    Levshina, Natalia, Savithry Namboodiripad, Marc Allassonni \`e re-Tang, Mathew Kramer, Luigi Talamo, Annemarie Verkerk, Sasha Wilmoth, Gabriela Garrido Rodriguez, Timothy Michael Gupton, Evan Kidd, Zoey Liu, Chiara Naccarato, Rachel Nordlinger, Anastasia Panova, & Natalia Stoy...

  2. [10]

    Greenhill, Christoph Rzymski, Johannes Englisch, & Russell D

    List, Johann-Mattis, Robert Forkel, Simon J. Greenhill, Christoph Rzymski, Johannes Englisch, & Russell D. Gray. 2022. Lexibank, a public repository of standardized wordlists with computed phonological and lexical features. Scientific Data, 9(1):316. ://doi.org/10.1038/s41597-...

  3. [11]

    Mayer, Thomas & Michael Cysouw. 2014. Creating a massively parallel B ible corpus. In Proceedings of The International Conference on Language Resources and Evaluation (LREC), pp. 3158--3163. Reykjavik

  4. [12]

    Ring, Hiram. 2025 a . The taggedPBC: Annotating a massive parallel corpus for crosslinguistic investigations. ://arxiv.org/abs/2505.12560. 2505.12560

  5. [13]

    Ring, Hiram. 2025 b . Word length predicts word order: "min-max"-ing drives language evolution. ://arxiv.org/abs/2505.13913. 2505.13913

  6. [14]

    Haynie, Dami \'a n E

    Skirg rd, Hedvig, Hannah J. Haynie, Dami \'a n E. Blasi, Harald Hammarstr \"o m, & et al. 2023. Grambank reveals global patterns in the structural diversity of the world's languages. Science Advances, 9

  7. [15]

    \"U st \"u n, Ahmet, Arianna Bisazza, Gosse Bouma, & Gertjan van Noord. 2020. UD apter: Language adaptation for truly U niversal D ependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2302--2315. Online: Asso...

  8. [16]

    Zeman, Daniel, Joakim Nivre, Mitchell Abrams, & et al. 2024. Universal dependencies 2.14. LINDAT / CLARIAH - CZ digital library at the Institute of Formal and Applied Linguistics ( \'U FAL ), Faculty of Mathematics and Physics, Charles University

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.