{"id":"8d3dddc9-dc01-422c-9fd1-5ba4b2a68c3a","arxiv_id":"2506.06785","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A dependency-projected version of the taggedPBC yields corpus word-order statistics that correlate with expert transitive word order classifications in WALS, Grambank, and Autotyp.","lead":"This paper adds automatic dependency annotations to a parallel Bible corpus covering more than 1,500 languages, then shows that a simple corpus statistic, the N1 ratio, matches expert classifications of transitive word order. The work supports the idea that noisy, automatically annotated data can still support large-scale crosslinguistic typology.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The described N1 ratio cannot distinguish verb-medial from verb-final orders, so the ANOVA result may not support the claimed four-way differentiation.","rationale":"The reader's weakest-assumption analysis focuses on unvalidated dependency transfer. That is a genuine risk, but it is not the single most load-bearing issue because the paper's own textual definition of the N1 ratio is insufficient for the claimed discrimination. Section 3 says N1 is 'a count of verses that start with an argument divided by those that start with a predicate.' In canonical verb-medial (SVO, OVS) and verb-final (SOV, OSV) orders, the first constituent is an argument in both cases; only verb-initial orders begin with the predicate. The ANOVA p < 0.003 is therefore likely driven by the VI vs non-VI contrast, and no post-hoc tests or effect sizes are reported to support the stronger claim that VM, VF and free are mutually separated. This concern is internal to the paper and does not depend on the accuracy of the transferred dependencies. It can be settled directly from the released repository by recomputing the ratio and running pairwise contrasts. I would keep the reader's conditional verdict: the manuscript would need to clarify the actual definition of N1 ratio and show post-hoc contrasts before the central validation claim can be accepted.","tokens_in":7335,"tokens_out":7666,"duration_ms":86025,"concrete_test":"Using the code and CoNLL-U files in the linked repository, recompute the N1 ratio for the 961 languages and run a one-way ANOVA with Tukey HSD post-hoc contrasts for VI vs VM, VI vs VF, VM vs VF, and each class vs free. Report group means, standard deviations, and 95% confidence intervals. Also inspect the implementation to determine whether N1 ratio is actually computed as argument-initial divided by predicate-initial or as some other quantity. If the VM-VF contrast is not significant, the central claim reduces to VI vs non-VI and should be revised accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing issue is not dependency accuracy but the definition of the N1 ratio itself. Section 3 describes N1 as 'a count of verses that start with an argument divided by those that start with a predicate.' For canonical VM orders (SVO/OVS) and canonical VF orders (SOV/OSV), the first constituent is an argument either way; only VI orders (VSO/VOS) are predicate-initial. Thus this ratio is structurally unable to separate VM from VF. The one-way ANOVA reported (p < 0.003) can therefore be significant only through the VI-versus-rest contrast, unless the actual implementation computes a different quantity than the prose states. The paper reports no post-hoc contrasts, effect sizes, or group means, so the central claim that 'N1 ratio clearly differentiates between these groups' is not established. This concern is independent of the unvalidated dependency transfer; even perfect dependency annotation would not make the stated ratio distinguish verb-medial from verb-final languages.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the taggedPBC, a massively parallel Bible corpus with automatically transferred POS tags, by adding dependency annotations transferred from English SpaCy parses through IBM Model 2 word alignments, and releases the result in CoNLL-U format. It then tests whether corpus-derived measures—notably the 'N1 ratio', defined as the count of verses starting with an argument divided by those starting with a predicate—correlate with expert word-order classifications (VI, VM, VF, free) in WALS, Grambank, and Autotyp for 961 overlapping languages. The paper reports a significant one-way ANOVA (p < 0.003) and interprets this as showing that the N1 ratio clearly differentiates the four word-order groups, arguing that noisy large-scale parallel data can support coarse-grained typological investigation.","tokens_in":7524,"tokens_out":2962,"duration_ms":33466,"significance":"If the central correlation were robustly established, the paper would make a useful contribution to computational typology by showing that automatically transferred dependency annotations over a very large parallel corpus can recover a coarse typological distinction, and the released dependency-annotated corpora would be a valuable resource. The paper also deserves credit for making code and data publicly available and for positioning the work as a validation exercise rather than a claim of gold-standard annotation. However, the current statistical and conceptual evidence does not support the strong claim that the N1 ratio differentiates among all four word-order classes, and the unvalidated dependency transfer leaves the measure's reliability open to question.","major_comments":[{"comment":"The N1 ratio, as defined in the text ('a count of verses that start with an argument divided by those that start with a predicate'), is structurally unable to distinguish verb-medial from verb-final orders. For canonical VM (SVO, OVS) and VF (SOV, OSV) orders, the clause-initial constituent is an argument, whereas for VI (VSO, VOS) orders it is a predicate. Thus the ratio can separate VI from non-VI languages, but it cannot by itself separate VM from VF. The one-way ANOVA with p < 0.003 across the four groups may therefore be driven entirely by the VI-versus-rest contrast. The paper reports no post-hoc pairwise tests, group means, standard deviations, or effect sizes, so the sentence 'the N1 ratio clearly differentiates between these groups' is not supported by the evidence presented. Please report the group-level statistics and post-hoc contrasts, and either reformulate the claim to 'differentiates VI from VM/VF' or provide a different measure that distinguishes VM from VF.","section":"Section 3, N1 ratio definition and ANOVA"},{"comment":"The N1 ratio depends on identifying which word at the start of a verse is an argument versus a predicate, and this identification comes from dependency relations transferred from English through automatic word alignments. The paper explicitly declines to validate the transferred dependencies (Section 2), yet the central correlation is based entirely on them. If the transferred subject/object labels are systematically biased—for example, if English SVO word order influences which token is labeled as the subject—the N1 ratio could reflect English or alignment artifacts rather than the target language's own word order. The paper should either provide direct validation of the transferred dependencies on a hand-annotated sample (as was done for POS tags in Ring 2025a) or conduct a robustness analysis showing that the reported ANOVA result is stable under plausible levels of dependency-label noise. Without this, the external correlation cannot be interpreted as evidence for the usefulness of the dependency annotations.","section":"Section 2 and Section 3, dependency transfer validity"},{"comment":"The paper states that 'free' word order languages are 'extremely difficult to distinguish' from other classes and that identifying free versus fixed order from corpus proportions 'does not seem to be possible.' Yet the ANOVA reported for the N1 ratio includes 'free' as one of the four groups. If free-order languages overlap heavily with all other groups, their inclusion can dilute or distort the test, and the claim of a four-way differentiation is misleading. The authors should run the analysis separately on fixed-order languages only, report pairwise comparisons between VI, VM, and VF, and state explicitly what the N1 ratio can and cannot distinguish. In addition, the manuscript does not discuss whether the 961-language subset is representative of the full taggedPBC or of the world's languages; this should be addressed, especially since the paper emphasizes the dataset's broad family and isolate coverage.","section":"Section 3, inclusion of 'free' word order languages and subset representativeness"}],"minor_comments":[{"comment":"There are several typographical errors and formatting issues: 'comparibility' should be 'comparability', 'classfications' should be 'classifications', 'ANOV A' and 'ANOV As' should be 'ANOVA' and 'ANOVAs', and the title page renders 'thetaggedPBC' without spacing.","section":"Throughout"},{"comment":"The figures are referenced but not fully described in the text; in particular, Figure 5 should include confidence intervals or error bars and a description of what the plotted values represent. As it stands, a reader cannot verify the reported ANOVA result from the figure alone.","section":"Section 3, Figures 2–5"},{"comment":"The sentence 'see stats at the linked repository' is not sufficient for a refereed paper. The paper should include a table with group sizes, means, standard deviations, F-statistic, degrees of freedom, p-value, and effect size (e.g., eta-squared) for the ANOVA, as well as post-hoc test results.","section":"Section 3, statistical reporting"},{"comment":"The N1 ratio was introduced in Ring (2025a) for intransitive SV/VS word order. The paper should explain more explicitly how the ratio is adapted to transitive clauses and why clause-initial position, rather than the full distribution of six order patterns, is the chosen operationalization.","section":"Section 3, N1 ratio definition"},{"comment":"The reference to Baylor et al. (2024) contains a duplicated URL fragment ('https://arxiv.org/abs/2402.01513.2402.01513'); it should be corrected to a single URL.","section":"References"},{"comment":"The term 'free' word order is used without operationalizing how it is defined across WALS, Grambank, and Autotyp; these databases may use different criteria for 'free', which could affect the pooled analysis. Please clarify how the classifications were combined.","section":"Section 3, 'free' classification"}],"recommendation":"major_revision","confidential_remarks":"The manuscript describes a useful resource and a plausible broad-brush validation idea, but the main quantitative claim—that the N1 ratio differentiates all four word-order classes—is not currently supported because the ratio cannot structurally separate VM from VF and because no post-hoc or effect-size information is provided. I would encourage the editor to request a revision that either narrows the claim to what the measure can actually separate or supplements the analysis with a measure that distinguishes verb-medial from verb-final orders, together with validation or robustness checks of the transferred dependencies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The CoNLL-U version of the taggedPBC is a genuinely useful resource, and I'd be glad to see it out there. But the paper's central claim about the N1 ratio differentiating four word-order classes doesn't hold up under inspection, and the ANOVA as reported can't carry the weight. What is new: a dependency-annotated, CoNLL-U formatted parallel corpus for 1,500+ languages, released on GitHub. That is a real contribution. The paper also does something right in correlating corpus-derived measures against three typological databases (WALS, Grambank, Autotyp), and it is honest about annotation noise and English-centric transfer. The 961-language matched subset is a decent size for this kind of work. The soft spot is not the unvalidated dependency transfer, though that is a secondary concern. The load-bearing issue is the N1 ratio itself. Section 3 describes it as a count of verses that start with an argument divided by those that start with a predicate. For canonical verb-medial (SVO/OVS) and verb-final (SOV/OSV) orders, the first constituent is an argument either way; only verb-initial orders (VSO/VOS) start with the predicate. So the ratio is structurally unable to separate VM from VF. The ANOVA p<0.003 is therefore likely driven by the VI-versus-rest contrast. No post-hoc tests, effect sizes, or group means are reported, so the claim of four-way differentiation is not established. This is independent of dependency accuracy; even perfect dependencies would not fix the problem. The paper's own Figures 2-4, which plot proportions of each verb-position pattern separately, might contain the real signal, but the ANOVA on N1 is not the right test. The subset of 961 languages is also not checked for representativeness, though that is a minor point. Who is this for? Computational typologists who want a large, noisy parallel corpus with dependency annotations, and anyone working on word order classification from corpus data. The dataset deserves serious referee time, but the paper needs major revision: either redefine the measure so it can actually separate VM from VF, or use the six-pattern proportions and report post-hoc contrasts. I would send it to review, but with a clear request for stronger statistics and a re-analysis of the N1 claim.","headline":"Useful dataset release, but the headline N1-ratio result cannot separate verb-medial from verb-final and needs post-hoc stats or a redefinition.","tokens_in":8012,"tokens_out":2698,"would_cite":true,"duration_ms":24099,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatically transferred dependency labels, summarized as an $N_1$ ratio, separate expert classifications of transitive word order into verb-initial, verb-medial, verb-final, and free categories across 961 languages.","keywords":["computational typology","word order","dependency projection","cross-lingual annotation transfer","parallel corpora","taggedPBC","CoNLL-U","under-resourced languages"],"falsifier":"Hand-annotate the subject, object, and verb in a stratified sample of verses from 25 to 30 languages spanning several families and word-order types, recompute the $N_1$ ratio from those gold labels, and check whether it still separates verb-initial, verb-medial, verb-final, and free languages at the reported significance level; if the separation disappears or diverges from the transferred-data ratios, the correlation is an artifact of the English-based transfer.","tokens_in":7138,"feed_emoji":"🌐","tokens_out":14184,"duration_ms":134702,"temperature":0.7,"pith_summary":"The paper extends the taggedPBC, a part-of-speech-tagged parallel corpus covering more than 1,500 languages, by projecting English dependency annotations through word alignments into every language and converting the result to CoNLL-U format. Its central claim is that one number computed from these noisy annotations, the $N_1$ ratio (verses whose first word is an argument divided by verses whose first word is the predicate), separates languages classified by experts as verb-initial, verb-medial, verb-final, or free word order; across 961 overlapping languages the separation is significant at $p < 0.003$. The paper does not directly validate the accuracy of the transferred dependency relations, yet argues that at this scale the aggregate signal still lines up with expert typological judgments. If the claim is right, automatic transfer from a single high-resource language can give coarse but real word-order typology for languages with no hand-built treebanks.","feed_headline":"Noisy auto-labels match expert word-order classes in 961 languages","feed_subtitle":"A single start-of-clause ratio separates the four basic transitive order types, giving noisy corpora typological bite.","key_machinery":"The load-bearing object is the $N_1$ ratio, defined as the number of transitive verses whose first word is labelled as an argument (subject or object) divided by the number whose first word is the predicate; it compresses the six-way word-order space into a single gradient that separates the four collapsed expert classes. The mechanism producing it is a transfer pipeline: English sentences are parsed with an automatic dependency parser, IBM Model 2 word alignments map the English words to words in each target language, and the English dependency and morphological labels are transferred across the alignment links into CoNLL-U format. The paper deliberately substitutes broad validation against typological databases for direct validation of the transferred dependency labels, treating the correlation as evidence that the labels carry usable signal despite their noise.","core_discovery":"The discovery, stated on the paper's own terms, is that corpus measures built from dependency relations that were never hand-annotated can reproduce expert judgments about basic transitive word order. In the CoNLL-U version of the taggedPBC, English sentences are parsed automatically and their subject and object labels are copied to aligned words in each target language through word alignments. For every language the paper counts verses containing a verb plus subject and object relations, computes proportions for the six transitive orders, collapses these to verb-initial, verb-medial, verb-final, and free, and finds that the $N_1$ ratio differentiates the four classes with a one-way analysis of variance at $p < 0.003$. The paper also observes that every language corpus contains at least one verse for each of the six orders, that free-order languages cannot be separated from fixed ones by the six proportions alone, and that these gradient patterns support the reality of discrete categories: languages grammaticalize one order while still showing all orders to differing degrees.","pith_inferences":["A quick check the paper does not run is a permutation baseline: shuffle the dependency labels across aligned words and recompute the $N_1$ ratio; if the analysis-of-variance separation survives the shuffling, the correlation is alignment noise rather than target-language word order.","The paper's finding that free-order languages resist detection through order proportions suggests an untested corollary: within-corpus variance of verse-level $N_1$ ratios, or the entropy of the six-order distribution, may separate free from fixed languages even when the mean proportions do not.","Because subject and object labels are transferred under a nominative-accusative reading, the method may systematically mislabel the single argument of ergative languages; a bounded test is to recompute the $N_1$-ratio separation within a sample of ergative languages once hand-annotated dependencies exist.","The same transfer pipeline could be applied to a non-religious parallel corpus with comparable sentence counts; if the verb-initial, verb-medial, verb-final, and free separation persists there, the result generalizes beyond Bible-verses translation style rather than reflecting a genre-specific artifact."],"forward_implications":["The $N_1$ ratio can be used to assign coarse transitive word-order classes to taggedPBC languages that are absent from the three expert typological databases, extending word-order typology to languages without hand-coded classifications.","Because all six transitive orders occur in every language's corpus, word order is better treated as gradient; the significant $N_1$-based separation suggests that discrete categories such as verb-initial, verb-medial, and verb-final emerge from the relative frequency of competing orders.","Free word order cannot be detected directly from the six order proportions, since free-order languages scatter across the verb-initial, verb-medial, and verb-final ranges; the $N_1$ ratio, not the proportions, is the measure that aligns with the free classification.","The English-centric annotation restricts the dataset's use: language-specific categories such as numeral classifiers are largely missing, no morphological annotations are present, and dependent clauses are not distinguished from main clauses, so the dataset supports coarse features like word order rather than fine-grained morphosyntactic analysis."],"supporting_citations":[{"why":"Describes the taggedPBC corpus, its POS-tag transfer, and the word alignments that this study reuses for dependency transfer.","marker":"Ring 2025a"},{"why":"Created the Parallel Bible Corpus from which the taggedPBC's aligned verses are drawn.","marker":"Mayer & Cysouw 2014"},{"why":"One of the three expert typological databases whose transitive word-order classifications form the validation target.","marker":"Dryer & Haspelmath 2013"},{"why":"Grambank, whose verb-position and fixed/free determinations are folded into the four-way classification.","marker":"Skirgård et al. 2023"},{"why":"Autotyp, the third expert database contributing basic transitive word-order labels.","marker":"Bickel et al. 2023"},{"why":"Supplies the gradient-approach argument that justifies measuring word-order proportions rather than relying on discrete labels alone.","marker":"Levshina et al. 2023"},{"why":"Defines Universal Dependencies and the CoNLL-U format into which the transferred annotations are converted.","marker":"Zeman et al. 2024"}],"fun_headline_variants":["Noisy auto-labels match expert classes in 961 languages","One start-of-clause ratio sorts four word-order types","Copied dependencies match expert typology in 961 languages","Single ratio splits transitive orders in 961 languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result collapses if the transferred subject and object labels are systematically wrong, because the $N_1$ ratio would then be measuring English word order or alignment noise rather than the target language's clause structure; the paper does not test the labels directly, only their downstream correlation, a gap it acknowledges in the methodology.","fun_headline_variants_meta":{"raw":{"variants":["Noisy auto-labels match expert classes in 961 languages","One start-of-clause ratio sorts four word-order types","Copied dependencies match expert typology in 961 languages","Single ratio splits transitive orders in 961 languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000999,"raw_usage":{"total_tokens":4238,"prompt_tokens":962,"completion_tokens":3276,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":3210}},"tokens_in":578,"tokens_out":3276,"duration_ms":24176,"temperature":1.0,"reasoning_tokens":3210,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:48:51.252949+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-annotate the subject, object, and verb in a stratified sample of verses from 25 to 30 languages spanning several families and word-order types, recompute the $N_1$ ratio from those gold labels, and check whether it still separates verb-initial, verb-medial, verb-final, and free languages at the reported significance level; if the separation disappears or diverges from the transferred-data ratios, the correlation is an artifact of the English-based transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Created the Parallel Bible Corpus from which the taggedPBC's aligned verses are drawn."},{"cited_title":"& Martin Haspelmath (eds.)","cited_arxiv_id":null,"evidence_quote":"One of the three expert typological databases whose transitive word-order classifications form the validation target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Autotyp, the third expert database contributing basic transitive word-order labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-approach argument that justifies measuring word-order proportions rather than relying on discrete labels alone."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Universal Dependencies and the CoNLL-U format into which the transferred annotations are converted."}],"review_version":1}