On the feasibility of dependency parsing of non-human sequences without a gold standard. Is evaluation possible in other species?

Catherine Hobaiter; Morgan Gustison; Ramon Ferrer-i-Cancho; Thore Bergman

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Random parsers hit 79%+ accuracy on chimp sequences

2026-07-08 02:16 UTC pith:ACCDVW3U

load-bearing objection A clean mathematical result (2/n expected edge accuracy for a random parser) applied to animal communication, but the feasibility claim is limited by trivially small tree spaces at short sequence lengths. the 1 major comments →

arxiv 2607.06542 v2 pith:ACCDVW3U submitted 2026-07-07 cs.CL

On the feasibility of dependency parsing of non-human sequences without a gold standard. Is evaluation possible in other species?

Ramon Ferrer-i-Cancho , Catherine Hobaiter , Thore Bergman , Morgan Gustison This is my paper

classification cs.CL

keywords dependency parsingunsupervised parsingnon-human primate communicationsequence length distributiongold standard evaluationrandom spanning treesgeometric distributiongelada vocalizations

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that dependency parsing of non-human primate vocal and gestural sequences can be evaluated without a gold standard because primate sequences are so short that even a random parser achieves high expected accuracy. The key result is that for a sequence of length n, a uniformly random parser correctly identifies 2/n of the edges on average, derived from a result on the intersection of random spanning trees. Since primate sequence length distributions decay exponentially (geometrically), most sequences are very short, yielding expected edge accuracies of 51% for geladas and over 79% for chimpanzees. In contrast, human sentence lengths follow an inverted-V distribution with much longer averages, giving a random parser only about 13-28% accuracy. The paper thus claims that evaluation without a gold standard is feasible for non-human primates but remains a hard problem for human language. The paper also outlines a practical methodology: verify statistical structure in the sequences, train an unsupervised parser, validate it against chance-level performance, and use the random-parser lower bound to certify that its accuracy must be high.

Core claim

The expected proportion of correct dependency edges retrieved by a random parser on a sequence of length n is 2/n (Eq. C.2), a direct consequence of the expected intersection size of two uniformly random labeled trees on n vertices. When sequence lengths follow a geometric distribution with rapid decay—as observed in gelada vocalizations and chimpanzee vocal and gestural sequences—the expectation E[Q] = 2E[1/n] becomes large because the distribution is dominated by short sequences. For geladas this yields approximately 51% expected edge accuracy and for chimpanzees over 79%, establishing a lower bound that any reasonable unsupervised parser must exceed. Human languages, whose sentence length

What carries the argument

The 2/n expected edge accuracy of a random parser on a sequence of length n, derived from the intersection of two uniformly random labeled trees (London and Pluhár 2023, Observation 3), combined with the exponentially decaying (geometric) distribution of primate sequence lengths.

Load-bearing premise

The paper assumes that all labeled trees of the same size are equally likely to be the correct dependency tree for a sequence. This uniform prior is what makes the 2/n expected accuracy hold. If real primate dependency structures are not uniformly distributed over possible trees—for example, if certain topologies are preferred due to cognitive constraints—the lower bound could change.

What would settle it

If empirical dependency structures in primate sequences were found to be non-uniformly distributed over labeled trees, such that the expected intersection of two random trees differs from 2(n-1)/n, then the 2/n accuracy bound would not hold and the lower-bound argument for gold-standard-free evaluation would collapse.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Unsupervised dependency parsers trained on primate vocal or gestural sequences can be evaluated even when no annotated gold-standard treebank exists, because the random-parser lower bound provides a floor on expected accuracy.
The contrast between primate and human sequence-length distributions implies that gold-standard-free evaluation is species-dependent: feasible for short-sequence species, hard for long-sequence species including humans.
The methodology outlined in Section 5.3 provides a concrete pipeline for other species: check for statistical structure, verify fast length decay, train, validate against chance, and apply the lower bound.
Species whose maximum sequence length is 2 or 3 (covering 25 of 31 primate species in the review) have trivially high random-parser accuracy, suggesting parsing is immediately feasible for them without any gold standard.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the uniform prior over tree structures is violated—say, primate sequences preferentially use certain dependency topologies due to cognitive or production constraints—the 2/n bound could shift. Testing whether actual primate dependency structures are uniformly distributed over labeled trees would require at least a small annotated corpus, creating a circular dependency with the gold-standard-fre
The argument could extend beyond primates: any species whose sequence-length distribution decays rapidly (e.g., birdsong with short motif sequences) would inherit the same lower-bound guarantee, broadening the scope of gold-standard-free parsing evaluation.
The high accuracy of the random parser on short sequences may paradoxically make it harder to demonstrate that an unsupervised parser is genuinely learning structure, since the margin between chance and learned performance is compressed—especially for chimpanzee gestures where the random floor exceeds 84%.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A clean mathematical result (2/n expected edge accuracy for a random parser) applied to animal communication, but the feasibility claim is limited by trivially small tree spaces at short sequence lengths.

read the letter

The main thing to know: this paper applies the tree intersection game value from London and Pluhár (2023) to get E[P_c^e|n] = 2/n for a random dependency parser on sequences of length n, then argues that because primate sequence lengths decay geometrically, the random baseline is already high (51% for geladas, >79% for chimpanzees with n_min=2), so gold-standard-free evaluation is feasible in non-human primates but not in humans. The math is correct and the synthesis is genuinely new — nobody has framed the evaluation problem this way before, and the contrast between primate and human sequence-length distributions is well-supported with real datasets from Gustison et al. (2016) and Girard-Buttoz et al. (2022). The appendices are thorough: exact closed-form expressions for uniform and geometric distributions, MLE derivations, and a numerical approximation scheme with explicit error bounds. That is solid work. The uniform-tree prior (Section 2.3) is load-bearing for the 2/n result, and the paper acknowledges it is chosen to be maximally agnostic. The reader flags this as a concern; I think it is secondary. For small n, the tree space is so constrained that even if the prior were non-uniform, the expected intersection would not move much. The stress-test note raises a sharper issue: for n=2, the random parser gets 100% (zero information), and for n=3, there are only 3 labeled trees, so 2/n ≈ 67% is trivially high. The paper does report n_min=3 results (E[Q] drops to 41% for geladas, ~57% for chimpanzee vocal), which is the right thing to do. But the gap between the random baseline and what a real parser would achieve on these short sequences is entirely uncharacterized — no parser is run. The paper's own discussion (Section 5.1) gestures at this, arguing that a good-enough parser is channeled toward high accuracy, but this is an assertion, not a demonstration. The feasibility claim is defensible as a theoretical argument: the lower bound is real, and the geometric-decay observation is empirically grounded. But the practical force — whether this lower bound is actually useful for distinguishing a parser that has learned structure from one that has not — remains open. The paper is honest about this limitation in places, though the abstract and conclusion state feasibility more strongly than the evidence supports. This is a short, theoretically clean paper aimed at researchers in animal communication and computational linguistics who care about the evaluation problem. It deserves a serious referee who can assess whether the gap between the theoretical bound and practical utility is bridgeable, and who can push the authors to either run a parser or explicitly scope down the claim. I would accept this for peer review.

Referee Report

1 major / 7 minor

Summary. This short paper addresses whether unsupervised dependency parsing can be evaluated without a gold standard for non-human primate sequences. The core argument is: (1) a random parser selecting uniformly among all labeled trees on n vertices has expected edge accuracy 2/n, derived from London and Pluhár (2023, Observation 3) on random spanning tree intersections in complete graphs; (2) primate vocal and gestural sequences have fast-decaying (geometric) length distributions, so the expected accuracy E[Q] is high (51% for geladas, >79% for chimpanzees with n_min=2); (3) any 'good-enough' parser that exploits statistical structure will exceed this baseline; (4) therefore, a useful lower bound on real parser accuracy exists, making gold-standard-free evaluation feasible in non-human primates but not in humans (where E[Q] ≈ 13% for unrestricted sentence lengths). The paper computes expectations under uniform, geometric, and empirical distributions (Appendices A–C), validates the geometric approximation on gelada and chimpanzee data (Section 4.1), and provides comparative results across 21 human languages from PUD.

Significance. The paper applies a result from network science (random spanning tree intersection sizes) to a genuinely novel problem: establishing a parameter-free lower bound on parser accuracy for animal communication sequences without any gold standard. The 2/n result (Eq. C.2) is elegant and the derivation through Appendices A–C is mathematically clean. The comparative analysis across primate species and human languages is well-motivated and the datasets are drawn from established empirical studies. The paper identifies a real and underexplored problem at the intersection of computational linguistics and animal communication.

major comments (1)

Section 5.1, final paragraph: The paper's central claim is that 'evaluation without a gold standard is feasible in non-human primates.' However, what the paper actually establishes is that a random parser baseline is high. The leap from 'the baseline is high' to 'evaluation is feasible' requires that the gap between the baseline and actual parser performance be large enough to distinguish a parser that has learned genuine structure from one that has not. For n=2, the baseline is 100% (zero information). For n=3, the baseline is 2/3 ≈ 66.7%, and there are only 3 possible free trees on 3 labeled vertices. The paper does not provide any evidence—empirical or theoretical—that the gap between 2/n and a real unsupervised parser's performance is large enough to be practically useful. The paper should either (a) temper the feasibility claim to match what is proven (i.e., 'a lower bound on parser

minor comments (7)

Section 2.3: The assumption that 'all labeled trees of the same size have the same probability of being the correct tree' is stated as a modeling choice. However, for the 2/n result (Eq. C.2), this assumption is not actually load-bearing: by the symmetry of the complete graph K_n, the expected number of shared edges between a fixed tree and a uniformly random tree is 2(n-1)/n regardless of which fixed tree is chosen. The paper could strengthen its argument by noting this, as it would remove the reader's concern that non-uniform tree topology distributions in real primate sequences could invalidate the baseline.
Table 5 and Table 6 present the same data in different sort orders; consider merging or clarifying why both are needed.
Section 4.1: The geometric distribution fit for chimpanzee gestural sequences is described as having 'a breakpoint' and a 'two-regime exponential distribution,' but no formal breakpoint analysis or goodness-of-fit test is reported. A brief note on how 'good approximation' is assessed would help.
Figure 4 caption references 'the best fit of a 2-parameter geometric distribution (Eq. 10)' but Eq. 10 is the displaced geometric distribution with parameters q and n_min; the caption could clarify this is the displaced geometric, not a separate 2-parameter family.
The abstract states 'the proportion of correct edges retrieved by a parser must be high' — this phrasing could be misread as a claim about real parsers rather than the random parser baseline. Consider rephrasing to 'the expected proportion of correct edges retrieved by a random parser is high, providing a lower bound on any good-enough parser.'
Section 5.3, methodology step 'Validation of parser': the mutual information criterion for distinguishing a good-enough parser from chance is mentioned but not developed. A brief reference to how this would work in practice would strengthen the proposed methodology.
Appendix G is referenced but appears to contain only tables (G.1, G.2) with no accompanying text; a brief introductory sentence would help.

Simulated Author's Rebuttal

1 responses · 1 unresolved

The referee raises a substantive concern: that the paper establishes a high random-parser baseline but does not demonstrate that the gap between this baseline and a real unsupervised parser's performance is large enough to distinguish structured from unstructured parsing. We agree this is a legitimate gap between what is proven (the lower bound) and what is claimed (feasibility of evaluation). We will revise to temper the feasibility claim and add explicit discussion of the gap issue, while defending the core contribution as establishing a parameter-free lower bound rather than claiming full evaluative power.

read point-by-point responses

Referee: Section 5.1, final paragraph: The paper's central claim is that 'evaluation without a gold standard is feasible in non-human primates.' However, what the paper actually establishes is that a random parser baseline is high. The leap from 'the baseline is high' to 'evaluation is feasible' requires that the gap between the baseline and actual parser performance be large enough to distinguish a parser that has learned genuine structure from one that has not. For n=2, the baseline is 100% (zero information). For n=3, the baseline is 2/3, and there are only 3 possible free trees on 3 labeled vertices. The paper does not provide any evidence—empirical or theoretical—that the gap between 2/n and a real unsupervised parser's performance is large enough to be practically useful. The paper should either (a) temper the feasibility claim to match what is proven (i.e., 'a lower bound on parser [exists

Authors: We thank the referee for this incisive comment, which correctly identifies a gap between what we prove and what we claim. We agree that the paper establishes a lower bound on parser accuracy but does not demonstrate that the gap between this lower bound and the performance of a real unsupervised parser is large enough to be practically useful for distinguishing structured from unstructured parsing. This is a genuine limitation of the current contribution, and we will revise the manuscript accordingly. Specifically, we will make the following changes: (1) We will temper the central claim in the abstract and Section 5.1 from 'evaluation without a gold standard is feasible' to a more precise statement: namely, that a parameter-free lower bound on parser accuracy exists and is high for non-human primate sequences due to their length distribution, which is a necessary (but not sufficient) condition for gold-standard-free evaluation. (2) We will add an explicit discussion in Section 5.1 of the gap problem the referee identifies. We acknowledge that for n=2 the baseline carries zero information, and for n=3 the space of possible trees is so small (only 3 labeled free trees) that the gap between random and structured parsing may be negligible. We will note that the practical utility of our lower bound depends on the existence of a detectable gap, which we have not demonstrated. (3) We will reframe the contribution as establishing a foundational building block—the existence of a high lower bound—rather than claiming that full evaluation is feasible. The question of whether the gap is large enough for practical evaluation is left as an open problem for future work. We believe this framing is honest about what is proven while preserving the novelty of the contribution: applying a revision: partial

standing simulated objections not resolved

The referee's concern is well-taken and we cannot fully resolve it within the scope of this short paper: we do not have empirical results from an actual unsupervised parser trained on primate sequences, nor do we have a theoretical argument bounding the gap between 2/n and the performance of a structure-exploiting parser. The gap question is a genuine open problem that requires either empirical experiments with real parsers on primate data or theoretical analysis of specific parser classes, both of which are beyond what this paper currently provides. We will acknowledge this explicitly in the revision.

Circularity Check

0 steps flagged

No circularity: the central 2/n result derives from an external theorem (London and Pluhár 2023), and all empirical inputs come from independent external datasets.

full rationale

The paper's central derivation chain is self-contained against external benchmarks. The key mathematical result — that the expected proportion of correct edges for a random parser on a sequence of length n is 2/n (Eq. C.2) — is derived from London and Pluhár (2023, Observation 3), an external result about random spanning tree intersections in complete graphs. This is not a self-citation: London and Pluhár are not among the paper's authors, and their result is a standalone mathematical theorem about the expected size of the intersection of two uniformly random spanning trees. The geometric distribution parameters (q) used to compute E[Q] for geladas and chimpanzees are estimated via standard maximum likelihood (Property 5, Appendix A) from external datasets: Gustison et al. (2016) for gelada vocal sequences and Girard-Buttoz et al. (2022) for chimpanzee vocal sequences. The human language data comes from PUD/UD (Zeman et al. 2020). The uniform prior over labeled trees (Section 2.3: 'we assume all labeled trees of the same size have the same probability of being the correct tree') is a modeling assumption, not a circular definition: it does not define the output in terms of itself. The feasibility claim (high E[Q] for primates due to short sequences) follows from combining the external 2/n theorem with externally estimated sequence length distributions. No step in the derivation reduces to its own inputs by construction. The paper does not fit a parameter to a subset of data and then 'predict' a closely related quantity. The self-citations present (e.g., Ferrer-i-Cancho et al. 2022 for preprocessing methods, Alemany-Puig and Ferrer-i-Cancho 2024 for treebank processing) are methodological and not load-bearing for the central mathematical argument. The derivation is genuinely self-contained against external evidence.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The paper introduces no new entities (particles, forces, dimensions, etc.). It applies existing mathematical results (Cayley's formula, London and Pluhár 2023 tree intersection game) to existing datasets. The free parameters are standard distribution parameters estimated from data. The axioms are domain assumptions about tree uniformity and parser behavior, plus one ad hoc assumption about good-enough parsers exceeding random performance.

free parameters (3)

q (geometric distribution parameter) = estimated via MLE per dataset (e.g., q = 1/<n> for n_min=1)
The geometric distribution parameter q is estimated from empirical data (Appendix A, Property 5). It is a fitted parameter, not derived from first principles.
n_max (uniform distribution parameter) = borrowed from Girard-Buttoz et al. 2022 for 31 species
The maximum sequence length n_max is taken from external data for the uniform distribution lower bound (Table 4).
epsilon (approximation error) = 1e-8
Used in Appendix E for truncating the infinite sum for the geometric distribution. A computational parameter, not fitted to data.

axioms (5)

domain assumption All labeled trees of the same size have equal probability of being the correct tree.
Section 2.3: 'we assume all labeled trees of the same size have the same probability of being the correct tree.' This uniform prior is load-bearing for the 2/n expected edge accuracy result.
domain assumption Each sequence has a single correct dependency structure.
Section 2.1: 'We assume that each sequence has a single correct dependency structure, the default assumption for the syntactic dependency structure of human language sentences.' Extended to non-human primates without independent justification.
domain assumption The parser always retrieves a free tree when supplied a sequence.
Section 2.1: 'we assume that the parser always retrieves a free tree when supplied a sequence.' Simplifies analysis but may not hold for all parsing methods.
domain assumption Non-human primate sequence length distributions are non-increasing.
Section 4.2: used to justify the uniform distribution as a lower bound for the 31 species where only n_max is known. Supported empirically for geladas and chimpanzees (Figure 4) but assumed for the other 29 species.
ad hoc to paper A good-enough parser will perform better than the random parser.
Section 1: 'any good-enough unsupervised parser, namely any parser theoretically able to improve over a random parser by exploiting the statistical structure of the sequences.' This is plausible but untested in the paper.

pith-pipeline@v1.1.0-glm · 34582 in / 3615 out tokens · 530571 ms · 2026-07-08T02:16:53.539401+00:00 · methodology

0 comments

read the original abstract

Dependency parsing consists of finding a tree representation for a sequence. Unsupervised dependency parsing aims to develop parsing methods without a gold standard during model training. In human languages, an unsupervised parser can be evaluated because some gold standard is usually available or can be created. For other species, a gold standard is unknown. Thus one may conclude that it is impossible to determine the accuracy of an unsupervised parser and, consequently, dependency parsing is unfeasible in other species. However, here we apply recent advances in network science to demonstrate that the proportion of correct edges retrieved by a parser must be high for the sequences of vocalizations or gestures that non-human primates produce due to the fast decay of the sequence length distribution. In contrast, human language sequences lack that property. Therefore, evaluation without a gold standard is feasible in non-human primates but a hard problem in humans.

On the feasibility of dependency parsing of non-human sequences without a gold standard. Is evaluation possible in other species?

Core claim

What carries the argument

Load-bearing premise

What would settle it

If this is right

Where Pith is reading between the lines

discussion (0)