Pith. sign in

REVIEW 3 major objections 4 minor 29 references

The paper claims that a decoupled agentic parser, paired with an ambiguity-aware metric, substantially improves dependency parsing of spoken code-switching without retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

A 126-sentence spoken code-switching benchmark, a flexible scoring metric, and an agentic LLM parser are introduced, but the paper's tables contradict its claimed accuracy gains.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection The resource is real but the evaluation is incoherent: the paper's headline numbers are contradicted by its own tables, so it needs major revision before it can be trusted. the 3 major comments →

arxiv 2602.06307 v2 pith:FK24VCWT submitted 2026-02-06 cs.CL

Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions

classification cs.CL
keywords spoken code-switchingdependency parsingUniversal Dependenciesparsing evaluationLLM agentsdisfluenciesbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard Universal Dependencies parsing and evaluation break down on spoken code-switched conversation, not because models lack capacity but because written-text assumptions penalize linguistically valid structure. It offers three coordinated pieces: SpokeBench, a gold benchmark of 126 expert-annotated English-Spanish utterances; Flex-UD, a severity-weighted metric that tolerates acceptable variation; and DECAP, a modular agentic parser that separates disfluency handling from core syntax. The authors claim DECAP, instantiated with a proprietary LLM, outperforms traditional and LLM baselines, with Flex-UD exposing gains that attachment metrics hide. If right, the result is a path to reliable spoken parsing without retraining, and a more honest way to measure it.

Core claim

The central claim is that parsing spoken code-switching improves decisively when the problem is decomposed: a spoken-phenomena handler flags disfluencies, repetitions, and discourse markers; a language resolver normalizes contractions and multiword expressions; a core UD assigner builds the tree under those constraints; and a verifier enforces single-root acyclicity. On the authors' SpokeBench, DECAP reaches the highest Flex-UD score (76.2) versus 70.7 for an LLM pipeline and 30.4 for a traditional bilingual parser, with large reductions in catastrophic structural errors. The paper further claims that standard LAS/UAS metrics compress these gains, and that Flex-UD's graded penalties reveal t

What carries the argument

The load-bearing machinery is the decoupled agent pipeline (SPH → LSR → Core → V/R) combined with Flex-UD's severity multiplier. The four agents turn spoken-language annotation rules into explicit constraints: reparandum relations for repetitions, INTJ/discourse for fillers, dotted MWE nodes, contraction splits with integer-shifted IDs. Flex-UD aggregates five component scores (split, ID, UPOS, HEAD, DEPREL) and multiplies by (1 − P), where P penalizes catastrophic errors such as missing dotted MWEs or invalid heads. This design is what lets the paper claim interpretability and no-retraining extensibility.

Load-bearing premise

That the authors' reformulated UD guidelines and hand-tuned severity weights define what a correct parse is, so the benchmark, metric, and parser prompts all encode the same standard rather than an independent ground truth.

What would settle it

If an independent set of trained annotators following conventional UD guidelines re-annotated the same 126 utterances, and DECAP's parses matched the authors' gold far better than the independent gold, the reported superiority would be an artifact of the annotation standard. Also, if DECAP underperformed a simple fine-tuned parser on a larger spoken code-switching corpus measured by human preference, the no-retraining advantage would be questioned.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If DECAP generalizes, spoken-language dependency parsing for code-switching can be improved purely via prompt/agent design, without fine-tuning on expensive spoken corpora.
  • Flex-UD provides a template for evaluating parsers on other ambiguous or non-canonical data, where single-gold matching underestimates performance.
  • The taxonomy of nine phenomena offers a checklist for future spoken treebanks and parser stress tests.
  • The reported gains imply that standard UD metrics may be systematically under-reporting progress on conversational speech, affecting how leaderboards are read.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A closed-loop risk is left implicit: because DECAP's prompts encode the same annotation rules that defined SpokeBench's gold, the comparison may reflect rule adherence rather than broad parsing competence; an independent gold standard would test this.
  • The benchmark's small size means the claimed difficulty hierarchies (e.g., repetition easiest under UPOS-LAS, ellipsis hardest under LAS) would be more convincing if replicated on a larger or multi-pair corpus.
  • Flex-UD's severity weights are hand-set; an empirical calibration against human acceptability judgments would make the metric portable to other languages and domains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper argues that standard Universal Dependencies (UD) parsing and evaluation are ill-suited to spoken code-switched conversation. It introduces a taxonomy of ten spoken-language phenomena; SpokeBench, an expert-annotated benchmark derived from the Miami Corpus; Flex-UD, a weighted ambiguity-aware evaluation metric; and DECAP, a four-stage LLM-based agentic parser. The authors claim that DECAP substantially improves LAS, UPOS-LAS, and Flex-UD over traditional and LLM baselines, with up to 52.6% improvement and a top Flex-UD score of 76.2. I read the full text, including the appendices, tables, and prompts.

Significance. The resource and system-design ideas are timely. The expert annotation and dispute-resolution protocol, the plan to release data and code, the deterministic agent interaction, and the separation of spoken-phenomena handling from core parsing are genuine strengths. If the claims were reliable, Flex-UD would be a useful diagnostic for spoken treebanks, and DECAP would demonstrate a no-retraining route to spoken code-switching parsing. However, the manuscript's empirical foundation is compromised by internal contradictions between the prose and the tables, and by the close coupling among the gold standard, the metric, and the system prompts. The headline results therefore cannot be accepted without substantial additional evidence and likely re-analysis.

major comments (3)
  1. [§5.2 and Tables 1–2] The headline quantitative claims are contradicted by the paper's own tables. §5.2 states that DECAP improves overall LAS from 0.31 (BiLingua) to 0.48 and UPOS-LAS from 0.70 to 0.87, and that DECAP achieves the highest overall FLEX-UD score (76.2) versus 70.7 for BiLingua and 30.4 for the traditional parser. Table 1 Overall shows DECAP LAS 0.26, below BiLingua's 0.32; DECAP U-LAS is 0.70, not 0.87; and no row in Table 1 contains a DECAP LAS of 0.48. Table 2 Overall shows DECAP's Final FLEX-UD as 66.6, below BiLingua's 72.2. This is not a rounding issue; the reported rankings are reversed by the tables. Because these numbers are the empirical basis for the abstract and conclusion, the central claim is unsupported as presented.
  2. [§3.2, Appendix C, Appendix E] The evaluation is coupled to the system prompt. SpokeBench was annotated with a 'reformulated version of UD guidelines adapted for spoken, disfluent bilingual data' (§3.2, Appendix C). The Core UD Assigner prompt in Appendix E instructs the model to follow 'Miami Gold Subset spoken-language rules' and to use 'Miami-approved lists' for UPOS and DEPREL. In effect, DECAP is prompted with the annotation conventions used to create the gold benchmark and to define Flex-UD's tolerance classes. The FLEX-UD gains may therefore measure adherence to the authors' own guidelines rather than general parsing ability. This concern should be addressed by evaluating on an independently created gold standard or on existing spoken-UD treebanks, and by reporting standard LAS/UAS on such data.
  3. [§4.2] Flex-UD is not specified sufficiently for reproduction. The formal definition leaves the component scores s_i, the weights w_i, and the severity penalty P to an appendix reference ('see App. B'), but Appendix B contains only illustrative examples of spoken phenomena, not the penalty formula or the values of w_i. The text gives per-issue penalty ranges (P=0.25–0.6 for catastrophic errors, 0.01–0.05 for minor errors) without a clipping rule or aggregation procedure. Without these details, the aggregated FLEX-UD scores in Table 2 cannot be independently computed, and the claimed 'ambiguity-aware' behavior is not checkable.
minor comments (4)
  1. [Table 3] The category counts sum to 127 sentences (10+15+10+10+10+15+10+15+12+20), but the text in §3.2 and §5.1 says 126. Please correct the arithmetic or the reported number.
  2. [Figures 3 and 4] The captions say 'DECAP ... performs the best across all categories,' but Table 1 shows several categories where BiLingua exceeds DECAP in LAS (e.g., Discourse and None). The captions/figures should be reconciled with the table values.
  3. [Abstract] The abstract claims 'over 60% improvements in UPOS-F1 Score over baselines,' but the experiments report UPOS-LAS (U-LAS) and not UPOS-F1. The metric name and the reported numbers should be aligned.
  4. [Overall presentation] There are numerous formatting inconsistencies, including irregular spacing in 'FLEX-UD' and 'DECAP' in the abstract and running text, and inconsistent category labels (e.g., 'Ellipsis' vs. 'Ellipses' in Table 2). These should be cleaned up.

Circularity Check

1 steps flagged

Evaluation loop is closed: DECAP's prompts import the same spoken-language annotation rules used to build SpokeBench, and the paper's own tables contradict its headline gains.

specific steps
  1. self definitional [Appendix E (Core UD Assigner prompt); §3.2; Appendix C]
    "Your task is to assign UD-style annotations (UPOS, HEAD_ID, DEPREL) following the Miami Gold Subset spoken-language rules. | Annotations in SpokeBench follow UD conventions with targeted extensions for spoken, disfluent, and code-switched data."

    SpokeBench's gold annotations are produced under the same reformulated UD rules that DECAP's Core agent is explicitly prompted to follow ('Miami Gold Subset spoken-language rules'). The prompt's label mappings (reparandum→rep, discourse→INTJ/discourse, dep for unresolved ellipsis) mirror the gold-annotation rules in Appendix C. Evaluating DECAP against this gold therefore measures prompt compliance with the authors' own annotation conventions; the comparison systems are not given these rules. Part of DECAP's claimed improvement is an artifact of sharing the answer standard, rather than an independent test of parsing quality.

full rationale

SpokeBench is expert-annotated and human adjudication provides some external linguistic grounding, so the benchmark is not vacuous. However, the specific evaluation loop is substantially closed: the gold annotations were created with the authors' reformulated UD rules, DECAP's Core prompt explicitly instructs the model to follow the same 'Miami Gold Subset spoken-language rules', and FLEX-UD's severity weights are hand-set by the same team. Consequently, DECAP's high reported scores partly measure instruction-following against the authors' own standard rather than an independent parsing capability. The comparison is uneven because BiLingua and Stanza baselines are not given the gold rules. Separately, the abstract and §5.2 headline numbers (FLEX-UD 76.2, LAS 0.48) are contradicted by the paper's own Table 1 (DECAP Overall LAS 0.26) and Table 2 (DECAP Overall Final 66.6, below BiLingua's 72.2); I treat that as a correctness and verification flaw rather than circularity. Self-citations such as Kellert et al. (2025) are used for baselines and framing but are not load-bearing for the circularity finding. Overall, the central 'improvement' claim is partially circular because the benchmark, metric, and system prompt share the same authored conventions.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

Every score in §5 depends on an in-house measurement stack: the SpokeBench gold was created with the authors' reformulated UD guidelines; FLEX-UD weights and closeness classes are hand-set; and DECAP prompts explicitly cite the same 'Miami Gold Subset' rules. These choices act as implicit parameters. No external benchmark, inter-annotator statistics, or human validation of the metric is provided, so the empirical claims are not independently anchored.

free parameters (4)
  • FLEX-UD component weights w_Split, w_ID, w_UPOS, w_HEAD, w_DEPREL
    Chosen by hand with only constraint sum=1; no fitting procedure or justification in §4.2. They directly determine the aggregate score used for rankings.
  • FLEX-UD severity penalty P and per-issue penalties = catastrophic 0.25–0.6; minor 0.01–0.05
    The formula final = round(raw×(1−P)) depends on an unspecified bounded function of 'catastrophic errors'; the appendix reference is to examples, not a definition. Values are author-chosen.
  • FLEX-UD tolerance classes (VERB↔AUX, obj↔obl, etc.)
    Hand-specified closeness classes that decide which mismatches are minor; no empirical justification is provided.
  • Single-root priority ordering in DECAP Verifier = finite VERB > AUX > NOUN/PRON > communicative token
    Hand-coded tie-breaking in the V/R prompt that decides the root in fragments; influences final parse and therefore all scores.
axioms (5)
  • domain assumption Gold annotations in SpokeBench are correct and valid ground truth for spoken CSW parse quality
    §3.2/Appendix C: gold created by authors with a reformulated UD standard; no independent inter-annotator agreement statistics reported.
  • ad hoc to paper Single-root UD constraint is appropriate for fragmentary and elliptical spoken utterances
    Appendix C.1 imposes exactly one root even on 'si mmh', a design choice that directly shapes what counts as a correct parse.
  • ad hoc to paper FLEX-UD severity weighting maps monotonically to linguistic quality
    §4.2 assigns large penalties to author-defined 'catastrophic' errors and small ones to 'acceptable variation'; no human-judgment validation.
  • domain assumption GPT-4.1 with temperature=0 reliably follows the multi-agent DECAP protocol
    §5.1 instantiates all agents with GPT-4.1 and assumes deterministic decoding yields consistent structured JSON outputs.
  • domain assumption Miami Corpus sample is representative of spoken English-Spanish CSW challenges
    §5.1 uses a curated 126-sentence subset; the paper acknowledges dialect and speaker diversity limits.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions." pith.science (2026). https://pith.science/paper/FK24VCWT

@misc{pith2026260206307,
  author       = {Pith},
  title        = {Pith review of: Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FK24VCWT}},
  note         = {Machine review of arXiv:2602.06307}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Spoken bilingual conversations pose substantial challenges for syntactic parsing because they often include disfluencies and discourse-driven structures that complicate dependency parsing under standard Universal Dependencies (UD) assumptions and evaluation practices. To systematically study these challenges, in this work, we first introduce a linguistically grounded taxonomy of conversational bilingual phenomena, together with SpokeBench, an expert-annotated English-Spanish benchmark for structurally complex speech. To address the limitations of existing evaluation practices, we propose Flex-UD, an ambiguity-aware evaluation metric that distinguishes catastrophic structural failures from linguistically acceptable variations. Finally, we introduce DECAP, a decoupled agentic parsing framework that separates spoken-phenomena handling from core syntactic analysis, enabling robust and interpretable dependency parsing without retraining. Experiments across both proprietary and open-weight LLMs show that DECAP substantially improves performance on complex conversational phenomena and achieves over 60% improvements in UPOS-F1 Score over baselines, while Flex-UD evaluations reveal gains that otherwise remain partially hidden under standard attachment-based metrics.

Figures

Figures reproduced from arXiv: 2602.06307 by Chitta Baral, Holly Hendrix, Justin Mackie, Megan Michelle Smith, Nelvin Licona-Guevara, Nemika Tyagi, Olga Kellert, Phanos Kareen, Samhitha Harish, Tatiana Gallego Hernande.

Figure 1
Figure 1. Figure 1: Illustration of the modality gap motivating [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the DECAP framework for spoken code-switching parsing, illustrated with a running example. The input utterance (“Entonces then I won’t I won’t buy anything uh”) is processed. The Spoken-Phenomena Handler (SPH) detects disfluencies (e.g., repetition, discourse markers, fillers) and provides tokenization hints. The Language-Specific Resolver (LSR) applies conservative, language-aware normalizatio… view at source ↗
Figure 3
Figure 3. Figure 3: UPOS-LAS by Category and Parser; DECAP is the GPT-4.1 agent and performs the best across all categories [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: LAS by Category and Parser. DECAP is the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 4 linked inside Pith

  1. [1]

    Xuefeng Bai, Jialong Wu, Yulong Chen, Zhongqing Wang, Kehai Chen, Min Zhang, and Yue Zhang. 2025. Constituency parsing using llms. IEEE Transactions on Audio, Speech and Language Processing

  2. [2]

    Damir Cavar and 1 others. 2024. Syntactic annotation of spoken and disfluent language. In Proceedings of LREC-COLING 2024

  3. [3]

    O zlem C etino g lu and C a g r C \

    \"O zlem C etino g lu and C a g r C \"o ltekin. 2019. Challenges of annotating a code-switching treebank. In Proceedings of the 18th international workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019), pages 82--90

  4. [4]

    Clark and Thomas Wasow

    Herbert H. Clark and Thomas Wasow. 1998. Using ``uh'' and ``um'' in spontaneous speaking. In David E. Meyer and Steven Kornblum, editors, Lexical and Syntactic Processing, pages 199--228. Lawrence Erlbaum Associates

  5. [5]

    Frances Adriana Laureano De Leon, Harish Tayyar Madabushi, and Mark Lee. 2024. Code-mixed probes show how pre-trained models generalise on code-switched text. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3457--3468

  6. [6]

    Parafita Couto, and Dan Carter

    Margaret Deuchar, Peter Davies, Judith Herring, María C. Parafita Couto, and Dan Carter. 2014. Building bilingual corpora. In Enlli M. Thomas and Ineke Mennen, editors, Advances in the Study of Bilingualism, pages 93--110. Multilingual Matters, Bristol

  7. [7]

    Kaja Dobrovoljc. 2022 a . Annotating ellipsis in dependency treebanks. Language Resources and Evaluation

  8. [8]

    Kaja Dobrovoljc. 2022 b . Spoken language treebanks in universal dependencies: An overview. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1798--1806

  9. [9]

    Kim Gerdes, Bruno Guillaume, Sylvain Kahane, and Guy Perrier. 2019. Improving surface-syntactic universal dependencies (sud): surface-syntactic relations and deep syntactic features. In TLT 2019-18th International Workshop on Treebanks and Linguistic Theories, pages 126--132. Association for Computational Linguistics

  10. [10]

    Kim Gerdes and Sylvain Kahane. 2016. Surface-syntactic universal dependencies. In Proceedings of COLING 2016, pages 223--235. ACL

  11. [11]

    Carlos G \'o mez-Rodr \' guez, Muhammad Imran, David Vilares, Elena Solera, and Olga Kellert. 2024. Dancing in the syntax forest: fast, accurate and explainable sentiment analysis with salsa. In SEPLN–CEDI-PD 2024. Seminar of the Spanish Society for Natural Language Processing: Projects and System Demonstrations, volume 3729 of CEUR Workshop Proceedings, ...

  12. [12]

    Daniel Hardt and Maribel Romero. 2004. Ellipsis and the structure of discourse. In Proceedings of SALT 14

  13. [13]

    Muhammad Huzaifah, Weihua Zheng, Nattapol Chanpaisit, and Kui Wu. 2024. Evaluating code-switching translation with large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6381--6394

  14. [14]

    Muhammad Imran, Olga Kellert, and Carlos G \'o mez-Rodr \' guez. 2024. A syntax-injected approach for faster and more accurate sentiment analysis. arXiv preprint arXiv:2406.15163

  15. [15]

    Muhammad Imran, Olga Zamaraeva, and Carlos G \'o mez-Rodr \' guez. 2025. Synner: Syntax-infused named entity recognition in the biomedical domain. JAMIA Open

  16. [16]

    Jesin James, Vithya Yogarajan, Isabella Shields, Catherine I Watson, Peter Keegan, Keoni Mahelona, and Peter-Lucas Jones. 2022. Language models for code-switch detection of te reo m \=a ori and english in a low-resource setting. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 650--660

  17. [17]

    Sylvain Kahane. 2019. Interpreting and defining connections in dependency structures. In 5th international conference on Dependency Linguistics (Depling), pages 89--99. Association for Computational Linguistics

  18. [18]

    Sylvain Kahane, Bernard Caron, Emmett Strickland, and Kim Gerdes. 2021. Annotation guidelines of ud and sud treebanks for spoken corpora. In Proceedings of the 20th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2021), pages pp--35. Association for Computational Linguistics

  19. [19]

    Olga Kellert, Nemika Tyagi, Muhammad Imran, Nelvin Licona-Guevara, and Carlos G \'o mez-Rodr \' guez. 2025. Parsing the switch: Llm-based universal dependency annotation for code-switched language. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics

  20. [20]

    Boda Lin, Xinyi Zhou, Binghao Tang, Xiaocheng Gong, and Si Li. 2023. Chatgpt is a potential zero-shot dependency parser. arXiv preprint arXiv:2310.16654

  21. [21]

    Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Deep learning for ellipsis resolution. In Proceedings of ACL 2016. ACL

  22. [22]

    Jason Merchant. 2001. The Syntax of Silence: Sluicing, Islands, and the Theory of Ellipsis. Oxford University Press

  23. [23]

    Leif Arda Nielsen. 2004. A Logical Approach to Ellipsis Resolution. Ph.D. thesis, University of London

  24. [24]

    Igor Sterner and Simone Teufel. 2025 a . Code-switching and syntax: A large-scale experiment. arXiv preprint arXiv:2506.01846

  25. [25]

    Igor Sterner and Simone Teufel. 2025 b . Minimal pair-based evaluation of code-switching. arXiv preprint arXiv:2506.01840

  26. [26]

    Yuanhe Tian, Fei Xia, and Yan Song. 2024. Large language models are no longer shallow parsers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7131--7142

  27. [27]

    Ziyan Zhang, Yang Hou, Chen Gong, and Zhenghua Li. 2025. Self-correction makes llms better parsers. arXiv preprint arXiv:2504.14165

  28. [28]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  29. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.