Pith. sign in

REVIEW 4 major objections 5 minor 67 references

Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned language models can predict which words speakers shorten or stress in spontaneous conversation, and conversational pretraining beats written pretraining in English, French, and Mandarin.

desk verdict A useful multilingual benchmark idea with an honest write-up, but the headline result may just be genre transfer, and the paper says so itself. read the letter →

arxiv 2505.16277 v1 pith:PMESZJGN submitted 2025-05-22 cs.CL

Sheng-Fu Wang , Laurent Prevot , Jou-an Chi , Ri-Sheng Huang , Shu-Kai Hsieh This is my paper
classification cs.CL
keywords cognitiveplausibilityspontaneousspeechreductionprosodicprominencelanguagemodelevaluationcross-linguisticbenchmarkingpretraininggenre
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that spontaneous speech corpora can supply new evaluation benchmarks for language models, derived from how humans actually produce speech rather than from grammatical judgments. It extracts two production variables — speech reductions (words shortened by dropping sounds or merging syllables) and prosodic prominences (words made to stand out through pitch, duration, or loudness) — from conversational corpora in English, French, and Taiwan Mandarin. Fine-tuned masked language models trained on conversational, written, or mixed-genre text are then tested on predicting these labels. The central claim is that models can predict both variables well above chance after fine-tuning, and that conversational pretraining yields more accurate predictions than written encyclopedic pretraining in all three languages. If true, this gives LLM evaluation a text-only, cross-linguistic window onto the surface traces of human speech production.

What carries the argument

The load-bearing machinery is a pair of token-level production labels. Speech reduction labels are binary: each word's actual duration is divided by an expected duration estimated from segmental or syllabic content, with one half of the corpus used to predict the other, and a token is labeled reduced if the ratio falls below a threshold (0.5 for English and French, 0.6 for Mandarin). Prosodic prominence labels are produced by a continuous-wavelet transform that combines fundamental frequency, energy, and duration, thresholded at a fixed score to mark prominent tokens. These labels serve as fine-tuning targets for masked language models in a token-classification setup with speaker-group cross-validation, hyperparameter search over learning rate and batch size, and label-distribution baselines.

What would settle it

A decisive check would build two pretraining corpora matched for sentence length, word frequency, and topic but differing only in whether the sentences come from conversation transcripts or from written text; if the conversational advantage in reduction or prominence F1 disappears under that match, the headline effect is explained by distributional overlap rather than by conversational genre per se.

Watch

Extended reading notes

Core claim

The paper's central claim is that surface traces of speech production — which words get shortened and which get emphasized — are predictable from text alone, and that the genre of pretraining data measurably changes how well a model predicts them. After fine-tuning on token-classification labels derived from three conversational corpora, all models exceed label-distribution baselines on both tasks. In every language, the model pretrained on written encyclopedic text is outperformed by at least the conversational model, and in most cases the conversational or mixed model leads; the gap is largest for Mandarin reduction. The authors further report that this fine-tuning advantage is not simply a perplexity effect: the written model has the lowest benchmark perplexity in Mandarin yet does not win the reduction task, indicating the task captures something beyond generic fluency.

Load-bearing premise

The central assumption is that the conversational and written pretraining corpora are different enough from the test corpora that the conversational model's better scores come from what it learned, not from simply being trained on text that looks more like the test data.

Editorial extensions

If this is right

  • Existing spontaneous-speech corpora with fine-grained alignment can be turned into language-model benchmarks without additional human annotation, so the cost of adding new languages is mainly corpus availability.
  • Genre of pretraining data matters for cognitively oriented evaluation: conversational transcripts support better prediction of production variables than written encyclopedic text across three typologically distinct languages.
  • Perplexity on a benchmark corpus is not a dependable predictor of fine-tuned task performance; in Mandarin reduction the lowest-perplexity model is not the best predictor.
  • The same two-label protocol can be applied to other languages and other spontaneous-speech genres, providing comparable contrastive results across language families.
  • Larger pretrained models do not consistently dominate smaller genre-controlled models on reduction, while they do on prominence, so conclusions about scale depend on which production variable is probed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the paper does not control for sentence-length and lexical-frequency overlap between conversational pretraining data and the benchmark corpora, a frequency-matched replication is the decisive test; if the conversational advantage vanishes, the result would be a distributional artifact rather than a cognitive-plausibility signal.
  • Editorial inference: the over-prediction of function words and discourse markers for reduction suggests models may be exploiting a lexical-frequency shortcut; a controlled set that swaps high- and low-frequency words in matched contexts could reveal whether models learn contextual predictability or a static frequency prior.
  • Editorial inference: the threshold asymmetry across languages (0.5 versus 0.6) means "reduced" is defined relatively within each corpus; pooling or aligning thresholds would be needed before treating cross-linguistic F1 differences as direct evidence about language differences.
  • Editorial inference: one can extend the same labeling pipeline to child-directed speech corpora in additional languages to test the paper's broader claim that developmentally plausible conversational input, not just any spoken genre, is what makes representations more human-like.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a new evaluation paradigm for language models based on spontaneous speech production variables. The authors extract speech-reduction and prosodic-prominence labels from spontaneous speech corpora in English (Buckeye), French (CID), and Taiwan Mandarin (Sinica MCDC), then fine-tune language models that were pretrained on conversational, written, or mixed-genre data for each language. The central empirical claim is that, after fine-tuning, models pretrained on conversational data achieve higher F1 scores than models pretrained on Wikipedia data for both token-level tasks, and that all models perform above random baselines. The paper also reports perplexity and surprisal-label correlation analyses, and it explicitly discusses limitations including the risk of surface-level shortcuts and the genre similarity between pretraining and benchmark corpora.

Significance. If the central claim is supported, the paper introduces a useful cross-linguistic benchmark derived from high-quality spontaneous speech corpora and contributes to the BabyLM-style agenda by showing that pretraining genre affects performance on production-oriented tasks. The strengths of the paper include the use of three typologically different languages, the fully reproducible pipeline (pretraining, fine-tuning, and error analysis notebooks are said to be released), and the transparent acknowledgment of the genre-similarity confound in Section 6. However, the headline interpretation of the results as evidence about models' 'cognitive plausibility' is not yet warranted because the main comparisons do not control for the admitted distributional overlap between conversational pretraining data and the benchmarks.

major comments (4)
  1. [Section 6, 'Triviality of the main result'] The paper concedes that the conversational pretraining datasets are 'by all aspects (sentence length distribution, lexical frequencies, etc) more similar to benchmarks than Wikipedia datasets are.' This concession directly bears on the central interpretation of Section 5 and Figure 1, where conversational models are reported to outperform Wikipedia models in F1. Because the fine-tuning comparisons do not include any covariate controls, the observed advantage could stem from distributional overlap rather than from a more general or more 'cognitively plausible' model representation. The Appendix C error analysis is consistent with this concern: the most over-predicted reduction words are frequent function words and discourse markers (e.g., 'youre', 'dont', 'jesuis', 'de', 'le0'), which are prevalent in both conversational pretraining and the benchmarks. To support the headline claim, the authors should add matched or covariate-adjusted analyses (e.g., comparing models within frequency and sentence-length strata, or using lexical-frequency and sentence-length as predictors in the Bayesian model) and show that the conversational advantage remains when these variables are controlled.
  2. [Section 5 (Figure 1) and Appendix B] The claim that models 'comfortably outperform the label distribution baselines' is central, but the random baselines are only shown as dotted lines in Figure 1; Appendix B reports F1, precision, and recall for the models but does not report the baseline values. Without numeric baseline F1 scores for each language and task, the reader cannot verify the magnitude of the improvement. Please include a table with the exact baseline values (e.g., majority-class baseline and equal-probability random baseline) alongside the model scores.
  3. [Section 5, first paragraph and Figure 1 caption] The Bayesian regression analysis that underlies the starred comparisons in Figure 1 is described only as 'MODEL is the fixed predictor and FOLD is a random intercept' with 'weak (uniform) priors.' This is insufficient for a quantitative claim. The manuscript should specify the model family and link function (are F1 scores modeled as Gaussian or beta-binomial?), the prior distributions, the exact formula, and the posterior estimates (medians and credible intervals) for the pairwise contrasts between the three small models. Without this information, the reported >95% posterior probability is not reproducible.
  4. [Section 4.2.1] The reduction-labeling pipeline for French and Mandarin relies on an expected-duration model trained on one half of the corpus and applied to the other half. The manuscript does not report the accuracy, bias, or error pattern of this duration model. If the expected-duration model predicts durations less accurately for rare or content words, the resulting binary reduction labels could be systematically noisy and could interact with pretraining genre (since conversational models have different frequency exposures). Please provide validation metrics for the duration model and discuss how labeling noise varies with lexical frequency, or justify the assumption that labeling noise is independent of the pretraining data condition.
minor comments (5)
  1. [Abstract] The abstract contains an incomplete sentence: 'More precisely, we extract.' This should be completed or removed.
  2. [Section 4.1, French data] The text states that 'the conversational data contains an additional 10 million tokens sourced from Wikipedia.' This contradicts the preceding sentence, which defines the conversational data as ORFEO and CHILDES-FR; presumably this should say 'the written data.' Please fix.
  3. [Figure 1 caption] The caption reads 'F-scores on the benchmarks as a function of model (x), task and language (prominence)' — the word 'prominence' at the end appears to be a leftover and is confusing. It should likely be removed or rephrased.
  4. [Section 4.2.1, English reduction] The description of the English half-split method is vague: 'expected duration in one half calculated with segment duration from the other.' Please state the exact formula or provide a citation to the method used.
  5. [Throughout] There are several typos and inconsistencies: 'Proceeddings' in the Adi et al. reference, 'feasability' in Section 4.3, 'developping' in Section 7, and inconsistent capitalization of 'ROBERTA' vs. 'RoBERTa.' A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark labels are externally derived, the fine-tuning evaluation uses held-out speaker folds, and the conversational advantage is an empirical result, not a definitional or self-citation-based reduction.

full rationale

The paper's derivation chain is self-contained in the relevant sense. Reduction and prominence labels are derived from acoustic annotations and thresholds in three spontaneous-speech corpora that are independent of the pretraining corpora; the fine-tuning experiments use speaker-group cross-validation on those labels, so the reported F1 scores are standard supervised generalization results rather than fitted inputs renamed as predictions. The headline conversational-model advantage is an empirical comparison between models pretrained on different genre corpora. Section 6 candidly concedes that the conversational pretraining data are 'by all aspects (sentence length distribution, lexical frequencies, etc) more similar to benchmarks than Wikipedia datasets are,' which is a genuine confound for the cognitive-plausibility interpretation and a correctness risk, but it does not make the result circular: the outcome is not equivalent to an input by construction, and no load-bearing premise is justified only by self-citation. The self-citations (e.g., Wang 2022 for the duration model and Prévot's earlier corpus work) are methodological or related-work references, and they are not used to import an unverified uniqueness claim or to define the target result into existence.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on two hand-set thresholds (reduction and prominence) that define the labels, and on the assumption that acoustic production variables are valid cognitive traces. No new theoretical entities are introduced. The genre-similarity confound is acknowledged in Section 6.

free parameters (3)
  • Reduction label threshold = 0.5 for English/French; 0.6 for Mandarin
    Chosen to keep the proportion of reduced tokens similar across languages (13.58%, 17.54%, 11.09%). Affects the binary labels that models are trained and evaluated on. Location: Section 4.2.1.
  • Prominence score threshold = 1.25
    Applied to Suni et al. (2017) wavelet prominence scores; yields 14%, 13.79%, and 12.77% prominent tokens across languages. Threshold choice affects labels; reference to Appendix A figure is mismatched. Location: Section 4.2.2.
  • SentencePiece vocabulary size = 10,000
    Determined empirically from preliminary experiments; affects tokenization and thus model input. Location: Section 4.1.
assumptions (3)
  • domain assumption Speech reduction and prosodic prominence are partial indicators of human language processing and can serve as cognitive evaluation targets.
    Background assumption from phonetics literature (Aylett and Turk 2004; Seyfarth 2014); motivates the benchmark but is not verified for LLMs in this paper.
  • domain assumption Text-only language models can predict these acoustically-derived production variables to a meaningful degree.
    Cites Wolf et al. (2023) and Kakouros and O'Mahony (2023) for prosody-text redundancy; the paper relies on this to justify the probing setup.
  • ad hoc to paper The expected-duration model trained on one half of a corpus generalizes to the other half, yielding unbiased reduction labels.
    The reduction labels are computed by applying a duration model trained on the same corpus (Section 4.2.1). This within-corpus normalization could introduce systematic bias in the labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility." pith.science (2026). https://pith.science/paper/PMESZJGN

@misc{pith2026250516277,
  author       = {Pith},
  title        = {Pith review of: Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMESZJGN}},
  note         = {Machine review of arXiv:2505.16277}
}
read the original abstract

The achievements of Large Language Models in Natural Language Processing, especially for high-resource languages, call for a better understanding of their characteristics from a cognitive perspective. Researchers have attempted to evaluate artificial models by testing their ability to predict behavioral (e.g., eye-tracking fixations) and physiological (e.g., brain responses) variables during language processing (e.g., reading/listening). In this paper, we propose using spontaneous speech corpora to derive production variables (speech reductions, prosodic prominences) and applying them in a similar fashion. More precisely, we extract. We then test models trained with a standard procedure on different pretraining datasets (written, spoken, and mixed genres) for their ability to predict these two variables. Our results show that, after some fine-tuning, the models can predict these production variables well above baselines. We also observe that spoken genre training data provides more accurate predictions than written genres. These results contribute to the broader effort of using high-quality speech corpora as benchmarks for LLMs.

Figures

Figures reproduced from arXiv: 2505.16277 by the authors.

Figure 1
Figure 1. F-scores on the benchmarks as a function of model (x), task and language (prominence). Model [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Perplexity on the benchmarks These observations, summarized in terms of the winner of the comparison between the conversa￾tional and Wiki models in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 7
Figure 7. Distribution of prominence scores as calcu [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Distribution reduction ratios as calculated in [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Distribution of reduction ratios as calculated [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 10
Figure 10. Figure 10: Predicted and gold reduction label rates as a function of log frequency (x axis, z-scored) and model type [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Predicted and gold prominence label rates as a function of log frequency (x axis, z-scored) and model [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 46 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceeddings of ICLR Conference Track, Toulon, France

    work page 2017

  4. [4]

    Matthew Aylett and Alice Turk. 2004. The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and speech, 47(1):31--56

    work page 2004

  5. [5]

    Lisa Beinborn and Yuval Pinter. 2023. Analyzing cognitive plausibility of subword tokenization. In The 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  6. [6]

    Alan Bell, Jason M Brenier, Michelle Gregory, Cynthia Girand, and Dan Jurafsky. 2009. Predictability effects on durations of content and function words in conversational English . Journal of Memory and Language, 60(1):92--111

    work page 2009

  7. [7]

    Christophe Benzitoun, Jeanne-Marie Debaisieux, and Henri-Jos \'e Deulofeu. 2016. Le projet orf \'e o: un corpus d’ \'e tude pour le fran c ais contemporain. Corpus, (15)

    work page 2016

  8. [8]

    Joachim Bingel, Maria Barrett, and Anders S gaard. 2016. Extracting token-level signals of syntactic processing from fmri-with an application to pos induction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 747--755

    work page 2016

Show all 67 references
  1. [9]

    Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120

    2023

  2. [10]

    Philippe Blache, Roxane Bertrand, Ga \"e lle Ferr \'e , Berthille Pallaud, Laurent Pr \'e vot, and St \'e phane Rauzy. 2017. The corpus of interactional data: A large multimodal annotated resource. Handbook of linguistic annotation, pages 1323--1356

    2017

  3. [11]

    Kübra Bodur, Corinne Fredouille, Stéphane Rauzy, and Christine Meunier. 2023. https://doi.org/10.31234/osf.io/sbz8q Exploring the nuances of reduction in conversational speech: Lexicalized and non lexicalized reductions . PsyArXiv

    2023 doi

  4. [12]

    Francesco Cabiddu, Mitja Nikolaus, and Abdellah Fourtassi. 2025. Comparing children and large language models in word sense disambiguation: Insights and challenges. Language Development Research, 5(1)

    2025

  5. [13]

    Lih-Huei Chang, Zhong-Ru Chang, Yan-Chang Ke, and Su-Hui Xiao. 2011. Taiwan child language corpus (tccm). National Science Council Research Project, NSC96-2420-H-002-030. Available at http://lope.linguistics.ntu.edu.tw/tccm/

    2011

  6. [14]

    Tyler A Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K Bergen. 2024. Goldfish: Monolingual language models for 350 languages. arXiv preprint arXiv:2408.10441

    2024 arXiv

  7. [15]

    Leshem Choshen, Ryan Cotterell, Michael Y Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang. 2024. [call for papers] the 2nd babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus. arXiv p...

    2024 arXiv

  8. [16]

    Morten H Christiansen and Nick Chater. 2022. The language game: How improvisation created language and changed the world. Random House

    2022

  9. [17]

    Kawai Chui and Huei-ling Lai. 2008. The nccu corpus of spoken chinese: Mandarin, hakka, and southern min. Taiwan Journal of Linguistics, 6(2)

    2008

  10. [18]

    Uriel Cohen Priva. 2012. Sign and signal: Deriving linguistic generalizations from information utility . Ph.D. thesis, Stanford University

    2012

  11. [19]

    Uriel Cohen Priva and T Florian Jaeger. 2018. The interdependence of frequency, predictability, and informativity in the segmental domain. Linguistics Vanguard, 4(s2)

    2018

  12. [20]

    Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, and Irwin King. 2025. Voxeval: Benchmarking the knowledge understanding capabilities of end-to-end spoken language models. arXiv preprint arXiv:2501.04962

    2025 arXiv

  13. [21]

    Allyson Ettinger. 2020. https://doi.org/10.1162/tacl_a_00298 What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models . Transactions of the Association for Computational Linguistics

    2020 doi

  14. [22]

    Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st workshop on evaluating vector-space representations for nlp, pages 134--139

    2016

  15. [23]

    Steven Feng, Noah Goodman, and Michael C Frank. 2024. Is child-directed speech effective training data for language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22055--22071

    2024

  16. [24]

    Susanne Gahl. 2008. Time and thyme are not homophones: The effect of lemma frequency on word durations in spontaneous speech. Language, 84(3):474--496

    2008

  17. [25]

    Susanne Gahl, Yao Yao, and Keith Johnson. 2012. Why reduce? phonological neighborhood density and phonetic reduction in spontaneous speech. Journal of memory and language, 66(4):789--806

    2012

  18. [26]

    Youssef Hmamouche, Magalie Ochs, Laurent Pr \'e vot, and Thierry Chaminade. 2024. Interpretable prediction of brain activity during conversations from multimodal behavioral signals. Plos one, 19(3):e0284342

    2024

  19. [27]

    Nora Hollenstein, Emmanuele Chersoni, Cassandra L Jacobs, Yohei Oseki, Laurent Pr \'e vot, and Enrico Santus. 2021. Cmcl 2021 shared task on eye-tracking prediction. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 72--78

    2021

  20. [28]

    Nora Hollenstein, Jonathan Rotsztejn, Marius Troendle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. 2018. Zuco, a simultaneous eeg and eye-tracking resource for natural sentence reading. Scientific data, 5(1):1--13

    2018

  21. [29]

    Michael Y Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the second babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. arX...

    2024 arXiv

  22. [30]

    Sofoklis Kakouros and Johannah O'Mahony. 2023. What does bert learn about prosody? In 20th International Congress of Phonetic Sciences ICPhS. International Phonetics Association

    2023

  23. [31]

    Carina Kauf and Anna Ivanova. 2023. A better way to do masked language model scoring. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 925--935

    2023

  24. [32]

    T Kudo. 2018 a . Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226

    2018 arXiv

  25. [33]

    Taku Kudo. 2018 b . Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959

    2018 arXiv

  26. [34]

    Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc...

    2021

  27. [35]

    interaction engine

    Stephen C Levinson. 2020. On the human "interaction engine". In Roots of human sociality, pages 39--69. Routledge

    2020

  28. [36]

    Jixing Li, Shohini Bhattasali, Shulin Zhang, Berta Franzluebbers, Wen-Ming Luh, R Nathan Spreng, Jonathan R Brennan, Yiming Yang, Christophe Pallier, and John Hale. 2022. Le petit prince multilingual naturalistic fmri corpus. Scientific data, 9(1):530

    2022

  29. [37]

    Pierre Lison and J \"o rg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles

    2016

  30. [38]

    Brian MacWhinney. 2014. The CHILDES project: Tools for analyzing talk, Volume I: Transcription format and programs. Psychology Press

    2014

  31. [39]

    Richard Diehl Martinez, Hope McGovern, Zebulon Goriely, Christopher Davis, Andrew Caines, Paula Buttery, and Lisa Beinborn. 2023. Climb--curriculum learning for infant-inspired model building. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natur...

    2023

  32. [40]

    Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. https://doi.org/10.18653/v1/P19-1334 Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3...

    2019 doi

  33. [41]

    Vincent Micheli, Martin d ' Hoffschmidt, and Fran c ois Fleuret. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.632 On the importance of pre-training data volume for compact language models . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro...

    2020 doi

  34. [42]

    Alexandre Pasquiou, Yair Lakretz, John T Hale, Bertrand Thirion, and Christophe Pallier. 2022. Neural language models are not born equal to fit brain data, but training helps. In International Conference on Machine Learning, pages 17499--17516. PMLR

    2022

  35. [43]

    Mark A Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond. 2005. The buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability. Speech Communication, 45(1):89--95

    2005

  36. [44]

    Mark Pluymaekers, Mirjam Ernestus, and R Harald Baayen. 2005. Lexical frequency and acoustic reduction in spoken dutch. The Journal of the Acoustical Society of America, 118(4):2561--2569

    2005

  37. [45]

    Birgit Rauchbauer, Bruno Nazarian, Morgane Bourhis, Magalie Ochs, Laurent Pr \'e vot, and Thierry Chaminade. 2019. Brain activity during reciprocal social interaction investigated using conversational robots as control condition. Philosophical Transactions of the Royal Society...

    2019

  38. [46]

    Yvan Rose and Brian MacWhinney. 2014. The phonbank project: Data and software-assisted methods for the study of phonology and phonological development

    2014

  39. [47]

    Scott Seyfarth. 2014. Word informativity influences acoustic duration: Effects of contextual predictability on lexical representation. Cognition, 133(1):140--155

    2014

  40. [48]

    Taiga Someya and Yohei Oseki. 2023. Jblimp: Japanese benchmark of linguistic minimal pairs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1581--1594

    2023

  41. [49]

    Blimp-nl

    Michelle Suijkerbuijk, Zo \"e Prins, Marianne de Heer Kloots, Willem Zuidema, and Stefan L Frank. Blimp-nl

  42. [50]

    Antti Suni, Juraj S imko, Daniel Aalto, and Martti Vainio. 2017. Hierarchical representation and estimation of prosody using continuous wavelet transform. Computer Speech & Language, 45:123--136

    2017

  43. [51]

    Ekaterina Taktasheva, Maxim Bazhukov, Kirill Koncha, Alena Fenogenova, and Ekaterina Artemova. 2024. Rublimp: Russian benchmark of linguistic minimal pairs. arXiv preprint arXiv:2406.19232

    2024 arXiv

  44. [52]

    Aarne Talman, Antti Suni, Hande Celikkanat, Sofoklis Kakouros, J \"o rg Tiedemann, and Martti Vainio. 2019. Predicting prosodic prominence from text with pre-trained contextualized word representations. In Proceedings of the 22nd Nordic Conference on Computational Linguistics,...

    2019

  45. [53]

    Kevin Tang and Ryan Bennett. 2018. Contextual predictability influences word and morpheme duration in a morphologically complex language (kaqchikel mayan). The Journal of the Acoustical Society of America, 144(2):997--1017

    2018

  46. [54]

    Shu-Chuan Tseng. 2013. Lexical coverage in taiwan mandarin conversation. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 18, Number 1, March 2013

    2013

  47. [55]

    RJJH Van Son and Louis CW Pols. 2003. How efficient is speech. In Proceedings of the institute of phonetic sciences, volume 25, pages 171--184

    2003

  48. [56]

    RJJH Van Son, Louis CW Pols, et al. 1999. Effects of stress and lexical structure on speech efficiency. In EUROSPEECH

    1999

  49. [57]

    Sheng-Fu Wang. 2022. The interaction between predictability and pre-boundary lengthening on syllable duration in taiwan southern min. Phonetica, 79(4):315--352

    2022

  50. [58]

    Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. 2023 a . Call for papers--the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus. arXiv preprint arXiv:2301.11796

    2023 arXiv

  51. [59]

    Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, et al. 2023 b . Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proce...

    2023

  52. [60]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng Fu Wang, and Samuel R. Bowman. 2019 a . https://doi.org/10.1162/tacl_a_00321 Blimp: The benchmark of linguistic minimal pairs for english . Transactions of the Association for Computational Linguistics

    2019 doi

  53. [61]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020 a . Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377--392

    2020

  54. [62]

    Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019 b . https://doi.org/10.1162/tacl_a_00290 Neural network acceptability judgments . Transactions of the Association for Computational Linguistics, 7:625--641

    2019 doi

  55. [63]

    Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020 b . https://doi.org/10.18653/v1/2020.emnlp-main.16 Learning which features matter: R o BERT a acquires a preference for linguistic generalizations (eventually) . In Proceedings of the 2020 Conferen...

    2020 doi

  56. [64]

    Lukas Wolf, Tiago Pimentel, Evelina Fedorenko, Ryan Cotterell, Alex Warstadt, Ethan Wilcox, and Tamar Regev. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.606 Quantifying the redundancy between prosody and text . In Proceedings of the 2023 Conference on Empirical Methods i...

    2023 doi

  57. [65]

    Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. https://doi.org/10.18653/v1/2021.eacl-main.242 CL i MP : A benchmark for C hinese language model evaluation . In Proceedings of the 16th Conference of the European Chapter of the Association for Comp...

    2021 doi

  58. [66]

    Ziqing Yang, Yiming Cui, and Zhigang Chen. 2022. https://aclanthology.org/2022.acl-demo.4 T ext P runer: A model pruning toolkit for pre-trained language models . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations,...

    2022

  59. [67]

    Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. https://doi.org/10.18653/v1/2021.acl-long.90 When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int...

    2021 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.