Pith. sign in

REVIEW 2 major objections 5 minor 85 references

The Harmonic Structure of Information Contours

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that word-level surprisal in texts oscillates periodically, with the dominant periods matching the spans of elementary discourse units (EDUs).

desk verdict Solid boundary-effects paper with a novel method, but the periodicity claim needs a within-unit position baseline. read the letter →

arxiv 2506.03902 v1 pith:AFKEJ5BV submitted 2025-06-04 cs.CL

Eleftheria Tsipidi , Samuel Kiegeland , Franz Nowak , Tianyang Xu , Ethan Wilcox , Alex Warstadt , Ryan Cotterell , Mario Giulianelli This is my paper
classification cs.CL
keywords harmonicsurprisaluniforminformationdensitycontoursdiscoursestructurerhetoricaltheoryregressiontimescalingrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the rate at which words convey information, measured as surprisal, does not merely drift around a mean but oscillates on regular cycles whose dominant lengths are set by discourse structure. It introduces the Harmonic Surprisal hypothesis: surprisal within a document varies periodically, with periods corresponding to the boundaries of structural units, most strongly elementary discourse units. Using harmonic regression with a new time-scaling step, the authors fit sinusoids scaled to document, paragraph, sentence, and EDU lengths in six languages and find that first-order EDU-scaled sinusoids are consistently significant predictors with the largest amplitudes. The result matters because it refines the uniform information density hypothesis: rather than flattening information globally, speakers modulate it in a structure-aligned rhythm, connecting information theory to discourse organization.

What carries the argument

Harmonic regression is a linear regression whose predictors are sine and cosine components at integer multiples of a fundamental frequency, allowing a signal to be approximated as a sum of periodic trends. Time scaling modifies the period of those sinusoids by replacing absolute time with position normalized by the length of the structural unit containing the token, so that a first-order sinusoid completes exactly one oscillation per unit span. This machinery lets the authors test whether observed periodicity aligns with specific annotated structures: an EDU-scaled first-order sinusoid models one full cycle per elementary discourse unit, and its amplitude measures how strongly that particular period shapes surprisal. Feature selection uses L1 regularization, with significance assessed through one-way ANOVA and paired t-tests against a boundary-feature baseline.

What would settle it

Fit the same EDU-scaled harmonic regression to surprisal contours of texts whose clause order has been shuffled or whose EDU boundaries have been replaced by same-length random spans; if the first-order EDU sinusoid keeps its amplitude and significance, the periodic structure is not specific to real discourse segmentation. Equivalently, recompute the analysis with several different LLM families and check whether the EDU harmonic persists across models with different architectures and training data.

Watch

Extended reading notes

Core claim

The central claim is Hypothesis 2.3: the components of a document's surprisal contour vary periodically, with periods that correspond to the boundaries of structural units within that contour. Operationally, the authors fit harmonic regression models whose sine and cosine predictors are time-scaled by the token length of the containing document, paragraph, sentence, or EDU, and evaluate them with ten-fold cross-validation and ANOVA significance tests. In the maximal model, low-order EDU-scaled sinusoids show the highest amplitudes across languages, they persist through feature selection in all folds, and they are the only structure type whose retained sinusoids are consistently significant everywhere, while surprisal reliably decreases before discourse boundaries and increases after them. The authors take this as evidence that discourse constituents, especially elementary discourse units, are the periods that organize information flow in text.

Load-bearing premise

The load-bearing premise is that the transformer language models used to compute surprisal are faithful proxies for the unknown human language model; if their position-dependent predictions or boundary artifacts create the fitted sinusoids, then the periodic alignment is a property of those models rather than of language itself.

Editorial extensions

If this is right

  • If the Harmonic Surprisal hypothesis holds, uniform information density is not contradicted but refined: surprisal regresses toward a mean while oscillating periodically around it, so global uniformity and structure-aligned modulation coexist.
  • Position within an EDU, not just distance from a boundary, carries predictive information about surprisal, making discourse segmentation a useful feature for surprisal and reading-time modeling.
  • The time-scaled harmonic regression framework is a general tool: any annotated structural unit, at any granularity, can be tested as a candidate period for information-rate fluctuations.
  • The cross-linguistic consistency across English, Spanish, German, Dutch, Basque, and Brazilian Portuguese suggests the periodic pattern is a general property of written discourse rather than an artifact of one language or corpus.
  • Surprisal's reliable dip before and rise after boundaries implies that speakers modulate information rate to align with processing costs that accumulate toward unit ends and reset at unit starts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The framework predicts a directly testable extension: applying time-scaled harmonic regression to syntactic constituents, multi-word expressions, or words should yield high-amplitude sinusoids at those finer periods, revealing a hierarchy of nested information cycles.
  • The method could be applied to dialogue turns, prosodic units in speech, or explicitly structured text such as code, and if analogous periodicity appears there, the harmonic-surprisal claim would generalize beyond written expository discourse.
  • A null test with randomly assigned same-length unit boundaries would isolate whether the EDU-aligned periodicity is genuine discourse alignment or merely an artifact of unit-length scaling; the paper's permutation checks rearrange surprisal values but not the structural segmentation itself.
  • Because surprisal estimates come from one LLM per language, comparing the fitted EDU harmonic across models with different architectures, context lengths, and training objectives would measure how much of the signal is a property of language versus a property of the estimator.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes the Harmonic Surprisal (HS) hypothesis, according to which surprisal contours vary periodically and the dominant periods align with the boundaries of structural discourse units (EDUs, sentences, paragraphs). To test this, the authors introduce time-scaled harmonic regression, in which sine/cosine predictors for each token are scaled by the length of the token's containing structural unit, and compare such models against a linear baseline that includes boundary-window flags, previous surprisal, token length, and document-level relative position. Using LLM-derived surprisal estimates on RST-annotated corpora in six languages, they report that EDU-scaled sinusoids yield the lowest cross-validated MSE in five of six languages and that low-order EDU harmonics have the largest amplitudes. The paper also documents a systematic boundary profile (low surprisal before, high after boundaries) and includes a permutation control.

Significance. If established, the HS hypothesis would give a concrete, falsifiable functional form to the link between global information contours and discourse structure, and the time-scaling regression would be a reusable tool for probing periodicity at other linguistic levels. The paper has clear strengths: six languages, cross-validated model comparison with L1 feature selection, Holm-corrected significance tests, a permutation control in App. G, and publicly available code. However, as detailed below, the current evidence does not yet distinguish the periodicity claim from a much weaker non-periodic within-unit position effect, so the central empirical contribution remains under-supported. The paper is well written and the computational framework is a useful methodological contribution regardless of the outcome of that control.

major comments (2)
  1. [§3.2, Eq. (8); §5, Tables 1–2; App. E] EDU-scaled sinusoids can absorb a non-periodic within-EDU trend; the missing baseline makes the periodicity claim indeterminate. For a token t in an EDU of length U_t, sin(2πt/U_t) and cos(2πt/U_t) are smooth functions of the relative position t/U_t, so a fitted linear combination is any smooth within-EDU ramp of the kind documented for boundaries in Sec. 6 and Table 12. The baseline in App. E includes boundary-window flags of width 1, 2, and 4 tokens, document-level relative position, token length, and previous surprisal, but not a continuous or discretized relative-position feature within the EDU, sentence, or paragraph. Therefore the significant MSE reductions in Table 1 and the dominant low-order EDU amplitudes in Table 2 do not discriminate between Hypothesis 2.3 and the weaker structured-context account that surprisal follows relative position within the unit. In addition, Eq. (8) imposes the period rather than estimating it, so the analysis tests the predictive value of a particular set of unit-anchored functions, not whether the contour is periodic at those periods. I request a control condition with non-periodic within-unit positional features (linear, quadratic, or spline terms in relative position) and, if possible, a free-period condition in which the sinusoid periods are estimated from the data; the main claim should be re-evaluated after those controls.
  2. [§2.2; §4.2; Tab. 6] The cross-linguistic evidence rests on a single LLM per language, and several of the selected models are instruction-tuned; without estimator-level controls, the documented boundary effects could be artifacts of those models' tokenization or positional biases. The permutation control in App. G destroys the temporal structure of the surprisal values and therefore cannot address this concern; it shows only that the harmonic fit is not an artifact of arbitrary permutations. Adding a second, architecturally different surprisal estimator (e.g., an LSTM, a non-instruction-tuned causal LM, or an n-gram model) for at least one corpus per language would substantially strengthen the claim that the pattern is a property of linguistic information contours rather than of a particular model family. If the paper is intended only as a claim about LLM-computed surprisal, this concern is less severe, but the abstract and Hypothesis 2.3 are framed in language-general terms.
minor comments (5)
  1. [App. F] The sentence 'To account for the multiple comparisons, we we the Holm–Bonferroni correction' contains a typo and should read 'we used the Holm–Bonferroni correction.'
  2. [Tables 4–5] The column header 'Variance' appears to contain standard deviations (e.g., mean 12.87 tokens per English EDU with value 9.13); if so, the header should be corrected to 'Standard deviation' to avoid misreading.
  3. [Table 2] Amplitudes are reported as means over cross-validation folds without a measure of variability; since the table is used to support the claim that EDU harmonics are strongest, standard errors or per-fold ranges would be informative.
  4. [§5.1, Table 2] The text says all EDU-scaled sinusoids that remain after feature selection are consistently significant across folds; this is true for the reported EDU rows, but the sentence immediately follows a table that also contains non-EDU rows with fold subscripts lower than 10, so a reader could misread the claim as applying to the whole table.
  5. [Abstract and §5] The phrase 'Many dominant frequencies align with discourse structure' overstates the design: the periods are imposed by time scaling rather than discovered, so a more precise formulation would be 'EDU-anchored sinusoids are predictive of surprisal contours.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the harmonic-surprisal test is an empirical held-out model comparison with negative controls.

full rationale

The paper's central claim (Hypothesis 2.3) is operationalized by fitting time-scaled sinusoids, but the evidence for it is not obtained by renaming the fit. The predictive claims rest on 10-fold cross-validated held-out MSE (Tab. 1), one-sided paired t-tests against a baseline, and ANOVA significance tests for individual harmonic components (Sec. 3.3, App. E, App. F). The baseline includes token length, previous surprisal, document relative position, and 1/2/4-token boundary indicators, so the reported improvement of EDU-scaled models is not a tautological consequence of the regression setup. The paper also includes a permuted-surprisal negative control (App. G), which shows the scaling differences disappear when surprisal is shuffled. The key choice of EDU, sentence, and paragraph periods is motivated by prior work, including the same authors' Structured Context hypothesis (Tsipidi et al., 2024), but that prior work is externally published and empirically testable, and the present comparisons would be meaningful even without those citations. A legitimate scientific concern is that the baseline does not include a smooth within-EDU relative-position trend, so the first-order EDU sinusoid may absorb a boundary ramp rather than demonstrate true oscillation; however, this is an omitted-variable/alternative-explanation issue, not a by-construction reduction of the result to its inputs. No equation in the paper equates a predicted quantity to a fitted input, and no self-citation is invoked as a uniqueness theorem to forbid alternatives. Therefore no significant circularity is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the chosen estimators of surprisal, the RST annotations, and the modeling choices of harmonic order and penalty. No new entities are introduced.

free parameters (4)
  • L1 penalty λ = 0.01 (0.1 considered for Brazilian Portuguese)
    Chosen by lowest MSE on one cross-validation fold per corpus (App. D).
  • Boundary window sizes = 1, 2, 4 tokens
    Hand-chosen widths for boolean boundary-proximity features in the baseline (Sec. 3.3, App. E).
  • Harmonic order cap K = length of the longest unit in the training fold for each granularity
    Modeling choice to bound the number of sinusoids (Sec. 3.2).
  • Significance level α = 0.001
    Chosen threshold for ANOVA and t-tests to reduce false positives (App. E).
assumptions (4)
  • domain assumption Neural language model surprisal is an adequate proxy for the human language model's information rate.
    Stated in Sec. 2.2: 'we use neural network models as a proxy for a particular hypothetical construct model, i.e., the human language model'. If LLM surprisal diverges from human processing, the periodic structure may not be a property of language but of the model.
  • domain assumption RST-based EDU, sentence, and paragraph annotations are accurate and meaningful for the information structure.
    All structural predictors are derived from these annotations (Sec. 4.1); errors or arbitrary segmentation would produce spurious alignment.
  • domain assumption Residuals are independent for significance testing.
    ANOVA and paired t-tests assume regression errors are independent, but token-level surprisal within a document is likely autocorrelated; no clustered or robust standard errors are reported (Sec. 3.3).
  • domain assumption BPE subword tokens are appropriate base units for surprisal contours.
    Surprisal is computed per BPE token (Sec. 3); the paper acknowledges alternative units in Sec. 2.3 and Limitations, but the analysis depends on this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Harmonic Structure of Information Contours." pith.science (2026). https://pith.science/paper/AFKEJ5BV

@misc{pith2026250603902,
  author       = {Pith},
  title        = {Pith review of: The Harmonic Structure of Information Contours},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AFKEJ5BV}},
  note         = {Machine review of arXiv:2506.03902}
}
read the original abstract

The uniform information density (UID) hypothesis proposes that speakers aim to distribute information evenly throughout a text, balancing production effort and listener comprehension difficulty. However, language typically does not maintain a strictly uniform information rate; instead, it fluctuates around a global average. These fluctuations are often explained by factors such as syntactic constraints, stylistic choices, or audience design. In this work, we explore an alternative perspective: that these fluctuations may be influenced by an implicit linguistic pressure towards periodicity, where the information rate oscillates at regular intervals, potentially across multiple frequencies simultaneously. We apply harmonic regression and introduce a novel extension called time scaling to detect and test for such periodicity in information contours. Analyzing texts in English, Spanish, German, Dutch, Basque, and Brazilian Portuguese, we find consistent evidence of periodic patterns in information rate. Many dominant frequencies align with discourse structure, suggesting these oscillations reflect meaningful linguistic organization. Beyond highlighting the connection between information rate and discourse structure, our approach offers a general framework for uncovering structural pressures at various levels of linguistic granularity.

Figures

Figures reproduced from arXiv: 2506.03902 by the authors.

Figure 1
Figure 1. Illustration of Harmonic Regression on Surprisal Contours. Surprisal contours, unit boundaries, and first-order sinusoids for the first 200 tokens from a Wall Street Journal article (document wsj_1111 in the English RST Discourse Bank). Time scaling (§3.2) is applied according to the lengths of elementary discourse units (EDUs), sentences, and paragraphs. Here, we set the coefficients of the sinusoids to 1 for illus… view at source ↗
Figure 2
Figure 2. Predicted vs. Observed Surprisal Curves for EDU-scaled Sinusoids. Each panel shows predictions for one document: English (wsj_0605), Spanish (as00007), German (maz-1818), Dutch (AD14_CarpeDiem), Basque (GMB0002-GS), Brazilian Portuguese (D2_C38_Estadao). For other scaling methods, see App. H.2. ously. All models include the baseline predictors, as described in §3.3. Predictive power is reported as the mean validatio… view at source ↗
Figure 3
Figure 3. Harmonic Structure in Surprisal Contours. Top three most dominant sinusoids (EDU-scaled) in the maximal model for a Spanish text (doc ma00059a). Amplitudes signify the contribution to the overall vari￾ation, with higher amplitudes indicating a larger effect. trend holds across languages and boundary types. To explore how this relates to periodicity, we focus on EDU boundaries, since they are the most predictive stru… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Top three most dominant sinusoids for English (doc [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Top three most dominant sinusoids for English (doc [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Predicted curves for all languages and all time-scaled sinusoids. Languages correspond to the following [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

85 extracted references · 37 canonical work pages

  1. [1]

    Christoph Aurnhammer and Stefan L. Frank. 2019. https://doi.org/https://doi.org/10.1016/j.neuropsychologia.2019.107198 Evaluating information-theoretic measures of word prediction in naturalistic sentence reading . Neuropsychologia, 134:107198

    work page arXiv 2019

  2. [2]

    Matthew Aylett and Alice Turk. 2004. https://doi.org/10.1177/00238309040470010201 The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech . Language and Speech, 47(1):31--56. PMID: 15298329

    doi:10.1177/00238309040470010201 2004

  3. [3]

    Matthew Aylett and Alice Turk. 2006. https://doi.org/10.1121/1.2188331 Language redundancy predicts syllabic duration and the spectral characteristics of vocalic syllable nuclei . The Journal of the Acoustical Society of America, 119(5):3048--3058

    doi:10.1121/1.2188331 2006

  4. [4]

    Matthew P. Aylett. 1999. Stochastic suprasegmentals: Relationships between redundancy, prosodic structure and syllabic duration. In Proceedings of the XIVth International Congress of Phonetic Sciences, pages 289--292. American Institute of Physics

    1999

  5. [5]

    Alan Bell, Daniel Jurafsky, Eric Fosler - Lussier, Cynthia Girand, Michelle Gregory, and Daniel Gildea. 2003. https://doi.org/10.1121/1.1534836 Effects of disfluencies, predictability, and utterance position on word form variation in E nglish conversation . The Journal of the Acoustical Society of America, 113(2):1001--1024

    work page doi:10.1121/1.1534836 2003

  6. [6]

    um" to "yeah

    Claire Augusta Bergey and Simon DeDeo. 2024. http://arxiv.org/abs/2403.08890 From "um" to "yeah": Producing, predicting, and regulating information flow in human conversation

    arXiv 2024

  7. [7]

    Paula C. F. Cardoso, Erick G. Maziero, Mara Luca Castro Jorge, Eloize R. M. Seno, Ariani Di Felippo, Lucia Helena Machado Rino, Maria das Graças Volpe Nunes, and Thiago A. S. Pardo. 2011. https://sites.icmc.usp.br/taspardo/RST2011-CardosoEtAl1.pdf Cstnews - a discourse-annotated corpus for single and multidocument summarization of news texts in brazilian ...

    2011

  8. [8]

    Lynn Carlson and Daniel Marcu. 2001. https://web.archive.org/web/20170808131213id_/https://www.isi.edu/ marcu/discourse/tagging-ref-manual.pdf Discourse tagging reference manual . Technical Report ISI-TR-545, University of Southern California Information Sciences Institute

    work page arXiv 2001

Show all 85 references
  1. [9]

    Lynn Carlson, Daniel Marcu, and Mary Ellen Okurovsky. 2001. https://aclanthology.org/W01-1605 Building a discourse-tagged corpus in the framework of R hetorical S tructure T heory . In Proceedings of the Second SIG dial Workshop on Discourse and Dialogue

    2001

  2. [10]

    Raoul Cervantes and Glenn Gainer. 1992. http://www.jstor.org/stable/3586886 The effects of syntactic simplification and repetition on listening comprehension . TESOL Quarterly, 26(4):767--770. Wiley

    1992

  3. [11]

    Clark and Deanna Wilkes-Gibbs

    Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. https://doi.org/https://doi.org/10.1016/0010-0277(86)90010-7 Referring as a collaborative process . Cognition, 22(1):1--39

    1986 doi

  4. [12]

    Uriel Cohen Priva. 2015. https://doi.org/10.1515/lp-2015-0008 Informativity affects consonant duration and deletion rates . Laboratory Phonology, 6(2)

    2015 doi

  5. [13]

    Michael Xavier Collins. 2014. https://doi.org/https://doi.org/10.1007/s10936-013-9273-3 Information density and dependency length as complementary cognitive models . Journal of Psycholinguistic Research, 43:651--681

    2014 doi

  6. [14]

    Iria da Cunha, Juan-Manuel Torres-Moreno, and Gerardo Sierra. 2011. https://aclanthology.org/W11-0401 On the development of the RST S panish treebank . In Proceedings of the 5th Linguistic Annotation Workshop, pages 1--10, Portland, Oregon, USA. Association for Computational L...

    2011

  7. [15]

    Jeffrey L. Elman. 1990. https://www.sciencedirect.com/science/article/pii/036402139090002E Finding structure in time . Cognitive Science, 14(2):179--211. Wiley Online Library

    1990

  8. [16]

    Julen Etxaniz, Oscar Sainz, Naiara Perez, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Ormazabal, Mikel Artetxe, and Aitor Soroa. 2024. http://arxiv.org/abs/2403.20266 L atxa: An open language model and evaluation suite for B asque

    2024 arXiv

  9. [17]

    a chtnis-konstanz im sprachlichen Informationsflu ? Zeitschrift f \

    August Fenk and Gertraud Fenk. 1980. https://wwwg.uni-klu.ac.at/mk0/personal/bedienst/Kurzzeitgedaechtnis1980.pdf Konstanz im Kurzzeitged \"a chtnis-konstanz im sprachlichen Informationsflu ? Zeitschrift f \"u r experimentelle und angewandte Psychologie , 27(3):400--414

    1980

  10. [18]

    Frank, and Gabriella Vigliocco

    Irene Fernandez Monsalve, Stefan L. Frank, and Gabriella Vigliocco. 2012. https://aclanthology.org/E12-1041 Lexical surprisal as a general predictor of reading time . In Proceedings of the 13th Conference of the E uropean Chapter of the Association for Computational Linguistic...

    2012

  11. [19]

    Austin F Frank and T Florain Jaeger. 2008. https://escholarship.org/content/qt7d08h6j4/qt7d08h6j4.pdf Speaking rationally: Uniform information density as an optimal strategy for language production . In Proceedings of the annual meeting of the cognitive science society, volume 30

    2008

  12. [20]

    Stefan L Frank, Leun J Otten, Giulia Galli, and Gabriella Vigliocco. 2015. https://doi.org/10.1016/j.bandl.2014.10.006 The ERP response to the amount of information conveyed by words in sentences . Brain and Language, 140:1--11

    2015 doi

  13. [21]

    Markus Frohmann, Igor Sterner, Ivan Vuli \'c , Benjamin Minixhofer, and Markus Schedl. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.665 Segment any text: A universal approach for robust, efficient and adaptable sentence segmentation . In Proceedings of the 2024 Conference...

    2024 doi

  14. [22]

    Richard Futrell. 2023. https://doi.org/10.1073/pnas.2220593120 Information-theoretic principles in incremental language production . Proceedings of the National Academy of Sciences, 120(39):e2220593120

    2023 doi

  15. [23]

    Dmitriy Genzel and Eugene Charniak. 2002. https://doi.org/10.3115/1073083.1073117 Entropy rate constancy in text . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 199--206, Philadelphia, Pennsylvania, USA. Association for Compu...

    2002

  16. [24]

    Dmitriy Genzel and Eugene Charniak. 2003. https://aclanthology.org/W03-1009 Variation of entropy and parse trees of sentences as a function of the sentence number . In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 65--72

    2003

  17. [25]

    Edward Gibson. 1998. https://doi.org/https://doi.org/10.1016/S0010-0277(98)00034-1 Linguistic complexity: locality of syntactic dependencies . Cognition, 68(1):1--76

    1998 doi

  18. [26]

    Edward Gibson. 2000. https://tedlab.mit.edu/tedlab_website/researchpapers/Gibson_2000_DLT.pdf The dependency locality theory: A distance-based theory of linguistic complexity . Image, language, brain/MIT Press

    2000

  19. [27]

    Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy

    Edward Gibson, Richard Futrell, Steven P. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. https://doi.org/https://doi.org/10.1016/j.tics.2019.02.003 How efficiency shapes human language . Trends in Cognitive Sciences, 23(5):389--407

    2019 doi

  20. [28]

    Mario Giulianelli and Raquel Fern \'a ndez. 2021. https://doi.org/10.18653/v1/2021.conll-1.50 Analysing human strategies of information transmission as a function of discourse context . In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 647...

    2021 doi

  21. [29]

    Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, and Ryan Cotterell. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.1032 On the proper treatment of tokenization in psycholinguistics . In Proceedings of the 2024 Conference on Empirical Meth...

    2024 doi

  22. [30]

    Mario Giulianelli, Andreas Opedal, and Ryan Cotterell. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.682 Generalized measures of anticipation and responsivity in online language processing . In Findings of the Association for Computational Linguistics: EMNLP 2024, p...

    2024 doi

  23. [31]

    Mario Giulianelli, Arabella Sinclair, and Raquel Fern \'a ndez. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.652 Is information density uniform in task-oriented dialogues? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 827...

    2021 doi

  24. [32]

    Mario Giulianelli, Sarenne Wallbridge, Ryan Cotterell, and Raquel Fernández. 2024 c . https://doi.org/10.31234/osf.io/fhp84 Incremental alternative sampling as a lens into the temporal and representational resolution of linguistic prediction

    2024 doi

  25. [33]

    Mario Giulianelli, Sarenne Wallbridge, and Raquel Fern \'a ndez. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.343 Information value: Measuring utterance predictability as distance from plausible alternatives . In Proceedings of the 2023 Conference on Empirical Methods in ...

    2023 doi

  26. [34]

    Adam Goodkind and Klinton Bicknell. 2018. https://doi.org/10.18653/v1/W18-0102 Predictive power of word surprisal for reading times is a linear function of language model quality . In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics ( CMCL 20...

    2018 doi

  27. [35]

    Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65--70

    1979

  28. [36]

    Eva Huber, Sebastian Sauppe, Arrate Isasi-Isasmendi, Ina Bornkessel-Schlesewsky, Paola Merlo, and Balthasar Bickel. 2024. https://doi.org/10.1162/nol_a_00121 Surprisal from language models can predict ERP s in processing predicate-argument structures only if enriched by an age...

    2024 doi

  29. [37]

    Aranzabe, Arantza Diaz de Ilarraza, Itziar Gonzalez-Dios, Mikel Lersundi, and Oier Lopez de la Calle

    Mikel Iruskieta, Mar\' i a J. Aranzabe, Arantza Diaz de Ilarraza, Itziar Gonzalez-Dios, Mikel Lersundi, and Oier Lopez de la Calle. 2013. https://www.ixa.eus/sites/default/files/dokumentuak/3960/2013RST-Basque-TB.pdf The rst basque treebank: An online search interface to check...

    2013

  30. [38]

    Florian Jaeger

    T. Florian Jaeger. 2010. https://doi.org/https://doi.org/10.1016/j.cogpsych.2010.02.002 Redundancy and reduction: Speakers manage syntactic information density . Cognitive Psychology, 61(1):23--62

    2010 doi

  31. [39]

    Marcel A Just and Patricia A Carpenter. 1980. https://psycnet.apa.org/doiLanding?doi=10.1037 theory of reading: from eye fixations to comprehension. Psychological review, 87(4):329

    1980

  32. [40]

    Gretchen Kambe Keith Rayner and Susan A. Duffy. 2000. https://doi.org/10.1080/713755934 The effect of clause wrap-up on eye movements during reading . The Quarterly Journal of Experimental Psychology Section A, 53(4):1061--1080. PMID: 11131813

    2000 doi

  33. [41]

    Frank Keller. 2004. https://aclanthology.org/W04-3241 The entropy rate principle as a predictor of processing effort: An evaluation against eye-tracking data . In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 317--324, Barcelona,...

    2004

  34. [42]

    Florian Jaeger

    Roger Levy and T. Florian Jaeger. 2006. https://proceedings.neurips.cc/paper_files/paper/2006/file/c6a01432c8138d46ba39957a8250e027-Paper.pdf Speakers optimize information density through syntactic reduction . In Advances in Neural Information Processing Systems, volume 19. MIT Press

    2006

  35. [43]

    Edwin Herbert Lewis. 1894. https://archive.org/details/historyenglishp00lewigoog The history of the English paragraph . University of Chicago Press

  36. [44]

    Jiaxuan Li and Richard Futrell. 2024. https://escholarship.org/content/qt1fd682nd/qt1fd682nd_noSplash_e733707513b8f4be3d407d3f029acd2b.pdf?t=sgric3 An information-theoretic model of shallow and deep language comprehension . In Proceedings of the Annual Meeting of the Cognitive...

    2024

  37. [45]

    Eliot Ma \"e s, Philippe Blache, and Leonor Becerra-Bonache. 2022. https://hal.science/hal-04151675/document Shared knowledge in natural conversations: c an entropy metrics shed light on information transfers? In 26th Conference on Computational Natural Language Learning, page...

    2022

  38. [46]

    Piantadosi, and Edward Gibson

    Kyle Mahowald, Evelina Fedorenko, Steven T. Piantadosi, and Edward Gibson. 2013. https://doi.org/https://doi.org/10.1016/j.cognition.2012.09.010 Info/information theory: Speakers choose shorter words in predictive contexts . Cognition, 126(2):313--318

    2013 doi

  39. [47]

    Mann and Sandra A

    William C. Mann and Sandra A. Thompson. 1988. https://doi.org/doi:10.1515/text.1.1988.8.3.243 Rhetorical structure theory: Toward a functional theory of text organization . Text - Interdisciplinary Journal for the Study of Discourse, 8(3):243--281

    1988 doi

  40. [48]

    Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor

    Mitchell P. Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. 1999. https://catalog.ldc.upenn.edu/LDC99T42 Treebank-3 . Linguistic Data Consortium, 14

    1999

  41. [49]

    Clara Meister, Mario Giulianelli, and Tiago Pimentel. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.921 Towards a similarity-adjusted surprisal theory . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16485--16498, Miami, Fl...

    2024 doi

  42. [50]

    Clara Meister, Tiago Pimentel, Patrick Haller, Lena J \"a ger, Ryan Cotterell, and Roger Levy. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.74 Revisiting the U niform I nformation D ensity hypothesis . In Proceedings of the 2021 Conference on Empirical Methods in Natural ...

    2021 doi

  43. [51]

    Benjamin Minixhofer, Jonas Pfeiffer, and Ivan Vuli \'c . 2023. https://aclanthology.org/2023.acl-long.398 Where ' s the point? self-supervised multilingual punctuation-agnostic sentence segmentation . In Proceedings of the 61st Annual Meeting of the Association for Computation...

    2023

  44. [52]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. https://openreview.net/forum?id=wHBfxhZu1u Ya RN : Efficient context window extension of large language models . In The Twelfth International Conference on Learning Representations

    2024

  45. [53]

    Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. 2023. https://doi.org/10.1007/978-3-031-45392-2_15 Sabi \'a : Portuguese large language models . In Intelligent Systems, pages 226--240, Cham. Springer Nature Switzerland

    2023 doi

  46. [54]

    Florian Jaeger

    Ting Qian and T. Florian Jaeger. 2011. https://escholarship.org/content/qt6b0712jg/qt6b0712jg.pdf Topic shift in efficient discourse production . In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33

    2011

  47. [55]

    Milena Rabovsky, Steven S Hansen, and James L McClelland. 2018. https://doi.org/https://www.nature.com/articles/s41562-018-0406-4 Modelling the N 400 brain potential as change in a probabilistic representation of meaning . Nature Human Behaviour, 2(9):693--705

    2018

  48. [56]

    Radvansky and David E

    Gabriel A. Radvansky and David E. Copeland. 2010. https://doi.org/10.1037/a0017258 Reading times and the detection of event shift processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(1):210--216. Place: US Publisher: American Psychological Association

    2010 doi

  49. [57]

    Gisela Redeker, Ildik \'o Berzl \'a novich, Nynke van der Vliet, Gosse Bouma, and Markus Egg. 2012. http://www.lrec-conf.org/proceedings/lrec2012/pdf/887_Paper.pdf Multi-layer discourse annotation of a D utch text corpus . In Proceedings of the Eighth International Conference ...

    2012

  50. [58]

    Hosseini, Nancy Kanwisher, Joshua B

    Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2021. https://doi.org/10.1073/pnas.2105646118 The neural architecture of language: Integrative modeling converges on predictive proce...

    2021 doi

  51. [59]

    Cory Shain, Idan Asher Blank, Marten van Schijndel , William Schuler, and Evelina Fedorenko. 2020. https://doi.org/https://doi.org/10.1016/j.neuropsychologia.2019.107307 fmri reveals language-specific predictive coding during naturalistic sentence comprehension . Neuropsycholo...

    2020

  52. [60]

    Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. https://doi.org/https://doi.org/10.1073/pnas.2307876121 Large-scale evidence for logarithmic effects of word predictability on reading time . Proceedings of the National Academy of Sciences, 121(1...

    2024 doi

  53. [61]

    Claude E. Shannon. 1948. https://ieeexplore.ieee.org/document/6773024 A mathematical theory of communication . The Bell System Technical Journal, 27(3):379--423

    1948

  54. [62]

    S kipper S eabold and J osef P erktold. 2010. https://doi.org/10.25080/Majora-92bf1922-011 S tatsmodels: E conometric and S tatistical M odeling with P ython . In P roceedings of the 9th P ython in S cience C onference , pages 92 -- 96

    2010 doi

  55. [63]

    Meyer, and Andrea E

    Sophie Slaats, Antje S. Meyer, and Andrea E. Martin. 2024. https://doi.org/10.1162/nol_a_00155 Lexical surprisal shapes the time course of syntactic structure building . Neurobiology of Language, 5(4):942--980

    2024 doi

  56. [64]

    Smith and Roger Levy

    Nathaniel J. Smith and Roger Levy. 2013. https://doi.org/https://doi.org/10.1016/j.cognition.2013.02.013 The effect of word predictability on reading time is logarithmic . Cognition, 128(3):302--319

    2013 doi

  57. [65]

    Speer and Jeffrey M

    Nicole K. Speer and Jeffrey M. Zacks. 2005. https://doi.org/https://doi.org/10.1016/j.jml.2005.02.009 Temporal changes as event boundaries: Processing and memory consequences of narrative time shifts . Journal of Memory and Language, 53(1):125--140

    2005 doi

  58. [66]

    Manfred Stede. 2004. https://aclanthology.org/W04-0213 The P otsdam commentary corpus . In Proceedings of the Workshop on Discourse Annotation, pages 96--102, Barcelona, Spain. Association for Computational Linguistics

    2004

  59. [67]

    Manfred Stede and Arne Neumann. 2014. http://www.lrec-conf.org/proceedings/lrec2014/pdf/579_Paper.pdf P otsdam commentary corpus 2.0: Annotation for discourse research . In Proceedings of the Ninth International Conference on Language Resources and Evaluation ( LREC '14) , pag...

    2014

  60. [68]

    Fatemeh Torabi Asr and Vera Demberg. 2015. https://aclanthology.org/W15-0117 Uniform surprisal at the level of discourse relations: Negation markers and discourse connective omission . In Proceedings of the 11th International Conference on Computational Semantics, pages 118--1...

    2015

  61. [69]

    Eleftheria Tsipidi, Franz Nowak, Ryan Cotterell, Ethan Wilcox, Mario Giulianelli, and Alex Warstadt. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1047 Surprise! U niform I nformation D ensity isn't the whole story: Predicting surprisal contours in long-form discourse . In...

    2024 doi

  62. [70]

    van der Vliet , I

    N.H. van der Vliet , I. Berzl \'a novich, G. Bouma, M. Egg, and G. Redeker. 2011. Building a discourse-annotated dutch text corpus. In Beyond Semantics, volume 3 of Bochumer Linguistische Arbeitsberichte, pages 157 -- 171. Ruhr-Universit \"a t Bochum. 2011/g.bouma/pub005, 2011...

    2011

  63. [71]

    Bram Vanroy. 2024. http://arxiv.org/abs/2412.04092 Geitje 7b ultra: A conversational model for dutch

    2024 arXiv

  64. [72]

    Vivek Verma, Nicholas Tomlin, and Dan Klein. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.1039 Revisiting entropy rate constancy in text . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15537--15549, Singapore

    2023 doi

  65. [73]

    O'Donnell, and Ryan Cotterell

    Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timothy J. O'Donnell, and Ryan Cotterell. 2024. http://arxiv.org/abs/2412.03719 From language models over tokens to language models over characters

    2024 arXiv

  66. [74]

    Sarenne Wallbridge, Peter Bell, and Catherine Lai. 2022. https://www.isca-archive.org/interspeech_2022/wallbridge22_interspeech.pdf Investigating perception of spoken dialogue acceptability through surprisal . In Interspeech 2022: The 23rd Annual Conference of the Internationa...

    2022

  67. [75]

    Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020. https://cognitivesciencesociety.org/cogsci20/papers/0375/0375.pdf On the predictive power of neural language models for human real-time comprehension behavior . In Proceedings of the 42nd Annu...

    2020

  68. [76]

    Ethan Gotlieb Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. https://doi.org/10.1162/tacl_a_00612 Testing the Predictions of Surprisal Theory in 11 Languages . Transactions of the Association for Computational Linguistics, 11:1451--1470

    2023 doi

  69. [77]

    Weijie Xu, Jason Chon, Tianran Liu, and Richard Futrell. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.1052 The linearity of the effect of surprisal on reading times across languages . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1571...

    2023 doi

  70. [78]

    Yang Xu and David Reitter. 2016. https://doi.org/10.18653/v1/P16-1051 Entropy converges between dialogue participants: Explanations from an information-theoretic perspective . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2016 doi

  71. [79]

    Yang Xu and David Reitter. 2018. https://doi.org/https://doi.org/10.1016/j.cognition.2017.09.018 Information density converges in dialogue: Towards an information-theoretic model . Cognition, 170:147--163

    2018 doi

  72. [80]

    Yang Xu, Yu Wang, Hao An, Zhichen Liu, and Yongyuan Li. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.564 Detecting subtle differences between human and model languages using spectrum of relative likelihood . In Proceedings of the 2024 Conference on Empirical Methods in Na...

    2024 doi

  73. [81]

    Zuhao Yang, Yingfang Yuan, Yang Xu, Shuo Zhan, Huajun Bai, and Kefan Chen. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/37094fdc81632915a5738293cf9b7ad4-Paper-Conference.pdf Face: E valuating natural language generation with F ourier analysis of cross-entro...

    2023

  74. [82]

    Sinclair

    Jun Sen Yee, Mario Giulianelli, and Arabella J. Sinclair. 2024. https://aclanthology.org/2024.lrec-main.494/ Efficiency and effectiveness in task-oriented dialogue: On construction repetition, information rate, and task success . In Proceedings of the 2024 Joint International ...

    2024

  75. [83]

    George Kingsley Zipf. 1949. Human Behavior and the Principle of Least Effort. Addison-Wesley Press, Oxford, England

    1949

  76. [84]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  77. [85]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.