Pith. sign in

REVIEW 3 major objections 4 minor 25 references

Locating the Leading Edge of Cultural Change

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a text's precocity — how much more it resembles the future than the past — aligns with citation counts and authorial youth across three corpora, and that this alignment is strongest when only the most forward-looking…

desk verdict A solid empirical benchmark for textual precocity, but the headline causal interpretation needs passage-level evidence and significance tests before I'd take the top-quartile claim as established. read the letter →

arxiv 2411.15068 v1 pith:EAWZJDJG submitted 2024-11-22 cs.CL

Sarah Griebel , Becca Cohen , Lucian Li , Jaihyun Park , Jiayu Liu , Jana Perkins , Ted Underwood This is my paper · ORCID
classification cs.CL
keywords precocityculturalchangetextualnoveltytopicmodelsdocumentembeddingsperplexitycitationsauthorialage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Researchers want to measure cultural change from text, but different similarity measures give different answers and no absolute ground truth exists. This paper tests three text representations — topic models, document embeddings, and word-level perplexity — against three social signals: citation counts, authorial youth, and critical discussion, across literary studies, economics, and fiction. Its central claim is that textual precocity, defined as looking more like later documents than earlier ones, aligns with these social signals in every corpus. The alignment is consistently strongest when a document is represented by the average precocity of its top quartile of passages rather than by all passages. That result suggests cultural impact is carried by a small forward-looking fraction of a text, not by sustained average novelty.

What carries the argument

The load-bearing quantity is precocity, defined as novelty minus transience: novelty measures divergence from documents in the preceding twenty years, transience measures divergence from documents in the following twenty years, and a text with high precocity simply "looks later than" peers published in the same year. For perplexity, precocity is computed as $2\cdot(\mathrm{ppl}_{\mathrm{past}}-\mathrm{ppl}_{\mathrm{future}})/(\mathrm{ppl}_{\mathrm{past}}+\mathrm{ppl}_{\mathrm{future}})$. Documents are cut into passages of about 512 tokens, each passage is characterized by its relation to the surrounding time window, and a document is represented either by the mean of all passages or by the mean of the top quartile of passages by precocity. The top-quartile operation is what carries the main conclusion: it concentrates the measure on a text's most forward-looking moments.

What would settle it

Take a corpus in which later adoption can be traced passage by passage, for example by recording which specific passages later articles quote or which specific books later authors imitate. If later works are not disproportionately influenced by the high-precocity chunks identified by the method, then the top-quartile alignment with citations and youth would not show that precocity locates the leading edge of cultural change. The paper reports the aggregate alignment but does not perform this passage-level check.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that precocity is a measurable property of documents and that it tracks independent social evidence of cultural change. For every corpus and every representation, works by highly cited authors and by younger authors are textually ahead of the curve, even though citations and youth are anti-correlated in the data. The paper's most specific finding is that this alignment is stronger when a text is summarized by the mean precocity of its top 25% of passages than when all passages are averaged; the authors report that this held "in all of the tests we ran." They also report no systematic advantage for neural embeddings over lexical topic models, and treat topic models as a strong baseline.

Load-bearing premise

The argument depends on treating citation counts, authorial youth, and critical discussion as genuine signals of being on the leading edge of cultural change; if those variables mostly track prestige or visibility rather than influence, the alignment would not demonstrate what the paper claims.

Editorial extensions

If this is right

  • Averaging all passages dilutes the signal; later studies of textual change should consider measuring documents at their most precocious passages.
  • Lexical topic models, not just neural embeddings, can capture the leading edge; researchers should compare against such models before adopting embeddings.
  • Because precocity correlates with both citations and youth despite the two being anti-correlated, the measure is apparently not just tracking prestige.
  • Including both past and future information matters; the future-only prescience measure achieved less than half the explanatory power of the expanded version.
  • Journal-level and author-level plots of precocity can reveal editors or authors who anticipate later trends even when citation counts do not distinguish them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that if influence concentrates in the most precocious passages, later works should quote or cite those passages disproportionately; a passage-level citation-tracing study could test this directly.
  • We infer the same top-quartile logic could be transferred to patents, scientific articles, or policy documents to ask whether breakthrough recognition is driven by a few forward-looking sections.
  • We infer that prestige-based evaluation may systematically underweight young authors whose innovation is concentrated in short anticipatory moments, since averaging hides the leading edge.
  • We infer a possible mechanism worth testing: the top-quartile passages may be structurally distinctive, such as introductions, conclusions, or theoretical framing, a pattern the paper did not analyze.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces a text-level measure called "precocity," defined as the degree to which a text resembles future documents more than past documents, and compares three textual representations (topic models, fine-tuned document embeddings, and word-level perplexity) across three corpora (literary studies, economics, and fiction). The authors regress five social-outcome variables (citation counts in two fields, author age in two fields, and critical discussion in fiction) on precocity, comparing two document-level aggregation strategies: the mean over all passages and the mean over the top quartile of passages by precocity. They report that the top-quartile strategy aligns better with social evidence in every test they ran, while no single text representation consistently outperforms the others, and they interpret this as evidence that cultural influence may be concentrated in a small forward-looking fraction of a text.

Significance. If the top-quartile finding is robust, it would be a useful methodological result for computational humanities and cultural analytics, because most prior work averages novelty or prescience over entire documents. The paper is commendably broad in scope, covering three corpora and three representation families, and it makes data and code publicly available. It also shows unusual transparency in Appendix H, disclosing departures from the preregistration and reporting paths not taken, which strengthens confidence in the authors' good faith. However, the central claim currently rests on small differences in document-level R-squared values without uncertainty quantification, and the mechanism invoked to interpret those differences is not directly tested. The paper's practical significance depends on whether these gaps can be closed with additional analysis.

major comments (3)
  1. [Section 4, Table 1] The claim that the top-quartile method "aligned better with social evidence than a method that averaged all passages" in all tests is not actually supported by Table 1. In the Perplexity rows for Age of literary scholars and Age of fiction writers, the reported R-squared values are identical under 0.25 and 1.0 aggregation (.024 vs .024 and .014 vs .014, respectively). Several other differences are extremely small, for example Topics on fiction-writer age (.051 vs .049) and Embeds on literary-scholar age (.035 vs .034). Because no confidence intervals, standard errors, or p-values are reported, the reader cannot tell whether these differences are anything beyond rounding or sampling noise. The claim of a consistent top-quartile advantage needs at least paired or clustered uncertainty estimates, and ideally a multiple-comparison correction across the 30 method-outcome combinations.
  2. [Section 4, Table 1] The entire method-comparison conclusion rests on R-squared values that are not accompanied by any measure of uncertainty. R-squared differences of 0.002 to 0.005 are reported as evidence for one aggregation strategy over another, but without error bars or significance tests these differences are not interpretable. This is load-bearing because the paper's headline conclusion is not that precocity has some association with social outcomes, but that the top-quartile representation is systematically better. Given that many modeling choices were explored and revised (Appendix H), the risk of overfitting to the reported outcome is real. I would like to see bootstrap confidence intervals for the R-squared differences, or an equivalent resampling procedure, applied to the central comparison in Section 4.
  3. [Section 4, paragraph beginning "So what did we learn"] The inference that citations and critical references are "motivated by innovations expressed in a relatively small part of a text" is not directly tested. Table 1 contains only document-level regressions, which cannot establish that the specific passages ranked as most precocious are the passages that elicited the social response. A document-level confound is plausible: highly cited or prominent documents may contain a few future-sounding passages for reasons unrelated to the content that actually drew attention, such as visibility, author networks, or length. Appendix C excludes same-author and verbatim-quote chunks, which addresses one mechanical circularity, but it does not identify the locus of social influence. A passage-level validation would be needed: for example, locate passages that citing or critical texts explicitly quote, paraphrase, or discuss, and compare the precocity scores of those passages with the precocity scores of non-discussed passages, ideally conditioning on document-level effects.
minor comments (4)
  1. [Section 2.1] The statement that "youth and citation frequency are negatively correlated in this corpus" is reported without a correlation coefficient or sample-size caveat; since the age data cover only 2,646 of 40,407 articles, a brief summary of the strength and robustness of this correlation would help the reader assess the tension the paper relies on.
  2. [Appendix D] The author-age inference procedure is described as achieving "overall accuracy of greater than 90%," but no confidence interval or validation-set details are provided; because age is one of the two primary social-outcome variables, a sentence on the precision of the VIAF matching model would be useful.
  3. [Section 3.4] The top-quartile strategy averages the 25% of chunks with the highest precocity within each document, but the number of chunks per document is not reported; for very short documents the quartile may be based on very few chunks, which could affect the comparison between the two aggregation strategies.
  4. [Appendix H] The appendix honestly notes that the embedding strategy changed several times and that authorial age was added after preregistration; however, the main text does not flag how these departures affect the strength of the evidence for the claim that no representation is preferred, and a sentence acknowledging the potential for selective reporting would improve the paper's transparency.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: precocity is computed purely from textual divergence, social evidence is external, and Appendix C removes text-reuse artifacts.

full rationale

The derivation chain is self-contained with respect to its social outcomes. Precocity is defined in Eq. 1 using only perplexity in past and future models; novelty, transience, and resonance are computed from textual divergence alone, with no citation, age, or critical-discussion variable entering the textual measure. The social variables (Semantic Scholar citation counts, VIAF/HathiTrust-based author ages, and mentions in the literary-studies corpus) are external to the precocity calculation, so the reported R2 values are not forced by construction. The most plausible mechanical circularity—chunks that quote one another or share an author—is explicitly addressed in Appendix C: "we avoided comparing any papers written by the same author" and removed chunks containing quoted matches, with the paper reporting that these exclusions made little difference. The top-quartile strategy is a comparison of two aggregation rules, not a parameter fitted to the social outcomes; no equation equates the top-quartile score to citations or age. The self-citations present, especially [8] for fiction author-age data and the similar-chunks baseline and [22] for preregistration, are not load-bearing: the age data are external metadata, and the similar-chunks baseline was tested and did not improve results. The main limitation—that document-level R2 does not establish that the specific passages labeled most precocious are the passages that elicited citation or critical attention—is an evidentiary gap about mechanism rather than a circularity. Overall, no significant circularity was found.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several hand-chosen measurement parameters, including window sizes, topic count, chunk size, and the top-quartile threshold, and on the domain assumption that citations and youth are valid proxies for cultural change. No new physical or conceptual entities are postulated. The threshold and fine-tuning choices are partly post hoc, which raises the risk that the top-quartile finding is overfit to this dataset.

free parameters (6)
  • top_quartile_threshold = 0.25
    The central finding selects the top 25 percent of passages by precocity. The threshold is reported as aligning best with social evidence, but no cross-validation or systematic search over other quantiles is described.
  • lookback_window_years = 20
    Precocity compares chunks to documents in the preceding and following 20 years. This window defines the past and future for all methods and directly shapes the measurement.
  • perplexity_period_years = 12-year windows with 4-year offset, spanning 36 years
    Perplexity uses RoBERTa models trained on 12-year periods with a 4-year offset. The window size and offset are hand-chosen to balance corpus size and temporal resolution.
  • chunk_size_tokens = 512
    Passages are limited to 512 tokens for embeddings and perplexity, and topic model chunks are at least 512 tokens. Passage size affects both the top-quartile aggregation and the comparison units.
  • topic_count = 250
    LDA models use 250 topics. The number of topics is a modeling choice that can affect KL divergence and thus precocity.
  • paraphrase_fraction = up to 18 percent
    During embedding fine-tuning, up to 18 percent of positive pairs were paraphrased or condensed with GPT-3.5. This percentage was tuned and improved alignment with social evidence.
assumptions (4)
  • domain assumption Citation count, authorial youth, and critical discussion are valid social evidence of being on the leading edge of cultural change.
    Section 1 defines the validation strategy: because no absolute ground truth exists, the authors combine prominence and youth as independent signals. If these variables measure prestige or visibility rather than cultural influence, the conclusions about method ranking do not follow.
  • domain assumption Comparing a text chunk to all chunks in the preceding and following 20 years captures corpus-level cultural change.
    Section 3.4 and Appendix H describe the comparison window. Appendix H notes that genre-specific comparison to similar chunks was tested and did not often improve results, but the default assumption is that global corpus trends are the right reference.
  • standard math The regression model with precocity, precocity squared, novelty, and publication date adequately separates time effects and citation opportunity.
    Section 3.5 uses multiple linear regression. The model assumes linearity and that the date control handles older works having more time to accumulate citations, as noted in Appendix E.
  • domain assumption Text reuse and same-author comparisons can be excluded by the procedures in Appendix C without distorting the measure.
    Appendix C excludes chunks containing six or more matching quoted words or the cited author's last name. The authors estimate the effects were small, but the exclusion rule is a modeling choice that could affect the precocity scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Locating the Leading Edge of Cultural Change." pith.science (2026). https://pith.science/paper/EAWZJDJG

@misc{pith2026241115068,
  author       = {Pith},
  title        = {Pith review of: Locating the Leading Edge of Cultural Change},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EAWZJDJG}},
  note         = {Machine review of arXiv:2411.15068}
}
read the original abstract

Measures of textual similarity and divergence are increasingly used to study cultural change. But which measures align, in practice, with social evidence about change? We apply three different representations of text (topic models, document embeddings, and word-level perplexity) to three different corpora (literary studies, economics, and fiction). In every case, works by highly-cited authors and younger authors are textually ahead of the curve. We don't find clear evidence that one representation of text is to be preferred over the others. But alignment with social evidence is strongest when texts are represented through the top quartile of passages, suggesting that a text's impact may depend more on its most forward-looking moments than on sustaining a high level of innovation throughout.

Figures

Figures reproduced from arXiv: 2411.15068 by the authors.

Figure 1
Figure 1. Literary studies journals and authors plotted by the average precocity and number of citations associated with their articles. Precocity is determined here by topic modeling. Since both axes are z-scores, the center of the whole corpus would be at 0, 0. We’re looking mostly at the upper right quadrant. Citation counts are already public. But precocity—vertical position in [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 17 canonical work pages

  1. [1]

    Zhang, Q

    X. Zhang, Q. Xie, M. Song, Measuring the Impact of Novelty, Bibliometric, and Academic- Network Factors on Citation Count Using a Neural Network, Journal of Informetrics 15 (2021) 101140. doi:https://doi.org/10.1016/j.joi.2021.101140

    work page arXiv 2021

  2. [2]

    Vicinanza, A

    P. Vicinanza, A. Goldberg, S. B. Srivastava, A Deep-Learning Model of Prescient Ideas Demonstrates That They Emerge from the Periphery, PNAS Nexus 2 (2022) pgac275. doi:10.1093/pnasnexus/pgac275

    work page doi:10.1093/pnasnexus/pgac275 2022

  3. [3]

    A. T. J. Barron, J. Huang, R. L. Spang, S. DeDeo, Individuals, Institutions, and Innovation in the Debates of the French Revolution, Proceedings of the National Academy of Sciences 115 (2018) 4607–4612. doi:10.1073/pnas.1717729115

    doi:10.1073/pnas.1717729115 2018

  4. [4]

    Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data

    L. Bornmann, A. Tekles, H. H. Zhang, F. Y. Ye, Do We Measure Novelty When We An- alyze Unusual Combinations of Cited References? A Validation Study of Bibliometric Novelty Indicators Based on F1000Prime Data, 2019. URL: https://arxiv.org/abs/1910.03233. arXiv:1910.03233

    work page Pith review arXiv 2019

  5. [5]

    R. K. Merton, The Matthew Effect in Science, Science 159 (1968) 56–63. doi: 10.1126/ science.159.3810.56

    work page 1968

  6. [6]

    J. M. Meisel, M. Elsig, E. Rinke, Language Acquisition and Change: A Morphosyntactic Perspective, Edinburgh University Press, Edinburgh, 2013

    work page 2013

  7. [7]

    W. E. Miller, J. M. Shanks, The New American Voter, Harvard University Press, Cambridge, MA, 1996

    work page 1996

  8. [8]

    Underwood, K

    T. Underwood, K. Kiley, W. Shang, S. Vaisey, Cohort Succession Explains Most Change in Literary Culture, Sociological Science 9 (2022) 184–205. doi: 10.15195/v9.a8

    work page doi:10.15195/v9.a8 2022

Show all 25 references
  1. [9]

    Burns, A

    J. Burns, A. Brenner, K. Kiser, M. Krot, C. Llewellyn, R. Snyder, JSTOR - Data for Research, in: M. Agosti, J. Borbinha, S. Kapidakis, C. Papatheodorou, G. Tsakonas (Eds.), Research and Advanced Technology for Digital Libraries, Springer Berlin Heidelberg, Berlin, Heidelberg, ...

    2009

  2. [10]

    R. M. Kinney, C. Anastasiades, R. Authur, I. Beltagy, J. Bragg, A. Buraczynski, I. Cachola, S. Candra, Y. Chandrasekhar, A. Cohan, M. Crawford, D. Downey, J. Dunkelberger, O. Et- zioni, R. Evans, S. Feldman, J. Gorney, D. W. Graham, F. Hu, R. Huff, D. King, S. Kohlmeier, B. Ku...

    2023 arXiv

  3. [11]

    J. Jett, B. Capitanu, D. Kudeki, T. Cole, Y. Hu, P. Organisciak, T. Underwood, E. Dick- son Koehl, R. Dubnicek, J. S. Downie, The HathiTrust Research Center Extracted Features Dataset (2.0), HathiTrust Research Center, 2020. doi:10.13012/R2TE-C227

    2020 doi

  4. [12]

    A. K. McCallum, MALLET: A Machine Learning for Language Toolkit, 2002. URL: http: //mallet.cs.umass.edu

    2002

  5. [13]

    D. M. Blei, A. Y. Ng, M. I. Jordan, Latent Dirichlet Allocation, Journal of Machine Learning Research 3 (2003) 993–1022. URL: https://www.jmlr.org/papers/volume3/blei03a/blei03a. pdf

    2003

  6. [14]

    Reimers, I

    N. Reimers, I. Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT- Networks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2019, pp. 3982–3992. URL: https://arxiv.org/abs/1908.10084

    2019 arXiv

  7. [15]

    D. Yin, Z. Wu, K. Yokota, K. Matsumoto, S. Shibayama, Identify Novel Elements of Knowledge with Word Embedding, PLOS ONE 18 (2023) 1–16. doi:10.1371/journal. pone.0284567

    2023 doi

  8. [16]

    Shibayama, D

    S. Shibayama, D. Yin, K. Matsumoto, Measuring Novelty in Science with Word Embedding, PLOS ONE 16 (2021) 1–16. doi:10.1371/journal.pone.0254034

    2021 doi

  9. [17]

    Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, M. Zhang, Towards General Text Embeddings with Multi-Stage Contrastive Learning, arXiv preprint arXiv:2308.03281 (2023)

    2023 arXiv

  10. [18]

    M. L. Henderson, R. Al-Rfou, B. Strope, Y. Sung, L. Lukács, R. Guo, S. Kumar, B. Miklos, R. Kurzweil, Efficient Natural Language Response Suggestion for Smart Reply, CoRR abs/1705.00652 (2017). URL: http://arxiv.org/abs/1705.00652. arXiv:1705.00652

    2017 arXiv

  11. [19]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, R. Lowe, Training Language Models to Follow Instructions with...

    2022 arXiv

  12. [20]

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoy- anov, RoBERTa: A Robustly Optimized BERT Pretraining Approach, CoRR abs/1907.11692 (2019). URL: http://arxiv.org/abs/1907.11692. arXiv:1907.11692

    2019 arXiv

  13. [21]

    Sobchuk, A

    O. Sobchuk, A. Šel,a, Computational Thematics: Comparing Algorithms for Clustering the Genres of Literary Fiction, Humanities and Social Sciences Communications 11 (2024) 438. doi:10.1057/s41599-024-02933-6

    2024 doi

  14. [22]

    Griebel, R

    S. Griebel, R. Cohen, L. Li, J. Liu, J. Park, J. M. Perkins, W. E. Underwood, Comparing Measures of Textual Innovation, 2023. doi:10.17605/OSF.IO/A3G6E. A. Topic models Topic granularity will vary if a corpus includes many more texts in some periods than others, and this could...

    2023 doi

  15. [23]

    Restrict the corpus to an even distribution across time

  16. [24]

    inferencer

    Generate a 250-topic model with MALLET, including an “inferencer. ”

  17. [25]

    document embeddings

    Use the inferencer to generate topic distributions for documents that had to be left out of the “flat” distribution in step 1. Using this model, we assessed novelty, transience, and precocity by measuring the K-L divergence between texts. K-L divergence is an asymmetric measur...

    1900

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.