Pith. sign in

REVIEW 5 major objections 6 minor 24 references

Freshness and Informativity Weighted Cognitive Extent and Its Correlation with Cumulative Citation Count

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes a quota-level title-vocabulary measure, FICE, that weights scientific entities by freshness and informativity, and reports a strong positive Spearman correlation with the logarithm of 5-year average cumulative citation…

desk verdict FICE's correlation with citations is likely inflated because the lifetime model is fit on the full corpus, making the metric retrospective rather than predictive. read the letter →

arxiv 2412.03557 v2 pith:ULJHWHED submitted 2024-12-04 cs.DL cs.IR

Zihe Wang , Jian Wu This is my paper · ORCID
classification cs.DLcs.IR
keywords cognitiveextentcitationimpactentityrecognitiondocumentfrequencyfreshnessinformativitypredictiontitleanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the collective title vocabulary of a body of papers carries information about how many citations those papers will receive over the following five years. It defines FICE, a quota-level statistic that weights each recognized scientific entity in a title by two quantities: freshness, the share of the entity's predicted lifetime still ahead of it, and informativity, how uncommon the entity is within its contemporary vocabulary. Using entities extracted from the titles of computational-linguistics papers, the authors report that average FICE per quota correlates strongly with the logarithm of the average 5-year cumulative citation count, with Spearman coefficients of 0.766 at quota size 125 and 0.748 at quota size 250. If the result holds, title-vocabulary dynamics can act as a collective early indicator of research impact, even though the correlation is not claimed to apply to individual papers.

What carries the argument

FICE (Freshness and Informativity Weighted Cognitive Extent) is the central object. It is a weighted sum over a quota of papers: for each recognized scientific entity in a title, multiply informativity $w(e,t_0)$ by freshness $1-r(e,t_0)$, and sum across titles. The load-bearing part is the lifetime ratio $r$, computed from a model of each entity's yearly document frequency as a composite of Gaussian profiles; the model supplies the extinction year beyond the observed period, so freshness is not just presence or absence but where the entity sits in its predicted career. Informativity $w$ is a normalized inverse document frequency taken across the entities in the same title, ensuring rare entities count more. FICE carries the correlation result because it converts a raw vocabulary count into a measure of how new and how rare a quota's vocabulary is.

What would settle it

Compare the fitted predictions against reality: take entities whose fitted extinction year falls inside the corpus window, so their later document frequencies are observable, and check whether the Gaussian-composite fit predicts the actual yearly counts. Then recompute FICE using observed lifetimes only and re-run the Spearman test; if the strong correlation disappears or drops sharply, the reported relation depends on the extrapolation rather than on title vocabulary.

Watch

Extended reading notes

Core claim

FICE extends the earlier notion of cognitive extent, which counted unique phrases per quota, by replacing the dichotomous count with a weighted sum. For each paper title in a quota, every scientific entity contributes $w(e,t_0)(1-r(e,t_0))$, where the lifetime ratio $r$ is cumulative document frequency up to publication divided by the total over the entity's entire modeled lifetime, and the informativity weight $w$ normalizes cumulative document frequency across all entities in the same title. The authors model each entity's yearly document frequency as a composite of Gaussian profiles, extrapolate the fit to find the extinction year, and use per-year citation records to compute the average 5-year cumulative citation count per quota. In the corpus studied, they find a strong positive Spearman correlation between average FICE and the logarithm of that citation count, and they show FICE beats three simpler variants: the dichotomous entity count, the informativity weight alone, and the freshness weight alone. They also reproduce the earlier observation that the unique entity count per quota grows more slowly than the paper count.

Load-bearing premise

The load-bearing premise is that the Gaussian-curve extrapolation used to predict when an entity stops appearing approximates the entity's real lifetime; if that predicted extinction year is wrong, the freshness weights are artifacts and the reported citation correlation would not be a property of the corpus.

Editorial extensions

If this is right

  • FICE can act as a collective, citation-free early indicator of which research topics are accumulating 5-year citation impact.
  • The dichotomous entity count has weak or negative correlation with the citation measure, so the signal comes from the freshness and informativity weights rather than from vocabulary size alone.
  • The slower-than-exponential growth of unique scientific entities per quota, previously observed in other fields, also holds in the computational-linguistics corpus studied here.
  • Because FICE is defined for any scholarly text, the same weighted cognitive extent can be applied to abstracts, full texts, or other document units without changing the formula.
  • The paper is explicit that the collective correlation does not imply that individual authors can raise citation counts by coining new entity names, since transient entities contribute little to FICE.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to control for publication year, because both FICE and citation counts move upward over time; a year-controlled version would show how much of the correlation is vocabulary signal rather than shared time trend.
  • The same lifetime-fitting machinery could be reused to map how individual scientific terms age and fall out of use, independent of citation prediction.
  • A cross-corpus replication on titles from other disciplines would show whether the strong correlation is a property of scientific titles in general or is specific to this corpus and its entity extractor.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript proposes a new bibliometric indicator, Freshness and Informativity Weighted Cognitive Extent (FICE), which weights the scientific entities appearing in paper titles by a freshness factor (1 minus a lifetime ratio) and by a frequency-based informativity factor. The lifetime of each entity is modeled by fitting its yearly document frequency with a composite of Gaussian profiles, and the lifetime ratio compares the cumulative document frequency up to a paper's publication year with the cumulative frequency over the entire modeled lifetime. Using the ACL Anthology, the authors report that entity-based cognitive extent grows more slowly than the number of papers, and that average FICE per quota correlates strongly with the log of a 5-year cumulative citation count (Spearman 0.766 and 0.748 for quota sizes 125 and 250 in Table 2). The paper also compares FICE with three simplified baselines and claims that FICE shows the strongest correlation.

Significance. If the correlation is robust, the result would be interesting: aggregate title-vocabulary freshness and informativity would predict collective 5-year citation impact, and FICE would extend the original cognitive extent in a principled way. The paper has several concrete strengths: the code is publicly available, three entity recognizers are compared on a manually annotated benchmark, several baseline formulations are tested, and the authors explicitly note that the correlation is collective rather than applicable to individual papers. The main significance risk is that the headline correlation may be an artifact of fitting the lifetime model on the full corpus: the freshness weights for papers published before the end of the corpus use document frequencies observed after the paper's publication year, so post-publication entity success leaks into the metric. The statistical evidence is also less robust than the text suggests, because the correlation at quota size 500 is not significant and one baseline outperforms FICE at that quota.

major comments (5)
  1. [§4.4–4.5, Eq. (1)] The lifetime ratio r(e,t0) is computed with ts and te obtained from a Gaussian fit to the full 1952–2020 document-frequency curve. For any paper published at t0 before the corpus end, the denominator in Eq. (1) includes observed documents published after t0 as well as the extrapolated tail beyond 2020. Consequently, freshness 1 − r(e,t0) encodes how successful the entity became after the paper was published, and because future entity adoption is plausibly correlated with future citations, the Spearman coefficients in Table 2 may be inflated. This is a real leakage problem, not merely a theoretical one. I request a truncated refit in which df(e,t) is fit using only years up to t0, or an explicit reinterpretation of FICE as a retrospective hybrid measure, with correspondingly tempered language about prediction.
  2. [Table 2, |Q|=500 row] The claim that "FICE exhibits the strongest correlation against all baseline models" is not supported at |Q|=500: in that row the lifetime-ratio-only baseline has ρ=0.744, which is larger than FICE's ρ=0.717, and FICE's p-value is 0.109, i.e., not significant at the 0.05 level. The headline strong correlation therefore rests almost entirely on the two smaller quota sizes. Please report the number of bins used for each correlation, provide confidence intervals, and also compute a per-paper rank correlation without binning into quota averages, so that the reader can distinguish a genuine aggregate association from an artifact of binning.
  3. [§4.4] The Gaussian composite fitting procedure is not validated. The paper does not report goodness-of-fit statistics, residual diagnostics, uncertainty in the fitted parameters, or a comparison with alternative lifetime models, and te is often located beyond the observable period by construction. Because r(e,t0) and hence FICE depend directly on te, a poor extrapolation would make the freshness weights artifacts of the fitting model rather than properties of the corpus. Please include a hold-out validation, a null or alternative model comparison, and a report of the distribution of te values across entities.
  4. [§4.6, §5.2] The definition of C5(2015) as the sum of citations received in years 2015–2019 appears to use a fixed calendar window, regardless of each paper's publication year. A paper published in 2019 thus contributes only one year of citations, while a paper published in 2010 contributes citations received well after its first five years. This mixes publication years and citation windows, confounding the relation between FICE at publication time and citation impact. Please clarify the window; if it is indeed fixed, compute per-paper first-five-year citation counts or include a publication-year control.
  5. [§5.1, Table 1] The statement that entity-based cognitive extent "increases at a slower rate" is contradicted by the |Q|=125 row, where the slope rises from 1.19 in 1980–2000 to 6.80 in 2000–2020. The decreasing slope holds only for |Q|=250 and |Q|=500, so the conclusion is quota-dependent as reported. Please either present a trend test that accounts for quota size and time period, or explicitly report that the slowdown is not uniform across quota sizes.
minor comments (6)
  1. [§4.2] There is a typo, "Sciientific" for "Scientific," in the last paragraph of Section 4.2.
  2. [Table 2] The header contains "Speareman" instead of "Spearman."
  3. [Figures] Figure 4 is used for both the entity-based cognitive extent growth plot and the FICE–C5 correlation plot, which is confusing; please renumber the figures.
  4. [Figure 4 caption] The caption says the FICE values are calculated using "undisambiguated entities," while Section 5.2 says disambiguation does not affect the correlation; please clarify which entity set is plotted and whether the reported coefficients are for disambiguated or undisambiguated entities.
  5. [§4.2] The reported F1 scores for SciBERT and SpaCy (0.05 and 0.07) are far below typical published NER results on scientific text; please include the annotation guideline details, the benchmark construction, and the exact prompts so readers can assess whether the comparison is fair.
  6. [§5.2] The error bars are described as Gaussian standard deviations, but citation distributions are highly skewed; please use quartiles or bootstrapped confidence intervals for the binned averages.

Circularity Check

1 steps flagged · score 6.0 of 10

FICE's freshness weight is a full-corpus-fitted future share, so the headline citation correlation is partially circular.

  1. fitted input called prediction [Section 4.4-4.5, Eq. (1)]
    "After fitting the document frequency chart for a scientific entity e, the starting point for ts is determined as the year when e first appeared and te is determined when the predicted document frequency is less than 1, which may be beyond the time span of the observable period. We calculated the lifetime ratio for each e in a title d using Eq. (1) and the informativeness weight using Eq. (2)."

    Eq. (1) defines r(e,t0) as cumulative df up to t0 divided by cumulative df over [ts,te]. The te value is obtained from a Gaussian fit to the entity's full 1952-2020 document-frequency series, so for any paper published before 2020 the denominator includes documents published after t0, plus a predicted tail beyond the corpus. Consequently 1-r(e,t0) is algebraically the fraction of the entity's total lifetime documents that appear after t0. FICE at publication time is therefore a retrospective, future-informed quantity, not a contemporaneous title property.

full rationale

The paper's central derivation is mostly empirical: FICE is not algebraically defined in terms of citation counts, and the Gaussian fits target document frequency, not citations. However, the freshness component in Eq. (3) depends on the full-corpus-fitted lifetime endpoint te, so for each paper FICE encodes entity success after publication. This makes the claimed correlation with C5 partially circular: the predictor already contains future information that is correlated with the outcome. The self-citations [10,22] are used only for annotation guidelines and are not load-bearing, and no uniqueness theorem or ansatz is smuggled via citation. Because the headline result does not reduce wholly to a citation fit, the circularity is partial, warranting a 6 rather than a higher score.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central metric is constructed from per-entity Gaussian fits and a calibrated disambiguation threshold; each fit introduces degrees of freedom that are not fixed by theory. The correlation with citations is then observed on the same corpus, so these fitting choices are part of the claim's support rather than external anchors.

free parameters (4)
  • Gaussian profile parameters (amplitude, mean, dispersion) for each entity document-frequency curve = Per entity; values not reported
    Section 4.4 uses these fitted profiles to extrapolate df beyond the observable period and to determine te for the lifetime ratio.
  • Number of Gaussian peaks per entity = Inferred automatically from scipy find_peaks
    No theoretical basis; chosen from data, affecting the shape of the extrapolated lifetime.
  • Entity disambiguation similarity threshold = 0.5
    Section 4.3 calibrates the Cross Encoder threshold on 180 manually labeled pairs; this threshold decides which entities are conflated.
  • Regularization coefficient in curve-fitting loss = Not reported
    Section 4.4 adds a penalty against excessively large or narrow Gaussian peaks; the coefficient is a tunable hyperparameter.
assumptions (6)
  • domain assumption All scientific entities have a finite lifetime with a first appearance ts and a last appearance te
    Stated in Section 3.2 and used to define the lifetime ratio; a term that slowly decays but never exactly reaches zero is excluded.
  • ad hoc to paper The time-dependent document frequency of an entity can be modeled as a composite of Gaussian profiles and extrapolated beyond the observed corpus
    Section 4.4 assumes this model to predict te and total lifetime df; no evidence is given that real term lifecycles are Gaussian.
  • domain assumption GPT-4 zero-shot extraction with F1=0.66 is a reliable entity extractor for this task
    Section 4.2 adopts GPT-4 based on a 200-title benchmark; errors in entity boundaries propagate into frequencies and FICE.
  • domain assumption Cross Encoder similarity with a threshold classifies semantically similar entities correctly
    Section 4.3 relies on 180 labeled pairs and a 0.5 threshold without reporting precision/recall.
  • domain assumption Semantic Scholar citation counts are complete and reliable for computing 5-year cumulative citations
    Section 4.6 obtains citation counts from the Semantic Scholar Graph API; coverage gaps would bias C5.
  • domain assumption Citations received in the fixed window 2015-2019 are a meaningful impact measure for papers from all publication years
    Section 4.6 defines C5(2015) with the same window for every paper; papers published in different eras are pooled in the correlation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Freshness and Informativity Weighted Cognitive Extent and Its Correlation with Cumulative Citation Count." pith.science (2026). https://pith.science/paper/ULJHWHED

@misc{pith2026241203557,
  author       = {Pith},
  title        = {Pith review of: Freshness and Informativity Weighted Cognitive Extent and Its Correlation with Cumulative Citation Count},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULJHWHED}},
  note         = {Machine review of arXiv:2412.03557}
}
abstract

In this paper, we revisit cognitive extent, originally defined as the number of unique phrases in a quota. We introduce Freshness and Informative Weighted Cognitive Extent (FICE), calculated based on two novel weighting factors, the lifetime ratio and informativity of scientific entities. We model the lifetime of each scientific entity as the time-dependent document frequency, which is fit by the composition of multiple Gaussian profiles. The lifetime ratio is then calculated as the cumulative document frequency at the publication time $t_0$ divided by the cumulative document frequency over its entire lifetime. The informativity is calculated by normalizing the document frequency across all scientific entities recognized in a title. Using the ACL Anthology, we verified the trend formerly observed in several other domains that the number of unique scientific entities per quota increased gradually at a slower rate. We found that FICE exhibits a strong correlation with the average cumulative citation count within a quota. Our code is available at \href{https://github.com/ZiheHerzWang/Freshness-and-Informativity-Weighted-Cognitive-Extent}{https://github.com/ZiheHerzWang/Freshness-and-Informativity-Weighted-Cognitive-Extent}

Figures

Figures reproduced from arXiv: 2412.03557 by the authors.

Figure 1
Figure 1. The number of papers, scientific entities (undisam￾biguated), and disambiguated entities in the ACL Corpus [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The document frequency chart (blue) of an entity named machine learning and a fitting with 4 Gaussian profiles. To compare the performance of these models, we built a small benchmark dataset by manually annotating 200 ti￾tles randomly selected from all ACL Anthology papers, fol￾lowing the annotation guidelines in Wu et al. [22]. The F1-scores achieved by GPT-4, SciBERT, and SpaCy are 0.66, 0.05, and 0.07, respective… view at source ↗
Figure 4
Figure 4. Average FICE calculated using undisambiguated enti￾ties per quota vs. the log 𝐶5. Paper titles are grouped into a bin size of 250. demonstrate the contribution of the lifetime ratio and infor￾mativity weight in the correlation above. These simplified versions are (1) Dichotomous Entity-based Cognitive Extent, calculated by adding the number of disambiguated unique scientific entities in a quota. (2) Weight Only, cal… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 16 canonical work pages

  1. [1]

    Milojević, Quantifying the cognitive extent of sci- ence, Journal of Informetrics 9 (2015) 962–973

    S. Milojević, Quantifying the cognitive extent of sci- ence, Journal of Informetrics 9 (2015) 962–973

    work page 2015

  2. [2]

    Bérubé, M

    N. Bérubé, M. Sainte-Marie, P. Mongeon, V. Lariv- ière, Words by the tail: Assessing lexical diver- sity in scholarly titles using frequency-rank distri- bution tail fits, PLOS ONE 13 (2018) 1–31. URL: https://doi.org/10.1371/journal.pone.0197775. doi:10. 1371/journal.pone.0197775

    work page doi:10.1371/journal.pone.0197775 2018

  3. [3]

    Jarvis, M

    S. Jarvis, M. Daller, Defining and measuring lexical diversity, Vocabulary knowledge: Human ratings and automated measures. Amsterdam, The Netherlands (2013)

    work page 2013

  4. [4]

    Malvern, B

    D. Malvern, B. Richards, N. Chipere, P. Durán, Lexical diversity and language development, Springer, 2004

    work page 2004

  5. [5]

    S. Milojević, The length and semantic struc- ture of article titles—evolving disciplinary prac- tices and correlations with impact, Frontiers in Research Metrics and Analytics 2 (2017). URL: https://www.frontiersin.org/journals/ research-metrics-and-analytics/articles/10.3389/ frma.2017.00002. doi:10.3389/frma.2017.00002

    arXiv 2017

  6. [6]

    Eykens, R

    J. Eykens, R. Guns, T. C. Engels, F. Vandermo- ere, Cognitive and interdisciplinary mobility in the social sciences and humanities: Traces of in- creased boundary crossing, Journal of Informa- tion Science (2023) 01655515231171086. URL: https: //doi.org/10.1177/01655515231171086. doi:10.1177/ 01655515231171086

    work page doi:10.1177/01655515231171086 2023

  7. [7]

    Waltman, A review of the literature on cita- tion impact indicators, Journal of Informetrics 10 (2016) 365–391

    L. Waltman, A review of the literature on cita- tion impact indicators, Journal of Informetrics 10 (2016) 365–391. URL: https://www.sciencedirect.com/ science/article/pii/S1751157715300900. doi:https:// doi.org/10.1016/j.joi.2016.02.007

    doi:10.1016/j.joi.2016.02.007 2016

  8. [8]

    Albarrán, J

    P. Albarrán, J. A. Crespo, I. Ortuño, J. Ruiz-Castillo, The skewness of science in 219 sub-fields and a number of aggregates, Scientometrics 88 (2011) 385–397

    work page 2011

Show all 24 references
  1. [9]

    Leydesdorff, L

    L. Leydesdorff, L. Bornmann, J. A. Comins, S. Milo- jević, Citations: Indicators of quality? the impact fallacy, Frontiers in Research Metrics and Analytics 1 (2016). URL: https://www.frontiersin.org/journals/ research-metrics-and-analytics/articles/10.3389/ frma.2016.00001. d...

    2016

  2. [10]

    J. Wu, M. R. Ul Hoque, G. W. Reiske, M. C. Weigle, B. T. Bradshaw, H. D. Gaff, J. Li, C. Kwan, A compar- ative study of sequence tagging methods for domain knowledge entity recognition in biomedical papers, in: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries ...

    2020

  3. [11]

    Z. Hong, R. Tchoua, K. Chard, I. Foster, Sciner: Ex- tracting named entities from scientific literature, in: V. V. Krzhizhanovskaya, G. Závodszky, M. H. Lees, J. J. Dongarra, P. M. A. Sloot, S. Brissos, J. Teixeira (Eds.), Computational Science – ICCS 2020, Springer Internatio...

    2020

  4. [12]

    J. Li, A. Sun, J. Han, C. Li, A survey on deep learning for named entity recognition, IEEE transactions on knowledge and data engineering 34 (2020) 50–70

    2020

  5. [13]

    S. Wang, X. Sun, X. Li, R. Ouyang, F. Wu, T. Zhang, J. Li, G. Wang, Gpt-ner: Named entity recognition via large language models, arXiv preprint arXiv:2304.10428 (2023)

    2023 arXiv

  6. [14]

    Sousa, N

    H. Sousa, N. Guimarães, A. Jorge, R. Campos, Gpt struct me: Probing gpt models on narrative entity extraction, in: 2023 IEEE/WIC International Confer- ence on Web Intelligence and Intelligent Agent Tech- nology (WI-IAT), 2023, pp. 383–387. doi: 10.1109/ WI-IAT59888.2023.00063

    2023

  7. [15]

    Y. Li, J. Li, Y. Suhara, A. Doan, W.-C. Tan, Deep entity matching with pre-trained language models, arXiv preprint arXiv:2004.00584 (2020)

    2020 arXiv

  8. [16]

    De Beaugrande, W

    R.-A. De Beaugrande, W. U. Dressler, Introduction to text linguistics, volume 1, longman London, 1981

    1981

  9. [17]

    J. J. Tocatlian, Are titles of chemical papers becoming more informative?, Journal of the American Society for Information Science 21 (1970) 345–350

    1970

  10. [18]

    D. R. Radev, P. Muthukrishnan, V. Qazvinian, A. Abu- Jbara, The acl anthology network corpus, Language Resources and Evaluation 47 (2013) 919–944

    2013

  11. [19]

    T. B. Brown, Language models are few-shot learners, arXiv preprint arXiv:2005.14165 (2020)

    2020 arXiv

  12. [20]

    Beltagy, K

    I. Beltagy, K. Lo, A. Cohan, SciBERT: A pretrained language model for scientific text, in: Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Associa...

    2019

  13. [21]

    Matthew Honnibal, spaCy’s NER Model, 2024

    2024

  14. [22]

    J. Wu, M. R. Ul Hoque, G. W. Reiske, M. C. Weigle, B. T. Bradshaw, H. D. Gaff, J. Li, C. Kwan, A comparative study of sequence tagging methods for domain knowl- edge entity recognition in biomedical papers, in: Pro- ceedings of the ACM/IEEE Joint Conference on Digital Librarie...

    2020

  15. [23]

    D. P. Kingma, J. Ba, Adam: A method for stochas- tic optimization, in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL: http: //arxiv.org/abs/1412.6980

    2015 arXiv

  16. [24]

    semanticscholar.org/product/api, 2024

    AllenAI, Semantic Scholar API, https://www. semanticscholar.org/product/api, 2024

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.