Pith.

Digital Libraries · arXiv:2611.17233 v1 · submitted 12 Nov 2026 · 5 references

Assessing the Reliability of Longitudinal Bibliometric Rankings under Incomplete Citation Histories

João Paulo Papa

The paper says

“Longitudinal bibliometric analyses rely on annual citation histories, yet missing annual observations are frequently difficult to distinguish from true zero-citation years.”

more of the abstract

We present a reproducible case-study framework for auditing citation-history gaps, tracing retrieval provenance, and quantifying the sensitivity of longitudinal indicators and rankings to explicit missing-data assumptions. The case study comprises 595 consolidated publications derived from 616 Google Scholar article records, and examines the completed calendar years 2024-2025. Three scenarios are compared: strictly observed annual counts (A), hypothetical zeros for internal gaps only (B), and hypothetical zeros for all missing years within a valid publication interval (C). The number of publications with computable growth and acceleration indicators increases from 196 (A) to 257 (B) and 490 (C); the corresponding counts for a revival indicator are 131, 152, and 177. Although the ordering of publications eligible under both compared scenarios is unchanged (Spearman and Kendall correlations equal to 1.0), complete-ranking positions and Top-K membership may change markedly. In particular, acceleration exhibits a Jaccard overlap of 0.563 at K=50 between A and C, while revival exhibits 0.667 at K=10. These findings demonstrate that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes. The zero assignments are hypothetical sensitivity assumptions, not evidence of provider semantics. The results are descriptive and based on a single researcher profile. A reusable Google Scholar Tracker software workflow supports the audit and sensitivity analysis; version 1.8.5 is publicly available on GitHub and archived on Zenodo (DOI: 10.5281/zenodo.24170285).

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: agrees with the reader. Referee: major revision.

A careful, well-scoped case study that cleanly demonstrates a specific failure mode in longitudinal bibliometric rankings; the central claim holds, the limitations are transparent, and the paper deserves peer review. This line is wrong

3 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 5 references say.

The paper says, at §3.5, Eq. (7)–(9); §5.2, Table 1

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The Revival indicator definition is incompletely specified. Section 3.5 states that 'the implementation's convention for a zero preceding mean, percentage scaling, and any boundary cases must be transcribed from the tested source code before submission; the current experiment report does not specify those details.'

more of objection 1

This is a load-bearing gap because Revival shows different sensitivity patterns (e.g., Jaccard@10 = 0.667) and the reader cannot assess whether the results are robust to reasonable alternative definitions of the indicator. The paper should either provide the full formula in the text or include a clear reference to the exact code location and version that defines it. Without this, the Revival results are not independently verifiable from the manuscript alone.

The paper says, at §5.5, Table 3; §6.3

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.82):

“A deterministic secondary 15 ordering makes the exported Top-K lists reproducible, but it does not eliminate the analytical arbitrariness of selecting only part of a tied group at a cutoff.”

Pith says: Objection 2 of 3 · major

The paper acknowledges extensive ties (94.4% of Acceleration publications under A) and that Top-K boundaries sometimes cut through tie groups, making membership partly dependent on the deterministic secondary ordering. However, the main ranking results (Tables 2 and 3) are reported without any sensitivity analysis over tie-breaking rules.

more of objection 2

Given that the central claim concerns ranking stability, and ties are pervasive, the paper should at least report the range of possible Jaccard overlaps under alternative admissible tie resolutions (e.g., best-case/worst-case for membership). The current presentation may overstate the precision of the overlap estimates.

The paper says, at §7.4; §3.4

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.67):

“Since complete historical provider responses were not systematically archived, the present dataset cannot establish whether omitted rows correspond to zero suppression, incomplete provider coverage, or other causes.”

Pith says: Objection 3 of 3 · major

The three scenarios (A, B, C) are nested and cover a plausible range, but the paper does not discuss a scenario where missingness is correlated with citation volume (e.g., missing years are more likely for low-citation publications). The authors acknowledge this limitation in §7.4 ('missingness may be associated with publication age, citation volume, or provider coverage'), but they do not test any alternative scenario that addresses this.

more of objection 3

While I agree that a full missing-data mechanism analysis is beyond the scope, the paper could strengthen its sensitivity analysis by including a scenario where missing years are filled with a small positive constant (e.g., 1) rather than zero, to test whether the results are driven by the zero assumption specifically. This would not require imputing ground truth, but would test the robustness of the ranking changes to the magnitude of the hypothetical fill value.

6 minor objections

The paper says, at §1, abstract

Pith says: minor objection 1 of 6

The abstract states 'fourth' contribution after 'third' (line 'Fourth, we document a reusable Google Scholar Tracker software workflow...'). The numbering should be consistent: either list four contributions or rephrase to avoid the mismatch.

The paper says, at §3.5, Eq. (8)

Pith says: minor objection 2 of 6

The Growth indicator is defined as the sum of citations in 2024 and 2025. This is a recent citation volume, not a growth rate. The paper should clarify this earlier to avoid confusion with percentage growth measures, especially since the Revival indicator uses percentage growth.

show 4 more

The paper says, at §5.3, Table 2

Pith says: minor objection 3 of 6

The Jaccard overlap values for Acceleration A–B at K=50 is reported as 0.667, but the text in §5.3 says 'A–B acceleration Top-50 comparison has J@50 = 0.667'. However, Table 2 shows Acceleration A–B J@50 = 0.667, which is consistent. But the table also shows Acceleration A–C J@50 = 0.563, while the text says '0.5625' (more precise). Please ensure consistency in decimal places throughout.

The paper says, at §6.5

Pith says: minor objection 4 of 6

The paper refers to 'Google Scholar Tracker skill' but earlier it is called 'Google Scholar Tracker software workflow' or 'Google Scholar Tracker framework'. Consider using a consistent term throughout.

The paper says, at §7.7

Pith says: minor objection 5 of 6

The reproducibility statement mentions that exact reproduction requires frozen study data and configuration, which are not public. This is a significant limitation for a paper that emphasizes reproducibility. While I understand data privacy constraints, the paper could provide a synthetic dataset that mimics the structure of the real data to allow code testing.

The paper says, at §4.3

Pith says: minor objection 6 of 6

The audit reports '38 possible inconsistency checks' but it is unclear what constitutes an inconsistency check. Please define this term or provide an example.

What would settle it

As a verification step, independently re-implement the ranking-robustness analysis using the same frozen dataset and scenario definitions, and confirm that the reported Jaccard overlaps and displacement statistics are reproducible. This would test the reproducibility of the computational workflow, which the authors have made publicly available (v1.8.5 on GitHub and Zenodo).

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 7 s · 17,125 tokens read, 712 written
  3. Editor wrote the verdict above 6 s · 17,937 tokens read, 437 written
  4. Referee major revision: 3 major objections, 6 minor 14 s · 17,421 tokens read, 1,471 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The paper makes a modest but clean point: perfect rank correlations on a common eligible subset do not guarantee stable rankings when eligibility changes. The demonstration is methodologically careful—they separate observed zeros from missing observations, use three nested sensitivity scenarios without persisting hypothetical values, and report eligibility counts, Top-K overlap, common-universe correlations, and full-ranking displacement as distinct diagnostics. The software workflow is public and versioned. The authors are transparent about the single-profile scope, the hypothetical nature of the zero assignments, and the inability to determine the true missingness mechanism. The central claim is well supported by the reported results: for example, acceleration shows a Jaccard overlap of 0.563 at K=50 between the strictest and most permissive scenarios, despite perfect Spearman and Kendall correlations on the shared universe.

The soft spots are real but acknowledged. The three scenarios do not exhaust the space of plausible missing-data mechanisms, and the paper says so (Section 7.4). The private dataset is not publicly available, so exact numerical replication is not possible from the code alone—again, stated. The tie-breaking policy affects Top-K membership, and the paper flags this. These are not hidden flaws; they are stated limitations. The weakest load-bearing assumption is that the three scenarios span a useful range of missing-data mechanisms, but the paper does not overclaim—it presents the results as descriptive sensitivity analyses, not as estimates of ground truth.

Who this is for: bibliometricians, research evaluators, and anyone building or using longitudinal citation rankings. The paper provides a reproducible framework and a concrete warning about interpreting rank correlations. It deserves a serious referee. I would accept the peer review assignment.

The strongest objection

The paper is a carefully scoped, methodologically transparent case study that makes a modest but well-supported claim about the sensitivity of longitudinal bibliometric rankings to missing-data assumptions. The authors explicitly acknowledge all key limitations: the single-profile scope, the hypothetical nature of the zero-assignment scenarios, the inability to determine the true missingness mechanism, the dependence on tie-breaking conventions, and the lack of public access to the private dataset. The central argument—that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes—is convincingly demonstrated through the reported results. The three scenarios (A, B, C) are clearly defined, nested, and used only as sensitivity analyses, not as imputations of ground truth. The distinction between eligibility stability, score stability, and ranking-membership stability is well articulated and operationalized through separate diagnostics (eligibility counts, common-universe correlations, Top-K overlap, full-ranking displacement). The paper does not overclaim; it explicitly states that results are descriptive and should not be generalized. No internal inconsistency or hidden assumption undermines the central claim. The weakest assumption identified by the reader—that the three scenarios span a useful range of plausible missing-data mechanisms—is acknowledged by the authors as a limitation (Section 7.4: 'The three scenarios span explicit policies for missing annual observations but do not exhaust the space of plausible missing-data mechanisms'). This is a structural limitation of any sensitivity analysis, not a flaw in the argument. The paper's conclusions are appropriately conditional on this limitation.

The reader's verdict is ACCEPT with high confidence, and the weakest assumption identified is that the three missing-data scenarios (A, B, C) are meaningful sensitivity analyses. The paper acknowledges this limitation explicitly in Section 7.4, stating that the scenarios 'do not exhaust the space of plausible missing-data mechanisms.' This is a structural limitation of any sensitivity analysis, not a flaw that undermines the central claim. The paper's central claim—that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes—is convincingly demonstrated within the scope of the three scenarios. The authors are transparent about all key limitations: single profile, hypothetical assumptions, inability to determine true missingness, tie-breaking dependence, and private dataset. The paper does not overclaim; it explicitly states that results are descriptive and should not be generalized. No internal inconsistency or hidden assumption undermines the argument. The reader's weakest assumption is correctly identified, but it does not constitute a load-bearing concern that would change the verdict. The paper's conclusions are appropriately conditional on this limitation. Therefore, the verdict should remain UNCHANGED (ACCEPT). Agreement with the reader is 'agree' because the reader correctly identified the weakest assumption, and the stress-test confirms that this assumption is acknowledged and does not invalidate the central claim.

The referee's report

If the central claim holds—that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes—this is a methodologically important demonstration for the bibliometrics community. The paper provides a clear, reproducible framework for separating observation provenance, indicator computability, and ranking robustness. It explicitly acknowledges all key limitations (single profile, hypothetical scenarios, no public dataset). The contribution is not a new indicator or a large-scale empirical finding, but a carefully scoped case study that operationalizes a distinction (eligibility stability vs. score stability vs. ranking-membership stability) that is often conflated in practice. The reusable software workflow (Google Scholar Tracker) is a strength, though exact reproduction requires private data. The paper is well-positioned within the literature on responsible bibliometrics (Leiden Manifesto, Waltman review).

Pith's reading of the paper (a machine's summary, not the authors')

This paper demonstrates a methodological trap in longitudinal bibliometric rankings: when missing citation years are handled differently, the set of publications eligible for ranking changes, yet the relative ordering of publications that remain eligible in both scenarios can stay perfectly correlated (Spearman and Kendall equal to 1.0). The paper shows that this perfect correlation does not guarantee stable rankings overall, because newly eligible publications can enter the ranking and displace previously ranked ones. Using a case study of 595 consolidated publications from a single Google Scholar profile, the author compares three scenarios for treating missing annual citation counts: using only observed counts, filling internal gaps with hypothetical zeros, and filling all missing years within a valid publication interval with zeros. While the common-subset correlations are perfect, Top-K membership changes markedly for some indicators—acceleration exhibits a Jaccard overlap of only 0.563 at K=50 between the strictest and most permissive scenarios. The paper argues that eligibility, score stability, relative ordering, and ranking membership are four separate dimensions that must be reported independently.

What carries the argument. The central machinery is a provenance-aware, read-only sensitivity framework that separates observed annual citation counts from absent records using an observation indicator $o_{i,t}$ and three nested missing-data scenarios ($A$, $B$, $C$) defined by availability predicates $a^{(s)}_{i,t}$. For each scenario, the framework computes indicator-specific eligibility $e^{(s)}_{i,m}$ and scores, then compares rankings using both common-universe correlations (Spearman's $\rho$, Kendall's $\tau_b$) and complete-ranking diagnostics (Jaccard overlap of Top-K sets, full-ranking displacement $\Delta r_{m,i}$). The key identity is that $\rho = \tau_b = 1.0$ on the common eligible set is compatible with large displacements and Top-K turnover when newly eligible publications enter the ranking.

Core claim. The paper establishes that unchanged indicator values and perfect rank correlations on a common subset of publications do not guarantee stable rankings when the eligible population changes. Specifically, under three different missing-data scenarios (observed-only, internal zeros, all valid zeros), the relative ordering of publications eligible under both scenarios remains identical (Spearman and Kendall correlations equal to 1.0), yet complete-ranking positions and Top-K membership can change markedly. For example, the acceleration indicator shows a Jaccard overlap of only 0.563 at K=50 between the conservative and most permissive scenarios, and a mean absolute full-ranking displacement of 172 positions. The paper demonstrates that ranking robustness must be assessed along four separate dimensions: coverage (which publications can be scored), score stability, relative-order stability within the common eligible set, and composition/positional stability in complete rankings.

The premise it rests on. The three missing-data scenarios (observed-only, internal zeros, all valid zeros) are assumed to span a useful range of plausible missing-data mechanisms, but the paper acknowledges that the true missingness mechanism could be different (e.g., correlated with citation volume) and is not identified by the available data.

If the paper is right (Pith's inferences, not the authors')

Longitudinal bibliometric rankings should report eligibility denominators alongside indicator values, so readers can see how many publications were excluded due to missing data.

A perfect rank correlation on a common subset should not be taken as evidence of overall ranking robustness; complete-ranking displacement and Top-K overlap must be reported separately.

The three missing-data scenarios provide a template for sensitivity analysis that other researchers can apply to their own citation datasets, making assumptions about zero-filling explicit and auditable.

Tie-breaking policies matter: with extensive ties (e.g., 94% of acceleration publications tied under scenario A), Top-K membership can be arbitrary at boundaries, so tie-inclusive or tie-averaged sensitivity analyses are needed before drawing substantive conclusions.

The framework's separation of observation provenance, indicator computability, and ranking robustness could inform a reporting standard for longitudinal bibliometrics, similar to the Leiden Manifesto's transparency principles.

The paper's findings suggest that any bibliometric ranking based on annual citation differences is inherently fragile when missing data are handled differently, because the set of eligible publications can shift dramatically. This fragility is not captured by standard correlation metrics and could mislead research evaluation exercises that rely on longitudinal indicators.

The methodology could be extended to other citation databases (Scopus, Web of Science) and to other longitudinal indicators (e.g., percentile-based trajectories, field-normalized growth rates). A multi-provider replication would reveal whether the observed patterns are specific to Google Scholar's coverage semantics or general across sources.

The paper implicitly challenges the practice of imputing zeros for missing citation years without provenance information. If adopted as a reporting standard, the framework would require data collectors to retain raw provider responses and timestamps, which many current workflows do not do.

The extreme displacement observed for acceleration (mean 172 positions) suggests that year-to-year differences are particularly sensitive to eligibility changes, because adding many zero-filled publications creates a dense cluster of low or zero scores that pushes previously mid-ranked publications far down. This mechanism could be tested with synthetic data.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Longitudinal bibliometric analyses rely on annual citation histories, yet missing annual observations are frequently difficult to distinguish from true zero-citation years. We present a reproducible case-study framework for auditing citation-history gaps, tracing retrieval provenance, and quantifying the sensitivity of longitudinal indicators and rankings to explicit missing-data assumptions. The case study comprises 595 consolidated publications derived from 616 Google Scholar article records, and examines the completed calendar years 2024-2025. Three scenarios are compared: strictly observed annual counts (A), hypothetical zeros for internal gaps only (B), and hypothetical zeros for all missing years within a valid publication interval (C). The number of publications with computable growth and acceleration indicators increases from 196 (A) to 257 (B) and 490 (C); the corresponding counts for a revival indicator are 131, 152, and 177. Although the ordering of publications eligible under both compared scenarios is unchanged (Spearman and Kendall correlations equal to 1.0), complete-ranking positions and Top-K membership may change markedly. In particular, acceleration exhibits a Jaccard overlap of 0.563 at K=50 between A and C, while revival exhibits 0.667 at K=10. These findings demonstrate that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes. The zero assignments are hypothetical sensitivity assumptions, not evidence of provider semantics. The results are descriptive and based on a single researcher profile. A reusable Google Scholar Tracker software workflow supports the audit and sensitivity analysis; version 1.8.5 is publicly available on GitHub and archived on Zenodo (DOI: 10.5281/zenodo.24170285).

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-14

The authors' main claim, as the classifier picked it out 1.00

“These findings demonstrate that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.98 · human participants 0.07 · controlled comparison 0.44 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.55 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Longitudinal bibliometric analyses rely on annual citation histories, yet missing annual observations are frequently difficult to distinguish from true zero-citation years. background 0.96 0.07 0.04 asserted 0.92
We present a reproducible case-study framework for auditing citation-history gaps, tracing retrieval provenance, and quantifying the sensitivity of longitudinal indicators and rankings to explicit missing-data assumptions. contribution 0.98 0.92 0.07 asserted 0.99
The case study comprises 595 consolidated publications derived from 616 Google Scholar article records, and examines the completed calendar years 2024-2025. method 0.97 0.91 0.02 asserted 1.00
Three scenarios are compared: strictly observed annual counts (A), hypothetical zeros for internal gaps only (B), and hypothetical zeros for all missing years within a valid publication interval (C). method 0.97 0.81 0.03 asserted 0.97
The number of publications with computable growth and acceleration indicators increases from 196 (A) to 257 (B) and 490 (C); the corresponding counts for a revival indicator are 131, 152, and 177. result 1.00 0.96 0.06 asserted 1.00
Although the ordering of publications eligible under both compared scenarios is unchanged (Spearman and Kendall correlations equal to 1.0), complete-ranking positions and Top-K membership may change markedly. result 0.86 0.94 0.24 hedged possibility 0.92
In particular, acceleration exhibits a Jaccard overlap of 0.563 at K=50 between A and C, while revival exhibits 0.667 at K=10. result 1.00 0.96 0.08 asserted 1.00
These findings demonstrate that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes. interpretation 1.00 0.89 0.81 asserted 1.00
The zero assignments are hypothetical sensitivity assumptions, not evidence of provider semantics. limitation 0.90 0.42 0.05 asserted 1.00
The results are descriptive and based on a single researcher profile. limitation 1.00 0.68 0.03 asserted 1.00
A reusable Google Scholar Tracker software workflow supports the audit and sensitivity analysis; version 1.8.5 is publicly available on GitHub and archived on Zenodo (DOI: 10.5281/zenodo.24170285). contribution 0.54 0.82 0.02 asserted 1.00

What kind of paper the classifier reports this is: method or tool 0.79 · evidence from analysis of existing data 0.93

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-14: 1 request, 8,428 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (2)
Figure 1 of the paper
Figure 1. Number of publications with computable longitudinal indicators under the three hypo
Figure 2 of the paper
Figure 2. Jaccard overlap for the Top-10, Top-20, and Top-50 sets under the strictly observed

Figures are the authors', reproduced from the paper.

References (5)
  1. [1] Papa, J. P. (2026).Google Scholar Tracker(Version 1.8.5) [Computer software]. Zenodo. doi:10.5281/zenodo.24170285 on Pithdoi:10.5281/zenodo.24170285
  2. [2] Harzing, A.-W. and van der Wal, R. (2008). Google Scholar as a new source for citation analysis. Ethics in Science and Environmental Politics, 8(1), 61–73. doi:10.3354/esep00076 on Pithdoi:10.3354/esep00076
  3. [3] Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., and Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics.Nature, 520, 429–431. doi:10.1038/520429a. Martín-Martín, A., Orduna-Malea, E., Thelwall, M., and Delgado López-Cózar, E. (2018). Google on Pithdoi:10.1038/520429a
  4. [4] Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories.Journal of Informetrics, 12(4), 1160–1177. doi:10.1016/j.joi.2018.09.002 on Pithdoi:10.1016/j.joi.2018.09.002
  5. [5] Waltman, L. (2016). A review of the literature on citation impact indicators.Journal of Informetrics, 10(2), 365–391. doi:10.1016/j.joi.2016.02.007. 20 on Pithdoi:10.1016/j.joi.2016.02.007

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment