REVIEW 1 major objections 2 minor 13 references
Excessive use, ill use and misuse of Bibliometrics
T0 review · 1 major / 2 minor · reviewed 2026-06-28 · grok-4.3
Pith's one-line read Bibliometric indices like impact factor and h-index lack any statistical basis yet are used to rank institutions, hire faculty, allocate funds, and select awards.
desk verdict This is a short compilation of existing documents criticizing heavy use of citation metrics, with no new analysis or data of its own. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Compilation of expert statements and policy documents from multiple disciplines that recommend limiting the role of bibliometric indices in favor of content-based expert review.
What would settle it
A controlled comparison of research outcomes, career trajectories, or institutional performance in settings that rely primarily on bibliometric indices versus settings that rely primarily on expert content review.
Extended reading notes
Core claim
Impact factor, H-index, citation index, and similar measures play an increasing role in scientific assessment of institutions, researchers, and funding decisions across the globe. These indices have no statistical basis, yet they determine rankings of institutions and departments, hiring and promotion of faculty, and selection for awards. Experts across disciplines have documented that the indices should have only a marginal role, if any, with judgments grounded in critical assessment of content by experts instead; this article compiles such documents to urge decision makers to ignore or minimize reliance on the indices.
Load-bearing premise
The cited expert documents across disciplines accurately reflect the right policy response and that their recommendations can be acted on without new evidence comparing outcomes under index-heavy versus content-only assessment systems.
Editorial extensions
If this is right
- Institutional and departmental rankings should shift primary weight to expert content assessment rather than index values.
- Hiring, promotion, and tenure decisions should rest on critical reading of work instead of citation counts or journal impact factors.
- Allocation of research funds and selection for awards should treat bibliometric indices as secondary or irrelevant inputs.
- Continued growth in index dependence will occur unless decision makers explicitly adopt the minimal-use stance urged by the cited experts.
Reading between the lines
- Widespread index use may create incentives for researchers to prioritize easily cited topics over longer-term or less visible contributions.
- If the expert consensus holds, training programs for evaluators would need to emphasize skills in direct content assessment rather than metric interpretation.
- Departments or funders that reduce index weight could test whether the change alters the distribution of supported projects or recognized researchers.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript claims that bibliometric indices including impact factor, h-index, and citation index lack any statistical basis yet are increasingly used for institutional rankings, departmental assessments, faculty hiring/promotions, and awards. It positions itself as a compilation of expert documents from multiple disciplines that recommend assigning these indices only a marginal role (if any) and basing judgments instead on expert content assessment, while observing that dependence on the indices continues to grow.
Significance. If the cited documents accurately represent expert consensus, the paper offers a consolidated reference list of critiques on bibliometric misuse that could serve as a resource for policy discussions in research evaluation. The contribution is documentary rather than analytic: no new statistical tests, field-normalization studies, or outcome comparisons are presented, so significance rests on the breadth of the cited sources rather than original evidence.
major comments (1)
- [Abstract / Introduction] Abstract and opening paragraphs: the claim that 'These indicies do not have any statistical basis' is asserted as established fact but receives no statistical demonstration, derivation, or targeted references to statistical critiques (e.g., distributional assumptions of the h-index or independence conditions for citation counts) within the manuscript. Support is said to derive from the cited expert documents, yet the text neither reproduces nor analyzes any such statistical arguments; this assertion is load-bearing for the paper's framing of misuse.
minor comments (2)
- [Abstract] Multiple spelling/typographical errors: 'indicies' (should be 'indices'), 'disciplies' (should be 'disciplines'), and the apparent fragment 's steadily increasing' (likely 'is steadily increasing').
- The manuscript would benefit from an explicit methods or selection section describing how the cited documents were identified and whether the compilation aims to be exhaustive or representative.
Simulated Author's Rebuttal
We thank the referee for the detailed review and for highlighting the need to better connect our central claim to its supporting literature. The manuscript is a documentary compilation of expert statements across disciplines rather than an original statistical study, but we agree the introduction can be strengthened to make the cited statistical critiques more explicit.
read point-by-point responses
-
Referee: [Abstract / Introduction] Abstract and opening paragraphs: the claim that 'These indicies do not have any statistical basis' is asserted as established fact but receives no statistical demonstration, derivation, or targeted references to statistical critiques (e.g., distributional assumptions of the h-index or independence conditions for citation counts) within the manuscript. Support is said to derive from the cited expert documents, yet the text neither reproduces nor analyzes any such statistical arguments; this assertion is load-bearing for the paper's framing of misuse.
Authors: The paper's purpose is to assemble and highlight expert documents that already contain statistical critiques of bibliometric indices and recommend limiting their role. The assertion is therefore presented as the consensus view expressed in those sources rather than a novel derivation. We acknowledge that the current text does not excerpt or summarize the specific statistical arguments (e.g., citation count distributions or h-index assumptions) from the cited works. To address this, we will revise the abstract and introduction to include brief, targeted references and short summaries of key statistical points drawn from the referenced documents, thereby making the evidential link more transparent while preserving the manuscript's documentary character. revision: yes
Circularity Check
No circularity; argument rests on external expert citations without internal derivation or self-referential reduction.
full rationale
The manuscript asserts that bibliometric indices lack statistical basis and should play only a marginal role, then positions itself as a compilation of documents from experts across disciplines urging minimal reliance on such indices. No equations, predictions, fitted parameters, or first-principles derivations appear in the provided text. The central claim is not justified via self-citation chains, ansatzes smuggled through prior work, or renaming of known results; it explicitly defers to cited external documents. This structure is self-contained against external benchmarks because the paper's contribution is the act of citation itself rather than any closed logical loop that reduces the conclusion to its own inputs by construction. No load-bearing step meets the criteria for circularity.
Assumptions & free parameters
Cite this review
Pith. "Pith review of Excessive use, ill use and misuse of Bibliometrics." pith.science (2026). https://pith.science/paper/OJFNQNDQ
@misc{pith2026260603117,
author = {Pith},
title = {Pith review of: Excessive use, ill use and misuse of Bibliometrics},
year = {2026},
howpublished = {\url{https://pith.science/paper/OJFNQNDQ}},
note = {Machine review of arXiv:2606.03117}
}
read the original abstract
Impact factor, H-index, citation index, and such other indices have been playing an increasing role in scientific assessment of institutions, researchers, allocation of research funds,... across the globe. These indicies do not have any statistical basis but lots of decisions such as ranking of institutions, ranking of departments, assessment of faculty members for hiring and for promotions as well as selection for various awards are being made by using these indices. Several experts across disciplines have been writing that these indicies should have a marginal role, if any, and judgements should be based on critical assessment of the content by experts. But the dependence on these s steadily increasing. This article cites various such documents being published across the globe and across the disciplies and urging decision makers to ignore or at best give a minimal weight to such indices.
Reference graph
Works this paper leans on
-
[1]
Robert Adler, John Ewing, and Peter Taylor, Citation statistics: a report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS), Statist. Sci. 24 (2009), no. 1, 1–14, DOI 10.1214/09-STS285
-
[2]
www.ascb.org/SFdeclaration.html 2013
2013
-
[3]
https://www.science.org/doi/10.1126/science.1240319
Bruce Alberts, Impact Factor Distortions, Editorial, Science, Vol 340, 2013. https://www.science.org/doi/10.1126/science.1240319
-
[4]
David Magnus Overthrowing the Tyranny of the Journal Impact Factor The Amer ican Journal of Bioethics, 13(7), 1-2, 2013 https://doi.org/10.1080/15265161. 2013.807709
-
[5]
Randy Schekman, How journals like Nature, Cell and Science are damaging science The Guardian 2014 https://www.theguardian.com/commentisfree/2013/dec/ 09/how-journals-nature-science-cell-damage-science
2014
-
[6]
https://doi.org/10.7551/mitpress/10719.001.0001
Yves Gingras Bibliometrics and Research Evaluation: Uses and Abuses 2016. https://doi.org/10.7551/mitpress/10719.001.0001
-
[7]
Stephen Curry, From Declaration to Implementation Nature 554, 147 (2018) https://www.nature.com/articles/d41586-018-01642-w
2018
-
[8]
COPE Discussion Document: Citation Manipulation
COPE Council. COPE Discussion Document: Citation Manipulation. July 2019. https://doi.org/10.24318/cope.2019.3.1
Show all 13 references
-
[9]
Mario Biagioli, Alexandra Lippman (eds.), Gaming the Metrics: Misconduct and Manipulation in Academic Research, The MIT Press, 2020 (OA)
2020
-
[10]
David Pendlebury, Highly Cited Researchers 2023: The evolution of our evaluation and selection policy to support a robust scholarly landscape
2023
-
[11]
https://www.chronicle.com/issue/2024/03-29
Domingo Docampo The Dark World of ‘Citation Cartels’ The Chronicle of Higher Education, Volume 70, 2024. https://www.chronicle.com/issue/2024/03-29
2024
-
[12]
Notices of the AMS, October 2025
Ilka Agricola, Lynn Heller, Wil Schilders, Moritz Schubotz, Peter Taylor, Luis Vega, Fraudulent Publishing in the Mathematical Sciences. Notices of the AMS, October 2025. Digital version with all references: arXiv, math.HO, September 2025, https://arxiv.org/abs/2509.07257
2025
-
[13]
Print version with a selection of most important references: October 2025, Notices of the AMS, https: //www.ams.org/journals/notices/202509/rnoti-p1038.pdf
Ilka Agricola, Lynn Heller, Wil Schilders, Moritz Schubotz, Peter Taylor, Luis Vega, Fraudulent Publishing in the Mathematical Sciences. Print version with a selection of most important references: October 2025, Notices of the AMS, https: //www.ams.org/journals/notices/202509/...
2025
Reviewed June 28, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.