REVIEW 4 major objections 5 minor 8 references
A meta-analysis of impact factors of particle physics journals using NASA/ADS
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read NASA/ADS citation data reproduce Clarivate's impact factors for 16 particle physics journals, with no systematic bias, but single outlier papers dominate the scores of two journals.
desk verdict Readable, transparent extension to particle physics journals; the outlier stories are real, but the no-bias claim needs a statistical test before you trust it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the impact factor ratio itself, computed in two versions. The traditional impact factor for year $n$ is the number of citations in year $n-1$ to papers published in years $n-2$ and $n-3$, divided by the number of refereed papers in those years; the median-based 'new' impact factor replaces the total citation count with the median number of citations per paper. The machinery is the comparison of these two ratios, both built from NASA/ADS citation records for journals tagged 'Physics', against Clarivate's official values, with the outlier papers identified by examining each journal's most-cited papers.
What would settle it
Compare ADS and Web of Science citation counts for each journal at the level of individual papers and check whether the per-journal difference between ADS-based and official impact factors correlates with journal size, age, or uncited-paper fraction; alternatively, recompute PTEP and ARNPS impact factors with their top-cited papers excluded and see whether the gap to the median-based values disappears.
Extended reading notes
Core claim
The central claim is that impact factors computed from NASA/ADS citations match the official Clarivate impact factors for these 16 journals without any systematic offset, while a median-based impact factor reveals that two journals' official scores are inflated by single outlier papers. For PTEP, the Particle Data Group review with 1412 citations over the two-year window raises the official score far above the median-based value; for ARNPS, a review of the Hubble tension and early dark energy with 107 citations does the same. The paper therefore argues that ADS-based impact factors are a usable independent approximation to the Clarivate values, and that conventional impact factors in this field are fragile in the presence of a few extremely cited papers.
Load-bearing premise
The no-bias conclusion assumes that NASA/ADS's 'Physics' tag collects the same population of citable papers and citations for these 16 journals as Clarivate's Web of Science, so that any differences are just random noise.
Editorial extensions
If this is right
- ADS-based impact factors can serve as an independent, freely accessible cross-check on Clarivate's official values for particle and nuclear physics journals.
- For small journals, a single highly cited paper such as the PDG review can change the impact factor by a factor of several, so mean-based rankings should be read with that fragility in mind.
- The median-based impact factor resists this outlier effect and may give a more stable picture of a journal's typical citation rate.
- The absence of systematic bias suggests that, in aggregate, ADS citation coverage for these journals is comparable to Web of Science coverage, at least for the 2022-2024 window studied.
Reading between the lines
- If the no-bias result holds across years, ADS data could be used to track year-to-year impact factor trends before official values are released, provided coverage differences by document type remain stable.
- A direct test of the outlier explanation would be to recompute PTEP and ARNPS impact factors with their top-cited papers removed; the authors' claim predicts the two values would then come close to the median-based estimates.
- Because cosmology and astrophysics papers are often cross-listed under the ADS 'Physics' tag, the comparison could be re-run using only INSPIRE-HEP citations to test whether database choice changes the conclusion for multi-disciplinary journals.
- The acknowledged false positives and incompleteness in ADS citations mean the apparent absence of bias could stem from compensating errors; per-journal citation-level matching against Web of Science would settle that.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper computes 2025 impact factors for 16 particle- and nuclear-physics journals using NASA/ADS citation data, compares them with Clarivate's official impact factors, and also compares them with a median-based impact factor introduced in the authors' earlier work. The central claims are that there is no systematic bias between the ADS-based and official impact factors, and that the largest deviations between the old and new (median-based) impact factors, for PTEP and ARNPS, are driven by single highly cited outlier papers (the PDG review and the Hubble-tension/early-dark-energy review).
Significance. If the central claims hold, the paper usefully demonstrates that ADS citation data can serve as a practical proxy for Clarivate impact factors for this journal set, and it provides a concrete illustration of how traditional impact factors are sensitive to rare high-citation papers. The paper's strengths are the transparent presentation of per-journal data in Tables I and II, the reproducible methodology based on the ADS API, and the use of externally reported Clarivate values as an independent benchmark; no fitted parameters enter the calculation. However, the headline 'no systematic bias' claim currently rests on an informal count of the direction of differences rather than a formal statistical test, and the ADS-versus-Clarivate coverage equivalence is not validated; these issues are load-bearing for the main conclusion.
major comments (4)
- [Section II, first bullet and Table I] The claim that there is no systematic bias because 'about half of the journals have higher ADS-based impact factors' is not a statistical test. With only 16 journals, an approximately even split has very low power to exclude a systematic offset, and Table I shows substantial relative differences for several journals (e.g., ARNPS 8.4 vs 10.97, EPJP 2.9 vs 1.75, NIMPA 1.4 vs 0.98). Please provide a paired significance test (e.g., Wilcoxon signed-rank test on the ratio or difference), confidence intervals for the median difference, or a regression of ADS-based on official impact factors. Without such a test, the central 'no systematic bias' statement is not supported by the evidence presented.
- [Section II, caveat sentence] The paper itself notes that 'NASA/ADS collection of citations is not always complete and is known to contain false positives [5]'. The conclusion that there is no systematic bias between ADS and official impact factors presupposes that, for all 16 journals, the ADS 'Physics' tag contains the same set of citable papers and receives the same citations as Clarivate's Web of Science, up to random noise. This assumption is never tested. If ADS coverage differs systematically by journal age, journal size, or document type, the apparent absence of bias could be an artifact of compensating coverage gaps. Please add a validation step, such as comparing the number of citable papers per journal-year between ADS and Clarivate, or showing that the per-journal discrepancies are uncorrelated with journal size or age.
- [Abstract and Section I (definition of New Impact Factor)] The definition of the New Impact Factor in Section I, as written, cannot reproduce the values in Table I. If one takes 'median number of citations ... divided by the total number of published papers' literally, then for JHEP (4997 papers) a table value of 6 would require a median citation count of about 30,000; for ARNPS (34 papers) a table value of 4.5 would be consistent only if the median citation count were about 153. The intended definition is presumably the median number of citations per paper (the median of the per-paper citation counts). Please correct the definition so that it matches the R25 definition and the numerical values in Table I, since the outlier analysis relies directly on this quantity.
- [Abstract and Section III] The abstract states that 'We find the maximum relative difference between official and median-based impact factors for PTEP and ARNPS' and that the outlier papers 'have significantly elevated the official impact factors above its median value'. This conflates two different comparisons. Table I shows that the large old-vs-new differences (553% for PTEP, 143% for ARNPS) are between the ADS-based old impact factor and the ADS-based new impact factor, not between the Clarivate official value and the median-based value. Moreover, under any standard definition of relative difference, the official-vs-new comparison in Table I would make IJMPA (1.2 vs 0) or EPJP (2.9 vs 1) at least as extreme as PTEP and ARNPS. Please revise the abstract and conclusions to state precisely which quantities are being compared, and either demonstrate the counterfactual for the Clarivate official values or describe the outlier effect as applying to the ADS-based old impact factor.
minor comments (5)
- [General] There are several typographical errors: 'refereed to as' should be 'referred to as' in Section I; 'F raction' in the Table II heading should be 'Fraction'; 'systemic' should be 'systematic' in Section II; and 'astophysics' should be 'astrophysics' in the PTEP bullet point.
- [Table I and Table III] The journal code for Journal of Physics G appears as 'JPhG' in the text and Table III but as 'JPHG' in Table I; please make the acronyms consistent.
- [Section II, journal list] The phrase 'Internal Journal of Modern Physics A' should be 'International Journal of Modern Physics A'.
- [Section I, style] The spelling 'Inspire-hep' should be standardized to 'INSPIRE-HEP' for consistency with the database's official name.
- [Table I header] The header 'Official Impact factor (Old) Impact F actor using ADS' is ambiguous; adding separate column headers or a separator would clarify that the first column is the Clarivate value and the second is the ADS-based old impact factor.
Circularity Check
No significant circularity: the ADS and official impact-factor comparison uses an external Clarivate benchmark, and the median-based statistic is recomputed rather than fitted.
full rationale
The paper's central comparison is between impact factors computed from NASA/ADS citations and the official Clarivate values. These are independent data sources: no parameter is fitted to the Clarivate numbers, and the ADS-based old and new impact factors are computed directly from the stated formulas in Section I. The 'new' median-based impact factor was introduced in the authors' earlier work, but it is restated in this paper and recomputed from current data; this is a self-citation but not a load-bearing circular argument. The claim of no systematic bias is a descriptive comparison of the two columns in Table I, not a prediction derived from the inputs. The attribution of the PTEP and ARNPS old-versus-median differences to the PDG and Hubble-tension outlier papers is an empirical observation about the highest-cited papers in the ADS data; the further inference that those same papers elevated the Clarivate official values is not directly demonstrated, but that gap affects the strength of the evidence, not whether the reasoning is circular. Section II's caveat that ADS citations are 'not always complete and is known to contain false positives [5]' is a data-quality limitation and a reason to treat the no-bias conclusion cautiously, but it does not make the comparison definitionally circular. The absence of formal significance testing is also a robustness concern, not a circularity concern. No step in the derivation chain reduces by construction to its own inputs, and no fitted parameter is renamed as a prediction.
Assumptions & free parameters
assumptions (3)
- domain assumption NASA/ADS tag collection "Physics" identifies all relevant particle physics publications and their citations for the 16 journals.
- domain assumption The official Clarivate impact factors are computed on a comparable citable-item basis to the ADS-based old impact factor.
- standard math Impact factor definitions in Eq. (1) and the median-based new impact factor from R25 are appropriate for this comparison.
Cite this review
Pith. "Pith review of A meta-analysis of impact factors of particle physics journals using NASA/ADS." pith.science (2026). https://pith.science/paper/GZQGHCGJ
@misc{pith2026250714806,
author = {Pith},
title = {Pith review of: A meta-analysis of impact factors of particle physics journals using NASA/ADS},
year = {2026},
howpublished = {\url{https://pith.science/paper/GZQGHCGJ}},
note = {Machine review of arXiv:2507.14806}
}
read the original abstract
In this work, we calculate the 2025 impact factors for 16 journals in Particle Physics (and related areas such as Nuclear Physics) using citations collated by NASA/ADS (Astrophysics Data System). We then compare them to the official impact factors calculated by Clarivate. We also compare these impact factors to the median-based impact factors introduced in a previous work. We do not find any systematic bias between the ADS and official impact factors. We find the maximum relative difference between official and median-based impact factors for PTEP and Annual Review of Nuclear and Particle Science. For PTEP, this difference is due to one outlier, viz Particle Data Group with over 1400 citations in the last two years. Similarly for Annual Review of Nuclear and Particle Science, the difference is due to a review paper on Hubble tension and early dark energy with over 100 citations in the past two years. Both these papers have significantly elevated the official impact factors above its median value.
Reference graph
Works this paper leans on
- [5]
-
[1]
R. V. S. Rithvik and S. Desai, European Physical Journal Plus 140, 439 (2025), 2410.03342
work page Pith review arXiv 2025
- [2]
-
[3]
How stable are the impact factors of major astrophysics journals
R. Venkat Sai Rithvik and S. Desai, arXiv e-prints arXiv:2503.12372 (2025), 2503.12372
work page Pith review arXiv 2025
-
[4]
A. Gentil-Beccot, S. Mele, A. Holtkamp, H. B. O’Connell, and T. C. Brooks, arXiv e-prints arXiv:0804.2701 (2008), 0804.2701
work page Pith review arXiv 2008
-
[6]
R. L. Workman, V. D. Burkert, V. Crede, E. Klempt, U. Thoma, L. Tiator, K. Agashe, G. Aielli, B. C. Allanach, C. Amsler, et al., Progress of Theoretical and Experimental Physics 2022, 083C01 (2022)
work page 2022
- [7]
-
[8]
M. Kamionkowski and A. G. Riess, Annual Review of Nuclear and Particle Science 73, 153 (2023), 2211.04492. Appendix Journal Code Journal Name ARNPS Nuclear and Particle Science EL Europhysics Letters EPJA The European Physical Journal A EPJC The European Physical Journal C EPJP European Physical Journal Plus IJMPA International Journal of Modern Physics A...
arXiv 2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.