{"id":"0e3418c5-10b4-489a-88b6-5c7fd4d5423a","arxiv_id":"2507.14806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ADS-based impact factors for 16 particle physics journals broadly track official Clarivate values, with PTEP and ARNPS inflated by single outlier papers.","lead":"The authors computed 2025 impact factors for 16 particle physics journals from NASA/ADS citation data and compared them with official Clarivate values. They report no systematic bias and show that the largest discrepancies for PTEP and ARNPS come from single highly cited papers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The no-bias claim rests on an untested assumption that ADS and Clarivate count the same citations; with 16 journals and no significance test, the data are too weak to rule out systematic coverage bias.","rationale":"The paper is clearly written, reproducible through the ADS API, and the outlier discussion is plausible and concrete. My stress-test does not reveal an internal arithmetic error in the tables; the problem is the strength of the inference. The no-bias conclusion is a null result from an informal 16-point comparison using a database the authors themselves describe as incomplete and noisy. The reader's conditional verdict is exactly right: before the headline claim is treated as established, the authors should provide a formal test with uncertainties and a coverage check that shows ADS and Clarivate sample the same citation universe. I therefore agree with the reader and recommend keeping the conditional verdict.","tokens_in":4023,"tokens_out":10182,"duration_ms":103697,"concrete_test":"Run a matched coverage audit for PTEP, ARNPS, and EPJP: retrieve all 2024 ADS 'Physics' citations to their 2022-2023 papers, compare against Web of Science or INSPIRE-HEP citation lists for the same papers, and compute per-journal precision and recall of ADS citations. If recall differs materially by journal, the apparent absence of bias is a coverage artifact rather than a property of the citation data. Separately, report a paired Wilcoxon signed-rank test and a 95% bootstrap confidence interval for the 16 official-vs-ADS differences so the null claim has a stated uncertainty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II explicitly concedes that 'NASA/ADS collection of citations is not always complete and is known to contain false positives [5]'. The central no-bias conclusion therefore requires that ADS's 'Physics' tag collection have the same citable-paper population and citation coverage as Clarivate's Web of Science for all 16 journals. This is never tested. The evidence offered is the Table I comparison, but the per-journal discrepancies are substantial in relative terms (ARNPS 8.4 vs 10.97, EPJP 2.9 vs 1.75, PTEP 8.3 vs 6.53), and no uncertainty, confidence interval, or formal test is reported. With 16 journals, a roughly even split between positive and negative differences has low power to exclude a systematic offset; it is compatible with a wide range of nonzero biases. The abstract also attributes the large old-vs-median differences for PTEP and ARNPS to 'official impact factors', but those old-vs-median differences are computed entirely from ADS data; whether the PDG and Hubble-tension papers elevate the Clarivate official values is inferred, not directly demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper computes 2025 impact factors for 16 particle- and nuclear-physics journals using NASA/ADS citation data, compares them with Clarivate's official impact factors, and also compares them with a median-based impact factor introduced in the authors' earlier work. The central claims are that there is no systematic bias between the ADS-based and official impact factors, and that the largest deviations between the old and new (median-based) impact factors, for PTEP and ARNPS, are driven by single highly cited outlier papers (the PDG review and the Hubble-tension/early-dark-energy review).","tokens_in":4226,"tokens_out":5218,"duration_ms":57148,"significance":"If the central claims hold, the paper usefully demonstrates that ADS citation data can serve as a practical proxy for Clarivate impact factors for this journal set, and it provides a concrete illustration of how traditional impact factors are sensitive to rare high-citation papers. The paper's strengths are the transparent presentation of per-journal data in Tables I and II, the reproducible methodology based on the ADS API, and the use of externally reported Clarivate values as an independent benchmark; no fitted parameters enter the calculation. However, the headline 'no systematic bias' claim currently rests on an informal count of the direction of differences rather than a formal statistical test, and the ADS-versus-Clarivate coverage equivalence is not validated; these issues are load-bearing for the main conclusion.","major_comments":[{"comment":"The claim that there is no systematic bias because 'about half of the journals have higher ADS-based impact factors' is not a statistical test. With only 16 journals, an approximately even split has very low power to exclude a systematic offset, and Table I shows substantial relative differences for several journals (e.g., ARNPS 8.4 vs 10.97, EPJP 2.9 vs 1.75, NIMPA 1.4 vs 0.98). Please provide a paired significance test (e.g., Wilcoxon signed-rank test on the ratio or difference), confidence intervals for the median difference, or a regression of ADS-based on official impact factors. Without such a test, the central 'no systematic bias' statement is not supported by the evidence presented.","section":"Section II, first bullet and Table I"},{"comment":"The paper itself notes that 'NASA/ADS collection of citations is not always complete and is known to contain false positives [5]'. The conclusion that there is no systematic bias between ADS and official impact factors presupposes that, for all 16 journals, the ADS 'Physics' tag contains the same set of citable papers and receives the same citations as Clarivate's Web of Science, up to random noise. This assumption is never tested. If ADS coverage differs systematically by journal age, journal size, or document type, the apparent absence of bias could be an artifact of compensating coverage gaps. Please add a validation step, such as comparing the number of citable papers per journal-year between ADS and Clarivate, or showing that the per-journal discrepancies are uncorrelated with journal size or age.","section":"Section II, caveat sentence"},{"comment":"The definition of the New Impact Factor in Section I, as written, cannot reproduce the values in Table I. If one takes 'median number of citations ... divided by the total number of published papers' literally, then for JHEP (4997 papers) a table value of 6 would require a median citation count of about 30,000; for ARNPS (34 papers) a table value of 4.5 would be consistent only if the median citation count were about 153. The intended definition is presumably the median number of citations per paper (the median of the per-paper citation counts). Please correct the definition so that it matches the R25 definition and the numerical values in Table I, since the outlier analysis relies directly on this quantity.","section":"Abstract and Section I (definition of New Impact Factor)"},{"comment":"The abstract states that 'We find the maximum relative difference between official and median-based impact factors for PTEP and ARNPS' and that the outlier papers 'have significantly elevated the official impact factors above its median value'. This conflates two different comparisons. Table I shows that the large old-vs-new differences (553% for PTEP, 143% for ARNPS) are between the ADS-based old impact factor and the ADS-based new impact factor, not between the Clarivate official value and the median-based value. Moreover, under any standard definition of relative difference, the official-vs-new comparison in Table I would make IJMPA (1.2 vs 0) or EPJP (2.9 vs 1) at least as extreme as PTEP and ARNPS. Please revise the abstract and conclusions to state precisely which quantities are being compared, and either demonstrate the counterfactual for the Clarivate official values or describe the outlier effect as applying to the ADS-based old impact factor.","section":"Abstract and Section III"}],"minor_comments":[{"comment":"There are several typographical errors: 'refereed to as' should be 'referred to as' in Section I; 'F raction' in the Table II heading should be 'Fraction'; 'systemic' should be 'systematic' in Section II; and 'astophysics' should be 'astrophysics' in the PTEP bullet point.","section":"General"},{"comment":"The journal code for Journal of Physics G appears as 'JPhG' in the text and Table III but as 'JPHG' in Table I; please make the acronyms consistent.","section":"Table I and Table III"},{"comment":"The phrase 'Internal Journal of Modern Physics A' should be 'International Journal of Modern Physics A'.","section":"Section II, journal list"},{"comment":"The spelling 'Inspire-hep' should be standardized to 'INSPIRE-HEP' for consistency with the database's official name.","section":"Section I, style"},{"comment":"The header 'Official Impact factor (Old) Impact F actor using ADS' is ambiguous; adding separate column headers or a separator would clarify that the first column is the Clarivate value and the second is the ADS-based old impact factor.","section":"Table I header"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the arXiv:2507.14806 piece on particle physics journal impact factors from NASA/ADS. Two things you should know: it extends an earlier ADS-based impact-factor analysis to 16 particle/nuclear journals, and the outlier story—PDG review (1412 citations) for PTEP, Hubble-tension review (107 citations) for ARNPS—is the real substance. That part checks out and is worth reading.\n\nWhat is actually new: the dataset, the comparison to Clarivate for these specific journals, and the demonstration that two single papers drive the gap between mean and median impact factors. The extraction is described, Tables I and II let you reproduce the numbers, and the authors flag the known ADS incompleteness and false positives. That is honest, reproducible work.\n\nThe soft spots are modest but real. First, the 'no systematic bias' claim is supported only by counting that 8 journals have ADS above official and 8 below. With 16 points, no error bars, and no test, that has essentially no power; a Wilcoxon signed-rank or even a sign test would have been easy. The mean difference is actually +0.07, so the data are consistent with a small offset as easily as with zero. Second, the abstract calls the mean-vs-median difference an effect on 'official impact factors,' but you never compute Clarivate's values excluding those two papers; you infer it from your ADS data. That needs a direct demonstration or rephrasing. Third, the definitional mismatch—refereed plus unrefereed citations over refereed publications—is acknowledged in passing but never quantified as a coverage bias.\n\nWho should read it: anyone building or using ADS-based citation metrics for physics journals, or anyone who wants a clean example of how vulnerable standard impact factors are to single high-citation papers. It is a solid, modest contribution. It deserves a serious referee, provided the referee asks for a statistical test of the no-bias claim and a clarification of the 'official' wording. I'd send it out.","headline":"Readable, transparent extension to particle physics journals; the outlier stories are real, but the no-bias claim needs a statistical test before you trust it.","tokens_in":4718,"tokens_out":4524,"would_cite":false,"duration_ms":48691,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NASA/ADS citation data reproduce Clarivate's impact factors for 16 particle physics journals, with no systematic bias, but single outlier papers dominate the scores of two journals.","keywords":["impact factor","NASA/ADS","particle physics journals","bibliometrics","citation analysis","median-based impact factor","Clarivate Web of Science","outlier citations"],"falsifier":"Compare ADS and Web of Science citation counts for each journal at the level of individual papers and check whether the per-journal difference between ADS-based and official impact factors correlates with journal size, age, or uncited-paper fraction; alternatively, recompute PTEP and ARNPS impact factors with their top-cited papers excluded and see whether the gap to the median-based values disappears.","tokens_in":3840,"feed_emoji":"📊","tokens_out":6339,"duration_ms":60483,"temperature":0.7,"pith_summary":"This paper tries to establish that the NASA/ADS bibliographic database can stand in for Clarivate's Web of Science when computing journal impact factors in particle and nuclear physics. The authors calculate 2025 impact factors for 16 journals from ADS citations and compare them with the official Clarivate values, finding no systematic bias: about half the ADS values are higher and half lower. They also compare mean-based with median-based impact factors and find that the two agree for most journals, with the largest gaps at Progress of Theoretical and Experimental Physics and Annual Review of Nuclear and Particle Science. In both cases a single highly cited paper accounts for most of the gap, so the paper's second claim is that standard impact factors for these journals are strongly sensitive to rare extremely cited papers.","feed_headline":"NASA/ADS citation counts match official impact factors for 16 journals","feed_subtitle":"No systematic bias found, yet two journals' scores hinge on a single highly cited paper each.","key_machinery":"The central object is the impact factor ratio itself, computed in two versions. The traditional impact factor for year $n$ is the number of citations in year $n-1$ to papers published in years $n-2$ and $n-3$, divided by the number of refereed papers in those years; the median-based 'new' impact factor replaces the total citation count with the median number of citations per paper. The machinery is the comparison of these two ratios, both built from NASA/ADS citation records for journals tagged 'Physics', against Clarivate's official values, with the outlier papers identified by examining each journal's most-cited papers.","core_discovery":"The central claim is that impact factors computed from NASA/ADS citations match the official Clarivate impact factors for these 16 journals without any systematic offset, while a median-based impact factor reveals that two journals' official scores are inflated by single outlier papers. For PTEP, the Particle Data Group review with 1412 citations over the two-year window raises the official score far above the median-based value; for ARNPS, a review of the Hubble tension and early dark energy with 107 citations does the same. The paper therefore argues that ADS-based impact factors are a usable independent approximation to the Clarivate values, and that conventional impact factors in this field are fragile in the presence of a few extremely cited papers.","pith_inferences":["If the no-bias result holds across years, ADS data could be used to track year-to-year impact factor trends before official values are released, provided coverage differences by document type remain stable.","A direct test of the outlier explanation would be to recompute PTEP and ARNPS impact factors with their top-cited papers removed; the authors' claim predicts the two values would then come close to the median-based estimates.","Because cosmology and astrophysics papers are often cross-listed under the ADS 'Physics' tag, the comparison could be re-run using only INSPIRE-HEP citations to test whether database choice changes the conclusion for multi-disciplinary journals.","The acknowledged false positives and incompleteness in ADS citations mean the apparent absence of bias could stem from compensating errors; per-journal citation-level matching against Web of Science would settle that."],"forward_implications":["ADS-based impact factors can serve as an independent, freely accessible cross-check on Clarivate's official values for particle and nuclear physics journals.","For small journals, a single highly cited paper such as the PDG review can change the impact factor by a factor of several, so mean-based rankings should be read with that fragility in mind.","The median-based impact factor resists this outlier effect and may give a more stable picture of a journal's typical citation rate.","The absence of systematic bias suggests that, in aggregate, ADS citation coverage for these journals is comparable to Web of Science coverage, at least for the 2022-2024 window studied."],"supporting_citations":[{"why":"It defines the old and median-based impact factor formulas and the comparison methodology that this paper extends to particle physics journals.","marker":"[1]"},{"why":"It supplies the NASA/ADS database and API that provide all citation and publication counts used in the analysis.","marker":"[2]"},{"why":"It documents that NASA/ADS citation collections are incomplete and contain false positives, framing the main caveat on the no-bias conclusion.","marker":"[5]"},{"why":"It is the Particle Data Group review with 1412 citations that drives PTEP's impact factor above its median-based value.","marker":"[6]"},{"why":"It is the Hubble tension and early dark energy review with 107 citations that drives ARNPS's impact factor above its median-based value.","marker":"[8]"}],"fun_headline_variants":["ADS citations mirror official impact factors in 16 journals","Impact factors from NASA/ADS agree with Clarivate's for 16 journals","Two journals' impact factors inflated by single outlier papers","Median impact factor flags outlier-inflated journal scores","ADS impact factors match official, but outliers skew two"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The no-bias conclusion assumes that NASA/ADS's 'Physics' tag collects the same population of citable papers and citations for these 16 journals as Clarivate's Web of Science, so that any differences are just random noise.","fun_headline_variants_meta":{"raw":{"variants":["ADS citations mirror official impact factors in 16 journals","Impact factors from NASA/ADS agree with Clarivate's for 16 journals","Two journals' impact factors inflated by single outlier papers","Median impact factor flags outlier-inflated journal scores","ADS impact factors match official, but outliers skew two"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":2943,"prompt_tokens":847,"completion_tokens":2096,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":2015}},"tokens_in":463,"tokens_out":2096,"duration_ms":16719,"temperature":1.0,"reasoning_tokens":2015,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:46:19.560770+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare ADS and Web of Science citation counts for each journal at the level of individual papers and check whether the per-journal difference between ADS-based and official impact factors correlates with journal size, age, or uncited-paper fraction; alternatively, recompute PTEP and ARNPS impact factors with their top-cited papers excluded and see whether the gap to the median-based values disappears.","supporting_citations":[{"cited_title":"A meta-analysis of impact factors of astrophysics journals","cited_arxiv_id":"2410.03342","evidence_quote":"It defines the old and median-based impact factor formulas and the comparison methodology that this paper extends to particle physics journals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the NASA/ADS database and API that provide all citation and publication counts used in the analysis."},{"cited_title":"Will, Physics Today 67, 10 (2014)","cited_arxiv_id":null,"evidence_quote":"It documents that NASA/ADS citation collections are incomplete and contain false positives, framing the main caveat on the no-bias conclusion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the Particle Data Group review with 1412 citations that drives PTEP's impact factor above its median-based value."}],"review_version":1}