REVIEW 3 major objections 6 minor 1 cited by
Evaluating authorship disambiguation quality through anomaly analysis on researchers' career transition
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that the fraction of researchers whose first last-author paper appears in their debut year can serve as a reliable warning signal for poor authorship disambiguation, and demonstrates the signal on 5.8 million biomedical…
desk verdict A genuinely new anomaly signal for disambiguation quality, but the causal claim is overconfident and the ORCID comparison undermines it; worth refereeing with a required validation step. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the "immediate-independence anomaly rate": the share of authors in a cohort whose first last-author paper is dated in the same calendar year as their debut publication. It is computed from the distance in years between first publication and first last-author publication, under the field-specific convention that last author means senior or corresponding author. The mechanism carries the argument because it converts an unobservable quality, disambiguation correctness, into an observable distribution that has a plausible real-world ceiling; any rate far above that ceiling signals that author profiles are contaminated. The authors use metadata subsets, such as the presence of affiliation or of a persistent researcher identifier, as controlled comparisons to show that the anomaly moves with disambiguation difficulty.
What would settle it
Build a curated set of biomedical author profiles whose true publication histories are known; if the true histories show the same roughly 60% debut-year last-author rate, then the anomaly is not a disambiguation signal. More directly, check whether authors flagged as anomalous have their earliest recorded papers missing from the database: if many anomalous authors have no publications before the first last-author paper despite known earlier work, the signal tracks coverage gaps rather than name conflation.
Extended reading notes
Core claim
On the paper's own terms, last-author position in biomedical publishing marks the principal investigator, so the time from a researcher's first publication to their first last-author publication estimates career independence. The authors show that more than half of the 5.8 million researchers in their cohort are recorded as reaching that milestone in their debut year, and that this share stays above 60% across entry cohorts. They interpret the anomaly as evidence of name-disambiguation error, because authors without affiliation metadata show rates above 80%, while authors with a persistent self-reported identifier show rates around 40%. On that basis they claim the anomaly rate can serve as a reliable warning signal for disambiguation quality, and they apply it to show that female authors were more likely to be affected than male authors before roughly 2010. The paper's central claim is not a new disambiguation algorithm but a diagnostic: a high anomaly rate indicates poor disambiguation, and group comparisons built on such data should be re-examined.
Load-bearing premise
The load-bearing premise is that the abnormally high first-year last-author rate is caused by disambiguation errors and not by other database problems, especially incomplete coverage of each author's early publications.
Editorial extensions
If this is right
- The anomaly rate can be computed from any large bibliographic database with author-order information, so it offers a scalable quality check without requiring manually curated gold standards.
- Datasets with high anomaly rates should be treated as risky for career-timing studies, because debut-year last-authorship likely reflects record contamination rather than unusually fast promotion.
- Gender comparisons based on pre-2010 biomedical publication data may be biased by differential disambiguation quality between female and male authors, and those results warrant rechecking.
- Improving metadata completeness, such as adding affiliation details or persistent researcher identifiers, should reduce the anomaly and strengthen downstream science-of-science analyses.
- Because the same signal can be stratified by any author attribute, it provides a general way to test whether disambiguation quality varies across subgroups.
Reading between the lines
- The same anomaly metric could be recalibrated for other fields where the corresponding-author convention differs, since disciplines with rapid PI transitions or non-standard author ordering may have higher natural baselines.
- A direct testable extension is to link anomalous author profiles to known-correct identity records and verify whether the apparent first last-author paper is in fact authored by a different person with the same name.
- If incomplete early-career coverage rather than name conflation drives part of the anomaly, the metric could also function as a coverage diagnostic, not just a disambiguation diagnostic.
- The paper's gender finding suggests that historical gender-disparity results may need to be re-examined even when disambiguation quality appears acceptable on average, because subgroup-specific bias can hide behind aggregate rates.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes OpenAlex metadata for roughly 5.8 million biomedical authors and defines each author's career debut as the year of their first recorded publication. It reports that over 60% of authors have their first last-author paper in the same year as their debut, which the authors call an anomaly. The paper proposes this anomaly rate as a warning signal for poor authorship disambiguation, supports this with higher anomaly rates among authors lacking affiliation or ORCID metadata, and applies the metric to argue that disambiguation quality was lower for female than male authors before roughly 2008. The central claim is that the anomaly is caused primarily by authorship name disambiguation errors.
Significance. If the causal interpretation were established, the anomaly metric would be a cheap, scalable diagnostic for disambiguation quality in large bibliometric databases, and the gender-specific application would have direct implications for reinterpreting pre-2010 gender disparity studies. The paper's use of open data at very large scale is a strength, and the proposed metric is simple enough to be applied across databases. However, the current evidence is correlational rather than causal: the anomaly is never validated against a manually curated gold standard, and the ORCID group still shows an anomaly rate near 40%, which is difficult to reconcile with the claim that the anomaly mainly reflects disambiguation errors. The framework is promising, but the core attribution needs substantially stronger support before the metric can be called reliable.
major comments (3)
- [Results, 'Poor authorship disambiguation is the cause of the anomaly'] The central causal claim is not established because the anomaly measure is never validated against a curated gold standard of disambiguated author profiles. The career debut year is defined as the year of the earliest publication in the database; if an author's early publications are missing, the recorded debut shifts later and can coincide with a genuine first last-author paper, inflating the anomaly without any disambiguation failure. This alternative mechanism is especially relevant for the authors without affiliation or ORCID, who are also more likely to have incomplete early-career records, so the associations in Fig. 2 are confounded. The Discussion lists other limitations but does not address incomplete early publication coverage.
- [Fig. 2b] The ORCID comparison undermines the paper's interpretation rather than supporting it. Authors with ORCID are treated as a gold standard for disambiguation, yet this group still shows an anomaly rate near 40%. If ORCID profiles are well disambiguated, 40% cannot be a disambiguation-error signal; if ORCID profiles are incomplete, then incomplete coverage, not disambiguation, is a first-order cause. Either way, the statement that 'errors in authorship disambiguation highly contribute to the observed anomalies' (Results) is not established.
- [Results, gender analysis (Fig. 3)] The claim that female authors exhibit 'minor yet statistically significant' discrepancies is unsupported by any reported statistical test. Materials and Methods describes the gender prediction procedure and thresholds but does not describe a hypothesis test, effect size, or confidence interval for the comparisons in Fig. 3. Given the small differences and the multiple subgroup, continent, and period comparisons, the statistical evidence for the gender claim needs to be reported explicitly.
minor comments (6)
- [Introduction, final paragraph] There is a typo: 'authorshup disambiguation' should be 'authorship disambiguation'.
- [Fig. 1 caption] The caption contains 'stiking anomaly'; this should be 'striking anomaly'.
- [Results, first paragraph] The phrase 'last author positon' should read 'last author position'.
- [Fig. 3 title] The title 'Discrepancy in anomaly occured between male and female authors before 2008' has a grammatical error; 'occured' should be 'occurred'.
- [Materials and Methods, 'Authorship Metadata'] The terms 'affiliation information' and 'country metadata' are used interchangeably (Fig. 2 and text); the paper should define what exactly is measured, since country and institution are not the same.
- [Throughout] No data or code availability statement is provided; for a methods-oriented claim, releasing the analysis scripts would substantially improve reproducibility.
Circularity Check
No circular derivation: the anomaly metric is defined independently of the disambiguation-quality conclusion, and no fitted parameter or self-citation is used as the load-bearing input.
full rationale
The paper's derivation chain is self-contained and non-circular at the level of construction. The anomaly rate is defined directly from OpenAlex publication metadata: the career debut year is the year of an author's first recorded publication, and the anomaly is the event that the first last-author paper appears in that same debut year. This definition does not presuppose disambiguation quality; it is an observable computed from author positions. The subsequent inference that the anomaly indicates poor disambiguation is an interpretive hypothesis, tested by comparing anomaly rates across groups with and without affiliation metadata or ORCID identifiers. Those comparisons are not equivalent to the definition: ORCID status and affiliation availability are separate metadata attributes, and the observed differences in anomaly rates are empirical findings rather than consequences of the metric's definition. There is no fitted parameter that is later renamed as a prediction, no equation in which the conclusion is substituted into the premise, and no load-bearing self-citation: the cited 'last author analysis' [11], ORCID [9], OpenAlex [12], and gender classification [19] are external methodological references, not prior work by the present authors. The paper's central weakness is that it does not validate the anomaly against a ground-truth set of disambiguated authors, and it does not rule out the confounder that authors with incomplete early-career records (e.g., missing early publications) would also show inflated debut-year last-author rates. That is a validity or causal-attribution concern, not a circularity: the anomaly measure is not defined in terms of disambiguation errors, and the paper's own results (e.g., the ~40% anomaly rate among ORCID authors) leave the causal interpretation open. Under the stated rules, the absence of a specific reduction of the conclusion to the inputs means no circular step is established; the honest finding is no significant circularity.
Assumptions & free parameters
free parameters (2)
- gender_classification_threshold =
p(gf) <= 0.2 for male, p(gf) >= 0.8 for female
- five_year_window =
5 years
assumptions (4)
- domain assumption Last-author position indicates PI status in biomedical research
- domain assumption Career debut year equals the year of earliest publication in OpenAlex
- ad hoc to paper The observed anomaly is caused by name disambiguation errors
- domain assumption Presence of affiliation and ORCID identifiers are valid proxies for disambiguation quality
Cite this review
Pith. "Pith review of Evaluating authorship disambiguation quality through anomaly analysis on researchers' career transition." pith.science (2026). https://pith.science/paper/D4N6XZEQ
@misc{pith2026241218757,
author = {Pith},
title = {Pith review of: Evaluating authorship disambiguation quality through anomaly analysis on researchers' career transition},
year = {2026},
howpublished = {\url{https://pith.science/paper/D4N6XZEQ}},
note = {Machine review of arXiv:2412.18757}
}
read the original abstract
Authorship disambiguation is crucial for advancing studies in science of science. However, assessing the quality of authorship disambiguation in large-scale databases remains challenging since it is difficult to manually curate a gold-standard dataset that contains disambiguated authors. Through estimating the timing of when 5.8 million biomedical researchers became independent Principal Investigators (PIs) with authorship metadata extracted from the OpenAlex -- the largest open-source bibliometric database -- we unexpectedly discovered an anomaly: over 60% of researchers appeared as the last authors in their first career year. We hypothesized that this improbable finding results from poor name disambiguation, suggesting that such an anomaly may serve as an indicator of low-quality authorship disambiguation. Our findings indicated that authors who lack affiliation information, which makes it more difficult to disambiguate, were far more likely to exhibit this anomaly compared to those who included their affiliation information. In contrast, authors with Open Researcher and Contributor ID (ORCID) -- expected to have higher quality disambiguation -- showed significantly lower anomaly rates. We further applied this approach to examine the authorship disambiguation quality by gender over time, and we found that the quality of disambiguation for female authors was lower than that for male authors before 2010, suggesting that gender disparity findings based on pre-2010 data may require careful reexamination. Our results provide a framework for systematically evaluating authorship disambiguation quality in various contexts, facilitating future improvements in efforts to authorship disambiguation.
Figures
Forward citations
Cited by 1 Pith paper
-
Persistence Paradox in Dynamic Science
During the deep learning revolution, researchers who persisted in their prior topics suffered a citation-impact penalty, while moderate, selective pivoting was associated with the largest gains.
Reference graph
Works this paper leans on
-
[1]
Santo Fortunato, Carl T Bergstrom, Katy B¨ orner, James A Evans, Dirk Helbing, Staˇ sa Milo- jevi´ c, Alexander M Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, et al. Science of science. Science, 359(6379):eaao0185, 2018
work page 2018
-
[2]
Age and productivity among scientists
Wayne Dennis. Age and productivity among scientists. Science, 123(3200):724–725, 1956
work page 1956
-
[3]
The structure of scientific collaboration networks
Mark EJ Newman. The structure of scientific collaboration networks. Proceedings of the national academy of sciences, 98(2):404–409, 2001
work page 2001
-
[4]
Effectiveness of journal ranking schemes as a tool for locating information
Michael J Stringer, Marta Sales-Pardo, and Lu ´ ıs A Nunes Amaral. Effectiveness of journal ranking schemes as a tool for locating information. Plos one, 3(2):e1683, 2008
work page 2008
-
[5]
Neil R Smalheiser, Vetle I Torvik, et al. Author name disambiguation. Annual review of information science and technology, 43(1):1, 2009
work page 2009
-
[6]
Ciriaco Andrea D’Angelo, Cristiano Giuffrida, and Giovanni Abramo. A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments. Journal of the American Society for Information Science and Technology, 62(2):257–269, 2011
work page 2011
-
[7]
Ciriaco Andrea D’Angelo and Nees Jan Van Eck. Collecting large-scale publication data at the level of individual researchers: A practical proposal for author name disambiguation. Sciento- metrics, 123:883–907, 2020
work page 2020
-
[8]
Author name disambiguation of bibliometric data: A comparison of several unsupervised approaches
Alexander Tekles and Lutz Bornmann. Author name disambiguation of bibliometric data: A comparison of several unsupervised approaches. Quantitative Science Studies, 1(4):1510–1528, 2020
work page 2020
Show all 26 references
-
[9]
Orcid: a system to uniquely identify researchers
Laurel L Haak, Martin Fenner, Laura Paglione, Ed Pentz, and Howard Ratner. Orcid: a system to uniquely identify researchers. Learned publishing, 25(4):259–264, 2012
2012
-
[10]
Exploring the relevance of orcid as a source of study of data sharing activities at the individual-level: a methodological discussion
Sixto-Costoya Andrea, Robinson-Garcia Nicolas, and Costas Rodrigo. Exploring the relevance of orcid as a source of study of data sharing activities at the individual-level: a methodological discussion. Scientometrics, 126(8):7149–7165, 2021
2021
-
[11]
Publication metrics and success on the academic job market
David Van Dijk, Ohad Manor, and Lucas B Carey. Publication metrics and success on the academic job market. Current Biology, 24(11):R516–R517, 2014
2014
-
[12]
Openalex: A fully-open index of scholarly works, authors, venues, institutions, and concepts
Jason Priem, Heather Piwowar, and Richard Orr. Openalex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv preprint arXiv:2205.01833, 2022
2022 arXiv
-
[13]
The distribution of the asymptotic number of citations to sets of publications by a researcher or from an academic department are consistent with a discrete lognormal model
Jo˜ ao AG Moreira, Xiao Han T Zeng, and Lu ´ ıs A Nunes Amaral. The distribution of the asymptotic number of citations to sets of publications by a researcher or from an academic department are consistent with a discrete lognormal model. PLOS one, 10(11):e0143108, 2015
2015
-
[14]
Quantifying the evolution of individual scientific impact
Roberta Sinatra, Dashun Wang, Pierre Deville, Chaoming Song, and Albert-L´ aszl´ o Barab´ asi. Quantifying the evolution of individual scientific impact. Science, 354(6312):aaf5239, 2016. 9
2016
-
[15]
Number of authors per medline ®/pubmed® citation
-
[16]
The postdoc queue: A labour force in waiting
Maryam A Andalib, Navid Ghaffarzadegan, and Richard C Larson. The postdoc queue: A labour force in waiting. Systems research and behavioral science, 35(6):675–686, 2018
2018
-
[17]
The possible role of resource requirements and academic career-choice risk on gender differences in publication rate and impact
Jordi Duch, Xiao Han T Zeng, Marta Sales-Pardo, Filippo Radicchi, Shayna Otis, Teresa K Woodruff, and Lu ´ ıs A Nunes Amaral. The possible role of resource requirements and academic career-choice risk on gender differences in publication rate and impact. PloS one, 7(12):e51332, 2012
2012
-
[18]
Biblio- metrics: Global gender disparities in science
Vincent Larivi` ere, Chaoqun Ni, Yves Gingras, Blaise Cronin, and Cassidy R Sugimoto. Biblio- metrics: Global gender disparities in science. Nature, 504(7479):211–213, 2013
2013
-
[19]
An open-source cultural consen- sus approach to name-based gender classification
Ian Van Buskirk, Aaron Clauset, and Daniel B Larremore. An open-source cultural consen- sus approach to name-based gender classification. In Proceedings of the International AAAI Conference on Web and Social Media, volume 17, pages 866–877, 2023
2023
-
[20]
Contributorship and division of labor in knowledge production
Vincent Larivi` ere, Nadine Desrochers, Beno ˆ ıt Macaluso, Philippe Mongeon, Ad` ele Paul-Hus, and Cassidy R Sugimoto. Contributorship and division of labor in knowledge production. Social studies of science, 46(3):417–435, 2016
2016
-
[21]
Scopus database: a review
Judy F Burnham. Scopus database: a review. Biomedical digital libraries, 3:1–8, 2006
2006
-
[22]
Dimensions: building context for search and evaluation
Daniel W Hook, Simon J Porter, and Christian Herzog. Dimensions: building context for search and evaluation. Frontiers in Research Metrics and Analytics, 3:23, 2018
2018
-
[23]
Citation indexes for science: A new dimension in documentation through association of ideas
Eugene Garfield. Citation indexes for science: A new dimension in documentation through association of ideas. Science, 122(3159):108–111, 1955
1955
-
[24]
Name-based demographic inference and the unequal distribution of misrecognition
Jeffrey W Lockhart, Molly M King, and Christin Munsch. Name-based demographic inference and the unequal distribution of misrecognition. Nature Human Behaviour, 7(7):1084–1095, 2023
2023
-
[25]
Historical comparison of gender inequality in scientific careers across countries and disciplines.Proceedings of the National Academy of Sciences, 117(9):4609–4616, 2020
Junming Huang, Alexander J Gates, Roberta Sinatra, and Albert-L´ aszl´ o Barab´ asi. Historical comparison of gender inequality in scientific careers across countries and disciplines.Proceedings of the National Academy of Sciences, 117(9):4609–4616, 2020
2020
-
[26]
The extent and drivers of gender imbalance in neuroscience reference lists
Jordan D Dworkin, Kristin A Linn, Erin G Teich, Perry Zurn, Russell T Shinohara, and Danielle S Bassett. The extent and drivers of gender imbalance in neuroscience reference lists. Nature neuroscience, 23(8):918–926, 2020. 10
2020
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.