Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Quantifying the Dynamics of Harm Caused by Retracted Research

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that retracted papers reduce the citation performance of papers that cite them through an 'attention escape' mechanism: harm is delayed, intensifies with indirect citation distance, and is concentrated in journals with…

desk verdict A large-scale descriptive study of citation deficits around retracted papers; the distance gradient survives the comparator-contamination worry, but the causal 'harm' framing is not supported by the design. read the letter →

arxiv 2501.00473 v2 pith:EKLUH5DV submitted 2024-12-31 cs.DL

Yunyou Huang , Jiahui Zhao , Dandan Cui , Zhengxin Yang , Bingjie Xia , Qi Liang , Wenjing Liu , Li Ma
show 5 more authors
Suqin Tang Tianyong Hao Zhifei Zhang Wanling Gao Jianfeng Zhan
This is my paper · ORCID
classification cs.DL
keywords retractedpapersattentionescapecitationharmindirectcitationsresearchintegrityimpactfactorscientometricsnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that retracted papers measurably reduce the citation performance of papers that cite them, through a mechanism it calls 'attention escape.' Defining harm as a citing paper's citation shortfall relative to comparable papers from the same journal, year, and field, it finds the shortfall is small in the first years after publication but grows to a median of about 40 percent by the tenth year for direct citers. The shortfall is larger for papers that cite those direct citers, and it grows as the citation distance increases. The effect is strongest for papers in journals with an impact factor below 10, and it persists even after retraction. If correct, this means current alerting tools that focus on direct citations are aimed at the wrong place, because the worst harm happens outside the attention of authors and publishers.

What carries the argument

The machinery is a citation-based harm metric. For each citing paper, harm is defined as $1$ minus the ratio of the paper's citation count to the average citation count of a comparator group, where the comparator group consists of papers from the same venue, published in the year before, the same year, or the year after the citing paper, and sharing at least one field of study. The metric is computed for total citations and for each of the ten years after publication. The paper then builds six citation-distance layers by iteratively expanding to papers that cite the previous layer, and compares median harm across layers, time, and impact-factor bands. The comparator-group ratio is what turns 'harm' into a measurable shortfall.

What would settle it

Take the set of papers that cite a retracted work and, for each one, construct a control paper from the same journal, year, and field with similar author-team size, topic keywords, open-access status, and reference-list length; if the citation shortfall disappears or reverses under this matching, the reported harm is an artifact of confounding rather than an effect of the retracted citation.

Watch

Extended reading notes

Core claim

The central claim is that retracted papers propagate harm through citation networks by an 'attention escape' mechanism: their most damaging effects occur where authors and publishers are least likely to notice. The paper reports three signature patterns in citation data. First, papers that directly cite a retracted work show no significant citation shortfall in their first four years, but the shortfall then grows steadily, reaching a median of about 40 percent by the tenth year. Second, papers connected only indirectly, through up to six citation steps, suffer larger shortfalls than direct citers, and the severity increases with each extra step. Third, the shortfall is concentrated in journals with impact factor below 10; papers in journals with impact factor above 20 may even show a surplus in early years. The paper measures harm as the ratio of a paper's citations to the average of papers in the same venue, a one-year publication window, and an overlapping field, and it concludes that retraction alone does not stop the damage because the harm escapes the attention these interventions rely on.

Load-bearing premise

The study assumes the citation shortfall it measures is caused by the fact that the paper cites a retracted work, rather than by other traits of the citing paper such as topic popularity, author team, article quality, or open-access status, and it applies causal language without controlling for those traits.

Editorial extensions

If this is right

  • Retraction-alerting tools that flag only direct citations will miss the majority of the harm, because indirect citations carry larger citation shortfalls than direct ones.
  • Promotion and evaluation criteria that emphasize recent publications will systematically overlook the delayed harm that retracted citations cause, matching the paper's finding that harm grows after the early years.
  • Journals with impact factor below 10 are the highest-priority target for retroactive citation screening, since their papers absorb the largest citation losses.
  • Post-retraction intervention is needed in addition to the retraction notice itself, because citing papers continue to experience significant harm after the retraction date.
  • The harm metric can be applied to any individual retracted article to estimate its citation cost to the papers that follow and to rank retractions by damage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the causal interpretation would be strengthened or weakened by matching citing papers to controls with similar author-team size, topic keywords, open-access status, and reference-list length; the paper does not perform this matching, so the reported effect sizes are upper bounds if confounding exists.
  • Editorial extension: the attention-escape mechanism predicts that high-visibility retractions, such as prominent fraud cases, should produce a different harm curve from quiet retractions; comparing those two groups would test the mechanism more directly.
  • Editorial extension: the same metric could be used as a pre-publication screening score that warns authors when their manuscript's reference chain reaches a retracted paper, but the paper does not assess how often such warnings would be false alarms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a citation-based framework to quantify the "harm" caused by retracted papers. Using Semantic Scholar and Retraction Watch data, the authors construct citation chains C1–C6 (papers directly or indirectly citing retracted work) and define harm as 1 minus the ratio of a citing paper's citation count to the average citation count of a comparator group defined by the same venue, publication year ± 1, and overlapping field of study (Eqs. 1–5). The main empirical claims are that harm to citing papers grows over time, increases with citation distance up to six steps, and is larger for papers published in journals with impact factor below 10; these patterns are interpreted as an "attention escape" mechanism through which retracted papers inflict damage that evades stakeholder attention. The manuscript also reports analyses of harm before versus after retraction.

Significance. If the central causal claim were supported, this would be an important contribution to the study of retracted research, providing quantitative evidence on how retracted papers affect the citation performance of citing papers and suggesting policy implications for tracking indirect citations. The study uses a very large corpus, and the authors make data and code publicly available, which is commendable. However, the validity of the central claim hinges on the harm metric being an unbiased measure of the effect of retracted papers, and the current design does not establish that. The comparator-based deficit is susceptible to a serious contamination bias that can generate the reported distance gradient even under a null hypothesis of no effect, so the significance of the findings is conditional on major re-analysis.

major comments (3)
  1. [Methods 4.2, Eq. (1) and Fig. 3 / Extended Data Table 3] This is the central load-bearing issue.
  2. [Eqs. (4)–(5) and Sections 2.2–2.3, Discussion]
  3. [Section 2.4 and Extended Data Table 6]
minor comments (5)
  1. [Section 2.4] There is a typo in "Figrue 5" that should be corrected to "Figure 5".
  2. [Methods 4.2] The text contains "citing prc" and "paperc" where "paper prc" or "paper c" is intended; please revise for clarity.
  3. [Extended Data Tables 3 and 4] The column headers list "Y8 Harm (IQR)" twice, with the first occurrence likely meant to be "Y9"; also, Table 3 and Table 4 headers should be checked for consistency.
  4. [Throughout] The term "harm" is used both as a metric (a real-valued ratio) and as a normative concept; the paper would benefit from explicitly distinguishing the operational definition from the interpretive label, and from noting that negative harm values are possible.
  5. [Results 2.2 and Fig. 3] The figures report median harm and interquartile ranges, but no confidence intervals or significance tests are provided for the comparison across citation distances; adding such measures would make the claims more robust.

Circularity Check

1 steps flagged · score 2.0 of 10

No statistically forced circularity; only mild interpretive renaming of the observed harm patterns as an 'attention escape' mechanism.

  1. renaming known result [Abstract; Section 3 Discussion]
    "We uncover an ''attention escape'' mechanism, wherein retracted papers postpone significant harm, more prominently affect indirectly citing papers, and inflict greater harm on citations in journals with an impact factor less than 10."

    The paper presents 'attention escape' as its central explanatory mechanism, but the mechanism is not independently operationalized; its three listed properties are exactly the three empirical regularities computed from the harm metric (Eqs. 4-5) in Section 2 (temporal delay, increasing harm with citation distance, and larger harm for IF<10 journals). Naming these results a mechanism and then stating that the mechanism 'allows' those same effects is a restatement of the findings rather than an independent derivation. The measurements themselves are not forced by the definition of harm, so this is mild interpretive circularity rather than a statistical tautology.

full rationale

The derivation chain is otherwise self-contained: Equations (4)-(5) define harm as 1 minus the ratio of a paper's citation count to the mean of a comparator set matched on venue, publication year +/-1, and overlapping field. Nothing in this definition forces median harm to rise with citation distance, to increase over time, or to vary by impact factor; those are empirical outputs of the constructed dataset. No fitted parameter is relabeled as a prediction, and the only self-citation (reference [36], an evaluatology framework by the corresponding authors) is not load-bearing for the measurement or statistics. The comparator-contamination concern raised by the skeptic is a real internal-validity threat, but it is a confound, not circularity. The sole mild circular element is the 'attention escape' label, which summarizes the observed patterns rather than testing an independent mechanism.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The ledger shows the causal interpretation rests on strong untested assumptions; the only invented entity is the interpretive 'attention escape' mechanism. The hand-chosen parameters affect the magnitude and shape of all reported harm values.

free parameters (5)
  • citation_distance_cap = 6
    The network is truncated at six indirect citation steps, motivated by the 'six degrees of separation' concept; results for larger distances are not examined.
  • comparator_year_window = ±1 year
    Papers are considered comparable if published in the same venue in the same year, one year earlier, or one year later; the width is chosen by the authors.
  • first_year_exclusion = year 1 omitted
    The first year after publication is excluded from the 10-year citation window to avoid publication-date bias, changing the time axis of all harm trends.
  • impact_factor_bins = [0,3), [3,5), [5,10), [10,20), [20,∞)
    Journal IF is categorized into five arbitrary bins before comparing median harm; the key '<10' threshold is one of these hand-chosen boundaries.
  • deduplication_priority = higher citation count, then more references
    When duplicate DOIs are found, records with higher citation counts and more references are kept; this hand-chosen rule shapes the underlying corpus.
assumptions (5)
  • domain assumption Citation deficit is a valid proxy for the harm caused by a retracted paper.
    The entire metric, Equation (4), interprets receiving fewer citations than a comparator group as 'harm' from retraction, without evidence of a causal pathway.
  • domain assumption The comparator group of same-venue, year±1, field-overlapping papers is an adequate counterfactual for expected citations.
    Equation (1) constructs this baseline and all harm scores are relative to it; if the baseline is biased, the magnitude of harm is biased.
  • domain assumption No unmeasured confounders explain the association between citing retracted papers and citation deficits.
    No controls for paper quality, topic age, author reputation, team size, or access status are included, but the deficit is attributed to the retracted citation.
  • domain assumption The Semantic Scholar, Retraction Watch, and SciSciNet data are sufficiently complete and correct after cleaning.
    All results depend on the matching of DOIs and citation records across these three databases; any mismatch or missing field removes records from analysis.
  • domain assumption Citation chains C1 to C6 represent propagation of influence from the retracted paper.
    Indirect citation distance is treated as a transmission pathway of harm, though a paper far down the chain may cite for unrelated reasons.
invented entities (1)
  • 'attention escape' mechanism
    purpose: Explains the observed pattern that harm is delayed, strongest at indirect distances, and concentrated in low-IF journals by positing it escapes stakeholder attention.
    The mechanism is a narrative label for the three observed correlational patterns; the paper provides no direct test of attention or intervention behavior, and it could be an artifact of the metric and sample selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quantifying the Dynamics of Harm Caused by Retracted Research." pith.science (2026). https://pith.science/paper/EKLUH5DV

@misc{pith2026250100473,
  author       = {Pith},
  title        = {Pith review of: Quantifying the Dynamics of Harm Caused by Retracted Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EKLUH5DV}},
  note         = {Machine review of arXiv:2501.00473}
}
read the original abstract

Despite enormous efforts devoted to understand the characteristics and impacts of retracted papers, little is known about the mechanisms underlying the dynamics of their harm and the dynamics of its propagation. Here, we propose a citation-based framework to quantify the harm caused by retracted papers, aiming to uncover why their harm persists and spreads so widely. We uncover an ''attention escape'' mechanism, wherein retracted papers postpone significant harm, more prominently affect indirectly citing papers, and inflict greater harm on citations in journals with an impact factor less than 10. This mechanism allows retracted papers to inflict harm outside the attention of authors and publishers, thereby evading their intervention. This study deepens understanding of the harm caused by retracted papers, emphasizes the need to activate and enhance the attention of authors and publishers, and offers new insights and a foundation for strategies to mitigate their harm and prevent its spread.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 41 canonical work pages

  1. [1]

    Accessed: January 31, 2025 (2025)

    Retraction Watch: Retraction Watch Database. Accessed: January 31, 2025 (2025). https://retractiondatabase.org/RetractionSearch.aspx

    work page 2025

  2. [2]

    Nature 624(7992), 479–481 (2023) 16

    Van Noorden, R.: More than 10,000 research papers were retracted in 2023—a new record. Nature 624(7992), 479–481 (2023) 16

    work page 2023

  3. [3]

    Accessed: 2024-11-20

    Staff, R.W.: Braggadacio, Information Control, and Fear: Life Inside a Brigham Stem Cell Lab Under Investigation. Accessed: 2024-11-20. https://retractionwa tch.com/2014/05/30/braggadacio-information-control-and-fear-life-inside-a-bri gham-stem-cell-lab-under-investigation/ Accessed 2024-11-20

    work page 2024

  4. [4]

    Piller, C.: Blots on a field? Science 377(6604), 358–363 (2022)

    work page 2022

  5. [5]

    Science 25(1), 1–5 (2018)

    Brainard, J., You, J., et al.: What a massive database of retracted papers reveals about science publishing’s ‘death penalty’. Science 25(1), 1–5 (2018)

    work page 2018

  6. [6]

    Else, H.: Biomedical paper retractions have quadrupled in 20 years—why? Nature 630(8016), 280–281 (2024)

    work page 2024

  7. [7]

    Accessed: 2024-11-22

    Retraction Watch Staff: Retraction Watch. Accessed: 2024-11-22. https://retrac tionwatch.com/ Accessed 2024-11-22

    work page 2024

  8. [8]

    Accessed: 2024-11-22

    PubPeer Foundation: PubPeer: The Online Journal Club. Accessed: 2024-11-22. https://pubpeer.com/ Accessed 2024-11-22

    work page 2024

Show all 42 references
  1. [9]

    Accessed: 2024-11-22

    Bik, E.: Science Integrity Digest. Accessed: 2024-11-22. https://scienceintegrity digest.com/ Accessed 2024-11-22

    2024

  2. [10]

    Med Ethics 9(4), 296–300 (2024)

    MAURYA, M., MUNSHI, R.: Reasons for retraction of clinical research articles in pubmed indexed medical journals from 2012 to 2022. Med Ethics 9(4), 296–300 (2024)

    2024

  3. [11]

    Humanities and Social Sciences Communications 10(1), 1–12 (2023)

    Sharma, P., Sharma, B., Reza, A., Inampudi, K.K., Dhamija, R.K.: A systematic review of retractions in biomedical research publications: reasons for retrac- tions and their citations in indian affiliations. Humanities and Social Sciences Communications 10(1), 1–12 (2023)

    2023

  4. [12]

    ACS omega 8(35), 31568–31574 (2023)

    Sevryugina, Y., Jimenez, R.: Analysis of retracted manuscripts in chemistry: Errors vs misconduct. ACS omega 8(35), 31568–31574 (2023)

    2023

  5. [13]

    Proceedings of the National Academy of Sciences 119(25), 2119086119 (2022)

    Peng, H., Romero, D.M., Horv´ at, E.- ´A.: Dynamics of cross-platform attention to retracted papers. Proceedings of the National Academy of Sciences 119(25), 2119086119 (2022)

    2022

  6. [14]

    bmj 379 (2022)

    Candal-Pedreira, C., Ross, J.S., Ruano-Ravina, A., Egilman, D.S., Fern´ andez, E., P´ erez-R ´ ıos, M.: Retracted papers originating from paper mills: cross sectional study. bmj 379 (2022)

    2022

  7. [15]

    Pain, 10–1097 (2022)

    Ferraro, M.C., Moore, R.A., C Williams, A.C., Fisher, E., Stewart, G., Ferguson, M.C., Eccleston, C., O’Connell, N.E.: Characteristics of retracted publications related to pain research: a systematic review. Pain, 10–1097 (2022)

    2022

  8. [16]

    Scientometrics 126(10), 17 8433–8470 (2021)

    Heibi, I., Peroni, S.: A qualitative and quantitative analysis of open citations to retracted articles: the wakefield 1998 et al.’s case. Scientometrics 126(10), 17 8433–8470 (2021)

    2021

  9. [17]

    Journal of Information Science 50(2), 531–545 (2024)

    Yang, S., Qi, F., Diao, H., Ajiferuke, I.: Do retraction practices work effectively? evidence from citations of psychological retracted articles. Journal of Information Science 50(2), 531–545 (2024)

    2024

  10. [18]

    Quantitative Science Studies 3(4), 953–975 (2022)

    Heibi, I., Peroni, S.: A quantitative and qualitative open citation analysis of retracted articles in the humanities. Quantitative Science Studies 3(4), 953–975 (2022)

    2022

  11. [19]

    JAMA Network Open 4(7), 2118263–2118263 (2021)

    Wadhwa, R.R., Rasendran, C., Popovic, Z.B., Nissen, S.E., Desai, M.Y.: Tem- poral trends, characteristics, and citations of retracted articles in cardiovascular medicine. JAMA Network Open 4(7), 2118263–2118263 (2021)

    2021

  12. [20]

    Journal of the Chinese Medical Association 87(10), 927–932 (2024)

    Lin, K.-C., Chen, Y.-C., Lin, M.-H., Chen, T.-J.: The trend and ripple effects of retractions in primary health care: A bibliometric analysis. Journal of the Chinese Medical Association 87(10), 927–932 (2024)

    2024

  13. [21]

    Accountability in Research 29(7), 442–459 (2022)

    Fanelli, D., Wong, J., Moher, D.: What difference might retractions make? an estimate of the potential epistemic cost of retractions on meta-analyses. Accountability in Research 29(7), 442–459 (2022)

    2022

  14. [22]

    Elife 3, 02956 (2014)

    Stern, A.M., Casadevall, A., Steen, R.G., Fang, F.C.: Financial costs and personal consequences of research misconduct resulting in retracted publications. Elife 3, 02956 (2014)

    2014

  15. [23]

    Cell death & disease 10(3), 189 (2019)

    Bucci, E.M.: On zombie papers. Cell death & disease 10(3), 189 (2019)

    2019

  16. [24]

    Nature 632(8027), 977–979 (2024)

    Cabanac, G.: Chain retraction: how to stop bad science propagating through the literature. Nature 632(8027), 977–979 (2024)

    2024

  17. [25]

    Journal of Electronic Resources in Medical Libraries 21(3), 153–161 (2024)

    Swogger, S.E.: The interactive web–identifying retracted publications. Journal of Electronic Resources in Medical Libraries 21(3), 153–161 (2024)

    2024

  18. [26]

    https://go.nature.com/3ysnj8f

    Cabanac, G.: Feet of Clay Detector. https://go.nature.com/3ysnj8f. Accessed: 2024-12-18 (2024)

    2024

  19. [27]

    Accessed: 2024-12-29 (2024)

    Guillaume Cabanac: Annulled Detector. Accessed: 2024-12-29 (2024). https://db rech.irit.fr/pls/apex/f?p=9999:33::IR citations

    2024

  20. [28]

    figshare (2019) https: //doi.org/10.6084/m9.figshare.6177014.v4

    Feakins, K., Irving, F., DeVito, N., Feakins, B., Curtis, H., Bacon, S., et al.: RetractoBot: a protocol for a randomised controlled trial to assess the impact of notifying authors that they have cited a retracted paper. figshare (2019) https: //doi.org/10.6084/m9.figshare.617...

    2019 doi

  21. [29]

    Nature 633(8028), 13–15 (2024) 18

    Van Noorden, R., Naddaf, M.: Exclusive: the papers that most heavily cite retracted studies. Nature 633(8028), 13–15 (2024) 18

    2024

  22. [30]

    Accountability in Research 31(1), 14–37 (2024)

    Avenell, A., Bolland, M.J., Gamble, G.D., Grey, A.: A randomized trial alerting authors, with or without coauthors or editors, that research they cited in system- atic reviews and guidelines has been retracted. Accountability in Research 31(1), 14–37 (2024)

    2024

  23. [31]

    Science 354(6312), 5239 (2016)

    Sinatra, R., Wang, D., Deville, P., Song, C., Barab´ asi, A.-L.: Quantifying the evolution of individual scientific impact. Science 354(6312), 5239 (2016)

    2016

  24. [32]

    Proceedings of the National Academy of Sciences 111(43), 15316–15321 (2014)

    Petersen, A.M., Fortunato, S., Pan, R.K., Kaski, K., Penner, O., Rungi, A., Ric- caboni, M., Stanley, H.E., Pammolli, F.: Reputation and impact in academic careers. Proceedings of the National Academy of Sciences 111(43), 15316–15321 (2014)

    2014

  25. [33]

    Current Biology 24(11), 516–517 (2014)

    Van Dijk, D., Manor, O., Carey, L.B.: Publication metrics and success on the academic job market. Current Biology 24(11), 516–517 (2014)

    2014

  26. [34]

    Nature biotechnology 35(9), 885–887 (2017)

    Wright, C.B., Vanderford, N.L.: What faculty hiring committees want. Nature biotechnology 35(9), 885–887 (2017)

    2017

  27. [35]

    arXiv preprint arXiv:2301.10140 (2023)

    Kinney, R., Anastasiades, C., Authur, R., Beltagy, I., Bragg, J., Buraczynski, A., Cachola, I., Candra, S., Chandrasekhar, Y., Cohan, A., et al.: The semantic scholar open data platform. arXiv preprint arXiv:2301.10140 (2023)

    2023 arXiv

  28. [36]

    Bench- Council Transactions on Benchmarks, Standards and Evaluations 4(1), 100162 (2024)

    Zhan, J., et al.: Evaluatology: The science and engineering of evaluation. Bench- Council Transactions on Benchmarks, Standards and Evaluations 4(1), 100162 (2024)

    2024

  29. [37]

    https://www.topuniversities

    Top Universities: World University Rankings 2024. https://www.topuniversities. com/world-university-rankings/2024. Accessed: 2024-12-22

    2024

  30. [38]

    https://www.webofscience.com

    Clarivate Analytics: Web of Science. https://www.webofscience.com. Accessed: 2024-12-22

    2024

  31. [39]

    Samoylenko, I., Aleja, D., Primo, E., Alfaro-Bittner, K., Vasilyeva, E., Kovalenko, K., Musatov, D., Raigorodskii, A.M., Criado, R., Romance, M., Papo, D., Perc, M., Barzel, B., Boccaletti, S.: Why are there six degrees of separation in a social network? Phys. Rev. X 13, 02103...

    2023 doi

  32. [40]

    Science 342(6154), 127–132 (2013)

    Wang, D., Song, C., Barab´ asi, A.-L.: Quantifying long-term scientific impact. Science 342(6154), 127–132 (2013)

    2013

  33. [41]

    Nature 441(7095), 792–793 (2006)

    Fuyuno, I., Cyranoski, D.: Cash for papers: putting a premium on publication. Nature 441(7095), 792–793 (2006)

    2006

  34. [42]

    Lin, Z., Yin, Y., Liu, L., Wang, D.: Sciscinet: A large-scale open data lake for the science of science research. Scientific Data 10(1), 315 (2023) 19 Extended Data T able 1 The median harm experienced by papers directly citing retracted articles across different fields varies...

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.