pith. sign in

arxiv: 1907.06999 · v1 · pith:67GRVHLXnew · submitted 2019-07-16 · 🧬 q-bio.QM · cs.DL

Evaluating the Reproducibility of Research in Obstetrics and Gynecology

Pith reviewed 2026-05-24 20:43 UTC · model grok-4.3

classification 🧬 q-bio.QM cs.DL
keywords reproducibilityobstetrics and gynecologyclinical trialsdata availabilitypreregistrationconflict of interestopen accessresearch waste
0
0 comments X

The pith

Trials in obstetrics and gynecology rarely include data availability statements, preregistration, protocols, or conflict disclosures.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper evaluates 300 trials in obstetrics and gynecology against fourteen indicators of reproducibility. It finds that none provided links to protocols or materials availability statements, none were replication studies, and only about eleven percent offered data availability information. Just over five percent mentioned preregistration, while roughly a quarter omitted conflict of interest reporting and over half did not state funding origins. If these patterns hold, the field faces risks of research waste and clinical decisions based on non-reproducible findings.

Core claim

By examining 300 trials, the authors found that none supplied links to protocols or statements on material availability, none were replication studies, only 10.58% provided data availability statements, 5.82% addressed preregistration, 25.85% omitted conflict of interest reporting, and 54.08% did not state funding origins. This leads to the conclusion that research in obstetrics and gynecology is not consistently reproducible and frequently lacks conflict of interest disclosure.

What carries the argument

A checklist of fourteen indicators including data availability, analysis scripts, pre-registration, study protocols, open access, conflict of interest statements, and funding sources, applied to a sample of 300 empirical trials.

Load-bearing premise

The fourteen chosen indicators adequately capture reproducibility and the three-hundred-trial sample represents the broader literature in the field.

What would settle it

A larger or differently selected sample of trials where most include data links, protocols, preregistration statements, and conflict disclosures would contradict the central claim.

Figures

Figures reproduced from arXiv: 1907.06999 by Aaron Bowers, Daniel Tritz, Drayton Rorah, Lance Frye, Matt Vassar, Shelby Rauh.

Figure 1
Figure 1. Figure 1: Prisma Diagram for included and excluded studies [PITH_FULL_IMAGE:figures/full_fig_p014_1.png] view at source ↗
read the original abstract

Objective: Reproducibility is a core tenet of scientific research. A reproducible study is one where the results can be recreated by different investigators in different circumstances using the same methodology and materials. Unfortunately, reproducibility is not a standard to which the majority of research is currently adherent. Methods: We objectively evaluated 300 trials in the field of Obstetrics and Gynecology for fourteen indicators of reproducibility. These indicators include availability of data, analysis scripts, pre-registration information, study protocols and whether or not the study was available via Open Access. We also assessed the trials for financial conflict of interest statements and source of funding. Results: Of the 300 trials in our sample, 208 contained empirical data that could be assessed for reproducibility. None of the trials in our sample provided a link to their protocols or provided a statement on availability of materials. None were replication studies. Just 10.58% provided a statement regarding their data availability, while only 5.82% provided a statement on preregistration. 25.85% failed to report the presence or absence of conflicts of interest and 54.08% did not state the origin of their funding. Conclusion: Research in the field of Obstetrics and Gynecology is not consistently reproducible and frequently lacks conflict of interest disclosure. Consequences of this could be far-reaching and include increased research waste, widespread acceptance of misleading results and erroneous conclusions guiding clinical decision-making.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The manuscript evaluates 300 trials in Obstetrics and Gynecology against 14 indicators of reproducibility (data availability, analysis scripts, preregistration, protocols, materials, open access, COI statements, funding source). Among the 208 empirical trials, none provided protocol links or materials-availability statements, none were replication studies, 10.58% included data-availability statements, 5.82% mentioned preregistration, 25.85% omitted COI statements, and 54.08% did not report funding origin. The paper concludes that research in the field is not consistently reproducible and frequently lacks conflict-of-interest disclosure.

Significance. If the chosen indicators are accepted as valid proxies, the concrete empirical counts (explicit n=300, 208 empirical subset, specific percentages) provide a field-specific snapshot of reporting practices that could motivate transparency initiatives in clinical research. The work supplies direct, countable data rather than fitted models or derivations.

major comments (2)
  1. [Abstract, Conclusion] Abstract, Conclusion: The central claim that 'Research in the field of Obstetrics and Gynecology is not consistently reproducible' is not directly supported by the reported data, which consist exclusively of presence/absence of reporting statements; the manuscript contains no reproduction attempts, no verification of actual data or protocol availability beyond statements, and no test of whether the 14 indicators predict replicability. This proxy-to-conclusion link is load-bearing for the title and final sentence.
  2. [Methods] Methods (implied by abstract and reader's assessment): The full selection criteria used to identify the 300 trials, the exact operational definitions and coding rules for each of the 14 indicators, and any inter-rater reliability statistics are not visible, preventing assessment of whether the reported percentages (e.g., 10.58% data statements) are reproducible or representative.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed review and the opportunity to clarify our work on reproducibility indicators in Obstetrics and Gynecology. We address each major comment below.

read point-by-point responses
  1. Referee: [Abstract, Conclusion] Abstract, Conclusion: The central claim that 'Research in the field of Obstetrics and Gynecology is not consistently reproducible' is not directly supported by the reported data, which consist exclusively of presence/absence of reporting statements; the manuscript contains no reproduction attempts, no verification of actual data or protocol availability beyond statements, and no test of whether the 14 indicators predict replicability. This proxy-to-conclusion link is load-bearing for the title and final sentence.

    Authors: We agree that the study measures the presence or absence of reporting statements and other indicators widely viewed as necessary for reproducibility, rather than conducting reproduction attempts or testing predictive validity of the indicators. These indicators are drawn from established reproducibility frameworks, and their near-absence supports the inference that most studies in the sample are not positioned for consistent reproduction. To strengthen precision, we will revise the abstract and conclusion to state that the research 'lacks key indicators of reproducibility' and 'frequently omits elements required for reproducibility' while retaining the overall assessment. This constitutes a partial revision. revision: partial

  2. Referee: [Methods] Methods (implied by abstract and reader's assessment): The full selection criteria used to identify the 300 trials, the exact operational definitions and coding rules for each of the 14 indicators, and any inter-rater reliability statistics are not visible, preventing assessment of whether the reported percentages (e.g., 10.58% data statements) are reproducible or representative.

    Authors: The Methods section describes the sample of 300 trials and the 14 indicators evaluated, but we accept that greater detail on selection criteria, exact operational definitions, and coding rules would improve transparency and allow readers to assess reproducibility of the percentages. We will expand the Methods section with these specifics. If multiple independent coders were employed, inter-rater reliability statistics will be added; if a single coder was used, this will be stated explicitly as a methodological detail. revision: yes

Circularity Check

0 steps flagged

Empirical indicator count with no derivation chain

full rationale

The paper conducts a direct empirical audit: it samples 300 trials, applies 14 pre-specified binary indicators (data availability, preregistration, COI statements, etc.), and reports observed frequencies. No equations, fitted parameters, model predictions, or self-citation chains are used to derive the central claims. The conclusion that reproducibility indicators are infrequently reported follows immediately from the tabulated counts in the 208 empirical trials; the analysis is therefore self-contained and contains no circular reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that the sampled trials and chosen indicators validly measure reproducibility; no free parameters, invented entities, or non-standard axioms are introduced.

axioms (1)
  • domain assumption The 300 trials constitute a representative sample of obstetrics and gynecology research and the 14 indicators are sufficient to assess reproducibility.
    Stated in the Methods description within the abstract; if the sampling frame or indicator set is biased, the percentages do not generalize.

pith-pipeline@v0.9.0 · 5796 in / 1224 out tokens · 19200 ms · 2026-05-24T20:43:14.576938+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages

  1. [1]

    Reproducibility vs

    Plesser HE. Reproducibility vs. Replicability: A Brief History of a Confused Terminology. Frontiers in Neuroinformatics ​ . 2018;11. doi: ​ 10.3389/fninf.2017.00076

  2. [2]

    Impact of data transparency: Scientific publications

    Joshi M, Bhardwaj P. Impact of data transparency: Scientific publications. ​ Perspect Clin Res ​ . 2018;9(1):31-36

  3. [3]

    Novelty in science should not come at the cost of reproducibility

    Holding AN. Novelty in science should not come at the cost of reproducibility. ​ FEBS J ​ . 2019;103:843

  4. [4]

    Increasing value and reducing waste in research design, conduct, and analysis

    Ioannidis JPA, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. ​ Lancet ​ . 2014;383(9912):166-175

  5. [5]

    a nice little earner

    Jack A. Tamiflu: “a nice little earner.” ​ BMJ ​ . 2014;348(apr09 2):g2524-g2524. doi: ​ 10.1136/bmj.g2524

  6. [6]

    Tamiflu: the battle for secret drug data

    Payne D. Tamiflu: the battle for secret drug data. ​ BMJ ​ . 2012;345:e7303

  7. [7]

    Complications: tracking down the data on oseltamivir

    Cohen D. Complications: tracking down the data on oseltamivir. ​ BMJ ​ . 2009;339:b5387

  8. [8]

    https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD008965.pub4/full ​

  9. [9]

    An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014-2017)

    Hardwicke TE, Wallach JD, Kidwell M, Ioannidis J. An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014-2017). April

  10. [10]

    doi: ​ 10.31222/osf.io/6uhg5

  11. [11]

    Genome-wide Estrogen Receptor-α activation is sustained, not cyclical

    Holding AN, Cullen AE, Markowetz F. Genome-wide Estrogen Receptor-α activation is sustained, not cyclical. ​ Elife ​ . 2018;7. doi: ​ 10.7554/eLife.40854

  12. [12]

    Cofactor Dynamics and Sufficiency in Estrogen Receptor–Regulated Transcription

    Shang Y, Hu X, DiRenzo J, Lazar MA, Brown M. Cofactor Dynamics and Sufficiency in Estrogen Receptor–Regulated Transcription. ​ Cell ​ . 2000;103(6):843-852

  13. [13]

    Estrogen receptor-alpha directs ordered, cyclical, and combinatorial recruitment of cofactors on a natural target promoter

    Métivier R, Penot G, Hübner MR, et al. Estrogen receptor-alpha directs ordered, cyclical, and combinatorial recruitment of cofactors on a natural target promoter. ​ Cell ​ . 2003;115(6):751-763

  14. [14]

    Guidelines for reporting meta-epidemiological methodology research

    Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med ​ . 2017;22(4):139-142

  15. [15]

    Repeatability of published microarray gene expression analyses

    Ioannidis JPA, Allison DB, Ball CA, et al. Repeatability of published microarray gene expression analyses. ​ Nat Genet ​ . 2009;41(2):149-155

  16. [16]

    Public availability of published research data in high-impact journals

    Alsheikh-Ali AA, Qureshi W, Al-Mallah MH, Ioannidis JPA. Public availability of published research data in high-impact journals. ​ PLoS One ​ . 2011;6(9):e24357

  17. [17]

    Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition

    Hardwicke TE, Mathur MB, MacDonald K, et al. Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. ​ R Soc Open Sci ​ . 2018;5(8):180448

  18. [18]

    Data sharing in PLOS ONE: An analysis of Data Availability Statements

    Federer LM, Belter CW, Joubert DJ, et al. Data sharing in PLOS ONE: An analysis of Data Availability Statements. ​ PLoS One ​ . 2018;13(5):e0194768

  19. [19]

    Data availability for industry sponsored trials: what should medical journals require? ​ BMJ ​

    Steinbrook R, Kassirer JP. Data availability for industry sponsored trials: what should medical journals require? ​ BMJ ​ . 2010;341:c5391

  20. [20]

    1,500 scientists lift the lid on reproducibility

    Baker M. 1,500 scientists lift the lid on reproducibility. ​ Nature ​ . 2016;533(7604):452-454

  21. [21]

    Preparing and Publishing a Scientific Manuscript

    Jirge PR. Preparing and Publishing a Scientific Manuscript. ​ J Hum Reprod Sci ​ . 2017;10(1):3-9

  22. [22]

    Tools and techniques for computational reproducibility

    Piccolo SR, Frampton MB. Tools and techniques for computational reproducibility. Gigascience ​ . 2016;5(1):30

  23. [23]

    The preregistration revolution

    Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. ​ Proc Natl Acad Sci U S A ​ . 2018;115(11):2600-2606

  24. [24]

    Systematic review: Outcome reporting bias is a problem in high impact factor neurology journals

    Howard B, Scott JT, Blubaugh M, Roepke B, Scheckel C, Vassar M. Systematic review: Outcome reporting bias is a problem in high impact factor neurology journals. ​ PLoS One ​ . 2017;12(7):e0180986

  25. [25]

    The Problem With P-Hacking

    Dodson TB. The Problem With P-Hacking. ​ Journal of Oral and Maxillofacial Surgery ​ . 2019;77(3):459-460. doi: ​ 10.1016/j.joms.2018.12.034

  26. [26]

    Congress

    United States. Congress. House. Committee on Energy and Commerce. ​ Food and Drug Administration Amendments Act of 2007: Report Together with Additional Views (to Accompany H.R

  27. [27]

    Registration, results reporting, and publication bias of clinical trials supporting FDA approval of neuropsychiatric drugs before and after FDAAA: a retrospective cohort study

    Zou CX, Becker JE, Phillips AT, et al. Registration, results reporting, and publication bias of clinical trials supporting FDA approval of neuropsychiatric drugs before and after FDAAA: a retrospective cohort study. ​ Trials ​ . 2018;19(1). doi: ​ 10.1186/s13063-018-2957-0

  28. [28]

    Association of the FDA Amendment Act with trial registration, publication, and outcome reporting

    Phillips AT, Desai NR, Krumholz HM, Zou CX, Miller JE, Ross JS. Association of the FDA Amendment Act with trial registration, publication, and outcome reporting. ​ Trials ​ . 2017;18(1):333

  29. [29]

    Under-reporting of conflicts of interest among trialists: a cross-sectional study

    Rasmussen K, Schroll J, Gøtzsche PC, Lundh A. Under-reporting of conflicts of interest among trialists: a cross-sectional study. ​ J R Soc Med ​ . 2015;108(3):101-107

  30. [30]

    Conflict of interest disclosure in biomedical research: A review of current practices, biases, and the role of public registries in improving transparency

    Dunn AG, Coiera E, Mandl KD, Bourgeois FT. Conflict of interest disclosure in biomedical research: A review of current practices, biases, and the role of public registries in improving transparency. ​ Res Integr Peer Rev ​ . 2016;1. doi: ​ 10.1186/s41073-016-0006-7

  31. [31]

    Conflict of interest reporting by authors involved in promotion of off-label drug use: an analysis of journal disclosures

    Kesselheim AS, Wang B, Studdert DM, Avorn J. Conflict of interest reporting by authors involved in promotion of off-label drug use: an analysis of journal disclosures. ​ PLoS Med ​ . 2012;9(8):e1001280

  32. [32]

    Reporting of conflicts of interest from drug trials in Cochrane reviews: cross sectional study

    Roseman M, Turner EH, Lexchin J, Coyne JC, Bero LA, Thombs BD. Reporting of conflicts of interest from drug trials in Cochrane reviews: cross sectional study. ​ BMJ ​ . 2012;345:e5155

  33. [33]

    Sunshine Act: shedding light on inaccurate disclosures at a gynecologic annual meeting

    Thompson JC, Volpe KA, Bridgewater LK, et al. Sunshine Act: shedding light on inaccurate disclosures at a gynecologic annual meeting. ​ Am J Obstet Gynecol ​ . 2016;215(5):661.e1-e661.e7

  34. [34]

    Transparency of industry payments needed in clinical practice guidelines

    Vassar M, Bibens M, Wayant C. Transparency of industry payments needed in clinical practice guidelines. ​ BMJ Evid Based Med ​ . 2019;24(1):8-9

  35. [35]

    Financial Conflicts of Interest Among Authors of Urology Clinical Practice Guidelines

    Carlisle A, Bowers A, Wayant C, Meyer C, Vassar M. Financial Conflicts of Interest Among Authors of Urology Clinical Practice Guidelines. ​ Eur Urol ​ . 2018;74(3):348-354

  36. [36]

    Evaluating Industry Payments Among Dermatology Clinical Practice Guidelines Authors

    Checketts JX, Sims MT, Vassar M. Evaluating Industry Payments Among Dermatology Clinical Practice Guidelines Authors. ​ JAMA Dermatol ​ . 2017;153(12):1229-1235

  37. [37]

    doi: ​ 10.1126/science.aab2374

  38. [38]

    Repeat: a framework to assess empirical reproducibility in biomedical research

    McIntosh LD, Juehne A, Vitale CRH, et al. Repeat: a framework to assess empirical reproducibility in biomedical research. ​ BMC Med Res Methodol ​ . 2017;17(1):143

  39. [39]

    doi: ​ 10.1097/SAP.0000000000001476

  40. [40]

    Reporting guideline and clinical trial registration requirements in gastroenterology and hepatology journals

    Jorski A, Scott J, Heavener T, Vassar M. Reporting guideline and clinical trial registration requirements in gastroenterology and hepatology journals. ​ Int J Evid Based Healthc ​ . 2018;16(2):119-127

  41. [41]

    Trial registration and adherence to reporting guidelines in cardiovascular journals

    Sims MT, Bowers AM, Fernan JM, Dormire KD, Herrington JM, Vassar M. Trial registration and adherence to reporting guidelines in cardiovascular journals. ​ Heart ​ . 2018;104(9):753-759

  42. [42]

    The availability of research data declines rapidly with article age

    Vines TH, Albert AYK, Andrew RL, et al. The availability of research data declines rapidly with article age. ​ Curr Biol ​ . 2014;24(1):94-97

  43. [43]

    Populating the Data Ark: An attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles

    Hardwicke TE, Ioannidis JPA. Populating the Data Ark: An attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. ​ PLoS One ​ . 2018;13(8):e0201856. Figure