Pith. sign in

REVIEW 4 major objections 5 minor 25 references

Computational Reproducibility of R Code Supplements on OSF

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that when computational environments are rebuilt automatically from source code, only 25.87% of 460 R scripts shared on OSF complete without error, with most failures traced to missing packages, invalid paths, and…

desk verdict A useful, honestly-built automated R execution study whose headline 25.87% rate is a container-execution success rate, not a validated reproducibility rate. read the letter →

arxiv 2505.21590 v1 pith:6AP2KLMN submitted 2025-05-27 cs.CY cs.SE

Lorraine Saju , Tobias Holtdirk , Meetkumar Pravinbhai Mangroliya , Arnim Bleier This is my paper
classification cs.CYcs.SE
keywords computationalreproducibilityRscriptsOpenScienceFrameworkcontainerisationdependencyinferenceDockerautomatedverificationcodesupplements
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how often R code supplements shared alongside social-science publications actually rerun when someone else tries to execute them. To answer it, the authors built a fully automated pipeline that reconstructs a computational environment from a project's source code alone, using static analysis to infer packages and containerisation to run the scripts in isolation. Applied to 296 OSF projects from the StatCodeSearch dataset, 264 of which were still retrievable, the pipeline executed 460 R scripts and found that only 119 (25.87%) completed without a critical error. The authors classify the failures and show they concentrate in missing packages, invalid file or directory paths, missing objects or functions, and environment-specific issues. If correct, the result implies that most shared R supplements will not rerun automatically, and that automated dependency inference plus containerisation can locate the bottlenecks at scale.

What carries the argument

The load-bearing mechanism is the osf-to-binder pipeline, which combines flowR, a static dataflow analyser that reads R source code and extracts package dependencies; an auto-generated DESCRIPTION file that declares those dependencies; repo2docker, a tool that turns a project directory into a Docker image; and execution of every R script inside the built container. The pipeline does the work of turning raw source code into an executable environment without a human-written dependency list, so the observed success and failure rates depend entirely on this automated reconstruction chain. It also logs execution results and publishes the resulting images and launch links, making each verification repeatable by an outside reader.

What would settle it

Run the 119 'successful' scripts again in their own containers, capture all output artifacts, and compare them with the numbers, tables, and figures in the corresponding published papers; if a non-trivial share of scripts produce different results or no reproducible artifact, the 25.87% success rate overstates computational reproducibility.

Watch

Extended reading notes

Core claim

The central discovery is a measured reproducibility rate for R code supplements on OSF: after automatic environment reconstruction, 119 of 460 executed R scripts (25.87%) ran to completion without a critical error, while 341 (74.13%) failed. Among 249 containerised projects, 40 projects had all of their scripts succeed (16.06% of projects), 34 were partially reproducible, and 175 produced no successful script. The paper further identifies that 98.8% of the analysed projects had no formal dependency description file (DESCRIPTION, renv.lock, Dockerfile, or similar), so the pipeline had to infer dependencies from code. Failure analysis attributes 26.1% of failing scripts to missing packages, 19.1% to invalid file or directory paths, and 18.2% to missing objects or functions, with the remaining failures spread across shared-library, installation, file-read, and other errors. The authors present these numbers as evidence that scalable automated verification of reproducibility is feasible and that specific, fixable practices—explicit dependency declarations, relative paths, non-interactive scripts—would remove most barriers.

Load-bearing premise

The load-bearing premise is that finishing a script without a critical error inside an automatically built container counts as computational reproducibility; the paper does not compare any successful script's outputs with the figures, tables, or results in the associated publication.

Editorial extensions

If this is right

  • If the 25.87% figure holds, researchers should expect roughly three of every four shared R scripts to fail when their environment is rebuilt automatically, so code supplements cannot be assumed reproducible without verification.
  • The failure categories imply that a small set of author-side fixes—declaring all packages with versions, using relative paths, and avoiding GUI-dependent functions—would address the majority of observed failures.
  • Automated dependency inference and containerisation can act as a screening tool at submission time, because the pipeline identifies which scripts fail and why, without manual environment setup.
  • The published containers and launch links make each verification repeatable by any reader, moving reproducibility checking from a one-off manual exercise to an open, inspectable artifact.
  • The comparison with earlier Jupyter notebook studies suggests that automated reproducibility success rates in R (25.87%) may currently be around double those for Python notebooks, though still low in absolute terms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's success criterion is error-free exit, not output verification; a natural next step would be to check whether the 119 'successful' scripts actually reproduce the figures or tables in their associated publications, and if output comparison lowers the rate, the true reproducibility rate would be below 25.87%.
  • Because the failure taxonomy is sharp (missing packages 26.1%, invalid paths 19.1%, missing objects or functions 18.2%), a reasonable inference is that author-side tooling—auto-declaring packages, rewriting absolute paths, and running scripts non-interactively—could recover a large fraction of failures before any human review.
  • The paper's finding that 63 referenced R files were already gone and only 49 of 296 projects had file-preserving registrations suggests that platform-level mandating of snapshot registrations would be a precondition for reliable retrospective reproducibility assessment.
  • The cross-domain comparison with Jupyter notebook studies invites a testable extension: run the same pipeline on R supplements from other repositories or journal replication archives to see whether the 25.87% figure is specific to OSF or general to R.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper develops osf-to-binder, an automated pipeline that downloads R projects from OSF using the StatCodeSearch dataset, statically infers dependencies with flowR, builds Docker containers via repo2docker, and executes the contained R scripts. Of 296 projects, 32 were no longer retrievable; of the remaining 264, 15 failed containerization, leaving 460 scripts from 249 projects that were executed. The paper reports that 119 scripts (25.87%) exited without critical errors and that failures cluster in missing packages (26.1%), invalid file or directory paths (19.1%), and missing objects or functions (18.2%). On this basis, it recommends best practices for dependency declaration, relative paths, and non-interactive execution, and it compares its success rate with earlier studies of Jupyter notebooks and Dataverse R code.

Significance. If the headline finding is taken at face value, it provides a useful large-scale snapshot of R supplements on OSF and a publicly available pipeline for automated re-execution, which is a concrete contribution to reproducibility tooling. The manuscript is candid about several pipeline limitations, including flowR's incomplete dynamic-import detection, single-error halting, and GitHub size constraints. However, because the success criterion is error-free execution in an automatically constructed container rather than agreement of outputs with the published analyses, the figure of 25.87% is at best an upper-bound estimate of computational reproducibility, not a direct measurement. This distinction is load-bearing for the paper's central claim and needs to be corrected in the framing and title.

major comments (4)
  1. [Results, Code execution within containerised environments] The paper's central claim that 25.87% of R scripts are computationally reproducible is not supported by the reported measurement. The success predicate is that a script completed successfully without critical errors inside a repo2docker container; no comparison is reported between any output artifact (figure, table, coefficient, or data file) and the corresponding output in the associated publication. Under this criterion, a script that silently produces wrong numbers or empty plots is counted as reproducible. This conflicts with the standard quoted in the Introduction, which requires the complete software development environment and the complete set of instructions that generated the figures. The abstract and conclusion should replace computational reproducibility with a narrower term such as exit-status reproducibility in an automatically reconstructed container, and the paper should report an output-validation subsample to calibrate this surrogate.
  2. [Results, Containerisation success and failures and Code execution within containerised environments] The headline rate conditions on successful containerization, and this conditioning should be explicit. The 460-script denominator excludes 35 scripts from 15 projects that failed to containerize, and the 264-project sample excludes 32 projects (63 files) that were no longer retrievable. Counting containerization failures as non-reproducible changes the rate to 119/495 = 24.0%; further counting the inaccessible files as failures gives 119/558 = 21.3%. At a minimum, the paper should report sensitivity analyses with these alternative denominators and state clearly that 25.87% is the success rate conditional on a successful automatic environment build.
  3. [Conclusion, Cross-domain comparisons] The comparison with Samuel and Mietchen (2024) is not statistically grounded. The 11.6% figure is conditional on notebooks for which all declared dependencies could be installed, while the 25.87% figure is conditional on successful containerization of OSF projects; the selection mechanisms are only loosely analogous. The paper gives no confidence intervals, no hypothesis test, and no adjustment for the different attrition funnels (retrieval, build, dependency installation). The qualitative statement that both studies point to persistent reproducibility barriers is safe, but the implied quantitative comparison should be removed unless accompanied by a proper statistical analysis with matched denominators.
  4. [Results, paragraph beginning To analyse and interpret script execution failures] The two-level error classification is produced by regular expressions followed by semantic grouping, but the paper reports no inter-rater reliability, no codebook, and no manual validation of a random sample. Since the percentage breakdown in Figure 2 is used to motivate best-practice recommendations, an unvalidated taxonomy is a load-bearing limitation. The authors should report at least a random-sample audit of the automatic labels and the level of agreement with human annotators.
minor comments (5)
  1. [Figure 1] The branch labeled Too large to publish=23 is ambiguous because the text reports that these scripts were nevertheless included in the execution analysis; the figure should clarify that this branch refers only to GitHub publication, not to script execution.
  2. [Abstract] The phrase only 264 were still retrievable reads as if 264 scripts were retrievable; the paper should say 264 projects.
  3. [Table 1] The table header would be clearer as Projects (Percentage) and should explicitly state that the denominator is the 264 analysed projects.
  4. [Results, Package Installation Failure paragraph] The quotation marks around error messages are typographically inconsistent, for example in unable to install packages, lazy loading failed, and package or namespace load failed.
  5. [Methodology, Code Execution] The manuscript does not state whether script execution had a timeout or resource limits; if scripts that hang were killed, the success/failure distinction depends on this operational detail and should be reported.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 25.87% figure is an empirical execution count, and no derived quantity is defined in terms of the inputs or justified solely by self-citation.

full rationale

No circularity found. The paper's central figure, 119/460 scripts (25.87%) completing without critical errors, is a raw count of execution outcomes produced by an independently described pipeline (flowR, repo2docker, container execution), not a quantity derived from an input by construction. The dependency-file statistics are descriptive observations about the sampled OSF projects, and the failure categories are post-hoc classifications of logged error messages rather than predictions fitted to the data. The paper does not rename a known result or import a uniqueness theorem from its own prior work. The self-citations that appear (Bleier 2025; Kohrs et al. 2023; Schoch et al. 2024) are general references on reproducibility practices and are not load-bearing for the empirical result. The main weakness is a measurement-validity concern: equating 'script exits without a critical error in the reconstructed container' with 'computational reproducibility' may overstate reproducibility because successful scripts' outputs are not compared with the figures or tables in the associated publications. That concern is about construct validity, not circularity, because the success criterion is not fitted to, or defined by, the reported 25.87% rate. External comparisons with Trisovic et al. (2022), Pimentel et al. (2019), and Samuel and Mietchen (2024) anchor the metric independently, and the limitations section explicitly acknowledges threats to the success criterion (e.g., flowR missing dynamic imports, single-error halting, version differences). Therefore the derivation chain is self-contained in the sense relevant to circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The study relies on several domain assumptions about sample representativeness, container-based execution, and the meaning of success. No free parameters or invented entities are introduced; the empirical rate is a direct count rather than a model fit.

assumptions (4)
  • domain assumption Execution in a repo2docker container with automatically generated DESCRIPTION dependencies is a representative proxy for the computational environment intended by the authors.
    Underlies all execution outcomes; if false, success and failure rates reflect pipeline artifacts rather than project reproducibility. See Methodology, Containerisation and Code Execution.
  • domain assumption StatCodeSearch code-comment pairs identify a representative sample of R projects on OSF.
    Generalization from 296 projects to OSF R supplements depends on this assumption. See Methodology, first two paragraphs.
  • domain assumption An R script that exits without a critical error has been computationally reproduced.
    No outputs are compared with the published results, yet the conclusion frames successful execution as reproducibility. See Results and Conclusion.
  • domain assumption Automatic dependency extraction via flowR can recover all required packages from static source code.
    If incomplete, container failures and package errors are overcounted; the paper's own limitations section admits this risk.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Computational Reproducibility of R Code Supplements on OSF." pith.science (2026). https://pith.science/paper/6AP2KLMN

@misc{pith2026250521590,
  author       = {Pith},
  title        = {Pith review of: Computational Reproducibility of R Code Supplements on OSF},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6AP2KLMN}},
  note         = {Machine review of arXiv:2505.21590}
}
read the original abstract

Computational reproducibility is fundamental to scientific research, yet many published code supplements lack the necessary documentation to recreate their computational environments. While researchers increasingly share code alongside publications, the actual reproducibility of these materials remains poorly understood. In this work, we assess the computational reproducibility of 296 R projects using the StatCodeSearch dataset. Of these, only 264 were still retrievable, and 98.8% lacked formal dependency descriptions required for successful execution. To address this, we developed an automated pipeline that reconstructs computational environments directly from project source code. Applying this pipeline, we executed the R scripts within custom Docker containers and found that 25.87% completed successfully without error. We conducted a detailed analysis of execution failures, identifying reproducibility barriers such as undeclared dependencies, invalid file paths, and system-level issues. Our findings show that automated dependency inference and containerisation can support scalable verification of computational reproducibility and help identify practical obstacles to code reuse and transparency in scientific research.

Figures

Figures reproduced from arXiv: 2505.21590 by the authors.

Figure 1
Figure 1. Our completely automated pipeline osf-to-binder (left) and the analysis results using this pipeline for the StatCodeSearch dataset (right). advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures. Following this goal, Chung-hong Chan, Tim Schatto￾Eckrodt, and Johannes Gruber (2024), among others, em￾phasi… view at source ↗
Figure 2
Figure 2. Breakdown of execution errors by category. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 20 canonical work pages

  1. [1]

    Arnold, B.; Bowler, L.; Gibson, S.; Herterich, P.; Higman, R.; Krystalli, A.; Morley, A.; O'Reilly, M.; Whitaker, K.; et al. 2019. The turing way: a handbook for reproducible data science. Zenodo

    work page 2019

  2. [2]

    Barba, L. A. 2018. Terminologies for Reproducible Research . ArXiv:1802.03311 [cs]

    arXiv 2018

  3. [3]

    Bleier, A. 2025. What is Computational Reproducibility? GESIS – Leibniz Institute for the Social Sciences, Cologne

    work page 2025

  4. [4]

    Boettiger, C. 2015. An introduction to Docker for reproducible research. ACM SIGOPS Operating Systems Review, 49(1): 71--79

    work page 2015

  5. [5]

    Chung-hong Chan ; Tim Schatto-Eckrodt ; and Johannes Gruber . 2024. What makes computational communication science (ir)reproducible? Computational Communication Research, 6(1): 1

    work page 2024

  6. [6]

    F.; and Karrenbach, M

    Claerbout, J. F.; and Karrenbach, M. 1992. Electronic documents give reproducible research a new meaning. In SEG technical program expanded abstracts 1992, 601--604. Society of Exploration Geophysicists

    work page 1992

  7. [7]

    Diera, A.; Dahou, A.; Galke, L.; Karl, F.; Sihler, F.; and Scherp, A. 2023. GenCodeSearchNet : A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding . ArXiv:2311.09707 [cs]

    arXiv 2023

  8. [8]

    Forde, J.; Head, T.; Holdgraf, C.; Panda, Y.; Nalvarete, G.; Ragan-Kelley, B.; and Sundell, E. 2018. Reproducible research environments with repo2docker

    work page 2018

Show all 25 references
  1. [9]

    E.; Wallach, J

    Hardwicke, T. E.; Wallach, J. D.; Kidwell, M. C.; Bendixen, T.; Cr \"u well, S.; and Ioannidis, J. P. 2020. An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014--2017). Royal Society open science, 7(2): 190806

    2020

  2. [10]

    E.; Auer, S.; Bannach-Brown, A.; Fiedler, S.; Haven, T

    Kohrs, F. E.; Auer, S.; Bannach-Brown, A.; Fiedler, S.; Haven, T. L.; Heise, V.; Holman, C.; Azevedo, F.; Bernard, R.; Bleier, A.; et al. 2023. Eleven strategies for making reproducible research and open science training the norm at research institutions. Elife, 12: e89736

    2023

  3. [11]

    Lasser, J. 2020. Creating an executable paper is a journey through Open Science. Communications Physics, 3(1): 143

    2020

  4. [12]

    National Academies of Sciences, E.; and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. ISBN 978-0-309-48616-3

    2019

  5. [13]

    N \"u st, D.; and Eglen, S. J. 2021. CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility. F1000Research, 10: 253

    2021

  6. [14]

    Peng, R. D. 2011. Reproducible research in computational science. Science, 334(6060): 1226--1227

    2011

  7. [15]

    F.; Murta, L.; Braganholo, V.; and Freire, J

    Pimentel, J. F.; Murta, L.; Braganholo, V.; and Freire, J. 2019. A large-scale study about quality and reproducibility of jupyter notebooks. In 2019 IEEE/ACM 16th international conference on mining software repositories (MSR), 507--517. IEEE

    2019

  8. [16]

    Ragan-Kelley, B.; Willing, C.; Akici, F.; Lippa, D.; Niederhut, D.; and Pacer, M. 2018. Binder 2.0-Reproducible, interactive, sharable environments for science at scale. In Proceedings of the 17th python in science conference, 113--120. F. Akici, D. Lippa, D. Niederhut, and M....

    2018

  9. [17]

    Rainey, C.; Roe, H.; Wang, Q.; and Zhou, H. 2025. Data and Code Availability in Political Science Publications from 1995 to 2022. Political Science & Politics, 1–7

    2025

  10. [18]

    Samuel, S.; and Mietchen, D. 2024. Computational reproducibility of Jupyter notebooks from biomedical publications. GigaScience, 13: giad113

    2024

  11. [19]

    K.; Nekrutenko, A.; Taylor, J.; and Hovig, E

    Sandve, G. K.; Nekrutenko, A.; Taylor, J.; and Hovig, E. 2013. Ten simple rules for reproducible computational research. PLoS computational biology, 9(10): e1003285

    2013

  12. [20]

    Schoch, D.; Chan, C.-h.; Wagner, C.; and Bleier, A. 2024. Computational reproducibility in computational social science. EPJ Data Science, 13(1): 75

    2024

  13. [21]

    Seibold, H.; Czerny, S.; Decke, S.; Dieterle, R.; Eder, T.; Fohr, S.; Hahn, N.; Hartmann, R.; Heindl, C.; Kopper, P.; et al. 2021. A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses. PLoS One, 16(6): e0251194

    2021

  14. [22]

    Sihler, F.; and Tichy, M. 2024. flowR : A Static Program Slicer for R . In Proceedings of the 39th IEEE / ACM International Conference on Automated Software Engineering , ASE '24, 2390--2393. New York, NY, USA: Association for Computing Machinery. ISBN 979-8-4007-1248-7

    2024

  15. [23]

    K.; Pasquier, T.; and Crosas, M

    Trisovic, A.; Lau, M. K.; Pasquier, T.; and Crosas, M. 2022. A large-scale study on research code quality and execution. Scientific Data, 9(1): 60

    2022

  16. [24]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  17. [25]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.