{"id":"577b060c-73d3-4b27-b53a-e035b8e172a8","arxiv_id":"2505.21590","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Only 25.87% of 460 R scripts from OSF projects executed without error in automatically reconstructed Docker environments.","lead":"This paper reports that when 460 R scripts from OSF social-science projects are run in automatically built Docker containers, only about 26% finish without error. It also introduces a pipeline that infers R dependencies from source code, and uses it to identify common reproducibility failures.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline rate equates 'script exits without error' with 'reproduced the analysis'; because no successful run's outputs are compared to the publication's reported figures or tables, 25.87% overstates computational reproducibility.","rationale":"The paper's contribution is an automated pipeline plus a reproducibility rate. The number 25.87% is the central empirical result and the basis of the conclusion. The pipeline itself and the error taxonomy are transparent and potentially reusable; I would not object to those. But the rate's interpretation depends entirely on equating successful execution with reproduction of results. The reader's weakest assumption identifies exactly this, and I agree. The proposed output-comparison test is decisive: if successful runs do not reproduce the papers' numbers, the headline claim is not supported; if they do, the concern is resolved. A secondary concern about sample representativeness (StatCodeSearch OSF projects in social sciences and psychology) is real but less central because the paper is reasonably careful to describe the sample and its retrieval losses. Similarly, flowR's dependency misdetection is acknowledged as a limitation and affects the failure taxonomy, not the definition of success. Therefore the verdict remains CONDITIONAL, with no change needed.","tokens_in":7247,"tokens_out":4082,"duration_ms":47831,"concrete_test":"Select 30 of the 119 'successful' scripts whose OSF projects are linked to published papers with at least one reported numerical result (e.g., a regression table or figure value). Re-run each project through osf-to-binder exactly as described, capture the complete output (console logs and all generated files/figures), and independently compare it to the paper's key reported result. Pre-register a matching criterion (e.g., same sign and within 5% for at least one key statistic, or exact match for a deterministic figure). If a large share of sampled runs fails to match (say, more than 20%), the no-error success criterion overstates reproducibility and the headline should be reframed as 'executed without error' rather than 'computationally reproducible.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central number—119/460 scripts (25.87%) 'completed successfully without error'—is presented in the abstract and conclusion as the computational reproducibility rate of the sampled R supplements. But the pipeline's success predicate is entirely execution-oriented: a script succeeds if it runs to completion without a critical error inside the automatically built repo2docker container. The Results section reports no comparison of any output artifact (figure, table, regression coefficient, data file) with the output reported in the associated publication. This is not a minor omission: with only an exit-status criterion, scripts that silently produce wrong numbers, empty plots, or no artifacts are counted as reproducible. The tension is internal: the Introduction adopts Claerbout and Barba's standard that reproducibility requires 'the complete software development environment and the complete set of instructions which generated the figures,' and the paper's own research question asks about 'computational reproducibility of the submissions.' A rate of error-free execution in one reconstructed environment is a different, weaker quantity. The pipeline may still be a useful screening tool, but the paper's framing treats 25.87% as the reproducibility rate itself. The acknowledged limitations—flowR misses dynamic imports; execution halts at the first error; container R and package versions differ from the original environment—all reinforce that even a successful run is not evidence that the authors' original results were regenerated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper develops osf-to-binder, an automated pipeline that downloads R projects from OSF using the StatCodeSearch dataset, statically infers dependencies with flowR, builds Docker containers via repo2docker, and executes the contained R scripts. Of 296 projects, 32 were no longer retrievable; of the remaining 264, 15 failed containerization, leaving 460 scripts from 249 projects that were executed. The paper reports that 119 scripts (25.87%) exited without critical errors and that failures cluster in missing packages (26.1%), invalid file or directory paths (19.1%), and missing objects or functions (18.2%). On this basis, it recommends best practices for dependency declaration, relative paths, and non-interactive execution, and it compares its success rate with earlier studies of Jupyter notebooks and Dataverse R code.","tokens_in":7406,"tokens_out":5867,"duration_ms":69617,"significance":"If the headline finding is taken at face value, it provides a useful large-scale snapshot of R supplements on OSF and a publicly available pipeline for automated re-execution, which is a concrete contribution to reproducibility tooling. The manuscript is candid about several pipeline limitations, including flowR's incomplete dynamic-import detection, single-error halting, and GitHub size constraints. However, because the success criterion is error-free execution in an automatically constructed container rather than agreement of outputs with the published analyses, the figure of 25.87% is at best an upper-bound estimate of computational reproducibility, not a direct measurement. This distinction is load-bearing for the paper's central claim and needs to be corrected in the framing and title.","major_comments":[{"comment":"The paper's central claim that 25.87% of R scripts are computationally reproducible is not supported by the reported measurement. The success predicate is that a script completed successfully without critical errors inside a repo2docker container; no comparison is reported between any output artifact (figure, table, coefficient, or data file) and the corresponding output in the associated publication. Under this criterion, a script that silently produces wrong numbers or empty plots is counted as reproducible. This conflicts with the standard quoted in the Introduction, which requires the complete software development environment and the complete set of instructions that generated the figures. The abstract and conclusion should replace computational reproducibility with a narrower term such as exit-status reproducibility in an automatically reconstructed container, and the paper should report an output-validation subsample to calibrate this surrogate.","section":"Results, Code execution within containerised environments"},{"comment":"The headline rate conditions on successful containerization, and this conditioning should be explicit. The 460-script denominator excludes 35 scripts from 15 projects that failed to containerize, and the 264-project sample excludes 32 projects (63 files) that were no longer retrievable. Counting containerization failures as non-reproducible changes the rate to 119/495 = 24.0%; further counting the inaccessible files as failures gives 119/558 = 21.3%. At a minimum, the paper should report sensitivity analyses with these alternative denominators and state clearly that 25.87% is the success rate conditional on a successful automatic environment build.","section":"Results, Containerisation success and failures and Code execution within containerised environments"},{"comment":"The comparison with Samuel and Mietchen (2024) is not statistically grounded. The 11.6% figure is conditional on notebooks for which all declared dependencies could be installed, while the 25.87% figure is conditional on successful containerization of OSF projects; the selection mechanisms are only loosely analogous. The paper gives no confidence intervals, no hypothesis test, and no adjustment for the different attrition funnels (retrieval, build, dependency installation). The qualitative statement that both studies point to persistent reproducibility barriers is safe, but the implied quantitative comparison should be removed unless accompanied by a proper statistical analysis with matched denominators.","section":"Conclusion, Cross-domain comparisons"},{"comment":"The two-level error classification is produced by regular expressions followed by semantic grouping, but the paper reports no inter-rater reliability, no codebook, and no manual validation of a random sample. Since the percentage breakdown in Figure 2 is used to motivate best-practice recommendations, an unvalidated taxonomy is a load-bearing limitation. The authors should report at least a random-sample audit of the automatic labels and the level of agreement with human annotators.","section":"Results, paragraph beginning To analyse and interpret script execution failures"}],"minor_comments":[{"comment":"The branch labeled Too large to publish=23 is ambiguous because the text reports that these scripts were nevertheless included in the execution analysis; the figure should clarify that this branch refers only to GitHub publication, not to script execution.","section":"Figure 1"},{"comment":"The phrase only 264 were still retrievable reads as if 264 scripts were retrievable; the paper should say 264 projects.","section":"Abstract"},{"comment":"The table header would be clearer as Projects (Percentage) and should explicitly state that the denominator is the 264 analysed projects.","section":"Table 1"},{"comment":"The quotation marks around error messages are typographically inconsistent, for example in unable to install packages, lazy loading failed, and package or namespace load failed.","section":"Results, Package Installation Failure paragraph"},{"comment":"The manuscript does not state whether script execution had a timeout or resource limits; if scripts that hang were killed, the success/failure distinction depends on this operational detail and should be reported.","section":"Methodology, Code Execution"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a descriptive measurement study rather than a technical AI contribution, so the editor may wish to consider scope fit for AAAI; the main technical risk is the overclaiming from a surrogate outcome, which is fixable by reframing and by adding sensitivity analyses. The related-work coverage and the public availability of the pipeline are adequate strengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead it. The genuinely new thing is the shipped infrastructure: osf-to-binder, an automated pipeline that pulls OSF projects, statically extracts R dependencies with flowR, builds Docker containers with repo2docker, runs the scripts, logs results, and publishes images and repos. Applied to 264 retrievable StatCodeSearch projects, it gives a measured execution-success rate of 119/460 scripts (25.87%) and a sensible failure taxonomy: missing packages, invalid paths, missing objects, shared-library errors. That is a real, transferable contribution and the first large-ish data point on R supplements from OSF in the social sciences. The paper earns credit for shipping code, data, and containers, and the limitations section is candid about flowR gaps, single-error halting, and size constraints.\n\nThe soft spot is framing. The success criterion is \"script exits without a critical error in the reconstructed container.\" No successful output is compared with the associated paper's figures, tables, or coefficients. So 25.87% is an execution-success rate under automated environment reconstruction, not the reproducibility rate the introduction defines with Claerbout and Barba. If anything, it likely overstates reproducibility because a script can silently produce wrong numbers and still be counted as successful. Prior work by Trisovic et al. used a similar completion-without-error definition, so the cross-study comparisons are fair enough on that operationalization, but the paper's own abstract and conclusion should not call it computational reproducibility without qualification.\n\nAlso worth noting: the headline number has no confidence intervals, no sensitivity analysis for flowR's missed imports, no inter-rater validation of the error taxonomy, and the StatCodeSearch sample is not shown to be representative of OSF R projects in general. These are measurement-validity concerns rather than fatal flaws. The arithmetic is internally consistent, and the circularity worry is not real: 25.87% is a raw count, not a fitted parameter.\n\nWho is this for? People building reproducibility tooling, journal and policy folks considering cheap automated verification, and empirical reproducibility researchers. It deserves a serious referee. The paper would be stronger with output-level validation on a subsample, uncertainty bounds, and careful wording that distinguishes execution success from output verification. I would cite it as an execution-reproducibility data point, with that caveat stated.\n\nRecommendation: send it to peer review; it is a solid empirical infrastructure paper that needs revision, not desk rejection.","headline":"A useful, honestly-built automated R execution study whose headline 25.87% rate is a container-execution success rate, not a validated reproducibility rate.","tokens_in":8009,"tokens_out":1707,"would_cite":true,"duration_ms":21850,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that when computational environments are rebuilt automatically from source code, only 25.87% of 460 R scripts shared on OSF complete without error, with most failures traced to missing packages, invalid paths, and…","keywords":["computational reproducibility","R scripts","Open Science Framework","containerisation","dependency inference","Docker","automated verification","code supplements"],"falsifier":"Run the 119 'successful' scripts again in their own containers, capture all output artifacts, and compare them with the numbers, tables, and figures in the corresponding published papers; if a non-trivial share of scripts produce different results or no reproducible artifact, the 25.87% success rate overstates computational reproducibility.","tokens_in":6991,"feed_emoji":"🐳","tokens_out":8029,"duration_ms":75550,"temperature":0.7,"pith_summary":"This paper asks how often R code supplements shared alongside social-science publications actually rerun when someone else tries to execute them. To answer it, the authors built a fully automated pipeline that reconstructs a computational environment from a project's source code alone, using static analysis to infer packages and containerisation to run the scripts in isolation. Applied to 296 OSF projects from the StatCodeSearch dataset, 264 of which were still retrievable, the pipeline executed 460 R scripts and found that only 119 (25.87%) completed without a critical error. The authors classify the failures and show they concentrate in missing packages, invalid file or directory paths, missing objects or functions, and environment-specific issues. If correct, the result implies that most shared R supplements will not rerun automatically, and that automated dependency inference plus containerisation can locate the bottlenecks at scale.","feed_headline":"Only 26% of shared R scripts rerun cleanly","feed_subtitle":"Testing 460 R supplements from OSF in Docker containers finds most fail on missing packages, bad paths, and missing objects","key_machinery":"The load-bearing mechanism is the osf-to-binder pipeline, which combines flowR, a static dataflow analyser that reads R source code and extracts package dependencies; an auto-generated DESCRIPTION file that declares those dependencies; repo2docker, a tool that turns a project directory into a Docker image; and execution of every R script inside the built container. The pipeline does the work of turning raw source code into an executable environment without a human-written dependency list, so the observed success and failure rates depend entirely on this automated reconstruction chain. It also logs execution results and publishes the resulting images and launch links, making each verification repeatable by an outside reader.","core_discovery":"The central discovery is a measured reproducibility rate for R code supplements on OSF: after automatic environment reconstruction, 119 of 460 executed R scripts (25.87%) ran to completion without a critical error, while 341 (74.13%) failed. Among 249 containerised projects, 40 projects had all of their scripts succeed (16.06% of projects), 34 were partially reproducible, and 175 produced no successful script. The paper further identifies that 98.8% of the analysed projects had no formal dependency description file (DESCRIPTION, renv.lock, Dockerfile, or similar), so the pipeline had to infer dependencies from code. Failure analysis attributes 26.1% of failing scripts to missing packages, 19.1% to invalid file or directory paths, and 18.2% to missing objects or functions, with the remaining failures spread across shared-library, installation, file-read, and other errors. The authors present these numbers as evidence that scalable automated verification of reproducibility is feasible and that specific, fixable practices—explicit dependency declarations, relative paths, non-interactive scripts—would remove most barriers.","pith_inferences":["The paper's success criterion is error-free exit, not output verification; a natural next step would be to check whether the 119 'successful' scripts actually reproduce the figures or tables in their associated publications, and if output comparison lowers the rate, the true reproducibility rate would be below 25.87%.","Because the failure taxonomy is sharp (missing packages 26.1%, invalid paths 19.1%, missing objects or functions 18.2%), a reasonable inference is that author-side tooling—auto-declaring packages, rewriting absolute paths, and running scripts non-interactively—could recover a large fraction of failures before any human review.","The paper's finding that 63 referenced R files were already gone and only 49 of 296 projects had file-preserving registrations suggests that platform-level mandating of snapshot registrations would be a precondition for reliable retrospective reproducibility assessment.","The cross-domain comparison with Jupyter notebook studies invites a testable extension: run the same pipeline on R supplements from other repositories or journal replication archives to see whether the 25.87% figure is specific to OSF or general to R."],"forward_implications":["If the 25.87% figure holds, researchers should expect roughly three of every four shared R scripts to fail when their environment is rebuilt automatically, so code supplements cannot be assumed reproducible without verification.","The failure categories imply that a small set of author-side fixes—declaring all packages with versions, using relative paths, and avoiding GUI-dependent functions—would address the majority of observed failures.","Automated dependency inference and containerisation can act as a screening tool at submission time, because the pipeline identifies which scripts fail and why, without manual environment setup.","The published containers and launch links make each verification repeatable by any reader, moving reproducibility checking from a one-off manual exercise to an open, inspectable artifact.","The comparison with earlier Jupyter notebook studies suggests that automated reproducibility success rates in R (25.87%) may currently be around double those for Python notebooks, though still low in absolute terms."],"supporting_citations":[{"why":"Supplies the StatCodeSearch dataset of R scripts from OSF projects, which defines the sample for the study.","marker":"Diera et al. 2023"},{"why":"Supplies flowR, the static dataflow analyser the pipeline uses to extract package dependencies from R source code.","marker":"Sihler and Tichy 2024"},{"why":"Supplies repo2docker, the tool that turns project directories into Docker images for script execution.","marker":"Forde et al. 2018"},{"why":"Earlier large-scale execution of R code from a public data repository; its 74% failure rate is the direct baseline this study extends.","marker":"Trisovic et al. 2022"},{"why":"Large-scale Jupyter notebook reproducibility study (11.6% error-free) used for the cross-domain comparison.","marker":"Samuel and Mietchen 2024"},{"why":"Large-scale Jupyter notebook study (24% error-free) cited alongside Samuel and Mietchen for comparison.","marker":"Pimentel et al. 2019"},{"why":"Manual restoration study of 30 papers showing partial reproducibility; the contrast motivates the automated approach.","marker":"Chung-hong Chan, Tim Schatto-Eckrodt, and Johannes Gruber 2024"},{"why":"Manual assessment showing few social-science articles share analysis scripts; establishes the need for large-scale reproducibility measurement.","marker":"Hardwicke et al. 2020"}],"fun_headline_variants":["Only 1 in 4 shared R scripts runs without error","R scripts on OSF: 74% fail automated reproducibility test","98.8% of R projects skip dependency tracking","Docker test: most R supplements can't be reproduced","Missing packages and paths doom most shared R code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that finishing a script without a critical error inside an automatically built container counts as computational reproducibility; the paper does not compare any successful script's outputs with the figures, tables, or results in the associated publication.","fun_headline_variants_meta":{"raw":{"variants":["Only 1 in 4 shared R scripts runs without error","R scripts on OSF: 74% fail automated reproducibility test","98.8% of R projects skip dependency tracking","Docker test: most R supplements can't be reproduced","Missing packages and paths doom most shared R code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2817,"prompt_tokens":944,"completion_tokens":1873,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1792}},"tokens_in":560,"tokens_out":1873,"duration_ms":13440,"temperature":1.0,"reasoning_tokens":1792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:33:33.330318+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 119 'successful' scripts again in their own containers, capture all output artifacts, and compare them with the numbers, tables, and figures in the corresponding published papers; if a non-trivial share of scripts produce different results or no reproducible artifact, the 25.87% success rate overstates computational reproducibility.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies flowR, the static dataflow analyser the pipeline uses to extract package dependencies from R source code."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies repo2docker, the tool that turns project directories into Docker images for script execution."},{"cited_title":"K.; Pasquier, T.; and Crosas, M","cited_arxiv_id":null,"evidence_quote":"Earlier large-scale execution of R code from a public data repository; its 74% failure rate is the direct baseline this study extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Large-scale Jupyter notebook reproducibility study (11.6% error-free) used for the cross-domain comparison."},{"cited_title":"F.; Murta, L.; Braganholo, V.; and Freire, J","cited_arxiv_id":null,"evidence_quote":"Large-scale Jupyter notebook study (24% error-free) cited alongside Samuel and Mietchen for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Manual restoration study of 30 papers showing partial reproducibility; the contrast motivates the automated approach."},{"cited_title":"E.; Wallach, J","cited_arxiv_id":null,"evidence_quote":"Manual assessment showing few social-science articles share analysis scripts; establishes the need for large-scale reproducibility measurement."}],"review_version":1}