{"id":"c1431349-2ff2-40fd-ae7d-b1f05b69b140","arxiv_id":"2412.17859","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A replication study of 22 highly cited AI papers finds that sharing both code and data is strongly associated with successful reproduction (86% vs 33% for data-only).","lead":"This paper attempted to re-run 22 highly cited AI studies using the original code and data when available, and found that sharing both code and data led to successful replication in 86% of cases versus 33% for data-only studies. If the finding holds, it gives concrete evidence for open science mandates in AI and shifts attention from code documentation to data documentation.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Internal contradiction in 'The Replication Study': R4 code either was or was not reimplemented; if reimplemented, the 86% vs 33% claim does not measure code sharing.","rationale":"The reader identified the internal contradiction but selected the 40-hour time cap as the weakest assumption, leading to a CONDITIONAL verdict. In my view, the R4 reimplementation contradiction is more load-bearing: it threatens the construct validity of the main independent variable. If the R4 reproductions did not use the shared code, the paper does not measure what it claims, and the 86% versus 33% difference becomes uninterpretable. The time cap, in contrast, is a methodological limitation that the authors openly acknowledge and that affects both groups; it would bias the result only if R4 and R3 studies differ systematically in the time needed, which is plausible but not directly fatal. The reimplementation contradiction is a direct logical inconsistency that cannot be repaired without redoing the study or reclassifying the results. Therefore the appropriate verdict is REJECT, unless a re-analysis of the repository shows that the shared code was indeed run for all R4 successes. This is a concrete, checkable condition, and the authors should be asked to provide this evidence before the claim is accepted.","tokens_in":15372,"tokens_out":3377,"duration_ms":28230,"concrete_test":"Inspect the public replication repository (https://github.com/AIReproducibility2018) for the seven R4 studies. For each, determine whether the executed code is the original authors' code or a fresh implementation by comparing file contents, commit history, and README descriptions against the original open-source repositories (e.g., XGBoost for Chen and Guestrin 2016). If any R4 study used a reimplementation rather than the shared code, the 86% result does not support the code-sharing benefit. A definitive check is to see whether the repository's code for Rodriguez and Laio 2014 matches the Science paper's published code exactly; if not, the contradiction is resolved in favor of 'reimplemented from scratch.'","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—that sharing code and data raises independent reproducibility from 33% to 86%—rests entirely on the comparison between R4 (code+data) and R3 (data only) studies. But the paper contradicts itself about what was actually done for R4 studies. In 'The Replication Study,' the authors state: 'Hence, for R4 articles, we did not reimplement the code, except if parts of the code were missing.' A few paragraphs later they write: 'For R4 articles, we implemented methods and experiments from scratch.' These are mutually exclusive. If the latter is true, then the R4 'reproductions' did not run the authors' shared code; they were reimplementations from the article and data alone. In that case, the 86% figure measures the success of reimplementation, not the value of code availability, and the difference between R3 and R4 could be driven by confounds such as study difficulty, data quality, or documentation rather than by code sharing. The paper's conclusion that 'sharing both code and data publicly is extremely important' would be unsupported. This is an internal inconsistency about the manipulation itself, not a statistical quibble, and it must be resolved before the headline claim can be accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a systematic replication study of 30 highly cited AI articles from 2012, 2014, and 2016, of which 22 were actually attempted (7 with shared code and data, 15 with shared data only). The authors classify each attempted replication as Success, Partial Success, Failure, or No Result, and relate these outcomes to the reproducibility type and to 20 catalogued problem types. The central empirical claim is that sharing both code and data is strongly associated with reproducibility: 86% of R4 studies were fully or partly reproduced versus 33% of R3 studies. The paper also claims that data documentation quality correlates with success while code documentation quality does not, and it presents a logistic regression analysis intended to identify which problem types distinguish reproducible from irreproducible studies.","tokens_in":15575,"tokens_out":7175,"duration_ms":58739,"significance":"If the estimates were reliable, this study would be a valuable addition to the empirical reproducibility literature for AI, complementing work such as Raff (2019) and the large-scale psychology and economics replication studies. The study's strengths include a publicly released replication repository, a transparent protocol with a defined time budget, a fine-grained taxonomy of 20 problem types, and an unusually candid Limitations section that acknowledges the small sample and the arbitrary 40-hour cap. However, the headline comparison rests on very small counts, the protocol contains an unresolved internal contradiction about whether R4 code was re-run or reimplemented, and the secondary logistic regression is fit in-sample without validation. As a result, the paper is more suggestive than definitive, and the strength of the causal language in the abstract and conclusion is not supported by the evidence presented.","major_comments":[{"comment":"The protocol description contains a direct internal contradiction about what was done for R4 studies. The text first states, 'Hence, for R4 articles, we did not reimplement the code, except if parts of the code were missing,' and two paragraphs later states, 'For R4 articles, we implemented methods and experiments from scratch.' These statements are mutually exclusive. This contradiction is load-bearing because the headline 86% versus 33% contrast is intended to isolate the effect of code sharing. If the second statement describes the actual procedure, then R4 studies were reimplemented from the article and data alone, and the comparison does not measure the value of executing shared code; the difference could be driven by confounds such as study difficulty, data quality, or documentation quality. The authors must clarify which protocol was actually followed, and if both procedures occurred in different cases, they should report the R4 outcomes separately for studies that ran the shared code and studies that were reimplemented.","section":"The Replication Study"},{"comment":"The central comparison in Table 3 is 6 out of 7 R4 successes versus 5 out of 15 R3 successes. A two-sided Fisher exact test on these counts gives p ≈ 0.064, and the 95% confidence intervals for the two proportions overlap (approximately 49–97% for R4 and 15–58% for R3). The paper reports neither a significance test nor confidence intervals, yet the abstract and conclusion describe the correlation as 'strong' and use causal language such as 'extremely important to ensure reproducibility.' Given the small sample and the overlapping intervals, the evidence is suggestive at best. The authors should report interval estimates or a formal test, and should soften the claims accordingly.","section":"Table 3 and Figure 2"},{"comment":"The outcome variable is partly time-censored. The paper states that 40 hours was not enough to complete all experiments for 19 of the 22 studies, and in the Limitations section it acknowledges that 'The allotted time was not enough even for R4 studies that shared both code and data.' Because 'No Result' and some 'Failure' classifications depend on what could be accomplished within the cap, the R4/R3 difference could reflect differences in the time required to reach a conclusion rather than intrinsic reproducibility. The authors should report a sensitivity analysis, for example treating time-capped studies as censored, restricting the analysis to studies that were completed, or showing how the 86% versus 33% gap changes under a different time budget.","section":"The Replication Study; Limitations (third paragraph)"},{"comment":"The logistic regression analysis is not statistically credible as presented. Twenty problem-type features are used with only 22 samples, and the paper states that the validation and test sets were dropped, so the reported accuracy of 0.91 is purely in-sample. With 20 predictors and 22 observations, the model is heavily overfit, and the claim that P15, P7, and P18 are 'the three most significant features' based on |wi| > 0.8 is unsupported because no standard errors, confidence intervals, or hold-out evaluation are provided. This part of the analysis should be re-done with regularization or cross-validation, or the weights should be explicitly described as descriptive and unstable.","section":"Results (Table 2 and surrounding text)"}],"minor_comments":[{"comment":"The text says that for aborted studies, 'Such situations were recorded as spending all the time (40 hours) even though this was not the case in practice.' This creates a discrepancy with the 'Time' column in Table 1, which is presented as observed time spent. Please add a footnote or clarify in the table caption that some time entries are caps rather than actual hours.","section":"The Replication Study; Table 1"},{"comment":"The sentence 'In total, 50% of the articles included was reproduced to some extent' contains a subject-verb agreement error; 'articles included was' should be 'articles included were.'","section":"Abstract"},{"comment":"The table caption uses 'the true positive rate (TPR)' but the definition of TPR is not stated in the text until the Results paragraph that discusses it. Please define TPR explicitly near the table (e.g., the fraction of irreproducible studies in which the problem was encountered).","section":"Table 2"},{"comment":"The text refers to 'the reproducibility checklist in ICML and NeutIPS'; 'NeutIPS' should be 'NeurIPS.'","section":"Limitations"},{"comment":"The statement that 'sharing experiment parameters and hyperparameters is not associated with successful replications' relies on the underpowered logistic regression and TPR analysis discussed in the major comments; this claim should be tempered or explicitly marked as preliminary.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the internal contradiction in the replication protocol, which directly affects the interpretation of the main 86% versus 33% result. If that contradiction can be resolved (for example, by confirming that R4 studies were run using the shared code and that the 'from scratch' sentence is a drafting error), and if the authors add interval estimates, a formal test, and a sensitivity analysis for the time cap, the paper could become acceptable. The current version overstates the strength and causal status of the evidence, but the underlying data collection effort is valuable and the topic is appropriate for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"X,\n\nThe headline number—86% vs 33%—is not something I'd repeat to a colleague yet. The paper has real material, but the central comparison is broken as it stands.\n\nWhat is actually new: the authors attempted 22 replications, coded the problems into a 20-type taxonomy, and found that data documentation problems track failure better than code documentation problems. That null result on code docs is the most interesting thing here. They are also transparent about the 40-hour cap and the small sample in the limitations.\n\nThe soft spots are load-bearing. First, the paper contradicts itself about the R4 protocol. In 'The Replication Study' it says 'For R4 articles, we did not reimplement the code, except if parts of the code were missing.' A few paragraphs later: 'For R4 articles, we implemented methods and experiments from scratch.' Both cannot be true. If the second is true, the R4 'reproductions' did not run the shared code, and the 86% measures reimplementation from the article and data, not the value of code. That kills the main causal claim.\n\nSecond, the numbers don't add up. Table 1 lists 1 Success, 9 Partial Successes, 7 Failures, and 5 No Results. The abstract and Table 3 say 6 Successes and 5 Partial Successes. This is not a rounding issue; it changes the headline percentages. Using Table 1, the inclusive success rate for R4 is 71%, not 86%. The abstract's 33% for R3 also matches Table 1 only if you count 5/15, but Table 1 actually shows 5 PS for R3, not 1 S and 4 PS as in Table 3. So the tables contradict each other.\n\nThird, the logistic regression has 20 features on 22 samples, no validation, and the authors state they dropped validation. The 0.91 accuracy is overfit, and the feature weights are not trustworthy evidence.\n\nThe self-citation of the R1-R4 taxonomy is not a problem. The authors' own limitations section is honest about the time cap and sample size, but it misses the two big issues above.\n\nWho should read this: anyone working on reproducibility infrastructure, and anyone who wants a good case study in how to read a replication study critically. The taxonomy and the replication notes are reusable.\n\nMy recommendation: yes, send it to a serious referee. The topic matters, the work is substantial, and the flaws are addressable in revision—the contradiction and count inconsistencies need to be fixed, and the causal language needs to come down to correlation. After that, the direction of the finding (open code and data helps) is plausible, but the exact magnitudes are uncertain.","headline":"The 86% vs 33% headline is not supported by the paper's own tables, and the R4 protocol contradiction about reimplementation makes the central claim uninterpretable as stated, but the replication dataset and problem taxonomy are worth engaging with.","tokens_in":16148,"tokens_out":7354,"would_cite":false,"duration_ms":54403,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that sharing both code and data raises independently verified reproducibility of AI studies from 33% to 86%, and that data documentation quality, not code documentation quality, is what predicts success.","keywords":["reproducibility","artificial intelligence","open science","code sharing","data sharing","replication study","machine learning","data documentation"],"falsifier":"Re-running the same 22 replication attempts with an 80-hour cap per article and checking whether the data-only success rate rises toward the code-and-data rate would test the claim; if the 86% versus 33% gap largely disappears, the headline effect is an artifact of the time limit rather than of code sharing.","tokens_in":15157,"feed_emoji":"🔬","tokens_out":7835,"duration_ms":62452,"temperature":0.7,"pith_summary":"This paper tries to establish, with direct evidence, what makes independently reproducible AI research possible: whether authors share code as well as data. The authors selected the ten most-cited empirical AI articles from each of 2012, 2014, and 2016, and attempted to reproduce the 22 of those 30 that relied on publicly available data. They report an overall partial-or-full reproduction rate of 50%, with 86% of studies sharing both code and data reproduced versus 33% of studies sharing only data. They also report that the quality of data documentation predicts reproduction success, while the quality of code documentation does not once code is shared. The paper reads the pattern as evidence that open science, and code sharing in particular, is a practical lever for verification.","feed_headline":"Sharing code lifts AI reproducibility from 33% to 86%","feed_subtitle":"A 22-study replication finds data-only papers succeed only a third of the time; data documentation matters most.","key_machinery":"The carrying machinery is the R1–R4 reproducibility-type classification, which sorts a study by which artifacts are public: report only (R1), report plus code (R2), report plus data (R3), and report plus code and data (R4). The study attempts each selected article at its highest available type, spends at most 40 focused hours per article, and classifies each experiment's outcome as identical, consistent, or failed, then aggregates to study-level Success, Partial Success, Failure, or No Result. A 20-type problem taxonomy logs every obstacle, and logistic regression with balanced 11-versus-11 classes weights the problem types to show that data-related problems, such as dataset mismatch and missing partition descriptions, are the strongest discriminators. The R4-versus-R3 contrast under a shared protocol is the engine that produces the headline percentages.","core_discovery":"On the paper's own terms, the central discovery is a large empirical gap in independent reproducibility between two artifact-sharing regimes: when both code and data were public, 6 of 7 studies (86%) were fully or partly reproduced, and when only data were public, 5 of 15 (33%) were. The same effort budget and protocols applied to both groups, so the authors attribute the gap to the code itself, not to study quality. A secondary discovery is that the failure signal sits in the data: dataset mismatches, unspecified subsets, and undescribed train/validation/test splits are the problem types that characterize irreproducible studies, whereas poor code documentation, partial code, and missing version tags do not predict failure. The paper frames this as an empirical case for open science: code disambiguates underspecified methods, and shared data without code leaves too much reimplementation and guesswork.","pith_inferences":["If the effect is causal, then the marginal reproducibility return on releasing code is roughly 0.53, from 0.33 to 0.86, which implies that journal and conference policies should weight code release more heavily than data release; this follows from the paper's numbers but is not a claim the paper states.","The code-documentation null result may partly reflect selection: every R4 study had inspectable code, so the comparison is between usable code with bad documentation and no code at all; a useful extension would randomly down-sample code quality in a larger sample and watch whether the gap changes.","A cost-accounting extension would estimate hours spent per successful reproduction by artifact type; the paper logs time but does not compute unit costs, and those numbers would sharpen the policy argument."],"forward_implications":["If the 86% versus 33% gap holds, venues that require code plus data as the default would more than double the share of independently verifiable AI results among open-data papers.","Peer review should treat dataset version, provenance, and train/validation/test partition documentation as first-class requirements, since those are the problem types that mark irreproducible studies.","Sharing code in any inspectable form is the decisive act; extensive code comments, version tags, and polished packaging are nice but not predictors of successful reproduction.","Reproducibility statistics for AI should be reported separately by artifact type (R3 versus R4), because pooling them hides a large practice-dependent gap.","Studies built on closed code and closed training data, such as many large-language-model results, sit outside the regime where independent verification was observed to be feasible."],"supporting_citations":[{"why":"Supplies the earlier 63.5% reimplementation-based reproducibility estimate that this study compares against for method and bias.","marker":"(Raff 2019)"},{"why":"Provides the earlier 26% estimate for AAAI/IJCAI articles that motivates this study's direct replication approach.","marker":"(Gundersen and Kjensmo 2018)"},{"why":"Defines the R1–R4 reproducibility types that structure the article selection and analysis.","marker":"(Gundersen, Shamsaliei, and Isdahl 2022)"},{"why":"Supplies the 41 design decisions and the outcome-versus-analysis reproducibility terminology used to build the problem taxonomy and classify results.","marker":"(Gundersen et al. 2022)"},{"why":"Is the recent NLP replication study with a 25% success rate on open code and data that the discussion uses as a comparison for the effectiveness of code sharing.","marker":"(Arvan, Pina, and Parde 2022)"},{"why":"Supports the limitation and discussion point that reproducibility checklists increased code sharing, which contextualizes why the sampled 2012–2016 period may understate current openness.","marker":"(Pineau et al. 2021)"}],"fun_headline_variants":["AI reproducibility jumps to 86% when code is shared","Code sharing makes AI studies 2.6x more reproducible","Missing code hobbles AI replication: 33% vs 86%","Open code, not just data, powers AI replication","For AI replications, data documentation is key"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 40-hour cap is a fair and sufficient amount of effort for judging whether a well-documented study can be reproduced.","fun_headline_variants_meta":{"raw":{"variants":["AI reproducibility jumps to 86% when code is shared","Code sharing makes AI studies 2.6x more reproducible","Missing code hobbles AI replication: 33% vs 86%","Open code, not just data, powers AI replication","For AI replications, data documentation is key"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":2972,"prompt_tokens":926,"completion_tokens":2046,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1964}},"tokens_in":542,"tokens_out":2046,"duration_ms":13649,"temperature":1.0,"reasoning_tokens":1964,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:02:39.761464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the same 22 replication attempts with an 80-hour cap per article and checking whether the data-only success rate rises toward the code-and-data rate would test the claim; if the 86% versus 33% gap largely disappears, the headline effect is an artifact of the time limit rather than of code sharing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the earlier 63.5% reimplementation-based reproducibility estimate that this study compares against for method and bias."},{"cited_title":"E.; and Kjensmo, S","cited_arxiv_id":null,"evidence_quote":"Provides the earlier 26% estimate for AAAI/IJCAI articles that motivates this study's direct replication approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the recent NLP replication study with a 25% success rate on open code and data that the discussion uses as a comparison for the effectiveness of code sharing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the limitation and discussion point that reproducibility checklists increased code sharing, which contextualizes why the sampled 2012–2016 period may understate current openness."}],"review_version":1}