Pith. sign in

REVIEW 4 major objections 5 minor 71 references

The Unreasonable Effectiveness of Open Science in AI: A Replication Study

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that sharing both code and data raises independently verified reproducibility of AI studies from 33% to 86%, and that data documentation quality, not code documentation quality, is what predicts success.

desk verdict The 86% vs 33% headline is not supported by the paper's own tables, and the R4 protocol contradiction about reimplementation makes the central claim uninterpretable as stated, but the replication dataset and problem taxonomy are worth engaging with. read the letter →

arxiv 2412.17859 v1 pith:Y2DO3HPZ submitted 2024-12-20 cs.AI cs.LGcs.SE

Odd Erik Gundersen , Odd Cappelen , Martin M{o}ln{aa} , Nicklas Grimstad Nilsen This is my paper · ORCID
classification cs.AIcs.LGcs.SE
keywords reproducibilityartificialintelligenceopensciencecodesharingdatareplicationstudymachinelearningdocumentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish, with direct evidence, what makes independently reproducible AI research possible: whether authors share code as well as data. The authors selected the ten most-cited empirical AI articles from each of 2012, 2014, and 2016, and attempted to reproduce the 22 of those 30 that relied on publicly available data. They report an overall partial-or-full reproduction rate of 50%, with 86% of studies sharing both code and data reproduced versus 33% of studies sharing only data. They also report that the quality of data documentation predicts reproduction success, while the quality of code documentation does not once code is shared. The paper reads the pattern as evidence that open science, and code sharing in particular, is a practical lever for verification.

What carries the argument

The carrying machinery is the R1–R4 reproducibility-type classification, which sorts a study by which artifacts are public: report only (R1), report plus code (R2), report plus data (R3), and report plus code and data (R4). The study attempts each selected article at its highest available type, spends at most 40 focused hours per article, and classifies each experiment's outcome as identical, consistent, or failed, then aggregates to study-level Success, Partial Success, Failure, or No Result. A 20-type problem taxonomy logs every obstacle, and logistic regression with balanced 11-versus-11 classes weights the problem types to show that data-related problems, such as dataset mismatch and missing partition descriptions, are the strongest discriminators. The R4-versus-R3 contrast under a shared protocol is the engine that produces the headline percentages.

What would settle it

Re-running the same 22 replication attempts with an 80-hour cap per article and checking whether the data-only success rate rises toward the code-and-data rate would test the claim; if the 86% versus 33% gap largely disappears, the headline effect is an artifact of the time limit rather than of code sharing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a large empirical gap in independent reproducibility between two artifact-sharing regimes: when both code and data were public, 6 of 7 studies (86%) were fully or partly reproduced, and when only data were public, 5 of 15 (33%) were. The same effort budget and protocols applied to both groups, so the authors attribute the gap to the code itself, not to study quality. A secondary discovery is that the failure signal sits in the data: dataset mismatches, unspecified subsets, and undescribed train/validation/test splits are the problem types that characterize irreproducible studies, whereas poor code documentation, partial code, and missing version tags do not predict failure. The paper frames this as an empirical case for open science: code disambiguates underspecified methods, and shared data without code leaves too much reimplementation and guesswork.

Load-bearing premise

The 40-hour cap is a fair and sufficient amount of effort for judging whether a well-documented study can be reproduced.

Editorial extensions

If this is right

  • If the 86% versus 33% gap holds, venues that require code plus data as the default would more than double the share of independently verifiable AI results among open-data papers.
  • Peer review should treat dataset version, provenance, and train/validation/test partition documentation as first-class requirements, since those are the problem types that mark irreproducible studies.
  • Sharing code in any inspectable form is the decisive act; extensive code comments, version tags, and polished packaging are nice but not predictors of successful reproduction.
  • Reproducibility statistics for AI should be reported separately by artifact type (R3 versus R4), because pooling them hides a large practice-dependent gap.
  • Studies built on closed code and closed training data, such as many large-language-model results, sit outside the regime where independent verification was observed to be feasible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the effect is causal, then the marginal reproducibility return on releasing code is roughly 0.53, from 0.33 to 0.86, which implies that journal and conference policies should weight code release more heavily than data release; this follows from the paper's numbers but is not a claim the paper states.
  • The code-documentation null result may partly reflect selection: every R4 study had inspectable code, so the comparison is between usable code with bad documentation and no code at all; a useful extension would randomly down-sample code quality in a larger sample and watch whether the gap changes.
  • A cost-accounting extension would estimate hours spent per successful reproduction by artifact type; the paper logs time but does not compute unit costs, and those numbers would sharpen the policy argument.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a systematic replication study of 30 highly cited AI articles from 2012, 2014, and 2016, of which 22 were actually attempted (7 with shared code and data, 15 with shared data only). The authors classify each attempted replication as Success, Partial Success, Failure, or No Result, and relate these outcomes to the reproducibility type and to 20 catalogued problem types. The central empirical claim is that sharing both code and data is strongly associated with reproducibility: 86% of R4 studies were fully or partly reproduced versus 33% of R3 studies. The paper also claims that data documentation quality correlates with success while code documentation quality does not, and it presents a logistic regression analysis intended to identify which problem types distinguish reproducible from irreproducible studies.

Significance. If the estimates were reliable, this study would be a valuable addition to the empirical reproducibility literature for AI, complementing work such as Raff (2019) and the large-scale psychology and economics replication studies. The study's strengths include a publicly released replication repository, a transparent protocol with a defined time budget, a fine-grained taxonomy of 20 problem types, and an unusually candid Limitations section that acknowledges the small sample and the arbitrary 40-hour cap. However, the headline comparison rests on very small counts, the protocol contains an unresolved internal contradiction about whether R4 code was re-run or reimplemented, and the secondary logistic regression is fit in-sample without validation. As a result, the paper is more suggestive than definitive, and the strength of the causal language in the abstract and conclusion is not supported by the evidence presented.

major comments (4)
  1. [The Replication Study] The protocol description contains a direct internal contradiction about what was done for R4 studies. The text first states, 'Hence, for R4 articles, we did not reimplement the code, except if parts of the code were missing,' and two paragraphs later states, 'For R4 articles, we implemented methods and experiments from scratch.' These statements are mutually exclusive. This contradiction is load-bearing because the headline 86% versus 33% contrast is intended to isolate the effect of code sharing. If the second statement describes the actual procedure, then R4 studies were reimplemented from the article and data alone, and the comparison does not measure the value of executing shared code; the difference could be driven by confounds such as study difficulty, data quality, or documentation quality. The authors must clarify which protocol was actually followed, and if both procedures occurred in different cases, they should report the R4 outcomes separately for studies that ran the shared code and studies that were reimplemented.
  2. [Table 3 and Figure 2] The central comparison in Table 3 is 6 out of 7 R4 successes versus 5 out of 15 R3 successes. A two-sided Fisher exact test on these counts gives p ≈ 0.064, and the 95% confidence intervals for the two proportions overlap (approximately 49–97% for R4 and 15–58% for R3). The paper reports neither a significance test nor confidence intervals, yet the abstract and conclusion describe the correlation as 'strong' and use causal language such as 'extremely important to ensure reproducibility.' Given the small sample and the overlapping intervals, the evidence is suggestive at best. The authors should report interval estimates or a formal test, and should soften the claims accordingly.
  3. [The Replication Study; Limitations (third paragraph)] The outcome variable is partly time-censored. The paper states that 40 hours was not enough to complete all experiments for 19 of the 22 studies, and in the Limitations section it acknowledges that 'The allotted time was not enough even for R4 studies that shared both code and data.' Because 'No Result' and some 'Failure' classifications depend on what could be accomplished within the cap, the R4/R3 difference could reflect differences in the time required to reach a conclusion rather than intrinsic reproducibility. The authors should report a sensitivity analysis, for example treating time-capped studies as censored, restricting the analysis to studies that were completed, or showing how the 86% versus 33% gap changes under a different time budget.
  4. [Results (Table 2 and surrounding text)] The logistic regression analysis is not statistically credible as presented. Twenty problem-type features are used with only 22 samples, and the paper states that the validation and test sets were dropped, so the reported accuracy of 0.91 is purely in-sample. With 20 predictors and 22 observations, the model is heavily overfit, and the claim that P15, P7, and P18 are 'the three most significant features' based on |wi| > 0.8 is unsupported because no standard errors, confidence intervals, or hold-out evaluation are provided. This part of the analysis should be re-done with regularization or cross-validation, or the weights should be explicitly described as descriptive and unstable.
minor comments (5)
  1. [The Replication Study; Table 1] The text says that for aborted studies, 'Such situations were recorded as spending all the time (40 hours) even though this was not the case in practice.' This creates a discrepancy with the 'Time' column in Table 1, which is presented as observed time spent. Please add a footnote or clarify in the table caption that some time entries are caps rather than actual hours.
  2. [Abstract] The sentence 'In total, 50% of the articles included was reproduced to some extent' contains a subject-verb agreement error; 'articles included was' should be 'articles included were.'
  3. [Table 2] The table caption uses 'the true positive rate (TPR)' but the definition of TPR is not stated in the text until the Results paragraph that discusses it. Please define TPR explicitly near the table (e.g., the fraction of irreproducible studies in which the problem was encountered).
  4. [Limitations] The text refers to 'the reproducibility checklist in ICML and NeutIPS'; 'NeutIPS' should be 'NeurIPS.'
  5. [Discussion] The statement that 'sharing experiment parameters and hyperparameters is not associated with successful replications' relies on the underpowered logistic regression and TPR analysis discussed in the major comments; this claim should be tempered or explicitly marked as preliminary.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the 86% vs 33% rates are observed frequencies from Table 3; self-cited R1-R4 taxonomy and terminology are classification tools, not inputs that force the conclusion.

full rationale

The central claim is a direct contingency-table observation, not a quantity derived by construction from the self-cited taxonomy. The R1-R4 classification (Gundersen, Shamsaliei, and Isdahl 2022) and the outcome/analysis reproducibility terminology are self-citations, but they operate as coding definitions; the success rates are empirical outcomes of 22 attempted replications. No parameter is fitted and then renamed as a prediction. The logistic regression is fit and evaluated in-sample, so its 0.91 accuracy is descriptive rather than predictive, but the paper does not present it as out-of-sample prediction. The authors explicitly acknowledge small sample size, the 40-hour cap, selection bias, and possible bugs in their implementations, all of which are limitations rather than circularity. One serious non-circular concern: the Replication Study says both 'for R4 articles, we did not reimplement the code, except if parts of the code were missing' and 'For R4 articles, we implemented methods and experiments from scratch.' If taken literally, the R4 success rate may reflect reimplementation rather than code sharing, which threatens the causal attribution but is not a case where the conclusion reduces to the input by definition.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

This is an empirical measurement paper, so there are no derived constants or invented entities. The central estimates depend on a hand-chosen time cap (a free parameter) plus a set of domain assumptions about how to select, classify, and judge replication outcomes. The most consequential assumptions are the 40-hour threshold, the Scopus citation proxy for quality, and the subjective verdicts of 'identical' versus 'consistent.'

free parameters (1)
  • time_cap_hours = 40
    The maximum time spent per replication study was set to 40 hours by the authors, who 'considered 40 hours a reasonable effort.' This hand-chosen threshold determines whether a study is classified as No Result or Failure, so it directly shapes the reported success rates.
assumptions (5)
  • domain assumption Scopus citation counts are a valid proxy for the importance and quality of AI research.
    The ten most cited empirical AI papers per year were selected; the authors acknowledge this biases toward high-quality research (Section 'Selecting and Retrieving Articles').
  • domain assumption The R1-R4 reproducibility taxonomy from the authors' prior work is a valid classification scheme.
    The paper relies on Gundersen, Shamsaliei, and Isdahl (2022) to classify studies; this is a self-cited framework but applied consistently.
  • domain assumption A result is 'consistent' if the analysis leads to the same conclusion, a judgment the authors make subjectively.
    The classification of experiments as identical, consistent, or failed depends on the team's interpretation of the original conclusions (Section 'Classifying Results').
  • domain assumption The 40-hour time cap is sufficient for a well-documented study to be reproduced.
    Stated in 'The Replication Study'; this assumption directly determines the outcome labels.
  • domain assumption Finding a dataset with the same name or referencing the same article means it is the same dataset.
    Data retrieval procedure in 'Data Retrieval' assumes name matching is sufficient.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Unreasonable Effectiveness of Open Science in AI: A Replication Study." pith.science (2026). https://pith.science/paper/Y2DO3HPZ

@misc{pith2026241217859,
  author       = {Pith},
  title        = {Pith review of: The Unreasonable Effectiveness of Open Science in AI: A Replication Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y2DO3HPZ}},
  note         = {Machine review of arXiv:2412.17859}
}
read the original abstract

A reproducibility crisis has been reported in science, but the extent to which it affects AI research is not yet fully understood. Therefore, we performed a systematic replication study including 30 highly cited AI studies relying on original materials when available. In the end, eight articles were rejected because they required access to data or hardware that was practically impossible to acquire as part of the project. Six articles were successfully reproduced, while five were partially reproduced. In total, 50% of the articles included was reproduced to some extent. The availability of code and data correlate strongly with reproducibility, as 86% of articles that shared code and data were fully or partly reproduced, while this was true for 33% of articles that shared only data. The quality of the data documentation correlates with successful replication. Poorly documented or miss-specified data will probably result in unsuccessful replication. Surprisingly, the quality of the code documentation does not correlate with successful replication. Whether the code is poorly documented, partially missing, or not versioned is not important for successful replication, as long as the code is shared. This study emphasizes the effectiveness of open science and the importance of properly documenting data work.

Figures

Figures reproduced from arXiv: 2412.17859 by the authors.

Figure 1
Figure 1. An R3 type reproducibility study is conducted [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the results. lem with a TPR of 1.0. The source of three of these problem types, P15, P16, and P18 is the data, while code being shared in compiled form (P6) and lack of access to specialized hardware or software are the two others (P20). Notably, P17, which is the fourth of the problem types with data as the source of problems, has a TPR of 0.8, indicating that problems with documenting the data work are… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 64 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Akay, B.; and Karaboga, D. 2012. A modified artificial bee colony algorithm for real-parameter optimization. Information sciences, 192: 120--142

    work page 2012

  4. [4]

    Alexe, B.; Deselaers, T.; and Ferrari, V. 2012. Measuring the objectness of image windows. IEEE transactions on pattern analysis and machine intelligence, 34(11): 2189--2202

    work page 2012

  5. [5]

    Arvan, M.; Pina, L.; and Parde, N. 2022. Reproducibility in computational linguistics: Is source code enough? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2350--2361

    work page 2022

  6. [6]

    Baker, M. 2016. Reproducibility crisis. Nature, 533(26): 353--66

    work page 2016

  7. [7]

    Ball, P. 2023. Is AI leading to a reproducibility crisis in science? Nature, 624(7990): 22--25

    work page 2023

  8. [8]

    Belz, A.; Agarwal, S.; Shimorina, A.; and Reiter, E. 2021. A systematic review of reproducibility research in natural language processing. In 16th Conference of the European Chapter of the Associationfor Computational Linguistics 2021, 381--393. Association for Computational Linguistics

    work page 2021

Show all 71 references
  1. [9]

    Bouthillier, X.; Laurent, C.; and Vincent, P. 2019. Unreproducible Research is Reproducible. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 725--734. PMLR

    2019

  2. [10]

    F.; Dreber, A.; Forsell, E.; Ho, T.-H.; Huber, J.; Johannesson, M.; Kirchler, M.; Almenberg, J.; Altmejd, A.; Chan, T.; et al

    Camerer, C. F.; Dreber, A.; Forsell, E.; Ho, T.-H.; Huber, J.; Johannesson, M.; Kirchler, M.; Almenberg, J.; Altmejd, A.; Chan, T.; et al. 2016. Evaluating replicability of laboratory experiments in economics. Science, 351(6280): 1433--1436

    2016

  3. [11]

    F.; Dreber, A.; Holzmeister, F.; Ho, T.-H.; Huber, J.; Johannesson, M.; Kirchler, M.; Nave, G.; Nosek, B

    Camerer, C. F.; Dreber, A.; Holzmeister, F.; Ho, T.-H.; Huber, J.; Johannesson, M.; Kirchler, M.; Nave, G.; Nosek, B. A.; Pfeiffer, T.; et al. 2018. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature human behaviour, ...

    2018

  4. [12]

    C.; et al

    Chen, B.; Xing, L.; Zhao, H.; Zheng, N.; Pr , J. C.; et al. 2016. Generalized correntropy for robust adaptive filtering. IEEE Transactions on Signal Processing, 64(13): 3376--3387

    2016

  5. [13]

    Chen, T.; and Guestrin, C. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 785--794

    2016

  6. [14]

    Chen, Y.; Lin, Z.; Zhao, X.; Wang, G.; and Gu, Y. 2014. Deep learning-based classification of hyperspectral data. IEEE Journal of Selected topics in applied earth observations and remote sensing, 7(6): 2094--2107

    2014

  7. [15]

    Cheng, G.; Zhou, P.; and Han, J. 2016. Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 54(12): 7405--7415

    2016

  8. [16]

    Collaboration, O. S. 2015. Estimating the reproducibility of psychological science. Science, 349(6251): aac4716

    2015

  9. [17]

    Collberg, C.; and Proebsting, T. A. 2016. Repeatability in computer systems research. Communications of the ACM, 59(3): 62--69

    2016

  10. [18]

    F.; Cremonesi, P.; and Jannach, D

    Dacrema, M. F.; Cremonesi, P.; and Jannach, D. 2019. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems, 101--109

    2019

  11. [19]

    Donahue, J.; Jia, Y.; Vinyals, O.; Hoffman, J.; Zhang, N.; Tzeng, E.; and Darrell, T. 2014. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, 647--655. PMLR

    2014

  12. [20]

    Goferman, S.; Zelnik-Manor, L.; and Tal, A. 2012. Context-aware saliency detection. IEEE transactions on pattern analysis and machine intelligence, 34(10): 1915--1926

    2012

  13. [21]

    Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. Advances in neural information processing systems, 27

    2014

  14. [22]

    Graves, A.; and Jaitly, N. 2014. Towards end-to-end speech recognition with recurrent neural networks. In International conference on machine learning, 1764--1772. PMLR

    2014

  15. [23]

    Guha, T.; and Ward, R. K. 2011. Learning sparse representations for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 34(8): 1576--1588

    2011

  16. [24]

    Gundersen, O. E. 2020. The reproducibility crisis is real. AI Magazine, 41(3): 103--106

    2020

  17. [25]

    E.; Coakley, K.; Kirkpatrick, C.; and Gil, Y

    Gundersen, O. E.; Coakley, K.; Kirkpatrick, C.; and Gil, Y. 2022. Sources of irreproducibility in machine learning: A review. arXiv preprint arXiv:2204.07610

    2022 arXiv

  18. [26]

    E.; and Kjensmo, S

    Gundersen, O. E.; and Kjensmo, S. 2018. State of the art: Reproducibility in artificial intelligence. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32

    2018

  19. [27]

    E.; Shamsaliei, S.; and Isdahl, R

    Gundersen, O. E.; Shamsaliei, S.; and Isdahl, R. J. 2022. Do machine learning platforms provide out-of-the-box reproducibility? Future Generation Computer Systems, 126: 34--47

    2022

  20. [28]

    E.; Shamsaliei, S.; Kj rnli, H

    Gundersen, O. E.; Shamsaliei, S.; Kj rnli, H. S.; and Langseth, H. 2023. On Reporting Robust and Trustworthy Conclusions from Model Comparison Studies Involving Neural Networks and Randomness. In Proceedings of the 2023 ACM Conference on Reproducibility and Replicability, 37--61

    2023

  21. [29]

    A.; Hosny, A.; Khodakarami, F.; of Directors Shraddha Thakkar 35 Kusko Rebecca 36 Sansone Susanna-Assunta 37 Tong Weida 35 Wolfinger Russ D

    Haibe-Kains, B.; Adam, G. A.; Hosny, A.; Khodakarami, F.; of Directors Shraddha Thakkar 35 Kusko Rebecca 36 Sansone Susanna-Assunta 37 Tong Weida 35 Wolfinger Russ D. 38 Mason Christopher E. 39 Jones Wendell 40 Dopazo Joaquin 41 Furlanello Cesare 42, M. A. Q. C. M. S. B.; Wald...

    2020

  22. [30]

    Henderson, P.; Islam, R.; Bachman, P.; Pineau, J.; Precup, D.; and Meger, D. 2018. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence. AAAI Press

    2018

  23. [31]

    N.; and Wu, C

    Huang, G.; Song, S.; Gupta, J. N.; and Wu, C. 2014. Semi-supervised and unsupervised extreme learning machines. IEEE transactions on cybernetics, 44(12): 2405--2417

    2014

  24. [32]

    Hutson, M. 2018. Artificial intelligence faces reproducibility crisis

    2018

  25. [33]

    Ioannidis, J. P. 2005. Why most published research findings are false. PLoS medicine, 2(8): e124

    2005

  26. [34]

    Jia, F.; Lei, Y.; Lin, J.; Zhou, X.; and Lu, N. 2016 a . Deep neural networks: A promising tool for fault characteristic mining and intelligent diagnosis of rotating machinery with massive data. Mechanical systems and signal processing, 72: 303--315

    2016

  27. [35]

    Jia, J.; Liu, Z.; Xiao, X.; Liu, B.; and Chou, K.-C. 2016 b . iSuc-PseOpt: identifying lysine succinylation sites in proteins by incorporating sequence-coupling effects into pseudo components and optimizing imbalanced training dataset. Analytical biochemistry, 497: 48--56

    2016

  28. [36]

    Kapoor, S.; and Narayanan, A. 2023. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9)

    2023

  29. [37]

    A.; Ratliff, K

    Klein, R. A.; Ratliff, K. A.; Vianello, M.; Adams Jr, R. B.; Bahn \' k, S .; Bernstein, M. J.; Bocian, K.; Brandt, M. J.; Brooks, B.; Brumbaugh, C. C.; et al. 2014. Investigating variation in replicability. Social psychology

    2014

  30. [38]

    Le, Q.; and Mikolov, T. 2014. Distributed representations of sentences and documents. In International conference on machine learning, 1188--1196. PMLR

    2014

  31. [39]

    Li, G.; Niu, P.; and Xiao, X. 2012. Development and investigation of efficient artificial bee colony algorithm for numerical function optimization. Applied soft computing, 12(1): 320--332

    2012

  32. [40]

    Li, W.; Zhao, R.; Xiao, T.; and Wang, X. 2014. Deepreid: Deep filter pairing neural network for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, 152--159

    2014

  33. [41]

    Li, X.; and Yao, X. 2011. Cooperatively coevolving particle swarms for large scale optimization. IEEE Transactions on Evolutionary Computation, 16(2): 210--224

    2011

  34. [42]

    Liu, T.; and Tao, D. 2015. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3): 447--461

    2015

  35. [43]

    Lucic, M.; Kurach, K.; Michalski, M.; Gelly, S.; and Bousquet, O. 2018. Are GANs created equal? A large-scale study. In Advances in neural information processing systems, 700--709

    2018

  36. [44]

    A.; and Dodge, J

    Magnusson, I.; Smith, N. A.; and Dodge, J. 2023. Reproducibility in NLP : What Have We Learned from the Checklist? In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Findings of the Association for Computational Linguistics: ACL 2023, 12789--12811. Toronto, Canada: Associa...

    2023

  37. [45]

    Makridakis, S.; Spiliotis, E.; and Assimakopoulos, V. 2018. Statistical and Machine Learning forecasting methods: Concerns and ways forward. PloS one, 13(3): e0194889

    2018

  38. [46]

    Melis, G.; Dyer, C.; and Blunsom, P. 2018. On the State of the Art of Evaluation in Neural Language Models. In International Conference on Learning Representations

    2018

  39. [47]

    Meng, X.; Bradley, J.; Yavuz, B.; Sparks, E.; Venkataraman, S.; Liu, D.; Freeman, J.; Tsai, D.; Amde, M.; Owen, S.; et al. 2016. Mllib: Machine learning in apache spark. The journal of machine learning research, 17(1): 1235--1241

    2016

  40. [48]

    J.; and Roggen, D

    Ord \'o \ n ez, F. J.; and Roggen, D. 2016. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1): 115

    2016

  41. [49]

    Peng, Y.; Ganesh, A.; Wright, J.; Xu, W.; and Ma, Y. 2012. RASL: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE transactions on pattern analysis and machine intelligence, 34(11): 2233--2246

    2012

  42. [50]

    V.; Qian, S.; Wang, J.; Lutellier, T.; Rosenthal, J.; Tan, L.; Yu, Y.; and Nagappan, N

    Pham, H. V.; Qian, S.; Wang, J.; Lutellier, T.; Rosenthal, J.; Tan, L.; Yu, Y.; and Nagappan, N. 2020. Problems and opportunities in training deep learning software systems: An analysis of variance. In Proceedings of the 35th IEEE/ACM International Conference on Automated Soft...

    2020

  43. [51]

    Pineau, J.; Vincent-Lamarre, P.; Sinha, K.; Larivi \`e re, V.; Beygelzimer, A.; d'Alch \'e Buc, F.; Fox, E.; and Larochelle, H. 2021. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). The Journal of Machine Learnin...

    2021

  44. [52]

    Prinz, F.; Schlange, T.; and Asadullah, K. 2011. Believe it or not: how much can we rely on published data on potential drug targets? Nature reviews Drug discovery, 10(9): 712--712

    2011

  45. [53]

    Raff, E. 2017. JSAT: Java statistical analysis tool, a library for machine learning. The Journal of Machine Learning Research, 18(1): 792--796

    2017

  46. [54]

    Raff, E. 2019. A step toward quantifying independently reproducible machine learning research. Advances in Neural Information Processing Systems, 32

    2019

  47. [55]

    Raff, E.; and Farris, A. L. 2023. A siren song of open source reproducibility, examples from machine learning. In Proceedings of the 2023 ACM Conference on Reproducibility and Replicability, 115--120

    2023

  48. [56]

    Rodriguez, A.; and Laio, A. 2014. Clustering by fast search and find of density peaks. science, 344(6191): 1492--1496

    2014

  49. [57]

    A.; Bovik, A

    Saad, M. A.; Bovik, A. C.; and Charrier, C. 2012. Blind image quality assessment: A natural scene statistics approach in the DCT domain. IEEE transactions on Image Processing, 21(8): 3339--3352

    2012

  50. [58]

    R.; Gao, M.; Lu, L.; Xu, Z.; Nogues, I.; Yao, J.; Mollura, D.; and Summers, R

    Shin, H.-C.; Roth, H. R.; Gao, M.; Lu, L.; Xu, Z.; Nogues, I.; Yao, J.; Mollura, D.; and Summers, R. M. 2016. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE transactions on medical imaging...

    2016

  51. [59]

    J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al

    Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; et al. 2016. Mastering the game of Go with deep neural networks and tree search. nature, 529(7587): 484--489

    2016

  52. [60]

    Stodden, V.; Seiler, J.; and Ma, Z. 2018. An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11): 2584--2589

    2018

  53. [61]

    Thomson, C.; Reiter, E.; and Belz, A. 2024. Common Flaws in Running Human Evaluation Experiments in NLP. Computational Linguistics, 1--10

    2024

  54. [62]

    Van Noorden, R.; Maher, B.; and Nuzzo, R. 2014. The top 100 papers. Nature News, 514(7524): 550

    2014

  55. [63]

    Varoquaux, G.; and Cheplygina, V. 2022. Machine learning for medical imaging: methodological failures and recommendations for the future. NPJ digital medicine, 5(1): 1--8

    2022

  56. [64]

    Wei, J.; Tay, Y.; Bommasani, R.; Raffel, C.; Zoph, B.; Borgeaud, S.; Yogatama, D.; Bosma, M.; Zhou, D.; Metzler, D.; et al. 2022. Emergent Abilities of Large Language Models. Transactions on Machine Learning Research

    2022

  57. [65]

    Werner, L.; Laya \" da, N.; Genev \`e s, P.; Euzenat, J.; and Graux, D. 2024. Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 15850--15858

    2024

  58. [66]

    Yin, X.-C.; Yin, X.; Huang, K.; and Hao, H.-W. 2013. Robust text detection in natural scene images. IEEE transactions on pattern analysis and machine intelligence, 36(5): 970--983

    2013

  59. [67]

    D.; and Fergus, R

    Zeiler, M. D.; and Fergus, R. 2014. Visualizing and understanding convolutional networks. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, 818--833. Springer

    2014

  60. [68]

    Zhang, D.; Shen, D.; Initiative, A. D. N.; et al. 2012 a . Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer's disease. NeuroImage, 59(2): 895--907

    2012

  61. [69]

    Zhang, K.; Gao, X.; Tao, D.; and Li, X. 2012 b . Single image super-resolution with non-local means and steering kernel regression. IEEE Transactions on Image Processing, 21(11): 4544--4556

    2012

  62. [70]

    C.; and Tang, X

    Zhang, Z.; Luo, P.; Loy, C. C.; and Tang, X. 2014. Facial landmark detection by deep multi-task learning. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, 94--108. Springer

    2014

  63. [71]

    Zhuang, D.; Zhang, X.; Song, S.; and Hooker, S. 2022. Randomness in neural network training: Characterizing the impact of tooling. Proceedings of Machine Learning and Systems, 4: 316--336

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.