Pith. sign in

REVIEW 3 major objections 4 minor 70 references

Addressing Reproducibility Challenges in HPC with Continuous Integration

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that continuous integration, backed by complete provenance, can substitute for direct access to HPC resources, and presents CORRECT, a GitHub Action that runs reproducibility tests on remote HPC sites and returns documented

desk verdict Solid feasibility study for a useful multi-site HPC CI action, but the paper's central claim that CI logs plus provenance can substitute for direct resource access is unsupported and contradicted by its own Section 7.4. read the letter →

arxiv 2508.21289 v1 pith:LNCSKCXY submitted 2025-08-29 cs.DC cs.SE

Valérie Hayot-Sasson , Nathaniel Hudson , André Bauer , Maxime Gonthier , Ian Foster , Kyle Chard This is my paper · ORCID
classification cs.DCcs.SE
keywords reproducibilitycontinuousintegrationhigh-performancecomputingprovenanceartifactevaluationGitHubActionsGlobusComputeremoteexecution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

High-performance computing results are hard to reproduce because the specialized systems they run on are tightly controlled and hard to access. The paper argues that this barrier can be lowered by continuous integration: if a project's tests are run automatically on remote HPC resources, with the full execution log and provenance saved, a reviewer can assess reproducibility without touching the original system. To make that work in practice, the authors built CORRECT, a GitHub Action that uses the Globus Compute function-as-a-service platform to execute reproducibility tests on arbitrary HPC and cloud endpoints from a standard GitHub workflow. They demonstrate it on three HPC applications: multi-site protein-docking tests, scheduler-abstraction software tests, and reproduction of a paper's artifacts. The paper's central claim is that better HPC-compliant CI solutions will improve reproducibility of applications.

What carries the argument

Key machinery: CORRECT, a GitHub Action (COntinuous Reproducibility with a Remote Execution Computing Tool) that lets a normal GitHub workflow execute tests on remote HPC sites through Globus Compute, a function-as-a-service platform. The action authenticates with Globus Auth secrets stored in GitHub environments, invokes an endpoint whose administrator can restrict which functions run, and returns stdout and stderr as workflow artifacts; provenance is meant to come from the captured command, logs, and in principle environment information. This combination—CI trigger, FaaS execution, and logged artifacts—is what carries the argument that reproducible evaluation can happen without direct acce

What would settle it

A concrete test: take the KaMPIng artifact workflow, which CORRECT passes on Chameleon Cloud, and have a fresh reviewer manually reproduce the same artifact on the same instance class using only the CORRECT-published logs and the authors' artifact instructions. If the manual reproduction diverges from the paper's reported trends, the claim that CI records can substitute for access fails.

Watch

Extended reading notes

Core claim

The paper's discovery is a working route from a GitHub repository to execution on restricted HPC systems for reproducibility checking. CORRECT wraps Globus Compute function invocation as a GitHub Action: a workflow triggers a run, the runner authenticates via Globus Auth, the function clones the repository and runs user-specified tests (a shell command or a pre-registered Python function) on an HPC endpoint, and stdout and stderr are returned to the GitHub interface and stored as artifacts. Security is handled by GitHub environment secrets with manual approval, by Globus Compute multi-user endpoints that map execution to the correct user identity, and by endpoint-approved function lists. The

Load-bearing premise

The paper's argument depends on the idea that an automated test run's saved log and history can stand in for being able to log in to the original supercomputer yourself; if that exchange is not valid, the main claim falls.

Editorial extensions

If this is right

  • Reproducibility reviewers for conferences could evaluate a paper by reading CI-generated execution logs and artifacts instead of obtaining an allocation on the original HPC site.
  • A single CORRECT workflow can run the same test suite on several HPC and cloud sites by swapping endpoint identifiers, giving cross-platform evidence of reproducibility without multiple manual setups.
  • Because GitHub environment secrets require human approval, HPC sites can keep their security boundaries while still allowing automated, documented test runs; routine tests can run on cloud endpoints while HPC runs are gated by review.
  • Execution results can be committed to the repository or uploaded as artifacts, creating a persistent historical record of reproducibility evaluations that can be inspected after the original run.
  • The pattern generalizes to other CI platforms such as GitLab, since the action runs on a standard CI runner and only needs Python installed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its substitution claim could be tested directly: pit CORRECT's CI logs against a blind hands-on reproduction by an independent reviewer, but the paper does not report such a comparison.
  • The scheme could be extended to capture full system provenance—for example scheduler, node type, compiler, and library versions—by adding a second CORRECT call that runs environment-reporting commands and stores the output; this would address the paper's stated limitation.
  • The same pattern could support a new artifact badge level for CI-verified reproducibility, where the automated remote run supplies the evidence; that would make badge evaluation faster but would inherit the substitution assumption.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper argues that HPC-specific reproducibility barriers can be lowered by substituting direct access to HPC resources with continuous integration (CI) plus complete provenance information. It surveys reproducibility initiatives and existing HPC CI frameworks, then presents CORRECT, a GitHub Action that uses Globus Compute to execute user-defined functions or shell commands on remote endpoints and return their stdout/stderr to the GitHub runner. The evaluation covers three cases: ParslDock tests run across three sites, PSI/J tests run on Purdue Anvil, and reproduction of the KaMPIng artifact evaluation on Chameleon Cloud.

Significance. If the central claim were established, the paper would offer a practical, low-friction path for reproducibility evaluation in HPC, and the survey of current reproducibility and CI efforts is useful. CORRECT builds on established platforms (Globus Compute, GitHub Actions) and the authors publish the action for community use. The paper honestly reports failures and limitations, including the PSI/J test failure and the lack of environment capture. However, the evaluation demonstrates remote dispatch and log retrieval, not the stronger claim that CI logs plus provenance can substitute for access to the original environment. The paper's own Section 7.4 concedes that the current tool cannot validate reproducibility without environment information, so the central premise is not yet supported. The tool is a promising prototype rather than a validated solution to the stated problem.

major comments (3)
  1. [Abstract, Section 5.3, Section 7.4] The load-bearing premise is that CI execution records plus 'complete provenance information' can substitute for direct resource access. CORRECT, as described in Section 5.3, returns only the standard output and standard error of the executed function and does not capture the remote environment, resource configuration, module versions, compiler flags, or hardware details. Section 7.4 explicitly states: 'Without information about the environment, users can only see the results of previous executions, but that alone cannot validate reproducibility without access to the environment.' This directly contradicts the abstract's substitution claim. Either CORRECT must be extended to capture and publish environment/provenance information, or the paper must be substantially reframed around the weaker, supported claim that CORRECT enables repeated remote execution with logged output.
  2. [Section 6] None of the three evaluations tests the central substitution scenario: that a reviewer with no access to the execution environment can judge reproducibility from CORRECT's artifacts alone. Section 6.1 records runtimes across sites but does not assess reproducibility of results; Section 6.2 reports that the PSI/J pytest run failed due to an error in the PSI/J codebase; Section 6.3 reproduces only the KaMPIng artifacts that were already reported reproducible on Chameleon Cloud. Section 6.3 states the authors 'compare our findings with those reported in the paper' and reproduce 'the artifacts that were reported to be reproducible on Chameleon Cloud.' This is a repeatability check by the tool developers, not an independent evaluation of whether logged outputs are sufficient for a no-access reproducibility verdict. The paper needs an evaluation that directly exercises the claimed use case, fo
  3. [Section 5.2] The security argument is not fully specified. The paper argues that GitHub environment secrets with required review 'ensures that the person authorizing the execution maps to a user at the site at which the code is executed,' but GitHub environment protection does not, by itself, bind a specific remote HPC account to a specific human reviewer; the mapping relies on the recommendation that 'there is only one reviewer per environment.' This is administrative guidance rather than an enforced mechanism. Since satisfying HPC security requirements is one of the two primary goals stated in Section 5, the paper should provide a concrete threat model and describe how the proposed configuration enforces the claimed identity mapping, including what happens when environments are misconfigured or when multiple reviewers exist.
minor comments (4)
  1. [Section 2] Typo: 'Juypter' should be 'Jupyter'.
  2. [References] Reference [5] lists the author as 'Association for Computing Machinert' (typo). Reference [24] is titled 'GitBucket' but the introduction text refers to 'Bitbucket'; please align the citation.
  3. [Section 6.2 / Figure 5] The figure caption/labels are confusing: the top panel is described as 'Error and full execution stdout' and the bottom as 'Execution stdout.' Clarify what each panel shows and how they differ.
  4. [Section 6.1] The sentence 'we installed via Conda the Protein Docking application...' is grammatically awkward. Also, the claim that 'short duration tests highlight the benefits of adopting a FaaS based model' is not directly evidenced by the presented runtime comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CORRECT is a systems/engineering contribution; its evaluation is a capability demonstration, not a fitted prediction or a derivation from self-cited results.

full rationale

This paper makes no mathematical derivation and fits no parameters; its central claim is an explicitly hedged argument that CI plus provenance can substitute for direct resource access (Abstract: "we believe that regular documented testing... can be used as a substitute"). CORRECT is then evaluated for usability across three applications. The ParslDock experiment measures execution times across sites, the PSI/J experiment reports a test failure, and the KaMPIng experiment reproduces artifacts already claimed reproducible on Chameleon Cloud. None of these is a prediction derived from fitted inputs; they are demonstrations that the action can invoke remote functions and return logs. The paper's own §7.4 limitation — "Without information about the environment, users can only see the results of previous executions, but that alone cannot validate reproducibility without access to the environment" — undercuts the substitution thesis, but that is a validity/correctness weakness, not a circular reduction: the conclusion is not assumed as an input, and no output is made equal to an input by construction. Self-citations to Globus Compute, Parsl, and PSI/J are used as engineering building blocks rather than as authorities that force the conclusion. There is no uniqueness theorem imported from the authors, no ansatz smuggled in via citation, and no renaming of a known result as a new derivation. The paper is therefore self-contained in the sense relevant to circularity: its claims are arguable and testable, even if the current evaluation does not fully support the strongest substitution claim.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new scientific entities. Its central premises are domain assumptions about the sufficiency of CI logs as provenance and the security of the Globus Compute and GitHub combination, neither of which is proven beyond the authors' own qualitative assessment. There are no fitted numerical parameters, so the free_parameters list is empty.

assumptions (4)
  • domain assumption Regular documented testing through CI, coupled with complete provenance information, can be used as a substitute for direct resource access in reproducibility evaluation.
    This is the foundational premise in the Abstract and Section 5. It is asserted rather than empirically demonstrated, and Section 7.4 notes that the current tool does not capture full environment provenance.
  • domain assumption Globus Compute endpoints, and multi-user endpoints in particular, provide execution security that meets HPC site requirements.
    Sections 5.1 and 5.2 describe the security model but provide no independent security evaluation or evidence that the test sites accepted this model beyond the authors' own deployments.
  • domain assumption GitHub Actions environment secrets with a single reviewer can map the human approver to the HPC identity and prevent unauthorized execution.
    Section 5.2 describes this as a recommendation, not a verified mechanism. The paper acknowledges it 'may limit complete automation' but does not demonstrate that all HPC sites will accept it.
  • domain assumption The three evaluated applications are representative of HPC software for assessing reproducibility tooling.
    The selection of ParslDock, PSI/J, and KaMPIng exercises different CORRECT features, but the paper provides no evidence that these generalize to the broader HPC software landscape.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Addressing Reproducibility Challenges in HPC with Continuous Integration." pith.science (2026). https://pith.science/paper/LNCSKCXY

@misc{pith2026250821289,
  author       = {Pith},
  title        = {Pith review of: Addressing Reproducibility Challenges in HPC with Continuous Integration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LNCSKCXY}},
  note         = {Machine review of arXiv:2508.21289}
}
read the original abstract

The high-performance computing (HPC) community has adopted incentive structures to motivate reproducible research, with major conferences awarding badges to papers that meet reproducibility requirements. Yet, many papers do not meet such requirements. The uniqueness of HPC infrastructure and software, coupled with strict access requirements, may limit opportunities for reproducibility. In the absence of resource access, we believe that regular documented testing, through continuous integration (CI), coupled with complete provenance information, can be used as a substitute. Here, we argue that better HPC-compliant CI solutions will improve reproducibility of applications. We present a survey of reproducibility initiatives and describe the barriers to reproducibility in HPC. To address existing limitations, we present a GitHub Action, CORRECT, that enables secure execution of tests on remote HPC resources. We evaluate CORRECT's usability across three different types of HPC applications, demonstrating the effectiveness of using CORRECT for automating and documenting reproducibility evaluations.

Figures

Figures reproduced from arXiv: 2508.21289 by the authors.

Figure 1
Figure 1. Reproducibility badges awarded by SC over time. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 4
Figure 4. Runtimes of ParslDock tests on different machines. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 3
Figure 3. Example Globus Compute Action step definition. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: PSI/J test invocation failure. Top: Error and full execution stdout reported back to GitHub runner. Bottom: Execution stdout stored within a GitHub Action artifact. would be expressed within the shell_cmd to pytest within the action. With CORRECT, the PSI/J pytest runs…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 66 canonical work pages

  1. [1]

    Software and computing for Run 3 of the ATLAS experiment at the LHC

    2025. Software and computing for Run 3 of the ATLAS experiment at the LHC. The European Physical Journal C 85, 3 (2025), 234

    work page 2025

  2. [2]

    Ryan Adamson, Paul Bryant, Dave Montoya, Jeff Neel, Erik Palmer, Ray Powell, Ryan Prout, and Peter Upton. 2024. Creating Continuous Integration Infrastruc- ture for Software Development on US Department of Energy High-Performance Computing System. Computing in Science & Engineering 26, 1 (2024), 31–39

    work page 2024

  3. [3]

    Katz, Kevin Hunter Kesling, Chris Janidlo, Reid Mello, and Lei Wang

    Rachana Ananthakrishnan, Yadu Babuji, Matt Baughman, Josh Bryan, Kyle Chard, Ryan Chard, Ben Clifford, Ian Foster, Daniel S. Katz, Kevin Hunter Kesling, Chris Janidlo, Reid Mello, and Lei Wang. 2024. Enabling Remote Management of FaaS Endpoints with Globus Compute Multi-User Endpoints. In Practice and Experience in Advanced Research Computing 2024: Human ...

    work page arXiv 2024

  4. [4]

    Apache. [n. d.]. Apache Airavata. https://airavata.apache.org. Accessed: 2025- 04-14

    work page 2025

  5. [5]

    Association for Computing Machinert. [n. d.]. Artifact Review and Badging - Cur- rent. https://www.acm.org/publications/policies/artifact-review-and-badging- current. Accessed: 2025-08-22

    work page 2025

  6. [6]

    Yadu Babuji, Anna Woodard, Zhuozhao Li, Daniel S Katz, Ben Clifford, Rohan Kumar, Lukasz Lacinski, Ryan Chard, Justin M Wozniak, Ian Foster, et al. 2019. Parsl: Pervasive parallel programming in Python. In Proceedings of the 28th Valérie Hayot-Sasson, Nathaniel Hudson, André Bauer, Maxime Gonthier, Ian Foster, and Kyle Chard International Symposium on Hig...

    work page 2019

  7. [7]

    Sean Bechhofer, David De Roure, Matthew Gamble, Carole Goble, and Iain Buchan. 2010. Research objects: Towards exchange and reuse of digital knowl- edge. Nature Precedings (2010), 1–1

    work page 2010

  8. [8]

    Robin M Betz and Ross C Walker. 2013. Implementing continuous integration software in an established computational chemistry software package. In 2013 5th International Workshop on Software Engineering for Computational Science and Engineering (SE-CSE). IEEE, 68–74

    work page 2013

Show all 70 references
  1. [9]

    Boerner, Stephen Deems, Thomas R

    Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. 2023. ACCESS: Advancing Innovation: NSF’s Advanced Cyberinfras- tructure Coordination Ecosystem: Services & Support. In Practice and Experience in Advanced Research Computing 2023: Computin...

    2023

  2. [10]

    CCGrid. [n. d.]. Call for Artifacts - CCGrid 2025. https://site.uit.no/ccgrid2025/ artifacts/. Accessed: 2025-04-14

    2025

  3. [11]

    Kyle Chard, Niall Gaffney, Mihael Hategan, Kacper Kowalik, Bertram Ludäscher, Timothy McPhillips, Jarek Nabrzyski, Victoria Stodden, Ian Taylor, Thomas Thelen, et al. 2020. Toward enabling reproducibility for data-intensive research using the Whole Tale platform. In Parallel C...

    2020

  4. [12]

    Kyle Chard, Steven Tuecke, and Ian Foster. 2014. Efficient and Secure Transfer, Synchronization, and Sharing of Big Data. IEEE Cloud Computing 1, 3 (2014), 46–55. doi:10.1109/MCC.2014.52

    2014 doi

  5. [13]

    Ryan Chard, Yadu Babuji, Zhuozhao Li, Tyler Skluzacek, Anna Woodard, Ben Blaiszik, Ian Foster, and Kyle Chard. 2020. funcX: A federated function serving fabric for science. In Proceedings of the 29th International symposium on high- performance parallel and distributed computi...

    2020

  6. [14]

    Barbara Cheifet. 2021. Promoting reproducibility with Code Ocean. 2 pages

    2021

  7. [15]

    Conda. [n. d.]. Conda Documentation — conda-docs documentation. https: //docs.conda.io. Accessed: 2025-04-14

    2025

  8. [16]

    CruiseControl. [n. d.]. CruiseControl. https://cruisecontrol.sourceforge.net/. Accessed: 2025-04-14

    2025

  9. [17]

    Ewa Deelman, Karan Vahi, Gideon Juve, Mats Rynge, Scott Callaghan, Philip J Maechling, Rajiv Mayani, Weiwei Chen, Rafael Ferreira Da Silva, Miron Livny, et al. 2015. Pegasus, a workflow management system for science automation. Future Generation Computer Systems 46 (2015), 17–35

    2015

  10. [18]

    Docker. [n. d.]. Docker: Accelerated Container Application Development. https: //www.docker.com. Accessed: 2025-04-14

    2025

  11. [19]

    Elizabeth DuPre, Chris Holdgraf, Agah Karakuzu, Loïc Tetrel, Pierre Bellec, Nikola Stikov, and Jean-Baptiste Poline. 2022. Beyond advertising: New infras- tructures for publishing integrated research objects. PLOS Computational Biology 18, 1 (2022), e1009651

    2022

  12. [20]

    Scott C Edmunds, Peter Li, Christopher I Hunter, Si Zhe Xiao, Robert L David- son, Nicole Nogoy, and Laurie Goodman. 2017. Experiences in integrated data and research object publishing using GigaDB. International Journal on Digital Libraries 18, 2 (2017), 99–111

    2017

  13. [21]

    Johannes Elmsheuser, A Krasznahorkay, E Obreshkov, Alexander Undrus, ATLAS Collaboration, et al. 2017. A Roadmap to Continuous Integration for ATLAS Software Development. In Journal of Physics: Conference Series , Vol. 898. IOP Publishing, 072009

    2017

  14. [22]

    Carles Fernández-Prades, Jordi Vilà-Valls, Javier Arribas, and Antonio Ramos

  15. [23]

    Todd Gamblin and Daniel S Katz. 2022. Overcoming challenges to continuous integration in HPC. Computing in Science & Engineering 24, 6 (2022), 54–59

    2022

  16. [24]

    GitBucket. [n. d.]. GitBucket: A Git platform. https://gitbucket.github.io. Ac- cessed: 2025-04-14

    2025

  17. [25]

    GitHub. [n. d.]. GitHub Actions. https://github.com/features/actions. Accessed: 2025-04-14

    2025

  18. [26]

    GitHub. [n. d.]. GitHub: Build and ship software on a single, collaborative platform. https://github.com. Accessed: 2025-04-14

    2025

  19. [27]

    GitHub. [n. d.]. Marketplace - Tools to improve your workflow. https://github. com/marketplace. Accessed: 2025-04-14

    2025

  20. [28]

    GitLab. [n. d.]. CI/CD Catalog: GitLab. https://gitlab.com/explore/catalog. Ac- cessed: 2025-04-14

    2025

  21. [29]

    GitLab. [n. d.]. Get started with GitLab CI/CD - GitLab Docs. https://docs.gitlab. com/ci/. Accessed: 2025-04-14

    2025

  22. [30]

    GitLab. [n. d.]. The most-comprehensive AI-powered DevSecOps platform: Git- Lab. https://gitlab.com. Accessed: 2025-04-14

    2025

  23. [31]

    Google. [n. d.]. Secret Manager: Google Cloud. https://cloud.google.com/security/ products/secret-manager?hl=en. Accessed: 2025-04-14

    2025

  24. [32]

    HashiCorp. [n. d.]. HashiCorp: An IBM Company. https://www.hashicorp.com/ en. Accessed: 2025-04-14

    2025

  25. [33]

    Wozniak, Kyle Chard, Ian Foster, Rafael Ferreira da Silva, Shantenu Jha, and Daniel Laney

    Mihael Hategan-Marandiuc, Andre Merzky, Nicholson Collier, Ketan Mahesh- wari, Jonathan Ozik, Matteo Turilli, Andreas Wilke, Justin M. Wozniak, Kyle Chard, Ian Foster, Rafael Ferreira da Silva, Shantenu Jha, and Daniel Laney

  26. [34]

    Demian Hespe, Lukas Hübner, Florian Kurpicz, Peter Sanders, Matthias Schimek, Daniel Seemaier, Christoph Stelz, and Tim Niklas Uhl. 2024. Kamping: Flexible and (near) zero-overhead c++ bindings for mpi. arXiv preprint arXiv:2404.05610 (2024)

    2024 arXiv

  27. [35]

    Torsten Hoefler and Roberto Belli. 2015. Scientific benchmarking of parallel computing systems: twelve ways to tell the masses when reporting performance results. In Proceedings of the international conference for high performance com- puting, networking, storage and analysis . 1–12

    2015

  28. [36]

    Jenkins. [n. d.]. Jenkins. https://www.jenkins.io. Accessed: 2025-04-14

    2025

  29. [37]

    Ivo Jimenez, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Jay Lofstead, Car- los Maltzahn, Kathryn Mohror, and Robert Ricci. 2017. PopperCI: Automated reproducibility validation. In 2017 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). IEEE, 450–455

    2017

  30. [38]

    Vasileios Karakasis, Theofilos Manitaras, Victor Holanda Rusu, Rafael Sarmiento- Pérez, Christopher Bignamini, Matthias Kraushaar, Andreas Jocksch, Samuel Omlin, Guilherme Peretti-Pezzi, João PSC Augusto, et al. 2020. Enabling contin- uous testing of HPC systems using ReFrame....

    2020

  31. [39]

    Kate Keahey, Pierre Riteau, Dan Stanzione, Tim Cockerill, Joe Mambretti, Paul Rad, and Paul Ruth. 2019. Chameleon: A scalable production testbed for computer science research. In Contemporary High Performance Computing . CRC Press, 123–148

    2019

  32. [40]

    Samuel Khuvis, Zhi-Qiang You, Heechang Na, Scott Brozell, Eric Franz, Trey Dockendorf, Judith Gardiner, and Karen Tomko. 2019. A continuous integration- based framework for software management. In Practice and Experience in Ad- vanced Research Computing 2019: Rise of the Machi...

    2019

  33. [41]

    Mallory C Kidwell, Ljiljana B Lazarević, Erica Baranski, Tom E Hardwicke, Sarah Piechowski, Lina-Sophia Falkenberg, Curtis Kennett, Agnieszka Slowik, Carina Sonnleitner, Chelsey Hess-Holden, et al . 2016. Badges to acknowledge open practices: A simple, low-cost, effective meth...

    2016

  34. [42]

    Matthew Krafczyk, August Shi, Adhithya Bhaskar, Darko Marinov, and Victoria Stodden. 2019. Scientific tests and continuous integration strategies to enhance reproducibility in the scientific software context. In Proceedings of the 2nd in- ternational workshop on practical repr...

    2019

  35. [43]

    Gregory M Kurtzer, Vanessa Sochat, and Michael W Bauer. 2017. Singularity: Scientific containers for mobility of compute. PloS one 12, 5 (2017), e0177459

    2017

  36. [44]

    Jason M Lohrey, Neil EB Killeen, and Gary F Egan. 2009. An integrated ob- ject model and method framework for subject-centric e-Research applications. Frontiers in Neuroinformatics 3 (2009), 532

    2009

  37. [45]

    Caetano Melone and Steve Jones. 2023. Verifying functionality and performance of HPC applications with continuous integration. In Practice and Experience in Advanced Research Computing 2023: Computing for the Common Good . 460–462

    2023

  38. [46]

    Microsoft. [n. d.]. Key Vault: Microsoft Azure. https://azure.microsoft.com/en- us/products/key-vault. Accessed: 2025-04-14

    2025

  39. [47]

    Felix Mölder, Kim Philipp Jablonski, Brice Letcher, Michael B Hall, Christo- pher H Tomkins-Tinch, Vanessa Sochat, Jan Forster, Soohyun Lee, Sven O Twardziok, Alexander Kanitz, et al. 2021. Sustainable data analysis with Snake- make. F1000Research 10 (2021), 33

    2021

  40. [48]

    NERSC. [n. d.]. GitHub - NERSC/shifter: Shifter - Linux Containers for HPC. https://github.com/NERSC/shifter. Accessed: 2025-04-14

    2025

  41. [49]

    Nextflow. [n. d.]. A DSL for parallel and scalable computational pipelines: Nextflow. https://www.nextflow.io. Accessed: 2025-04-14

    2025

  42. [50]

    Benjamin Pachev, Georgia Stuart, and Clint Dawson. 2022. Continuous inte- gration for HPC with Github Actions and Tapis. In Practice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You . 1–4

    2022

  43. [51]

    Alessandro Vittorio Papadopoulos, Laurens Versluis, André Bauer, Nikolas Herbst, Jóakim Von Kistowski, Ahmed Ali-Eldin, Cristina L Abad, José Nel- son Amaral, Petr Tuma, and Alexandru Iosup. 2019. Methodological principles for reproducible performance evaluation in cloud compu...

    2019

  44. [52]

    Sam Parsons, Flávio Azevedo, Mahmoud M Elsherif, Samuel Guay, Owen N Shahim, Gisela H Govaart, Emma Norris, Aoife O’mahony, Adam J Parker, Ana Todorovic, et al. 2022. A community-sourced glossary of open scholarship terms. Nature human behaviour 6, 3 (2022), 312–318

    2022

  45. [53]

    Reid Priedhorsky and Tim Randles. 2017. Charliecloud: Unprivileged containers for user-defined software stacks in HPC. In Proceedings of the international conference for high performance computing, networking, storage and analysis . 1–10

    2017

  46. [54]

    John Raicu, Valerie Hayot-Sasson, Kyle Chard, and Ian Foster. 2023. Navigating the molecular maze: A Python-powered approach to virtual drug screening. Addressing Reproducibility Challenges in HPC with Continuous Integration In SC23: International Conference for High Performan...

    2023

  47. [55]

    Zebula Sampedro, Aaron Holt, and Thomas Hauser. 2018. Continuous integration and delivery for HPC: using singularity and Jenkins. InProceedings of the Practice and Experience on Advanced Research Computing: Seamless Creativity . 1–6

    2018

  48. [56]

    Geir Kjetil Sandve, Anton Nekrutenko, James Taylor, and Eivind Hovig. 2013. Ten simple rules for reproducible computational research. PLoS computational biology 9, 10 (2013), e1003285

    2013

  49. [57]

    Jacob Sanz-Robinson, Arman Jahanpour, Natalie Phillips, Tristan Glatard, and Jean-Baptiste Poline. 2022. NeuroCI: Continuous Integration of Neuroimaging Results Across Software Pipelines and Datasets. In 2022 IEEE 18th International Conference on e-Science (e-Science) . IEEE, 105–116

    2022

  50. [58]

    Tarek Sherif, Pierre Rioux, Marc-Etienne Rousseau, Nicolas Kassis, Natacha Beck, Reza Adalat, Samir Das, Tristan Glatard, and Alan C Evans. 2014. CBRAIN: a web-based, distributed computing platform for collaborative neuroimaging research. Frontiers in neuroinformatics 8 (2014), 54

    2014

  51. [59]

    Fernández, Daniel Garijo, Björn Grüning, Marco La Rosa, Si- mone Leo, Eoghan Ó Carragáin, Marc Portier, Ana Trisovic, RO-Crate Com- munity, Paul Groth, and Carole Goble

    Stian Soiland-Reyes, Peter Sefton, Mercè Crosas, Leyla Jael Castro, Frederik Coppens, José M. Fernández, Daniel Garijo, Björn Grüning, Marco La Rosa, Si- mone Leo, Eoghan Ó Carragáin, Marc Portier, Ana Trisovic, RO-Crate Com- munity, Paul Groth, and Carole Goble. 2022. Packagi...

    2022 doi

  52. [60]

    Spack. [n. d.]. Spack. https://spack.io. Accessed: 2025-04-14

    2025

  53. [61]

    Joe Stubbs, Richard Cardone, Mike Packard, Anagha Jamthe, Smruti Padhy, Steve Terry, Julia Looney, Joseph Meiring, Steve Black, Maytal Dahan, et al . 2021. Tapis: An API platform for reproducible, distributed computational research. In Advances in Information and Communication...

    2021

  54. [62]

    Tapis Project. [n. d.]. Welcome to Tapis v3 — Tapis documentation. https: //tapis.readthedocs.io/. Accessed: 2025-04-14

    2025

  55. [63]

    Travis CI. [n. d.]. Simple, Flexible, Trustworthy CI/CD Tools - Travis CI. https: //www.travis-ci.com. Accessed: 2025-04-14

    2025

  56. [64]

    Nicolas Vallet, David Michonneau, and Simon Tournier. 2022. Toward practical transparent verifiable and long-term reproducible research using Guix. Scientific Data 9, 1 (2022), 597

    2022

  57. [65]

    John Vivian, Arjun Arkal Rao, Frank Austin Nothaft, Christopher Ketchum, Joel Armstrong, Adam Novak, Jacob Pfeil, Jake Narkizian, Alden D Deran, Audrey Musselman-Brown, et al . 2017. Toil enables reproducible, open source, big biomedical data analyses. Nature biotechnology 35,...

    2017

  58. [66]

    Adina S Wagner, Laura K Waite, Małgorzata Wierzba, Felix Hoffstaedter, Alexan- der Q Waite, Benjamin Poldrack, Simon B Eickhoff, and Michael Hanke. 2022. FAIRly big: A framework for computationally reproducible processing of large- scale data. Scientific data 9, 1 (2022), 80

    2022

  59. [67]

    Mark D Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Apple- ton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E Bourne, et al. 2016. The FAIR Guiding Principles for scientific data management and stewardship...

    2016

  60. [68]

    Michael Woelfle, Piero Olliaro, and Matthew H Todd. 2011. Open science is a research accelerator. Nature chemistry 3, 10 (2011), 745–748

    2011

  61. [2018]

    Continuous reproducibility in GNSS signal processing.IEEE Access 6 (2018), 20451–20463

    2018

  62. [2023]

    In IEEE 19th International Conference on e-Science (e-Science)

    PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs. In IEEE 19th International Conference on e-Science (e-Science) . IEEE, 1–10. doi:10.1109/e-science58273.2023.10254912

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.