{"id":"a15afa77-444f-4c7d-a6d4-eb7d97f1235a","arxiv_id":"2508.21289","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new GitHub Action, CORRECT, uses Globus Compute to run continuous integration tests on remote HPC systems, aiming to make reproducibility evaluation easier.","lead":"The authors built CORRECT, a GitHub Action that runs software tests on remote supercomputers through the Globus Compute platform and logs the results. The paper argues that automated, documented testing can take the place of direct access when judging whether HPC research is reproducible.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CORRECT's substitution claim is undercut by its own §7.4: without environment capture, logged runs cannot validate reproducibility; evaluation never tests the no-access judgment scenario.","rationale":"Good-faith reading: the paper is a survey plus a prototype (CORRECT) that securely dispatches tests to remote HPC via GitHub Actions and Globus Compute. The prototype works for the demonstrated dispatch use case, and the survey is a useful synthesis. The load-bearing concern is not about the tool's operation but about the paper's central claim: that CI with complete provenance can substitute for direct resource access when evaluating reproducibility. This claim requires that the CI artifact set include enough environmental/provenance detail for an independent judge to assess reproducibility. CORRECT currently returns only stdout/stderr, and §7.4 concedes that without environment information, the recorded runs cannot validate reproducibility. The evaluation never tests the no-access judgment scenario; it tests execution feasibility and cross-site runtime differences. Thus the central argument rests on an assumption the authors themselves identify as unmet. This does not negate the value of the tool or survey, but it means the paper should be read as a feasibility study, not as a demonstration of the substitution claim. The reader's CONDITIONAL verdict already captures this, so no verdict change is needed.","tokens_in":19380,"tokens_out":4206,"duration_ms":41074,"concrete_test":"Have an independent reproducibility reviewer (not an author) attempt to validate a specific numerical result from the KaMPIng artifact evaluation using only the CORRECT workflow artifacts from Section 6.3—stdout, stderr, and run logs—with no access to the Chameleon Cloud instance or the original environment. The reviewer must state whether the result is reproducible. If the reviewer cannot make that determination from the captured artifacts alone, the paper's claim that CI plus provenance can substitute for resource access is falsified; if they can, the claim gains support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that CI plus 'complete provenance information' can substitute for direct resource access in reproducibility evaluation. But CORRECT (§5.3) returns only standard output/error to the GitHub runner, and does not capture resource configuration, module versions, compiler flags, or hardware details of the remote endpoint. Section 7.4 explicitly concedes: 'Without information about the environment, users can only see the results of previous executions, but that alone cannot validate reproducibility without access to the environment.' This admission directly contradicts the load-bearing premise. The evaluation does not exercise the substitution scenario: ParslDock measures runtime differences across sites, PSI/J's pytest failed, and KaMPIng reproduced only the already-claimed Cloud VM results. None tests whether a reviewer can judge reproducibility from CORRECT's artifacts alone. Therefore the central claim is unsubstantiated, and the paper's own limitation statement makes the gap explicit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that HPC-specific reproducibility barriers can be lowered by substituting direct access to HPC resources with continuous integration (CI) plus complete provenance information. It surveys reproducibility initiatives and existing HPC CI frameworks, then presents CORRECT, a GitHub Action that uses Globus Compute to execute user-defined functions or shell commands on remote endpoints and return their stdout/stderr to the GitHub runner. The evaluation covers three cases: ParslDock tests run across three sites, PSI/J tests run on Purdue Anvil, and reproduction of the KaMPIng artifact evaluation on Chameleon Cloud.","tokens_in":19540,"tokens_out":2939,"duration_ms":31791,"significance":"If the central claim were established, the paper would offer a practical, low-friction path for reproducibility evaluation in HPC, and the survey of current reproducibility and CI efforts is useful. CORRECT builds on established platforms (Globus Compute, GitHub Actions) and the authors publish the action for community use. The paper honestly reports failures and limitations, including the PSI/J test failure and the lack of environment capture. However, the evaluation demonstrates remote dispatch and log retrieval, not the stronger claim that CI logs plus provenance can substitute for access to the original environment. The paper's own Section 7.4 concedes that the current tool cannot validate reproducibility without environment information, so the central premise is not yet supported. The tool is a promising prototype rather than a validated solution to the stated problem.","major_comments":[{"comment":"The load-bearing premise is that CI execution records plus 'complete provenance information' can substitute for direct resource access. CORRECT, as described in Section 5.3, returns only the standard output and standard error of the executed function and does not capture the remote environment, resource configuration, module versions, compiler flags, or hardware details. Section 7.4 explicitly states: 'Without information about the environment, users can only see the results of previous executions, but that alone cannot validate reproducibility without access to the environment.' This directly contradicts the abstract's substitution claim. Either CORRECT must be extended to capture and publish environment/provenance information, or the paper must be substantially reframed around the weaker, supported claim that CORRECT enables repeated remote execution with logged output.","section":"Abstract, Section 5.3, Section 7.4"},{"comment":"None of the three evaluations tests the central substitution scenario: that a reviewer with no access to the execution environment can judge reproducibility from CORRECT's artifacts alone. Section 6.1 records runtimes across sites but does not assess reproducibility of results; Section 6.2 reports that the PSI/J pytest run failed due to an error in the PSI/J codebase; Section 6.3 reproduces only the KaMPIng artifacts that were already reported reproducible on Chameleon Cloud. Section 6.3 states the authors 'compare our findings with those reported in the paper' and reproduce 'the artifacts that were reported to be reproducible on Chameleon Cloud.' This is a repeatability check by the tool developers, not an independent evaluation of whether logged outputs are sufficient for a no-access reproducibility verdict. The paper needs an evaluation that directly exercises the claimed use case, fo","section":"Section 6"},{"comment":"The security argument is not fully specified. The paper argues that GitHub environment secrets with required review 'ensures that the person authorizing the execution maps to a user at the site at which the code is executed,' but GitHub environment protection does not, by itself, bind a specific remote HPC account to a specific human reviewer; the mapping relies on the recommendation that 'there is only one reviewer per environment.' This is administrative guidance rather than an enforced mechanism. Since satisfying HPC security requirements is one of the two primary goals stated in Section 5, the paper should provide a concrete threat model and describe how the proposed configuration enforces the claimed identity mapping, including what happens when environments are misconfigured or when multiple reviewers exist.","section":"Section 5.2"}],"minor_comments":[{"comment":"Typo: 'Juypter' should be 'Jupyter'.","section":"Section 2"},{"comment":"Reference [5] lists the author as 'Association for Computing Machinert' (typo). Reference [24] is titled 'GitBucket' but the introduction text refers to 'Bitbucket'; please align the citation.","section":"References"},{"comment":"The figure caption/labels are confusing: the top panel is described as 'Error and full execution stdout' and the bottom as 'Execution stdout.' Clarify what each panel shows and how they differ.","section":"Section 6.2 / Figure 5"},{"comment":"The sentence 'we installed via Conda the Protein Docking application...' is grammatically awkward. Also, the claim that 'short duration tests highlight the benefits of adopting a FaaS based model' is not directly evidenced by the presented runtime comparison.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The authors are closely affiliated with the development of Globus Compute and Parsl, which are central to the proposed tool. This is not disqualifying, but it makes independent validation more important. The manuscript would be significantly stronger with an evaluation performed by an outside group, or at minimum a scenario in which reviewers unfamiliar with the tool attempt to use the artifacts to judge reproducibility. The paper does honestly disclose its own limitations, which is to its credit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a useful systems paper that should be published as a feasibility study, but its main conceptual claim—that CI logs plus provenance can substitute for direct access to HPC resources—is not backed by the experiments and is contradicted by the authors' own Section 7.4. The stress-test note is right: CORRECT returns stdout/stderr from the remote site, does not capture the resource configuration, and Section 7.4 says that without environment information, users can only see execution results, which \"alone cannot validate reproducibility without access to the environment.\" That is the load-bearing premise, and the paper itself concedes it.\n\nWhat is genuinely new: CORRECT is a real artifact, a GitHub Action that uses Globus Compute to dispatch tests from GitHub-hosted runners to multiple HPC endpoints. That is a clean integration, and the three case studies show it can distribute ParslDock tests across sites, return logs from a failing PSI/J pytest, and rerun KaMPIng artifacts. The survey of HPC reproducibility initiatives and existing CI frameworks is competent and useful, especially the comparison of Jacamar, Tapis, and others. The tool is publicly available, which earns real credit.\n\nWhere the soft spots are, in proportion: the evaluation never actually tests the substitution scenario. ParslDock measures runtime differences; the PSI/J case failed; KaMPIng only reproduced results already known to work on Chameleon. No reviewer is shown using CORRECT's artifacts alone to judge reproducibility. Security is described by design, not demonstrated against a concrete threat model. These are real gaps, and the authors are honest about at least one of them. The reliance on Globus Compute, Parsl, and PSI/J—all from the same group—is not a flaw by itself, since the components are public, but the evaluation is dogfooding and should be read that way.\n\nMinor issues: output files are not copied back to the runner, GitHub artifacts expire after 90 days, and environment capture is absent. The authors mention potential fixes, but as written the tool is a harness for running tests, not a complete provenance system.\n\nFor readers working on HPC reproducibility or CI for scientific software, the survey and the prototype are worth a look. I would treat this as a conditional accept: reframe the contribution as a multi-site CI harness for HPC, explicitly defer or implement environment capture, and test the reviewer-without-access workflow. That is a strong paper. It deserves peer review, not a desk reject.","headline":"Solid feasibility study for a useful multi-site HPC CI action, but the paper's central claim that CI logs plus provenance can substitute for direct resource access is unsupported and contradicted by its own Section 7.4.","tokens_in":20056,"tokens_out":2209,"would_cite":true,"duration_ms":23257,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that continuous integration, backed by complete provenance, can substitute for direct access to HPC resources, and presents CORRECT, a GitHub Action that runs reproducibility tests on remote HPC sites and returns documented","keywords":["reproducibility","continuous integration","high-performance computing","provenance","artifact evaluation","GitHub Actions","Globus Compute","remote execution"],"falsifier":"A concrete test: take the KaMPIng artifact workflow, which CORRECT passes on Chameleon Cloud, and have a fresh reviewer manually reproduce the same artifact on the same instance class using only the CORRECT-published logs and the authors' artifact instructions. If the manual reproduction diverges from the paper's reported trends, the claim that CI records can substitute for access fails.","tokens_in":19275,"feed_emoji":"🔁","tokens_out":8505,"duration_ms":80564,"temperature":0.7,"pith_summary":"High-performance computing results are hard to reproduce because the specialized systems they run on are tightly controlled and hard to access. The paper argues that this barrier can be lowered by continuous integration: if a project's tests are run automatically on remote HPC resources, with the full execution log and provenance saved, a reviewer can assess reproducibility without touching the original system. To make that work in practice, the authors built CORRECT, a GitHub Action that uses the Globus Compute function-as-a-service platform to execute reproducibility tests on arbitrary HPC and cloud endpoints from a standard GitHub workflow. They demonstrate it on three HPC applications: multi-site protein-docking tests, scheduler-abstraction software tests, and reproduction of a paper's artifacts. The paper's central claim is that better HPC-compliant CI solutions will improve reproducibility of applications.","feed_headline":"CI plus provenance can substitute for direct HPC access in reviews","feed_subtitle":"New GitHub Action CORRECT runs reproducibility checks on remote HPC systems and logs the evidence.","key_machinery":"Key machinery: CORRECT, a GitHub Action (COntinuous Reproducibility with a Remote Execution Computing Tool) that lets a normal GitHub workflow execute tests on remote HPC sites through Globus Compute, a function-as-a-service platform. The action authenticates with Globus Auth secrets stored in GitHub environments, invokes an endpoint whose administrator can restrict which functions run, and returns stdout and stderr as workflow artifacts; provenance is meant to come from the captured command, logs, and in principle environment information. This combination—CI trigger, FaaS execution, and logged artifacts—is what carries the argument that reproducible evaluation can happen without direct acce","core_discovery":"The paper's discovery is a working route from a GitHub repository to execution on restricted HPC systems for reproducibility checking. CORRECT wraps Globus Compute function invocation as a GitHub Action: a workflow triggers a run, the runner authenticates via Globus Auth, the function clones the repository and runs user-specified tests (a shell command or a pre-registered Python function) on an HPC endpoint, and stdout and stderr are returned to the GitHub interface and stored as artifacts. Security is handled by GitHub environment secrets with manual approval, by Globus Compute multi-user endpoints that map execution to the correct user identity, and by endpoint-approved function lists. The","pith_inferences":["The paper leaves implicit that its substitution claim could be tested directly: pit CORRECT's CI logs against a blind hands-on reproduction by an independent reviewer, but the paper does not report such a comparison.","The scheme could be extended to capture full system provenance—for example scheduler, node type, compiler, and library versions—by adding a second CORRECT call that runs environment-reporting commands and stores the output; this would address the paper's stated limitation.","The same pattern could support a new artifact badge level for CI-verified reproducibility, where the automated remote run supplies the evidence; that would make badge evaluation faster but would inherit the substitution assumption."],"forward_implications":["Reproducibility reviewers for conferences could evaluate a paper by reading CI-generated execution logs and artifacts instead of obtaining an allocation on the original HPC site.","A single CORRECT workflow can run the same test suite on several HPC and cloud sites by swapping endpoint identifiers, giving cross-platform evidence of reproducibility without multiple manual setups.","Because GitHub environment secrets require human approval, HPC sites can keep their security boundaries while still allowing automated, documented test runs; routine tests can run on cloud endpoints while HPC runs are gated by review.","Execution results can be committed to the repository or uploaded as artifacts, creating a persistent historical record of reproducibility evaluations that can be inspected after the original run.","The pattern generalizes to other CI platforms such as GitLab, since the action runs on a standard CI runner and only needs Python installed."],"supporting_citations":[{"why":"Defines reproducibility and the ACM artifact review and badging levels that motivate the paper.","marker":"[5]"},{"why":"Shows conferences like CCGrid recommend CI as part of the badge-awarding process, the gap CORRECT targets.","marker":"[10]"},{"why":"Supplies the Globus Compute function-as-a-service platform that CORRECT builds on for remote execution.","marker":"[13]"},{"why":"Provides the multi-user endpoint model and identity mapping that let CORRECT run securely on HPC resources.","marker":"[3]"},{"why":"Coins the term continuous reproducibility and demonstrates CI-based reproducibility as a conceptual predecessor.","marker":"[22]"},{"why":"Prior automated reproducibility validation via CI, supporting the paper's claim that CI can aid reproducibility.","marker":"[37]"},{"why":"Shows how scientific tests and CI strategies can enhance computational reproducibility.","marker":"[42]"},{"why":"Provides NeuroCI, an HPC CI framework that captures task provenance and serves as a comparison point.","marker":"[57]"},{"why":"Describes Jacamar CI, an HPC CI framework addressing security, which CORRECT compares against.","marker":"[2]"},{"why":"Describes a closely related GitHub Actions plus Tapis approach for HPC CI, the nearest existing baseline.","marker":"[50]"}],"fun_headline_variants":["CORRECT GitHub Action runs HPC tests via remote execution","CI plus full provenance replaces direct HPC access for reviews","CORRECT automates reproducibility checks on restricted HPC systems","Restricted HPC access? CORRECT runs CI tests remotely"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper's argument depends on the idea that an automated test run's saved log and history can stand in for being able to log in to the original supercomputer yourself; if that exchange is not valid, the main claim falls.","fun_headline_variants_meta":{"raw":{"variants":["CORRECT GitHub Action runs HPC tests via remote execution","CI plus full provenance replaces direct HPC access for reviews","CORRECT automates reproducibility checks on restricted HPC systems","Restricted HPC access? CORRECT runs CI tests remotely"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000712,"raw_usage":{"total_tokens":3003,"prompt_tokens":670,"completion_tokens":2333,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":2265}},"tokens_in":414,"tokens_out":2333,"duration_ms":18024,"temperature":1.0,"reasoning_tokens":2265,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:24:43.395338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: take the KaMPIng artifact workflow, which CORRECT passes on Chameleon Cloud, and have a fresh reviewer manually reproduce the same artifact on the same instance class using only the CORRECT-published logs and the authors' artifact instructions. If the manual reproduction diverges from the paper's reported trends, the claim that CI records can substitute for access fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines reproducibility and the ACM artifact review and badging levels that motivate the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows conferences like CCGrid recommend CI as part of the badge-awarding process, the gap CORRECT targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Globus Compute function-as-a-service platform that CORRECT builds on for remote execution."},{"cited_title":"Katz, Kevin Hunter Kesling, Chris Janidlo, Reid Mello, and Lei Wang","cited_arxiv_id":null,"evidence_quote":"Provides the multi-user endpoint model and identity mapping that let CORRECT run securely on HPC resources."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Coins the term continuous reproducibility and demonstrates CI-based reproducibility as a conceptual predecessor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior automated reproducibility validation via CI, supporting the paper's claim that CI can aid reproducibility."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows how scientific tests and CI strategies can enhance computational reproducibility."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides NeuroCI, an HPC CI framework that captures task provenance and serves as a comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes Jacamar CI, an HPC CI framework addressing security, which CORRECT compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes a closely related GitHub Actions plus Tapis approach for HPC CI, the nearest existing baseline."}],"review_version":1}