{"id":"75721306-230f-4430-80df-6f53dad8523b","arxiv_id":"2412.10080","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Stimela2 introduces a YAML-based workflow framework that combines readable linear recipes with containerization and Kubernetes or Slurm backends for reproducible, scalable radio astronomy data reduction.","lead":"This paper describes Stimela2, a new framework for writing radio astronomy data processing workflows as simple YAML recipes. It aims to make data reduction pipelines easier to write, more reproducible, and able to run on local clusters or in the cloud.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reproducibility pillar is explicitly unproven: §10.2 says 'this still needs to be proven in practice,' while the abstract promises 'fully reproducible workflows'; no cross-architecture or bitwise reproducibility test is reported.","rationale":"The reader's weakest assumption targets cloud scalability and the Kubernetes autoscaler's economic viability. That is a legitimate concern, but it is partly buttressed by prior papers in the series reporting AWS-based benchmark runs, and §10.1 is explicitly modest ('we don't claim... cost-effective'). The reproducibility pillar is more directly load-bearing because the paper's own §10.2 states that reproducibility 'still needs to be proven in practice' and that total replicability may be unattainable, yet the abstract and conclusion assert 'fully reproducible workflows' and 'practical reproducibility (provided by containerization).' The framework's deliberate allowance of native backends, local binaries, and per-step backend overrides means reproducibility is opt-in rather than guaranteed. The proposed concrete test—re-running a published recipe on different architectures and comparing bitwise outputs—would settle whether containerization alone delivers the claimed reproducibility. If it does, the concern is rebutted; if it does not, the central claim needs to be qualified. Either way, the reader's CONDITIONAL verdict remains appropriate, so no verdict change is needed.","tokens_in":30040,"tokens_out":4463,"duration_ms":53247,"concrete_test":"Reproduce the published PARROT recipes (Smirnov et al., 2024) using pinned cult-cargo image digests on two different CPU architectures (e.g., x86_64 Intel vs AMD, or x86_64 vs ARM), and also in native vs Apptainer mode. Compute SHA-256 hashes of the final FITS images and the maximum absolute pixel difference between runs. Additionally, inspect the generated dependencies file to confirm it records image digests and recipe/config hashes. If outputs are bit-identical, the practical-reproducibility claim survives this check; if they differ materially, the abstract's 'fully reproducible' promise is falsified and the conclusion should be softened to conditional reproducibility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §10—that Stimela2 achieves 'practical reproducibility (provided by containerization)'—requires that containerized workflows actually reproduce results in practice. The paper's own §10.2 ('Wither reproducibility?') concedes the opposite: 'this still needs to be proven in practice,' 'total replicability may always remain hostage to the robustness of algorithm implementations,' and the proposed 'certifiable workflow' mechanism is future work. The abstract's 'fully reproducible workflows' is therefore stronger than anything demonstrated. Containerization plus versioned cult-cargo images addresses software availability, but not numerical/architectural variation, and the framework deliberately permits mixing native and containerized backends (e.g., §4.4 and Appendix B), which can silently make a published recipe non-reproducible. No reproducibility benchmark, bitwise comparison across hosts, or cross-architecture test is provided. This is not an external disagreement with consensus; it is an internal gap admitted by the authors, and it directly weakens the strongest claim's reproducibility component.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes Stimela2, a workflow framework for radio-astronomy data reduction that represents pipelines as linear YAML recipes composed of typed \"cab\" definitions. The paper presents the architecture: schema-based parameter validation, a substitution/evaluation language, composability via _include/_use constructs, cab flavours for native executables, Python functions, CASA tasks, and inline Python code, and execution backends for native, Apptainer, Slurm, and Kubernetes environments. It also covers logging, profiling, and resource-management features, and illustrates the design with a MeerKAT simulation recipe and a mixed local/Kubernetes deployment example in Appendix B. The paper is an overview and design document; quantitative benchmarks are not reported here but are attributed to companion papers in the same Africanus series.","tokens_in":30216,"tokens_out":7731,"duration_ms":86352,"significance":"If the design performs as described, Stimela2 addresses a real community need: modular, human-readable, portable recipes that can run on a laptop, an HPC cluster, or a cloud deployment, with containerized dependencies curated through cult-cargo. The framework is open source, the companion package provides versioned container images, and the paper includes substantial runnable examples. The claimed value is therefore credible as a software-engineering contribution, and the previously published RATT PARROT recipes give some external evidence that the system is usable in real science analysis. However, the paper's own text shows that the reproducibility pillar is not yet demonstrated, and the cloud cost/scalability claims rest on companion papers rather than on measurements reported here. The contribution is best characterized as a well-specified design with partial validation, not a fully evaluated framework as the abstract and conclusions sometimes imply.","major_comments":[{"comment":"The abstract promises \"fully reproducible workflows\" and §8 states that the Apptainer backend \"allows for true zero-install, fully reproducible workflows\", but §10.2 explicitly concedes that reproducibility \"still needs to be proven in practice\", that \"total replicability may always remain hostage to the robustness of algorithm implementations\", and that a formal \"certifiable workflow\" mechanism is future work. In addition, §4.4 and Appendix B deliberately permit per-cab and per-step switching to native backends, which §8 itself notes \"certainly does not promote reproducibility\". This is an internal inconsistency in one of the three pillars of the central claim in §10. The manuscript should either report a concrete reproducibility experiment (for example, a containerized recipe executed on two different hosts or architectures, compared bitwise or with a stated numerical tolerance) or qualify the abstract and conclusions so that they claim reproducible software environments rather than fully reproducible results. As written, the abstract overstates what the paper demonstrates.","section":"Abstract; §8; §10.2"},{"comment":"The central cloud-computing claim is that Kubernetes autoscaling with predefined pod CPU/RAM requests can resolve the \"thick-thin\" resource-profile problem by bringing virtual machines up and down on demand, allocating small cheap instances to thin steps and large instances to thick steps. The paper gives no measurements of autoscaling overhead, cold-start latency, scheduling delay, or cost versus static allocation to support this. §9.2 states that the Kubernetes backend currently collects only basic metrics (CPU usage, RAM usage, pod count, elapsed time) and explicitly leaves \"dollars per run\" as future work. The anecdotal totals in §10.1 ($25,000 for the whole paper series, under $1,000 for a workshop) do not quantify the marginal cost of autoscaling and cannot substitute for a controlled comparison. If scalable cloud deployment is a central contribution, the paper should include such measurements or clearly mark the autoscaling discussion as expected behavior rather than demonstrated capability; the current wording is stronger than the evidence presented in this manuscript.","section":"§10.1; §9.2"},{"comment":"The paper frames Stimela2's aim as occupying the middle ground between casual scripting and dataflow frameworks such as CWL and DALiuGE, and §10 concludes that Stimela2 \"has largely achieved\" this aim. However, the paper provides no evidence about the ease-of-use side of that trade-off: no usability study, no recipe-authoring time comparison, no assessment of recipe readability against CWL or DALiuGE, and no systematic demonstration that the scatter-gather constructs scale beyond the examples in companion papers. This does not invalidate the architecture, but it means the \"largely achieved\" conclusion is an assertion rather than a finding. Since companion papers (Kenyon et al. 2024; Bester et al. 2024; Smirnov et al. 2024) contain empirical demonstrations, the authors could include a short synthesis of those results and explicitly delineate which claims are established there and which remain design goals. In the absence of that, the conclusion overstates the evidence available in this manuscript.","section":"§1; §10"}],"minor_comments":[{"comment":"The word \"incoperating\" should be \"incorporating\".","section":"§1"},{"comment":"In the sentence \"A cab definition is simply a section of YAML, Fig. 2 being an example\", the reference should be to Listing 2 rather than Fig. 2.","section":"§4.1"},{"comment":"The word \"informaton\" should be \"information\".","section":"§4.3"},{"comment":"The backend heading \"Kkubernetes\" should be \"Kubernetes\".","section":"§8"},{"comment":"The heading \"Wither reproducibility?\" appears to be a typo for \"Whither reproducibility?\", unless the decay sense of \"wither\" is intentionally intended.","section":"§10.2"},{"comment":"In the `compare_models` step, `epsilon` is set to `=recipe.steps.sara.tol`, but the `sara` step shown in the same augmentation file does not define a `tol` parameter; if `tol` is provided by the base recipe of Bester et al. (2024), this should be stated explicitly in the text.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The main risk is not the architecture, which appears coherent and is supported by runnable examples, but the gap between the abstract/conclusion claims and the evidence inside the manuscript, particularly on reproducibility. The authors themselves concede the reproducibility claim is unproven, so this is an internal fixable issue rather than an external disagreement. I would advise the editor to request either a concrete reproducibility demonstration or a consistent softening of the claims, plus at least a summary of the companion-paper benchmarks that support the cloud-scalability assertions. This is not rejectable as long as the authors are willing to align their claims with their evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Stimela2 is a genuine redesign, not a rebrand. The move from a Python API to YAML recipes with _use/_include composition, a formula language, dynamic schemas, and per-step backend selection is substantive, and the paper explains it clearly. The authors also do the honest thing in §10.2: they state that reproducibility \"still needs to be proven in practice\" and that total replicability may be hostage to algorithm implementation. That concession is more credible than the abstract's \"fully reproducible workflows\" — the abstract oversells what the paper shows.\n\nWhat's missing is measurement. There are no benchmarks against CWL or DALiuGE, no performance numbers for the slurm/Kubernetes backends, and no reproducibility test across architectures or even across two runs. The cloud claims rest on prior papers in the same series plus anecdotal costs. For a framework paper, that may be okay if the design is the contribution and the code is public, but the title promises \"scalable and reproducible workflows,\" and the paper does not demonstrate either. The weakest pillar is reproducibility: containerization ensures software availability, not numerical or bitwise reproducibility, and the framework deliberately lets users mix native and containerized backends, which silently breaks the guarantee. The authors know this; they propose a certification mechanism as future work.\n\nThe citation pattern is heavy on the authors' own series, but that is natural for a paper describing the fourth part of an ecosystem. The related-work discussion of CWL and DALiuGE is fair. The math is not the point — there is no derivation, and none is needed.\n\nWho is this for? Radio astronomers who build or use data reduction pipelines, and software engineers in astronomy who want a readable, container-native workflow system. It deserves serious refereeing: the design is coherent, the implementation is open source, and the claims are falsifiable once benchmarks exist. I would ask the authors to either soften the abstract or add a measured reproducibility demonstration and at least one benchmark against an existing manager.","headline":"A clear, honest design overview of a real framework; the reproducibility claim is oversold relative to what is demonstrated, but the paper deserves refereeing.","tokens_in":30774,"tokens_out":1418,"would_cite":true,"duration_ms":15586,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stimela2 is a workflow framework that lets radio astronomy reduction recipes be written once as YAML and run unchanged on a laptop, a Slurm cluster, or a Kubernetes cloud, with containerized, versioned software for reproducibility.","keywords":["workflow management","radio astronomy data reduction","reproducibility","containerization","Kubernetes","Slurm","YAML recipes","cloud computing"],"falsifier":"Run a representative end-to-end MeerKAT reduction, expressed once as a Stimela2 recipe, on a local cluster and on an AWS EKS cluster with autoscaling, recording elapsed time, instance-hours, storage and egress costs, and the numerical agreement of the final images; if autoscaler cold starts or instance spin-up delays make the cloud run slower or more expensive than a fixed on-premises allocation, the central claim of practical and economical cloud deployment is not supported.","tokens_in":29876,"feed_emoji":"📡","tokens_out":5934,"duration_ms":62968,"temperature":0.7,"pith_summary":"This paper claims that a data-reduction workflow framework can occupy the middle ground between hand-written scripts and formal dataflow systems: radio astronomy reductions are described as linear, human-readable YAML recipes, while the framework silently maps them onto containers, Slurm, or Kubernetes so the same recipe runs locally or in the cloud. The authors state that Stimela2 has largely achieved this aim, combining ease of development, scalability, and practical reproducibility, and they point to cloud deployments on AWS in earlier papers of the same series as evidence. If the claim is right, astronomers can publish not only results but executable, reproducible recipes that others can rerun without installing arcane software stacks.","feed_headline":"Stimela2: one recipe syntax runs from laptop to cloud","feed_subtitle":"Wrap radio tasks once, then run the same recipe on a laptop, Slurm cluster, or Kubernetes cloud.","key_machinery":"The central object is the nested YAML namespace: recipes, cab definitions, schemas, and configuration all merge into one ordered dictionary, so workflow logic and local environment specifics can be composed from separate files. The load-bearing mechanism is the schema-plus-substitution system, where parameters can be typed, validated, aliased, and combined via formulas like \"=recipe.ms\" and string substitutions like \"{recipe.image_name}\", keeping parameter bookkeeping tractable in long reductions. On top of this, the backend layer executes the same abstract steps natively, in an Apptainer container, through the Slurm scheduler, or as Kubernetes pods with per-cab CPU and memory requests, which is what allows a workflow to scale from a laptop to a cloud cluster without changing the recipe.","core_discovery":"Stimela2's central claim is that a concise linear recipe language, backed by formal typed schemas, can be turned into distributed execution without requiring users to become workflow specialists. A recipe is a YAML sequence of steps; each step invokes a cab, an atomic task whose cargo can be a binary executable, a Python function, a CASA task, or inline Python code, and whose inputs and outputs are described by a schema. Because cabs carry schemas, Stimela2 can prevalidate parameters, chain them through formulas and string substitutions, scatter loop iterations across machines, and detect failures by parsing console output. Because the same step can be selected to run natively, under Apptainer, through a Slurm wrapper, or inside a Kubernetes pod, the logical recipe stays unchanged while the execution environment changes, and cloud autoscaling can in principle match cheap small instances to thin steps and large expensive instances to thick ones.","pith_inferences":["Beyond the paper: if the recipe language is as portable as claimed, the same cab-and-schema design should transfer to any containerized data-processing domain, since nothing in the core framework is radio-specific.","The paper leaves cost and performance unmeasured; a natural next experiment is a head-to-head dollar-per-image comparison of the same recipe on an in-house cluster, fixed cloud nodes, and autoscaling Kubernetes.","The proposed \"certifiable workflow\" idea, hashing container images, configuration, and input data into a workflow signature, could generalize into a reproducibility standard for containerized pipelines generally, provided that image registries remain accessible and immutable.","An open recipe competition, as sketched in the paper, would make algorithm comparisons more quantitative than side-by-side images, because swapping an imaging step inside a published recipe gives a direct, reproducible basis for comparison."],"forward_implications":["The same recipe file can run unchanged under the native, Apptainer, Slurm, or Kubernetes backends, so workflow logic does not need to be rewritten when moving from a laptop to a cluster to the cloud.","Containerized cab collections such as cult-cargo make workflows effectively zero-install: a host needs only Stimela2 and a container engine, while versioned images are downloaded on demand.","Scatter-gather loop constructs let parallelizable steps fan out over cluster nodes while serial steps remain serial, which is the paper's proposed answer to the \"thick-thin\" resource profile of radio astronomy reductions.","Published science can be accompanied by its actual reduction recipes, as in the RATT PARROT example, so other groups can rerun the exact processing rather than approximate it by hand.","Because Kubernetes pods carry predefined CPU and memory requests, autoscaling can in principle allocate small cheap instances to light steps and spin up large instances only for heavy steps, making cloud execution economically plausible."],"supporting_citations":[{"why":"Defines the first-generation Stimela package whose limitations, serial execution, Python-based recipe API, and weak support for native binaries, Stimela2 is explicitly designed to overcome.","marker":"Makhathini, 2018"},{"why":"Introduces Dask-MS and the storage-layer argument that replacing Measurement Set storage with parallel-I/O-friendly backends removes a major bottleneck to economical cloud deployment.","marker":"Perkins et al., 2024"},{"why":"Presents QuartiCal, a Dask-based calibration package whose workflows run through Stimela2 and supply the scalable, cloud-capable workloads the framework is built to carry.","marker":"Kenyon et al., 2024"},{"why":"Presents pfb-imaging and the Kubernetes/AWS benchmark configuration that Appendix B of this paper augments, demonstrating actual Stimela2 cloud deployment.","marker":"Bester et al., 2024"},{"why":"Publishes a science paper together with complete Stimela2 recipes, serving as the paper's central example of reproducible, non-standard data reduction.","marker":"Smirnov et al., 2024"},{"why":"Supplies the formal reproducibility tenets that the paper draws on when discussing certifiable versus non-certifiable workflows.","marker":"Pritchard and Wicenec, 2024"}],"fun_headline_variants":["Write once, run anywhere: Stimela2 recipes scale to cloud","Stimela2: YAML recipes that go from laptop to Kubernetes","One recipe, many runtimes: Stimela2's scalable workflows","Stimela2 turns simple YAML into cloud-scale radio astronomy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cloud-scalability claim depends on the assumption that Kubernetes autoscaling, bringing virtual machines up and down on demand, can economically handle workflows whose steps alternate between thin, serial tasks and thick, parallel, memory-hungry tasks; the paper gives no cost or performance measurements for this.","fun_headline_variants_meta":{"raw":{"variants":["Write once, run anywhere: Stimela2 recipes scale to cloud","Stimela2: YAML recipes that go from laptop to Kubernetes","One recipe, many runtimes: Stimela2's scalable workflows","Stimela2 turns simple YAML into cloud-scale radio astronomy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000499,"raw_usage":{"total_tokens":2466,"prompt_tokens":993,"completion_tokens":1473,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":1397}},"tokens_in":609,"tokens_out":1473,"duration_ms":9459,"temperature":1.0,"reasoning_tokens":1397,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:21:51.818279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a representative end-to-end MeerKAT reduction, expressed once as a Stimela2 recipe, on a local cluster and on an AWS EKS cluster with autoscaling, recording elapsed time, instance-hours, storage and egress costs, and the numerical agreement of the final images; if autoscaler cold starts or instance spin-up delays make the cloud run slower or more expensive than a fixed on-premises allocation, the central claim of practical and economical cloud deployment is not supported.","supporting_citations":[{"cited_title":", year 2018","cited_arxiv_id":null,"evidence_quote":"Defines the first-generation Stimela package whose limitations, serial execution, Python-based recipe API, and weak support for native binaries, Stimela2 is explicitly designed to overcome."},{"cited_title":"Africanus I. Scalable, distributed and efficient radio data processing with Dask-MS and Codex Africanus","cited_arxiv_id":"2412.12052","evidence_quote":"Introduces Dask-MS and the storage-layer argument that replacing Measurement Set storage with parallel-I/O-friendly backends removes a major bottleneck to economical cloud deployment."},{"cited_title":"Formal Definition and Implementation of Reproducibility Tenets for Computational Workflows","cited_arxiv_id":"2406.01146","evidence_quote":"Supplies the formal reproducibility tenets that the paper draws on when discussing certifiable versus non-certifiable workflows."}],"review_version":1}