{"id":"d1a1da1b-4b66-44c8-ab07-b11b5d450cc4","arxiv_id":"2607.02925","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An agentic team using Gemini successfully builds multi-platform containers, deploys and optimizes four HPC applications across 21 AWS instance types in Kubernetes, and produces expert-approved linear scaling studies.","lead":"Researchers built LLM agents that build containers, deploy HPC apps on Kubernetes, pick cloud instances, optimize performance, and run scaling studies for apps like LAMMPS and AMG. The work shows agents can automate much of the experimental lifecycle with human checks, cutting manual effort for cloud HPC runs.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Human gates and unreleased plans make the claimed semi-autonomous lifecycle success hard to attribute to the agentic design itself.","rationale":"The reader correctly identified the central soft spot: reliance on carefully engineered prompts, attempt limits, and human validation of regexes and hard dependencies. My concern is the same point sharpened—without the artifacts it is impossible to quantify how much of the “successful linear scaling approved by experts” is produced by the agentic system versus by the authors’ continuous intervention. The paper is honest about these gates (Discussion, Sections 3.3–3.5) and contains no logical contradiction, so the verdict remains CONDITIONAL rather than REJECT. Releasing the code and plans is the single concrete check that would either confirm or weaken the architectural claim. No stronger internal inconsistency or circularity is present; the evaluation is simply not yet independently reproducible at the level the strongest claim requires.","tokens_in":17298,"tokens_out":577,"duration_ms":6770,"concrete_test":"Publicly release the agent framework, all plan YAML files, and the exact prompt templates used for the four applications. An independent group re-runs the single-node LLM-decision optimization and the 5-node scaling study for LAMMPS and AMG with only the released artifacts and no additional human injection of versions or problem-size advice. If success rates, attempt counts, and final FOMs remain within 15 % of Table 2 / Figure 8, the claim holds; larger degradation shows the results depend on unreleased human scaffolding.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim is that an agentic team can deliver the full HPC experimental lifecycle (multi-platform build, K8s deploy, optimization across 21 instances, scaling) and produce expert-approved linear scaling. That claim rests on the assumption that Gemini plus the authors’ scoped prompts, max-attempt limits, and occasional human validation will converge without catastrophic drift (reader’s weakest_assumption). The paper itself shows that this assumption is only partially true: Laghos builds required human-supplied version/flag expertise (Section 3.3); FOM regexes always needed human yes/no/feedback (Section 2.2); AMG problem-size and OpenMP-thread advice was injected after observation; scaling was limited to 5 nodes rather than the planned 32 (Section 3.5); and the Discussion explicitly notes premature STOP decisions, temporal pinning of dependency versions, and the necessity of human gates. Without the plan YAMLs, exact prompts, and agent code, it is impossible to separate the contribution of the agentic architecture from the authors’ iterative prompt engineering and domain knowledge. The reported FOMs and “approved patterns” are therefore under-specified as evidence for the architectural claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper presents a prototype agentic system (manager + step agents for build/deploy/optimize/scale + helper agents for debug/result-parsing) that uses Gemini to execute the full experimental lifecycle of HPC applications in Kubernetes on AWS: multi-platform container builds, Job/MiniCluster deployment, instance selection and FOM optimization across 21 instance types, and strong-scaling studies. Experiments cover LAMMPS, AMG2013, Kripke, Laghos (and OSU for multi-node), reporting best FOMs (Table 2), instance histograms (Fig. 5), token patterns (Fig. 3), scaling curves (Fig. 8), and human-expert review of design choices (Table 3). The authors claim successful linear scaling patterns approved by experts, reduced time-to-completion designs, and best practices for agentic collaboration.","tokens_in":17602,"tokens_out":1079,"duration_ms":17666,"significance":"If the architecture and results hold under reduced human gating, this is a useful early systems contribution to agentic orchestration for scientific computing. Concrete strengths include measured FOMs and scaling curves for real proxy apps, explicit comparison of three optimization strategies (LLM decision vs. user-function variants), documentation of failure modes (OOM, timeouts, temporal pinning of dependency versions), and practical lessons on prompt emphasis, self-healing via debug agents, and controlled interfaces. The work is timely given national initiatives (FG-HPCC, Genesis) and the gap between commercial agent frameworks and institutional constraints. Reproducible artifacts and clearer autonomy metrics would raise impact further.","major_comments":[{"comment":"Sections 2.2–2.3 and 3.3–3.5: The central claim of an agentic team delivering the “entire life-cycle” rests on Gemini converging under scoped prompts and max-attempt limits. Yet the manuscript documents load-bearing human interventions that are not quantified: Laghos builds required explicit human-supplied version/flag expertise after N=10 failures; every FOM regex required human yes/no/feedback validation before caching; AMG problem-size and OpenMP-thread advice was injected after observation; scaling was limited to 5 nodes rather than the planned 32. Without a clear metric of autonomy (e.g., fraction of decisions made without human edit, or ablation removing the gates), it is impossible to attribute success to the agentic graph versus iterative human prompt engineering and domain knowledge.","section":null},{"comment":"Section 3.5 and Figure 8: The scaling study is presented as evidence of “successful linear scaling with patterns approved by human experts.” Only five hpc7g.16xlarge nodes could be provisioned; the planned N=32 never occurred. While the observed efficiency curves (including super-linear cache effects for Kripke) are interesting, the limited scale and the fact that problem size was pinned after a human-validated single-node optimization weaken the claim that the scaling agent autonomously orchestrated a full study. A stronger experiment or explicit discussion of the gap is needed.","section":null},{"comment":"Reproducibility of the architectural claim: The manager, step agents, helper agents, plan YAML schema, exact prompts, and controlled interfaces are described at a high level (Section 2) but neither code nor the concrete plan files used for the reported runs are released. Given that “stuck sequences,” premature STOP decisions, and temporal pinning of dependency versions are acknowledged in the Discussion, independent verification of the FOMs in Table 2 and the design choices in Table 3 is currently impossible. Release of the prototype and the exact plans used would make the contribution falsifiable.","section":null}],"minor_comments":[{"comment":"Abstract and Section 1 claim “four well-known HPC applications” while Methods (2.1) and multi-node experiments include five (adding OSU). Align the counts.","section":null},{"comment":"Table 1 lists 21 instance types; Figure 5 caption and text occasionally refer to hpc7g.12xlarge (non-existent). Correct the typo.","section":null},{"comment":"Figure 3 caption says “LAMMPS (green)” while the body text attributes the highest token counts to Laghos (green). Resolve the color/label inconsistency.","section":null},{"comment":"Section 2.3: “user guided function” vs. “user-guided function” and “llm decision” capitalization are inconsistent; standardize terminology.","section":null},{"comment":"Discussion notes that multi-stage Docker builds made the build agent’s task harder, yet Table 3 scores “Monolithic layers” as good practice. Clarify the recommendation.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid early systems prototype and fits a systems/HPC venue, but the autonomy claim is currently overstated relative to the documented human gates. Requiring artifact release and a quantitative autonomy metric would strengthen it without changing the core experimental narrative. Scope is appropriate; novelty is incremental but useful."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is simple: they built a manager-plus-step-agents system that can take an HPC app from Dockerfile through multi-platform build, Flux MiniCluster deploy on Kubernetes, instance selection across 21 types, FOM optimization, and a strong-scaling study, and they got linear patterns that domain experts signed off on for LAMMPS, AMG, Kripke (and Laghos on single node). That full pipeline is new; prior agent and Flux work does not ship this end-to-end loop with the reported measurements.\n\nWhat they did well is the empirics and the honesty. Tables 2–3 and Figures 3–8 give real FOMs, instance histograms, token counts, and scaling curves. They document good versus bad Dockerfile choices, premature STOP decisions, temporal pinning of dependency versions, and the need for human yes/no on regexes and Laghos flags. The self-healing design (debug helper, max attempts, return to manager) and the controlled interfaces (no free shell, only config files) are sensible engineering. Best-practice notes on prompt emphasis, problem-size advice, and caching base images are useful for anyone trying the same thing.\n\nSoft spots are real but proportionate. Laghos only succeeded after human-supplied versions; FOM parsing always needed human validation; AMG needed injected OpenMP and size advice; scaling stopped at five nodes instead of the planned 32. Without the plan YAMLs, exact prompts, and agent code, it is hard to separate the architecture from iterative prompt engineering and domain knowledge. Free parameters (max attempts, timeouts, initial size advice) and Gemini-specificity limit generality. None of this is a load-bearing flaw—the paper never claims pure autonomy—but it does mean the “semi-autonomous lifecycle” claim rests partly on unreleased artifacts and human gates.\n\nThis is for HPC systems people and cloud-native workflow researchers who want concrete agent experience rather than another survey. It deserves a serious referee; the measurements and design discussion are solid enough to improve under review. I would bring it to reading group as a practical systems case study, cite the empirical patterns if I am working on agentic HPC tooling, and recommend accept-with-revision rather than desk reject.","headline":"Working multi-agent prototype that actually runs the full HPC cloud lifecycle for four apps, with concrete FOMs and expert-checked scaling, but still gated by humans and unreleased plans.","tokens_in":18197,"tokens_out":558,"would_cite":true,"duration_ms":14129,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"An agentic team of step experts can run the full HPC cloud lifecycle—build, deploy, optimize, and scale—producing linear scaling patterns human experts accept.","keywords":["agentic AI","HPC applications","Kubernetes","container build","autoscaling","strong scaling","LLM orchestration","cloud HPC"],"falsifier":"Run the same agentic plan on a fifth application whose build or scaling behavior is outside the training distribution; if the agents cannot produce a working multi-platform image and a linear strong-scaling curve that experts accept within the stated attempt budgets, the claim fails.","tokens_in":18160,"feed_emoji":"⚙️","tokens_out":630,"duration_ms":7402,"temperature":0.7,"pith_summary":"This paper shows that a small team of specialized LLM agents, coordinated by a manager, can carry an HPC application through the whole experimental cycle in the cloud: writing multi-platform Dockerfiles, deploying them as Kubernetes Jobs or Flux MiniClusters, choosing among 21 instance types, optimizing a figure of merit, and deciding when a strong-scaling study has finished. The agents succeed on four standard applications (LAMMPS, AMG, Kripke, Laghos, plus OSU benchmarks for multi-node work), returning linear scaling curves that human experts approve. The design keeps each agent narrowly scoped, feeds only controlled configuration files and return codes rather than free-form shell commands, and inserts human checks only for regular-expression log parsing and hard dependency versions. The result is a working prototype that shortens wall-clock time for what used to be manual, multi-day setup and tuning work, together with concrete prompt and orchestration practices that keep the agents from drifting into unrecoverable loops.","feed_headline":"Agents build, deploy and scale HPC apps in cloud","feed_subtitle":"Four applications reach linear scaling that human experts accept, across 21 instance types","key_machinery":"The agentic graph: a shared context object that carries container URIs, manifests, metrics, and error summaries between narrowly scoped step agents, with each agent returning only structured decisions (RETRY/STOP) or validated configuration files and a manager that restarts or advances on return codes and human-approved regexes.","core_discovery":"A manager-orchestrated graph of step agents—build, deploy, optimize, scale, plus debug and result-parser helpers—can autonomously deliver container creation, multi-platform build, Kubernetes deployment, instance selection across 21 types, and a full strong-scaling study for four well-known HPC applications, producing linear scaling patterns that human experts accept as correct.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Manager agents automate HPC build deploy optimize and scale in cloud","Step-agent graph delivers expert-accepted linear HPC scaling on Kubernetes","Agents create multi-platform images and scale four HPC apps across 21 types","Autonomous agents complete full HPC lifecycle with linear scaling patterns","Orchestrated agents handle containerization to scaling studies for HPC apps"],"cache_read_input_tokens":4096,"weakest_assumption_plain":"That carefully scoped prompts, attempt limits, and occasional human validation of log parsers and dependency versions are enough for the LLM to converge on correct Dockerfiles, manifests, and stop decisions without catastrophic drift.","fun_headline_variants_meta":{"raw":{"variants":["Manager agents automate HPC build deploy optimize and scale in cloud","Step-agent graph delivers expert-accepted linear HPC scaling on Kubernetes","Agents create multi-platform images and scale four HPC apps across 21 types","Autonomous agents complete full HPC lifecycle with linear scaling patterns","Orchestrated agents handle containerization to scaling studies for HPC apps"]},"model":"grok-4.5","effort":"low","cost_usd":0.007444,"raw_usage":{"total_tokens":1766,"prompt_tokens":692,"num_sources_used":0,"completion_tokens":90,"cost_in_usd_ticks":74440000,"prompt_tokens_details":{"text_tokens":692,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":984,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":692,"tokens_out":90,"duration_ms":8115,"temperature":1.0,"reasoning_tokens":984,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T06:03:26.592402+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same agentic plan on a fifth application whose build or scaling behavior is outside the training distribution; if the agents cannot produce a working multi-platform image and a linear strong-scaling curve that experts accept within the stated attempt budgets, the claim fails.","supporting_citations":[],"review_version":1}