Pith. sign in

REVIEW 3 major objections 5 minor 32 references

An agentic team of step experts can run the full HPC cloud lifecycle—build, deploy, optimize, and scale—producing linear scaling patterns human experts accept.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:03 UTC pith:4SOPDVS6

load-bearing objection Working multi-agent prototype that actually runs the full HPC cloud lifecycle for four apps, with concrete FOMs and expert-checked scaling, but still gated by humans and unreleased plans. the 3 major comments →

arxiv 2607.02925 v1 pith:4SOPDVS6 submitted 2026-07-03 cs.DC

Agentic Orchestration of HPC Applications in Cloud

Vanessa Sochat , Daniel Milroy This is my paper
classification cs.DC
keywords agentic AIHPC applicationsKubernetescontainer buildautoscalingstrong scalingLLM orchestrationcloud HPC
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper shows that a small team of specialized LLM agents, coordinated by a manager, can carry an HPC application through the whole experimental cycle in the cloud: writing multi-platform Dockerfiles, deploying them as Kubernetes Jobs or Flux MiniClusters, choosing among 21 instance types, optimizing a figure of merit, and deciding when a strong-scaling study has finished. The agents succeed on four standard applications (LAMMPS, AMG, Kripke, Laghos, plus OSU benchmarks for multi-node work), returning linear scaling curves that human experts approve. The design keeps each agent narrowly scoped, feeds only controlled configuration files and return codes rather than free-form shell commands, and inserts human checks only for regular-expression log parsing and hard dependency versions. The result is a working prototype that shortens wall-clock time for what used to be manual, multi-day setup and tuning work, together with concrete prompt and orchestration practices that keep the agents from drifting into unrecoverable loops.

Core claim

A manager-orchestrated graph of step agents—build, deploy, optimize, scale, plus debug and result-parser helpers—can autonomously deliver container creation, multi-platform build, Kubernetes deployment, instance selection across 21 types, and a full strong-scaling study for four well-known HPC applications, producing linear scaling patterns that human experts accept as correct.

What carries the argument

The agentic graph: a shared context object that carries container URIs, manifests, metrics, and error summaries between narrowly scoped step agents, with each agent returning only structured decisions (RETRY/STOP) or validated configuration files and a manager that restarts or advances on return codes and human-approved regexes.

Load-bearing premise

That carefully scoped prompts, attempt limits, and occasional human validation of log parsers and dependency versions are enough for the LLM to converge on correct Dockerfiles, manifests, and stop decisions without catastrophic drift.

What would settle it

Run the same agentic plan on a fifth application whose build or scaling behavior is outside the training distribution; if the agents cannot produce a working multi-platform image and a linear strong-scaling curve that experts accept within the stated attempt budgets, the claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a prototype agentic system (manager + step agents for build/deploy/optimize/scale + helper agents for debug/result-parsing) that uses Gemini to execute the full experimental lifecycle of HPC applications in Kubernetes on AWS: multi-platform container builds, Job/MiniCluster deployment, instance selection and FOM optimization across 21 instance types, and strong-scaling studies. Experiments cover LAMMPS, AMG2013, Kripke, Laghos (and OSU for multi-node), reporting best FOMs (Table 2), instance histograms (Fig. 5), token patterns (Fig. 3), scaling curves (Fig. 8), and human-expert review of design choices (Table 3). The authors claim successful linear scaling patterns approved by experts, reduced time-to-completion designs, and best practices for agentic collaboration.

Significance. If the architecture and results hold under reduced human gating, this is a useful early systems contribution to agentic orchestration for scientific computing. Concrete strengths include measured FOMs and scaling curves for real proxy apps, explicit comparison of three optimization strategies (LLM decision vs. user-function variants), documentation of failure modes (OOM, timeouts, temporal pinning of dependency versions), and practical lessons on prompt emphasis, self-healing via debug agents, and controlled interfaces. The work is timely given national initiatives (FG-HPCC, Genesis) and the gap between commercial agent frameworks and institutional constraints. Reproducible artifacts and clearer autonomy metrics would raise impact further.

major comments (3)
  1. Sections 2.2–2.3 and 3.3–3.5: The central claim of an agentic team delivering the “entire life-cycle” rests on Gemini converging under scoped prompts and max-attempt limits. Yet the manuscript documents load-bearing human interventions that are not quantified: Laghos builds required explicit human-supplied version/flag expertise after N=10 failures; every FOM regex required human yes/no/feedback validation before caching; AMG problem-size and OpenMP-thread advice was injected after observation; scaling was limited to 5 nodes rather than the planned 32. Without a clear metric of autonomy (e.g., fraction of decisions made without human edit, or ablation removing the gates), it is impossible to attribute success to the agentic graph versus iterative human prompt engineering and domain knowledge.
  2. Section 3.5 and Figure 8: The scaling study is presented as evidence of “successful linear scaling with patterns approved by human experts.” Only five hpc7g.16xlarge nodes could be provisioned; the planned N=32 never occurred. While the observed efficiency curves (including super-linear cache effects for Kripke) are interesting, the limited scale and the fact that problem size was pinned after a human-validated single-node optimization weaken the claim that the scaling agent autonomously orchestrated a full study. A stronger experiment or explicit discussion of the gap is needed.
  3. Reproducibility of the architectural claim: The manager, step agents, helper agents, plan YAML schema, exact prompts, and controlled interfaces are described at a high level (Section 2) but neither code nor the concrete plan files used for the reported runs are released. Given that “stuck sequences,” premature STOP decisions, and temporal pinning of dependency versions are acknowledged in the Discussion, independent verification of the FOMs in Table 2 and the design choices in Table 3 is currently impossible. Release of the prototype and the exact plans used would make the contribution falsifiable.
minor comments (5)
  1. Abstract and Section 1 claim “four well-known HPC applications” while Methods (2.1) and multi-node experiments include five (adding OSU). Align the counts.
  2. Table 1 lists 21 instance types; Figure 5 caption and text occasionally refer to hpc7g.12xlarge (non-existent). Correct the typo.
  3. Figure 3 caption says “LAMMPS (green)” while the body text attributes the highest token counts to Laghos (green). Resolve the color/label inconsistency.
  4. Section 2.3: “user guided function” vs. “user-guided function” and “llm decision” capitalization are inconsistent; standardize terminology.
  5. Discussion notes that multi-stage Docker builds made the build agent’s task harder, yet Table 3 scores “Monolithic layers” as good practice. Clarify the recommendation.

Circularity Check

0 steps flagged

No significant circularity: experimental systems evaluation whose FOMs, scaling curves, and instance choices are external measurements, not tautological reductions of inputs.

full rationale

The paper is an empirical systems prototype and evaluation, not a derivation of a first-principles result. Its strongest claims (successful multi-platform builds, Kubernetes deploys, optimization across 21 instance types, and linear scaling patterns for LAMMPS/AMG/Kripke/Laghos/OSU) are judged by external application metrics (atom-steps/s, nnz/s, grind time, megadofs/s) that the agents do not define or fit by construction. Human-expert approval of patterns and occasional human gates (regex validation, Laghos version flags, problem-size advice) are acknowledged limitations of autonomy, not circular definitions. Self-citations ([25] Flux Operator MiniCluster CRD, [26] prior cloud-HPC usability) supply infrastructure and application baselines; they do not supply uniqueness theorems, fitted parameters renamed as predictions, or load-bearing premises that force the reported FOMs or stop/retry decisions. No equations equate outputs to inputs; no ansatz is smuggled via self-citation; no known empirical pattern is merely renamed. The work is therefore self-contained against external benchmarks, warranting only a minimal score for ordinary self-reference that is not load-bearing.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central empirical claims rest on a small set of engineering choices (retry limits, human validation gates, Gemini as the sole LLM, a fixed set of 21 instance types, and application-specific FOM definitions) plus standard assumptions about Kubernetes and container tooling. No new physical entities are postulated; free parameters are operational thresholds chosen by the authors.

free parameters (4)
  • max_attempts_per_step
    Hard-coded retry ceilings (typically 10 for step agents, 15 for manager, raised to 20 for Laghos) that determine when the system aborts or returns to the manager; chosen by hand and load-bearing for reported success rates.
  • pod_timeout_seconds
    User-set maximum wait (e.g., 300 s) after which a run is declared timed-out and fed to the debug agent; directly affects STOP/RETRY decisions.
  • instance_cost_cap
    Instances restricted to roughly ≤ $3/h; defines the 21-type search space and therefore the optimization surface.
  • initial_problem_size_advice
    Authors inject ‘use small problem size ≤ 10 for first deploy’ guidance; without it agents repeatedly OOM, so the advice is an essential free parameter of the experimental protocol.
axioms (4)
  • domain assumption Gemini API responses plus conversational memory are sufficient to generate correct Dockerfiles, Kubernetes YAML, and optimization JSON when given structured MUST/MUST-NOT prompts and filtered error logs.
    Invoked throughout Section 2.2 and the experimental design; never independently verified against other LLMs.
  • domain assumption Controlled subprocess interfaces (docker buildx, kubectl apply, log scraping) plus return-code checks constitute a safe and complete environment for agent action.
    Stated in ‘Controlled Interfaces’; underpins the claim that agents can act without free-form shell execution.
  • ad hoc to paper Human validation of a single regular expression for FOM extraction is an acceptable and non-circular gate for subsequent autonomous optimization.
    Described in Optimization Agent and Scaling Study sections; required for every successful run.
  • standard math Standard Kubernetes Job/MiniCluster semantics and AWS EFA networking behave as documented.
    Background infrastructure assumption used for all multi-node and scaling experiments.
invented entities (2)
  • step-agent / helper-agent / manager graph no independent evidence
    purpose: Decomposes the HPC lifecycle into narrowly scoped LLM roles with shared context and controlled edges.
    Core architectural invention of the paper; no independent evidence outside the reported prototype runs.
  • optimization agent returning RETRY/STOP + JSON manifest patch no independent evidence
    purpose: Closes the performance-improvement loop without returning to a human for every iteration.
    Defined in Section 2.2; success judged only by the authors’ own FOM trajectories.

pith-pipeline@v1.1.0-grok45 · 21381 in / 3150 out tokens · 32193 ms · 2026-07-12T06:03:26.592402+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) are serving as a catalyst of change for research practices, touching the daily lives of staff scientists, software engineers, and system administrators. The developments promise new degrees of autonomy, where categories of human work and decision making are replaced by autonomous, goal-oriented systems. This transition necessitates novel architectural paradigms and solid understanding of the strengths and limitations of LLMs. In this work, we design agents to intelligently deliver the entire life-cycle of an HPC application experimental run in cloud -- creation and build of a container, deployment in Kubernetes, optimization, and orchestration of a scaling study. We pursue this task for four well-known HPC applications to build multi-platform images and optimize across 21 instance types in Kubernetes. We demonstrate successful linear scaling with patterns approved by human experts, designs that improve work time to completion, and review suggested best practices for agentic design and collaboration.

Figures

Figures reproduced from arXiv: 2607.02925 by Daniel Milroy, Vanessa Sochat.

Figure 1
Figure 1. Figure 1: Agentic AI System. The manager receives a plan from the user, a YAML specification with a series of steps and context for each. The manager orchestrates executing steps defined in the plan, allowing step agents to work independently up to a maximum number of attempts, and return to the manager if decided by a helper agent or on success. The optimization works with the deploy agent until a decision is made … view at source ↗
Figure 2
Figure 2. Figure 2: Scaling Study Agentic Team for a hypothetical scaling study for four to 64 nodes (maroon boxes). The deploy agent (1) generates a manifest in a working state. The scaling agent (2) receives instruction to start the study, and updates the prompt to deploy at each requested size. At the smallest size, the scale agent directs execution to the optimization agent (3), which decides on a configuration for subseq… view at source ↗
Figure 3
Figure 3. Figure 3: Gemini Token Counts for Build Agent. The linear pattern of receiving back a similar or equivalent output token count reflects a likely convergence on response, either closing in on a solution or getting stuck in a broken state. The ease of generating a working build for LAMMPS (green) is reflected in requiring fewer total tokens. 3.2 Single Node Experiments We asked build, deploy, optimization, debugging, … view at source ↗
Figure 4
Figure 4. Figure 4: Agent Status. A failed run indicates that the application did not make it to a successful completion (non-zero exit code). A successful run (green) indicates that a FOM was generated. LAMMPS had more runs due to initial testing [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Application best FOM by Experiment Type. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: OSU Benchmarks AllReduce (left) and Latency (right) [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Application Scaling Efficiency hpc7g.16xlarge Instances for each of Kripke, AMG, and LAMMPS. An initial successful deployment using a small problem size is done first, then an optimization on one node, and pinning of the decided problem size up to the maximum size. For all applications, higher is better, with the exception of Kripke, where lower is better. The decision to stop/proceed is done by an agent. … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 2 canonical work pages

  1. [1]

    Ahn, Ned Bass, Albert Chu, Jim Garlick, Mark Grondona, Stephen Herbein, Helgi I

    Dong H. Ahn, Ned Bass, Albert Chu, Jim Garlick, Mark Grondona, Stephen Herbein, Helgi I. Ingólfsson, Joseph Koning, Tapasya Patki, Thomas R.W. Scogland, Becky Springmeyer, and Michela Taufer. 2020. Flux: Overcoming scheduling challenges for exascale workflows.Future Generation Computer Systems110 (2020), 202–213. doi:10.1016/j.future.2020.04.006

    work page doi:10.1016/j.future.2020.04.006 2020

  2. [2]

    Amazon Web Services. 2022. The Elastic Fabric Adapter (EFA). https://docs.aws. amazon.com/AWSEC2/latest/UserGuide/efa.html Accessed: 2024-12-12

    2022

  3. [3]

    Anthropic. 2024. Introducing the Model Context Protocol. https://www.anthropic. com/news/model-context-protocol. Accessed: 2025-8-31

    2024

  4. [4]

    Zain Asgar. [n. d.]. Efficient and Scalable Agentic AI with Heterogeneous Systems. https://arxiv.org/html/2507.19635v1. doi:10.48550/arXiv.2507.19635 Accessed: 2025-8-31

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.19635 2025

  5. [5]

    Y Combinator. 2025. Andrej Karpathy: Software Is Changing (Again)

    2025

  6. [6]

    Kubernetes Community. 2019. Metric-server cpu and memory units - General Discussions. https://discuss.kubernetes.io/t/metric-server-cpu-and-memory- units/7497. Accessed: 2025-9-1

    2019

  7. [7]

    DAIR.AI. 2025. LLM Agents. https://www.promptingguide.ai/research/llm- agents. Accessed: 2025-9-1

    2025

  8. [8]

    Aseem Datar. 2025. Transforming R&D with agentic AI: Introducing Mi- crosoft Discovery. https://azure.microsoft.com/en-us/blog/transforming-rd- with-agentic-ai-introducing-microsoft-discovery/. Accessed: 2025-8-31

    2025

  9. [9]

    Wensheng Gan, Zhenyao Ning, Zhenlian Qi, and Philip S Yu. 2025. Mixture of experts (MoE): A big data perspective.arXiv [cs.LG](Jan. 2025)

    2025

  10. [10]

    Gil and K

    D. Gil and K. A. Moler. 2025. Accelerating science with AI.Science390 (2025),

    2025

  11. [11]

    doi:10.1126/science.aee0605

    work page doi:10.1126/science.aee0605

  12. [12]

    Google. 2025. Agent Development Kit. https://google.github.io/adk-docs. Ac- cessed: 2025-9-1

    2025

  13. [13]

    Mourad Gridach, Jay Nanavati, Khaldoun Zine El Abidine, Lenon Mendes, and Christina Mack. 2025. Agentic ai for scientific discovery: A survey of progress, challenges, and future directions.arXiv preprint arXiv:2503.08979(2025)

    Pith/arXiv arXiv 2025

  14. [14]

    Honeypot. 2022. Kubernetes: The Documentary [PART 1]

    2022

  15. [15]

    Google Inc. 2025. Understand and count tokens. https://ai.google.dev/gemini- api/docs/tokens. Accessed: 2025-11-7

    2025

  16. [16]

    2023.Integrating Artificial Intelligence into Sci- ence Gateways

    Biggs Jr and Brandon Samuel. 2023.Integrating Artificial Intelligence into Sci- ence Gateways. Technical Report INL/CON-23-72098-Rev000. Idaho National Laboratory (INL), Idaho Falls, ID (United States)

    2023

  17. [17]

    Martin Laun and Fabian Wolff. 2025. Chatbots in education: hype or help? A meta-analysis.Learning and Individual Differences119 (2025), 102646

    2025

  18. [18]

    2025.Future Generation High Perfor- mance Computing Center (FG-HPCC) Request for Information (RFI) No

    Lawrence Livermore National Security, LLC. 2025.Future Generation High Perfor- mance Computing Center (FG-HPCC) Request for Information (RFI) No. HPC-007. Technical Report HPC-007. Lawrence Livermore National Laboratory (LLNL), Livermore, CA. https://hpc.llnl.gov/fg-hpcc-rfi Attachment (1) RFI Technical Considerations

    2025

  19. [19]

    Peter Martin. 2025. Context Drift: Why AI Loses Coherence Over Time and How to Fix It. https://www.linkedin.com/pulse/context-drift-why-ai-loses-coherence- over-time-how-fix-martin--zlfuf/. Accessed: 2025-9-10

    2025

  20. [20]

    Microsoft. 2025. Azure AI Foundry. https://azure.microsoft.com/en-us/products/ ai-foundry. Accessed: 2025-9-1

    2025

  21. [21]

    Felix Mölder, Kim Philipp Jablonski, Brice Letcher, Michael B Hall, Christopher H Tomkins-Tinch, Vanessa Sochat, Jan Forster, Soohyun Lee, Sven O Twardziok, Alexander Kanitz, Andreas Wilm, Manuel Holtgrewe, Sven Rahmann, Sven Nahnsen, and Johannes Köster. 2021. Sustainable data analysis with Snakemake. F1000Res.10 (Jan. 2021), 33. doi:10.12688/f1000resear...

    doi:10.12688/f1000research.29032.3 2021

  22. [22]

    Pavan Nutalapati. 2022. Self-Healing Cloud Systems: Designing Resilient and Autonomous Cloud Services.International Journal of Science and Research11, 8 (2022), 1173–1187

    2022

  23. [23]

    OpenAI. 2025. API Reference - OpenAI API. https://platform.openai.com/docs/ api-reference/introduction. Accessed: 2025-8-31

    2025

  24. [24]

    Panda Pranoy, Raghav Magazine, Devaguptapu Chaitanya, Takemori Sho, and Sharma Vishal. 2025. Adaptive LLM Routing under Budget Constraints.arXiv [cs.LG](Aug. 2025)

    2025

  25. [25]

    D Salomone and Aishwarya Prabhat. 2025. Build KYC agentic workflows with Google’s ADK. https://cloud.google.com/blog/products/ai-machine-learning/ build-kyc-agentic-workflows-with-googles-adk. Accessed: 2025-8-31

    2025

  26. [26]

    Vanessa Sochat, Aldo Culquicondor, Antonio Ojea, and Daniel Milroy. 2024. The Flux Operator.F1000Res.13, 203 (March 2024), 203

    2024

  27. [27]

    Vanessa Sochat, Daniel Milroy, Abhik Sarkar, Aniruddha Marathe, and Tapasya Patki. 2025. Usability Evaluation of Cloud for HPC Applications. InProceedings of the SC ’25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC Workshops ’25). Association for Computing Machinery, New York, NY, USA, 135–1...

    doi:10.1145/3731599 2025

  28. [28]

    Renan et al Souza. [n. d.]. PROV-AGENT: Unified Provenance for Tracking AI Agent Interactions in Agentic Workflows. https://arxiv.org/html/2508.02866v2. Accessed: 2025-9-1

    Pith/arXiv arXiv 2025

  29. [29]

    Qiqi Wang, Hongjie Zhang, Cheng Qu, Yu Shen, Xiaohui Liu, and Jing Li. 2021. RLSchert: An HPC job scheduler using deep reinforcement learning and remain- ing time prediction.Appl. Sci. (Basel)11, 20 (Oct. 2021), 9448

    2021

  30. [30]

    Qiuhong Wei, Zhengxiong Yao, Ying Cui, Bo Wei, Zhezhen Jin, and Ximing Xu

  31. [31]

    Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis.Journal of biomedical informatics151 (2024), 104620

    2024

  32. [32]

    Adar Weidman. 2025. Regular expression Denial of Service - Re- DoS. https://owasp.org/www-community/attacks/Regular_expression_Denial_ of_Service_-_ReDoS. Accessed: 2025-9-1

    2025