The National Research Platform: Stretched, Multi-Tenant, Scientific Kubernetes Cluster

Ashton Graves; Derek Weitzel; Dmitry Mishin; Elham E Khoda; Frank W\"urthwein; Huijun Zhu; John Graham; Larry Smarr; Mahidhar Tatineni; Mohammad Firas Sada

REVIEW 2 major objections 5 minor 19 references

The National Research Platform: Stretched, Multi-Tenant, Scientific Kubernetes Cluster

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues for a 'stretched' single Kubernetes cluster as national scientific infrastructure, citing 670 namespaces, 5M GPU-hours, and 50M CPU core-hours in 2024.

desk verdict A credible practice-and-experience paper on a real distributed Kubernetes platform; the usage numbers are self-reported and the §3.1 utilization gain is under-specified, but the architecture and operational detail carry the paper. read the letter →

arxiv 2505.22864 v1 pith:4W2MEI2K submitted 2025-05-28 cs.DC

Derek Weitzel , Ashton Graves , Sam Albin , Huijun Zhu , Frank Würthwein , Mahidhar Tatineni , Dmitry Mishin , John Graham

show 4 more authors

Elham E Khoda Mohammad Firas Sada Larry Smarr Thomas DeFanti

This is my paper

classification cs.DC

keywords distributedcomputingKubernetesmulti-tenantclusterscientificcyberinfrastructureGPUallocationhighthroughputAIandMLworkloadsfederatedinfrastructure

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports on the National Research Platform, a Kubernetes installation that spans more than 75 U.S. and international sites and presents itself to users as a single batch cluster. The authors' central operating claim is that 'stretched' federation at the hardware, operating-system, and Kubernetes-peering levels is a viable way to build national-scale scientific cyberinfrastructure, especially for AI and machine learning. They point to 670 active research namespaces consuming 5M GPU-hours and 50M CPU core-hours in calendar year 2024, with more than three-quarters of the GPU usage coming from 525 individual campus-researcher namespaces across 50 campuses. The paper also describes the mechanisms that make the federation practical: an A100 GPU reservation system, regionally distributed storage, Kubernetes-integrated security monitoring, and preemptible background work from the high-throughput computing community. A sympathetic reader would take away that distributed, multi-tenant Kubernetes can serve researchers across the spectrum from large universities to community colleges.

What carries the argument

The central object is the stretched Kubernetes cluster itself, with everything else in service of making that federation manageable. Nodes join at one of three levels: hardware-level control through IPMI (server management hardware), operating-system-level onboarding via an automation tool that installs the container runtime, Kubernetes, and system settings, or peering with independently operated Kubernetes clusters. Multi-tenancy is enforced through namespaces, with users getting RBAC-scoped credentials from a federated identity system. Storage is provided by a distributed filesystem replicated across five U.S. regions, and security monitoring sits inside Kubernetes through a runtime threat-detection tool that captures process-level detail on pods. The A100 reservation system is the load-balancing mechanism that the paper credits with the utilization improvement.

What would settle it

Recompute average GPU utilization before and after the reservation rollout with one fixed formula, such as pod GPU-seconds divided by available GPU-seconds on the same node set; if the 21.49% to 31.37% gap disappears under that consistent measure, the reservation system's benefit is not established.

Watch

Extended reading notes

Core claim

The central discovery is that a single Kubernetes cluster can be stretched across more than 75 administrative domains and still behave as one batch cluster from the user's point of view. As of spring 2025 the platform includes over 1,400 GPUs, 28,000 CPUs, and 161 TB of RAM spread across more than 420 nodes, with sites contributing hardware at the IPMI, operating-system, or Kubernetes-peering level. The paper offers 2024 accounting as evidence that this model works at national scale: 670 namespaces used 5M GPU-hours and 50M CPU core-hours, and 525 campus-researcher namespaces from 50 campuses consumed over three-quarters of the GPU-hours and two-thirds of the CPU-hours. It further reports that reserving A100 GPUs raised average cluster GPU utilization from 21.49% to 31.37%, that regionally replicated storage keeps data close to compute, and that preemptible work from a high-throughput federation contributed 6.5M CPU hours in 2024.

Load-bearing premise

The load-bearing premise is that the reported rise in average GPU utilization from 21.49% to 31.37% is real and caused by the new A100 reservation system, but the paper never defines how utilization was measured.

Editorial extensions

If this is right

If the platform performs as reported, a campus can contribute as little as one node and still draw on the full national resource pool.
The 525 individual campus-researcher namespaces from 50 campuses suggest the model spreads AI/ML capability beyond large research universities.
The reported jump in average GPU utilization after A100 reservations indicates scheduling policy can materially improve scarce-GPU use without new hardware.
Routing preemptible work from a high-throughput computing community through idle cycles contributed 6.5M CPU-hours in 2024 without displacing primary workloads.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: a natural test of the stretched-cluster model is whether queue wait times and failure rates stay acceptable as sites and namespaces grow; the paper does not report queue-level quality-of-service data.
Editorial inference: if the utilization claim survives a fixed measurement definition, the reservation mechanism could be extended to other high-demand GPU models.
Editorial inference: the education-facing use, including community colleges, points to a workforce-training benefit that may grow faster than research workloads and become the platform's dominant justification.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a credible practice-and-experience paper about the National Research Platform, a stretched Kubernetes cluster federating nodes across more than 70 sites. The genuinely new content is the operational description of NRP itself—how it integrates hardware at the IPMI, OS, and Kubernetes layers; how it manages inventory with NetBox and deployment with Ansible; how it uses Rook/Ceph for regional storage, Falco for container threat detection, and a custom Prometheus/Thanos/Redis accounting stack—plus the 2024 usage statistics. It is a direct continuation of the Pacific Research Platform, and the paper frames that lineage honestly.

What it does well: the contrast with OSG and ACCESS is clear. NRP federates at the node level rather than overlaying batch schedulers, and the multi-tenancy story (670 namespaces, 525 campus researcher namespaces, 23 MSIs, EPSCoR states) is useful evidence of the platform's reach. The security section, though short, makes a concrete point about Falco being chosen after Prometheus overload alerts proved insufficient. The acknowledgment that the paper was edited with LLMs hosted on NRP is a nice honest footnote, not a flaw.

Soft spots: the utilization improvement claim in Section 3.1—21.49% to 31.37%—is presented without defining what 'utilization' means or how it was measured. The stress-test note is correct that demand was growing and the hardware pool was changing, so a simple before/after comparison cannot isolate the reservation system's causal effect. This should be fixed with a definition and a caveat, but it is one metric, not the paper's backbone. The headline numbers (5M GPU-hours, 50M CPU core-hours) come from the accounting pipeline described in Section 5, and the paper does not report validation or error bars for that pipeline. For a PEARC practice paper, self-reported usage numbers are normal; a sentence on how the totals are derived would be enough. There is also a minor inconsistency: the abstract says 'over 75 locations' while Section 2 says 'more than 70 sites.'

Overall, the central descriptive claim—that NRP operates as a distributed multi-tenant Kubernetes cluster at national scale and supports a real research community—holds up. The math is not the point; the operational evidence is. This paper deserves a serious referee and likely acceptance after minor revision. I would ask the authors to clarify the utilization metric and to add a brief note on the accounting pipeline's provenance and any known error margins.

Recommendation: send it to peer review. It is exactly the kind of experience report PEARC is for.

Referee Report

2 major / 5 minor

Summary. The paper describes the National Research Platform (NRP), a distributed multi-tenant Kubernetes cluster that federates compute and storage nodes across more than 70 sites in the U.S. and internationally. It reports a cluster of 420+ nodes, 1,400+ GPUs, 28,000+ CPUs, and 161 TB of RAM, presenting the whole federation as a single batch cluster to users. The paper details the deployment toolchain (Ansible, NetBox), three integration layers (IPMI hardware-level, OS-level, and Kubernetes peering via Admiralty), user interfaces (JupyterHub, Coder, direct kubectl), storage (regionally distributed Ceph via Rook), security (Falco-based threat detection), accounting and monitoring (Prometheus/Thanos with custom caching, sFlow, PerfSONAR), and future scheduler work with YuniKorn. It also gives 2024 usage statistics: 670 namespaces, 5M GPU-hours, 50M CPU core-hours, and a claimed improvement in average GPU utilization from 21.49% to 31.37% after introducing an A100 reservation system.

Significance. If the operational claims are accurate, the NRP represents a notable and rare instance of a stretched, multi-tenant Kubernetes federation that gives researchers a unified cluster view across many autonomous sites. The paper's value is primarily descriptive: it documents a real system with specific engineering choices—layered integration, regional Ceph distribution, Falco-based intrusion detection, and a custom accounting layer—that are directly useful to operators of similar community infrastructures. The 2024 scale figures are striking and plausible given the NSF funding and prior Pacific Research Platform lineage, but they are self-reported and lack metric definitions; the utilization improvement claim is also methodologically under-specified. The paper includes a transparent disclosure that LLMs hosted on NRP assisted in editing, which is commendable. This is not a paper with machine-checked proofs or reproducible experiments, but as a practice/experience case study it has clear value to the PEARC audience.

major comments (2)

[Section 3.1] The reported utilization improvement from 21.49% to 31.37% is presented without defining "average GPU utilization": it is not stated whether this is request-time from allocated reservations, device busy-time as measured by GPU telemetry, or something else, nor is the denominator or the measurement window specified. The surrounding text also notes that demand for high-end GPUs was growing (see Fig. 2) and that the hardware pool was changing as new models arrived, so a simple before/after comparison cannot identify the reservation system's causal effect. Please define the metric and its source, and either temper the claim that the improvement "underscores the effectiveness of the reservation system" or provide additional controls (e.g., matching by GPU type, time period, workload mix).
[Section 5 (and Section 1)] The headline 2024 totals—5M GPU-hours, 50M CPU core-hours, and 670 namespaces—depend entirely on the accounting pipeline described in Section 5 as using Prometheus, Thanos, custom caching, query segmentation, and a Redis layer. No definition is given for how GPU-hours and core-hours are computed (e.g., allocated pod request-time multiplied by duration, versus measured busy-time), what query granularity is used, or how the pipeline is validated against ground truth. Without this information, the central scale claims are not interpretable or reproducible. Please add a short paragraph in Section 5 that states the exact metric definitions, the data sources, and any known limitations or approximations.

minor comments (5)

[Abstract and Section 2] The abstract says "over 75 locations" while Section 2 says "more than 70 sites"; reconcile these counts or use a single approximate figure consistently.
[Section 2, first sentence] "The Network Research Platform" should be "The National Research Platform" (the word "Network" is a typo).
[Section 4] "Cloud Native Computing Federation" should be "Cloud Native Computing Foundation (CNCF)".
[References] Reference [6] contains a capitalization typo: "Red HAt" should be "Red Hat".
[Figure 2 caption and Section 1] The figure caption reports a 12-month total of 6,605,042 GPU hours from 412 research groups, while Section 1 reports 5M GPU-hours for calendar year 2024 from 670 namespaces; clarify the time window of the figure and how these numbers relate to the 2024 totals and the namespace/group counts.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning identified; the paper is an operational description with no derivation chain that reduces to its own inputs.

full rationale

The paper is a systems and operations description of the National Research Platform, not a derivation of a theoretical result from fitted parameters or axioms. Its central claims are existential and observational: that NRP spans over 75 locations, presents as a single Kubernetes-based batch cluster, and supported the reported 2024 usage totals. No equation is derived, no parameter is fitted and then renamed as a prediction, and no uniqueness theorem is invoked. The only self-reference is the acknowledgment footnote stating that the paper was edited using LLMs hosted on the NRP, but that statement is not load-bearing for any quantitative or architectural claim. The utilization improvement reported in Section 3.1 (21.49% to 31.37%) is asserted without a defined measurement methodology and without a causal analysis, which is a correctness limitation, not a circularity: the number is not constructed to equal its own input and the claim does not reduce to a definition or a self-citation. Likewise, the accounting pipeline described in Section 5 is the source of the usage totals, but the paper does not claim to predict those totals from the pipeline; it reports them as measurements. Any concerns about reproducibility, missing metric definitions, or confounding factors belong to correctness and validation risk, not circularity. Accordingly, the paper is self-contained with respect to the kind of derivation-chain circularity this review targets, and the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No free parameters or invented entities appear in this system description. The platform relies on standard open-source components (Kubernetes, Ceph, Falco, Ansible) and treats their documented behavior as background; no new physical or mathematical postulates are introduced.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The National Research Platform: Stretched, Multi-Tenant, Scientific Kubernetes Cluster." pith.science (2026). https://pith.science/paper/4W2MEI2K

@misc{pith2026250522864,
  author       = {Pith},
  title        = {Pith review of: The National Research Platform: Stretched, Multi-Tenant, Scientific Kubernetes Cluster},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4W2MEI2K}},
  note         = {Machine review of arXiv:2505.22864}
}

read the original abstract

The National Research Platform (NRP) represents a distributed, multi-tenant Kubernetes-based cyberinfrastructure designed to facilitate collaborative scientific computing. Spanning over 75 locations in the U.S. and internationally, the NRP uniquely integrates varied computational resources, ranging from single nodes to extensive GPU and CPU clusters, to support diverse research workloads including advanced AI and machine learning tasks. It emphasizes flexibility through user-friendly interfaces such as JupyterHub and low level control of resources through direct Kubernetes interaction. Critical operational insights are discussed, including security enhancements using Kubernetes-integrated threat detection, extensive monitoring, and comprehensive accounting systems. This paper highlights the NRP's growing importance and scalability in addressing the increasing demands for distributed scientific computational resources.

Figures

Figures reproduced from arXiv: 2505.22864 by the authors.

**Figure 2.** GPU hours by research group on the NRP over time showing growth of GPU use. For the 12 month [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages

[1]

2025.Multi-Cluster Kubernetes

Admirality. 2025.Multi-Cluster Kubernetes. Simplified. | Admiralty. Admirality. https://admiralty.io/

work page 2025
[2]

Boerner, Stephen Deems, Thomas R

Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. 2023. ACCESS: Advancing Innovation: NSF’s Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support. InPractice and Ex- perience in Advanced Research Computing 2023: Computing for the Common Good(New York, NY, USA, 2023-09-10) (PEARC ’23). Association for...

arXiv 2023
[3]

2025.Coder - Cloud Development Environment: Remote & Self Hosted

Coder. 2025.Coder - Cloud Development Environment: Remote & Self Hosted. https://coder.com/

work page 2025
[4]

2025.Falco

Falco. 2025.Falco. https://falco.org/

work page 2025
[5]

Granger and Fernando Pérez

Brian E. Granger and Fernando Pérez. 2021. Jupyter: Thinking and Storytelling With Code and Data. 23, 2 (2021), 7–14. doi:10.1109/MCSE.2021.3059263 Conference Name: Computing in Science & Engineering

arXiv 2021
[6]

2025.Ansible Collaborative

Red HAt. 2025.Ansible Collaborative. https://www.redhat.com/en/ansible-collaborative

work page 2025
[7]

Adventures with Grace Hopper AI Super Chip and the National Research Platform

J. Alex Hurt, Grant J. Scott, Derek Weitzel, and Huijun Zhu. 2024. Adventures with Grace Hopper AI Super Chip and the National Research Platform. doi:10.48550/arXiv.2410.16487 arXiv:2410.16487 [cs]

work page Pith review arXiv doi:10.48550/arxiv.2410.16487 2024
[8]

2025.Kubernetes

Kubernetes. 2025.Kubernetes. https://kubernetes.io/

work page 2025

Show all 19 references

[9]

NetBox Labs. [n. d.].Enterprise-Grade, SaaS Network Source of Truth. https://netboxlabs.com/
[10]

2025.NRP-Managed LLMs

NRP. 2025.NRP-Managed LLMs. https://nrp.ai/documentation/userdocs/ai/llm-managed/

2025
[11]

2025.Campus Cyberinfrastructure (CC*) | NSF - National Science Foundation

NSF. 2025.Campus Cyberinfrastructure (CC*) | NSF - National Science Foundation. https://www.nsf.gov/funding/ opportunities/cc-campus-cyberinfrastructure , Vol. 1, No. 1, Article . Publication date: September 2025. The National Research Platform: Stretched, Multi-Tenant, Scient...

2025
[12]

2025.Established Program to Stimulate Competitive Research (EPSCoR) | NSF - National Science Foundation

NSF. 2025.Established Program to Stimulate Competitive Research (EPSCoR) | NSF - National Science Foundation. https://www.nsf.gov/funding/initiatives/epscor

2025
[13]

2025.Observable: Build expressive charts and dashboards with code

ObservableHQ. 2025.Observable: Build expressive charts and dashboards with code. https://observablehq.com/

2025
[14]

The Open Science Grid Executive Board on behalf of the Osg Consortium:Ruth Pordes, Don Petravick, Bill Kramer, Doug Olson, Miron Livny, Alain Roy, Paul Avery, Kent Blackburn, Torre Wenaus, Frank Würthwein, Ian Foster, Rob Gardner, Mike Wilde, Alan Blatecky, John McGee, and Rob...

2007 doi
[15]

Rajesh Raman, Miron Livny, and Marvin Solomon. 2000. Resource management through multilateral matchmaking. In Proceedings the Ninth International Symposium on High-Performance Distributed Computing. IEEE, 290–291

2000
[16]

Larry Smarr, Camille Crittenden, Thomas DeFanti, John Graham, Dmitry Mishin, Richard Moore, Philip Papadopoulos, and Frank Würthwein. 2018. The Pacific Research Platform: Making High-Speed Networking a Reality for the Scientist. InProceedings of the Practice and Experience on ...

2018
[17]

2025.Solvik/netbox-agent

Solvik. 2025.Solvik/netbox-agent. https://github.com/Solvik/netbox-agent

2025
[18]

Sage Weil, Scott A Brandt, Ethan L Miller, Darrell DE Long, and Carlos Maltzahn. 2006. Ceph: A scalable, high- performance distributed file system. InProceedings of the 7th Conference on Operating Systems Design and Implementation (OSDI’06). 307–320

2006
[19]

2025.Welcome to Apache YuniKorn Apache YuniKorn

YuniKorn. 2025.Welcome to Apache YuniKorn Apache YuniKorn. https://yunikorn.apache.org/ , Vol. 1, No. 1, Article . Publication date: September 2025

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

2025.Multi-Cluster Kubernetes

Admirality. 2025.Multi-Cluster Kubernetes. Simplified. | Admiralty. Admirality. https://admiralty.io/

work page 2025

[2] [2]

Boerner, Stephen Deems, Thomas R

Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. 2023. ACCESS: Advancing Innovation: NSF’s Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support. InPractice and Ex- perience in Advanced Research Computing 2023: Computing for the Common Good(New York, NY, USA, 2023-09-10) (PEARC ’23). Association for...

arXiv 2023

[3] [3]

2025.Coder - Cloud Development Environment: Remote & Self Hosted

Coder. 2025.Coder - Cloud Development Environment: Remote & Self Hosted. https://coder.com/

work page 2025

[4] [4]

2025.Falco

Falco. 2025.Falco. https://falco.org/

work page 2025

[5] [5]

Granger and Fernando Pérez

Brian E. Granger and Fernando Pérez. 2021. Jupyter: Thinking and Storytelling With Code and Data. 23, 2 (2021), 7–14. doi:10.1109/MCSE.2021.3059263 Conference Name: Computing in Science & Engineering

arXiv 2021

[6] [6]

2025.Ansible Collaborative

Red HAt. 2025.Ansible Collaborative. https://www.redhat.com/en/ansible-collaborative

work page 2025

[7] [7]

Adventures with Grace Hopper AI Super Chip and the National Research Platform

J. Alex Hurt, Grant J. Scott, Derek Weitzel, and Huijun Zhu. 2024. Adventures with Grace Hopper AI Super Chip and the National Research Platform. doi:10.48550/arXiv.2410.16487 arXiv:2410.16487 [cs]

work page Pith review arXiv doi:10.48550/arxiv.2410.16487 2024

[8] [8]

2025.Kubernetes

Kubernetes. 2025.Kubernetes. https://kubernetes.io/

work page 2025