Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

TENT claims that choosing paths at slice time with live link telemetry, rather than at startup, lifts LLM serving throughput by up to 1.36x and cuts P90 TTFT by 26%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A telemetry-driven slice-spraying transfer engine with late-binding path selection improves LLM serving throughput and self-heals from link failures in tens of milliseconds.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection TENT is a genuine step forward for data movement in disaggregated LLM serving, but its grey-failure resilience claim outruns the evidence. the 4 major comments →

arxiv 2604.00368 v2 pith:5MYGF7ZJ submitted 2026-04-01 cs.DC

TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving

Feng Ren , Ruoyu Qin , Teng Ma , Shangming Cai , Zheng Liu , Chao Lei , Dejiang Zhu , Ke Yang
show 11 more authors
Zheming Li Jialei Cui Weixiao Huang Yikai Zhao Yineng Zhang Hao Wu Xiang Gao Yuhao Fu Jinlei Jiang Yongwei Wu Mingxing Zhang
This is my paper
classification cs.DC
keywords declarative data movementslice sprayingmulti-rail RDMAself-healing transporttelemetry-driven schedulingLLM servingKVCache migrationcheckpoint engine
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the dominant obstacle to fast, resilient data movement in disaggregated LLM serving is not raw link speed but early, static path binding: today's transfer engines commit to a transport at initialization and stripe blindly, so a single slow or flapping rail stalls the whole elephant flow. TENT claims to fix this by making data movement declarative — applications say what to move, and the engine decides how at slice granularity, using live queue depths and completion latencies to send each 64 KB slice to the rail that will finish it soonest. The payoff, if true, is that multi-rail bandwidth is actually aggregated instead of strangled by the slowest link, and that routine hardware churn becomes an invisible, sub-50 ms performance blip rather than an application-visible failure. A sympathetic reader should care because KVCache migration and model-weight updates sit directly on the serving critical path, so transport-level wins translate into user-visible TTFT, throughput, and RL checkpoint times.

Core claim

On the paper's own terms, the central discovery is that the transfer engine should own path selection, multi-rail scheduling, and resilience, and should express them declaratively: a request names source and destination segments, and the runtime builds a ranked transport plan at request time, decomposes the flow into fine-grained slices, and dispatches each slice by a predictive cost model that estimates completion time from queued bytes, bandwidth, and a topology penalty. The engine continuously feeds observed completions back into the model, so it steers around congested or failing rails and re-admits recovered ones. The paper reports that this design, deployed as a production data plane,

What carries the argument

The load-bearing mechanism is Algorithm 1, a telemetry-driven slice scheduler. For each candidate device it estimates slice completion time as t_hat = beta0 + beta1*(A_d + L)/B_d, where A_d is queued bytes, B_d bandwidth, and beta coefficients are EWMA-corrected from observed completions; a tier penalty P_tier = {1,3,∞} encodes topology affinity, and a tolerance window gamma=0.05 prevents one NIC from being overused. This single scoring function simultaneously replaces round-robin striping, implements soft priority spillover from tier-1 to tier-2 rails, and drives soft exclusion and re-admission of failed rails — turning path selection from a static binding into a continuous control loop.

Load-bearing premise

The performance and self-healing gains rest on the assumption that live telemetry (queued bytes and completion latency) reflects rail health promptly and accurately enough to reroute slices before a degraded rail dominates end-to-end latency; the paper validates hard-failure recovery but does not measure detection latency for gradual grey failures.

What would settle it

On an eight-rail 200 Gbps fabric, gradually cap one rail's bandwidth from 200 to 50 Gbps over 500 ms while running 64 MB elephant flows, and record when throughput and P99 first deviate from baseline and how long until aggregate bandwidth returns. If the recovery window exceeds ~50 ms or the tail stays elevated while telemetry is stale, the sub-50 ms self-healing claim fails for grey failures; alternatively, if the EWMA feedback adapts within one or two slice completions, the claim holds.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • KVCache reuse in LLM serving can be accelerated by routing intra-node KV flows over fast GPU fabrics and inter-node flows over multi-rail RDMA, without application changes; the paper measures 3.79x throughput vs the no-HiCache baseline and 1.36x vs the predecessor engine.
  • Reinforcement-learning weight refresh windows shrink by 20-26%, enabling more frequent model updates without pausing rollouts.
  • Multi-rail fabrics stop being bottlenecked by the slowest rail: host-to-host throughput rises ~33% and P99 latency drops to ~27.6% of baseline.
  • Routine NIC and link failures become sub-50 ms throughput dips, with recovered rails reintegrated in ~26 ms, eliminating application-level retry and checkpoint logic.
  • A single engine can span RDMA, NVLink, MNNVL, Ascend UB, and io_uring with backends under 800 LOC each and near-native peak bandwidth.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same slice-level control loop were exported as a general data-movement substrate, other critical-path workloads such as disaggregated memory, storage tiering, and multi-tenant NIC sharing could inherit the same latency and resilience properties; the paper's portability results hint at this but do not demonstrate it.
  • A natural extension is to make the topology penalties self-tuning: the paper fixes P1=3 by hand, but the EWMA feedback could in principle learn tier penalties online, removing the need for operator configuration.
  • The sub-50 ms self-healing claim is validated for hard failures; testing gradual grey failures (e.g., a rail degrading from 200 to 50 Gbps over hundreds of milliseconds) would show whether telemetry updates fast enough to prevent stragglers before timeout.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. TENT is a data-movement engine for disaggregated LLM serving that replaces static transport binding with declarative transfer intents. Elephant flows are split into fine-grained slices (64 KB by default) and scheduled at runtime by a telemetry-driven linear cost model (Eq. 1) with EWMA-corrected coefficients and topology penalties. The paper claims that this design eliminates head-of-line blocking, unifies heterogeneous fabrics, and provides transparent sub-50 ms self-healing around failures and degradations. The evaluation covers SGLang HiCache KVCache reuse (up to 1.36× throughput and 26% lower P90 TTFT versus Mooncake TE), Moonshot Checkpoint Engine weight updates (20–26% faster), microbenchmarks against Mooncake TE, NIXL, and UCCL-P2P, portability across RDMA/NVLink/MNNVL/Ascend/io_uring, and a single NIC-failure injection experiment. Production deployment claims include clusters with up to thousands of GPUs.

Significance. If the claims hold, this is a useful systems contribution: it reframes multi-rail data movement as a late-bound, telemetry-driven scheduling problem, and the engineering artifacts—a small backend interface (<800 LOC per transport), a shared 31K-line core, and a declarative API—are credible and valuable to the LLM-serving community. The paper is strongest where it is concrete: open-source artifacts, reproducible H800 benchmarks, and a clean architectural separation between intent and execution. The main algorithmic novelty is modest (a linear completion-time model with online correction), so the contribution rests on the evaluation and production evidence. The production-scale and self-healing claims are not yet fully supported by the presented data, which is why the paper needs revision rather than acceptance as-is.

major comments (4)
  1. [§4.3/§5.3, Fig. 10] The headline 'sub-50 ms self-healing by rerouting slices around failures or degradations' is not validated for the degradation case. §2.2 motivates grey failures—rails whose effective bandwidth degrades without hard errors—but Fig. 10 only injects a hard NIC shutdown at t=1000 ms and recovery at t=3000 ms. In that experiment detection can ride on explicit RDMA completion errors/timeouts; the measured <50 ms dip is a hard-failure reroute, not a telemetry-driven degradation reroute. The paper never (i) injects gradual bandwidth degradation, (ii) measures detection-to-reroute latency, or (iii) shows that throughput/P99 do not degrade while the EWMA corrections in Algorithm 1 catch up. Because the abstract and §1 generalize the resilience claim to 'degradations,' this gap is load-bearing. Please add a grey-failure experiment reporting detection-to-reroute latency, or soften the claim to hard
  2. [§5, baseline configuration] The comparison appears to understate the baselines on multi-rail fabrics. The testbed paragraph says NIXL uses its default UCX policy, which selects a small set of best NICs (two by default), and the microbenchmark text says UCCL-P2P binds each registered memory region to a single NIC and performs no cross-NIC aggregation. On an eight-rail 200 Gbps fabric, these baselines are limited to one or two NICs, so the headline 'outperforms state-of-the-art baselines' conflates multi-rail enablement with smarter scheduling. Please add NIXL configured with all eight NICs (if supported) and UCCL-P2P with multi-region striping, or provide a clear justification that the chosen configurations are the vendors' intended production settings. This affects the central performance claims in §5.1.1, §5.1.3, and Figures 5–6.
  3. [§5.1.1–§5.1.3, Tables 2–3, Figures 5–10] All quantitative claims—for example 1.36× throughput, 26.4% lower P90 TTFT, 19.7%/26.1% checkpoint speedups, 33.7% higher write throughput, 27.6% lower P99—are reported as single point values with no error bars, confidence intervals, or number of runs. P90/P99 latencies are inherently noisy, and single-shot measurements do not establish that TENT's margins exceed run-to-run variation. Given that the paper's evidence is entirely empirical, repeated-run statistics are necessary for the reader to assess the strength of the claims. Please report distributions or at least means with ranges for the principal comparisons.
  4. [§4.2, Algorithm 1, and §5.1.3 Fig. 8] The scheduler depends on several free parameters—P_tier, the tolerance window γ, slice size, the EWMA update rate for β0,d and β1,d, the state-reset interval, and the optional global diffusion weight ωd—but sensitivity is explored only for the tier-1 penalty P1 (Fig. 8). In particular, the EWMA update rate and the 30 s state-reset interval are never varied, although these parameters directly determine how quickly the model reacts to grey failures, which is the scenario missing from §5.3. The EWMA update itself is described only in prose; no equation or hyperparameter values are given. Please add a sensitivity analysis for these parameters, including the detection-latency implications, or state explicitly which values come from production tuning and with what margins.
minor comments (4)
  1. [Eq. (1) and Algorithm 1] The notation is inconsistent: the completion-time estimate is written ^t_d in Eq. (1) and ^t_d in Algorithm 1, and the symbol for the EWMA-corrected coefficients is not defined with a consistent subscript style. Please harmonize the typesetting.
  2. [Fig. 8] The figure is very small, and the legend labels 'P1 = 1, P2 = 3, P3 = 5, P4 = 10' conflict with the P_tier = {1,3,∞} notation in Eq. (2). Please clarify whether these are values assigned to the tier-1 penalty or to different tiers, and enlarge the figure.
  3. [§5.2, Table 4] The portability table reports TENT's measured bandwidth against 'theoretical' peaks but does not show native-API baseline measurements for NVLink, MNNVL, or Ascend; only io_uring has a native dagger. Adding native numbers would make the 'negligible overhead' claim easier to verify.
  4. [§1 and §5.3] Production-scale statements ('billions of tokens per day,' 'clusters with up to thousands of GPUs,' 'a thousand-GPU cluster running for over a year') are operator-reported and not independently verifiable from the paper. Consider marking them explicitly as production experience rather than benchmark evidence, or moving them to a separate operational-report section.

Circularity Check

0 steps flagged

No significant circularity: TENT's headline results are external measurements, not outputs of the fitted cost model.

full rationale

The paper's central claims—up to 1.36× SGLang HiCache throughput, 26% lower P90 TTFT, 20–26% faster checkpoint updates, and sub-50 ms rerouting—are empirical benchmark and fault-injection results measured against Mooncake TE, NIXL, UCCL-P2P, and a testbed NIC failure, not quantities derived from TENT's own model. Algorithm 1 and Equations (1)–(2) define a telemetry-driven scheduling heuristic: predicted completion time is formed from bandwidth, queued bytes, and EWMA-updated correction factors, and is used to choose rails. This is an online control loop, not a first-principles derivation, and the paper does not present the model's predictions themselves as the validation target. The EWMA-fitted β0/β1 coefficients are not renamed predictions of the paper's results; the results are measured externally. Self-citations to Mooncake TE [25], Moonshot Checkpoint Engine [1], and SMART [26] identify the authors' prior systems used as baselines, predecessors, integration targets, or related-work context, and none of these citations substitutes for the benchmarks or supplies the load-bearing argument. The grey-failure detection latency for gradual degradation is indeed not directly measured, and the <50 ms recovery figure is demonstrated only for a hard NIC shutdown in Figure 10; however, this is an evidential gap in validating the self-healing claim, not a circular reduction of the claim to its inputs. No step in the paper reduces a prediction to a fitted parameter by construction or imports a uniqueness/ansatz conclusion through self-citation.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The central performance claims rest on several hand-tuned scheduling parameters (slice size, γ, topology penalties) and on assumptions that telemetry signals reflect true path quality and that per-slice retries are idempotent. The topology penalty P1 is tuned on the same testbed used for evaluation.

free parameters (6)
  • Topology penalty set P_tier = {1, 3, ∞} (P1=3 default)
    Chosen by hand and tuned on the H800 testbed; sensitivity analysis in Figure 8 shows P1=3 yields best P99 latency.
  • Tolerance window γ = 0.05
    Algorithm 1, default; determines how much suboptimal devices are tolerated before round-robin tie-break.
  • Slice size = 64 KB
    Configurable minimum slice size; small enough to reduce HoL, large enough to amortize costs (Section 4.2).
  • EWMA correction factors β0,d, β1,d = online learned
    Fitted online from observed completion times to absorb congestion effects; not fixed in the paper.
  • State reset interval = 30 s
    Periodically resets learned penalties to reintegrate recovered rails (Section 4.2).
  • Global load diffusion weight ωd = disabled by default
    Optional cross-process balancing weight; disabled in evaluation.
axioms (4)
  • domain assumption Instantaneous fabric state is observable via per-device queue length and completion latency.
    Section 4.2 Algorithm 1 assumes these signals predict slice completion time; grey-failure detection depends on this.
  • domain assumption Per-slice retries are idempotent because slices are written to absolute destination offsets.
    Section 4.3 relies on this for transparent re-execution without ordering or deduplication.
  • domain assumption Workloads are dominated by elephant flows large enough to amortize slicing.
    Section 4.2 splits transfers into 64 KB slices; the benefits are demonstrated for large KVCache/checkpoint transfers, not small messages.
  • ad hoc to paper The linear completion-time model Eq. (1) with online β correction approximates real service time well enough for scheduling.
    The model is introduced for this paper; no convergence or optimality guarantee is given.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving." pith.science (2026). https://pith.science/paper/5MYGF7ZJ

@misc{pith2026260400368,
  author       = {Pith},
  title        = {Pith review of: TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5MYGF7ZJ}},
  note         = {Machine review of arXiv:2604.00368}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Modern GPU clusters rely on complex, heterogeneous interconnects. As large language model (LLM) serving shifts toward agentic reasoning, KVCache becomes a first-class mobile asset, driving frequent migrations and massive elephant flows that dominate the execution critical path. Operating Mooncake Transfer Engine (TE) on thousands of GPUs exposed a fundamental flaw in existing frameworks: imperative, early-binding path selection. This rigidity results in state-blind striping that ignores congestion and grey failures, leading to bandwidth stranding. This also produces operational fragility where routine faults require manual intervention. We present TENT, a declarative orchestration engine that decouples transfer intent from physical execution. By abstracting interconnects into a unified resource pool, TENT shifts path resolution from initialization to slice-time late binding. Applications simply declare transfer intents, while TENT dynamically "sprays" fine-grained slices across rails based on real-time telemetry and predictive cost modeling. This orchestration eliminates head-of-line (HoL) blocking and enables transparent, sub-50 ms self-healing by rerouting slices around failures or degradations without application-level intervention. TENT serves as the production data plane for LLM inference and reinforcement learning (RL) pipelines at multiple industrial clusters. Our evaluation shows that TE+ outperforms state-of-the-art baselines, including Mooncake TE, NIXL, and UCCL. In LLM inference with SGLang HiCache, TE+ achieves up to 1.36x higher throughput and 26% lower P90 time-to-first-token (TTFT) than Mooncake TE. In RL pipelines, TENT accelerates parameter updates in Moonshot Checkpoint Engine by 20-26%.

Figures

Figures reproduced from arXiv: 2604.00368 by Chao Lei, Dejiang Zhu, Feng Ren, Hao Wu, Jialei Cui, Jinlei Jiang, Ke Yang, Mingxing Zhang, Ruoyu Qin, Shangming Cai, Teng Ma, Weixiao Huang, Xiang Gao, Yikai Zhao, Yineng Zhang, Yongwei Wu, Yuhao Fu, Zheming Li, Zheng Liu.

Figure 2
Figure 2. Figure 2: Per-rail average latency. TENT eliminates the HoL [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Logical and metadata view of a memory segment. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: TENT’s architecture and execution pipeline. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Host-to-host read/write throughput and P99 latency between two nodes, with memory allocated per socket and transfers [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Write throughput and P99 latency for point-to-point [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: GPU-to-GPU read performance measured with var [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: GPU-to-GPU P99 latency under different P1 val￾ues ( [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Impact of manual rail shutdown (at 1000 ms) and [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inference

    cs.DC 2026-05 unverdicted novelty 7.0

    EEP makes wide expert-parallel MoE serving survive single-rank failures with an 11s recovery pause, 8s reintegration pause, and throughput restored to 95% of pre-fault level within 52s while staying within 4.4% of a f...

Reference graph

Works this paper leans on

39 extracted references · 7 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Checkpoint engine

    Moonshot AI. Checkpoint engine. https://github. com/MoonshotAI/checkpoint-engine, 2025

    2025

  2. [2]

    Amd infinity fabric™ link user guide

    AMD. Amd infinity fabric™ link user guide. https: //www.amd.com/content/dam/amd/en/documents/ instinct-tech-docs/other/56978.pdf, 2020

    2020

  3. [3]

    Rocm documentation

    AMD. Rocm documentation. https://rocm.docs. amd.com/en/latest/, 2025

    2025

  4. [4]

    Lmcache: An efficient kv cache layer for enterprise-scale llm inference.arXiv preprint arXiv:2510.09665, 2025

    Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, and Junchen Jiang. Lmcache: An efficient kv cache layer for enterprise-scale llm inference.arXiv preprint arXiv:2510.09665, 2025

    arXiv 2025

  5. [5]

    Dlslime: Flexible & efficient hetero- geneous transfer toolkit

    DeepLink. Dlslime: Flexible & efficient hetero- geneous transfer toolkit. https://github.com/ DeepLink-org/DLSlime, 2025. BSD-3-Clause Li- cense; toolkit for peer-to-peer communication via RDMA, NVLink, NVShmem etc.; supports heteroge- neous transfer engines and slicing behaviour

    2025

  6. [6]

    Farm: Fast remote mem- ory

    Aleksandar Dragojevi’c, Dushyanth Narayanan, Miguel Castro, and Orion Hodson. Farm: Fast remote mem- ory. In11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), pages 401–414, 2014

    2014

  7. [7]

    The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    2024

  8. [8]

    Demystifying nccl: An in-depth analysis of gpu communication protocols and algorithms.arXiv preprint arXiv:2507.04786, 2025

    Zhiyi Hu, Siyuan Shen, Tommaso Bonato, Sylvain Jeaugey, Cedell Alexander, Eric Spada, James Dinan, Jeff Hammond, and Torsten Hoefler. Demystifying nccl: An in-depth analysis of gpu communication protocols and algorithms.arXiv preprint arXiv:2507.04786, 2025

    arXiv 2025

  9. [9]

    Huawei xfer library (hixl)

    Huawei. Huawei xfer library (hixl). https://gitcode. com/cann/hixl, 2025

    2025

  10. [10]

    Datacenter rpcs can be general and fast

    Anuj Kalia, Michael Kaminsky, and David Andersen. Datacenter rpcs can be general and fast. In16th USENIX Symposium on Networked Systems Design and Imple- mentation (NSDI 19), pages 1–16, 2019

    2019

  11. [11]

    Design guidelines for high performance rdma systems

    Anuj Kalia, Michael Kaminsky, and David G Andersen. Design guidelines for high performance rdma systems. In2016 USENIX annual technical conference (USENIX ATC 16), pages 437–450, 2016

    2016

  12. [12]

    Understanding rdma microarchitecture resources for performance isola- tion

    Xinhao Kong, Jingrong Chen, Wei Bai, Yechen Xu, Mahmoud Elhaddad, Shachar Raindel, Jitendra Padhye, Alvin R Lebeck, and Danyang Zhuo. Understanding rdma microarchitecture resources for performance isola- tion. In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 31–48, 2023

    2023

  13. [13]

    Gonza- lez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonza- lez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with page- dattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  14. [14]

    Ub-mesh: a hi- erarchically localized nd-fullmesh datacenter network architecture.arXiv preprint arXiv:2503.20377, 2025

    Heng Liao, Bingyang Liu, Xianping Chen, Zhigang Guo, Chuanning Cheng, Jianbing Wang, Xiangyu Chen, Peng Dong, Rui Meng, Wenjie Liu, et al. Ub-mesh: a hi- erarchically localized nd-fullmesh datacenter network architecture.arXiv preprint arXiv:2503.20377, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Deepseek-v3 techni- cal report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 techni- cal report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Lmcache.https://lmcache.ai/, 2025

    LMCache. Lmcache.https://lmcache.ai/, 2025

    2025

  17. [17]

    Birds of a feather flock together: Scaling rdma rpcs with flock

    Sumit Kumar Monga, Sanidhya Kashyap, and Chang- woo Min. Birds of a feather flock together: Scaling rdma rpcs with flock. InProceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, pages 212–227, 2021. 13

    2021

  18. [18]

    Gpudirect storage

    NVIDIA. Gpudirect storage. https://docs.nvidia. com/gpudirect-storage/, 2025

    2025

  19. [19]

    Nccl (nvidia collective communications li- brary).https://github.com/NVIDIA/nccl, 2025

    NVIDIA. Nccl (nvidia collective communications li- brary).https://github.com/NVIDIA/nccl, 2025

    2025

  20. [20]

    Nvidia dynamo

    NVIDIA. Nvidia dynamo. https://developer. nvidia.com/dynamo, 2025

    2025

  21. [21]

    Nvidia inference xfer library (nixl)

    NVIDIA. Nvidia inference xfer library (nixl). https: //github.com/ai-dynamo/nixl, 2025

    2025

  22. [22]

    Nvlink and nvlink switch

    NVIDIA. Nvlink and nvlink switch. https://www. nvidia.com/en-us/data-center/nvlink/, 2025

    2025

  23. [23]

    NVIDIA. Nvshmem. https://developer.nvidia. com/nvshmem, 2025

    2025

  24. [24]

    Gloo: Collective communications library with various primitives for multi-machine training

    PyTorch. Gloo: Collective communications library with various primitives for multi-machine training. https: //github.com/pytorch/gloo, 2025

    2025

  25. [25]

    Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In23rd USENIX Confer- ence on File and Storage Technologies (FAST 25), pages 155–170, Santa Clara, CA, February 2025. USENIX Association

    2025

  26. [26]

    Scaling up memory disaggregated applications with smart

    Feng Ren, Mingxing Zhang, Kang Chen, Huaxia Xia, Zuoning Chen, and Yongwei Wu. Scaling up memory disaggregated applications with smart. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pages 351–367, 2024

    2024

  27. [27]

    SGLang. Sglang. https://github.com/ sgl-project/sglang, 2025

    2025

  28. [28]

    Msccl++: Rethinking gpu communication abstrac- tions for cutting-edge ai applications.arXiv preprint arXiv:2504.09014, 2025

    Aashaka Shah, Abhinav Jangda, Binyang Li, Caio Rocha, Changho Hwang, Jithin Jose, Madan Musu- vathi, Olli Saarikivi, Peng Cheng, Qinghua Zhou, et al. Msccl++: Rethinking gpu communication abstrac- tions for cutting-edge ai applications.arXiv preprint arXiv:2504.09014, 2025

    arXiv 2025

  29. [29]

    Kimi k2: Open agen- tic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Ji- ahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agen- tic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Unified communication x

    UCX. Unified communication x. https://openucx. org/, 2025

    2025

  31. [31]

    vLLM. vllm. https://github.com/vllm-project/ vllm, 2025

    2025

  32. [32]

    Srnic: A scalable archi- tecture for rdma nics

    Zilong Wang, Layong Luo, Qingsong Ning, Chaoliang Zeng, Wenxue Li, Xinchen Wan, Peng Xie, Tao Feng, Ke Cheng, Xiongfei Geng, et al. Srnic: A scalable archi- tecture for rdma nics. In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 1–14, 2023

    2023

  33. [33]

    Characterizing and optimizing remote persistent memory with rdma and nvm

    Xingda Wei, Xiating Xie, Rong Chen, Haibo Chen, and Binyu Zang. Characterizing and optimizing remote persistent memory with rdma and nvm. In2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 523–536, 2021

    2021

  34. [34]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chen- gen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales.arXiv preprint arXiv:2308.01320, 2023

    Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales.arXiv preprint arXiv:2308.01320, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Glm-4.5: Agentic, reasoning, and coding (arc) foundation models.arXiv preprint arXiv:2508.06471, 2025

    Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models.arXiv preprint arXiv:2508.06471, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Sglang: Efficient execution of structured language model programs.Advances in neural information pro- cessing systems, 37:62557–62583, 2024

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs.Advances in neural information pro- cessing systems, 37:62557–62583, 2024

    2024

  38. [38]

    Dist- serve: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- serve: Disaggregating prefill and decoding for goodput- optimized large language model serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210, 2024

    2024

  39. [39]

    An exten- sible software transport layer for gpu networking.arXiv preprint arXiv:2504.17307, 2025

    Yang Zhou, Zhongjie Chen, Ziming Mao, ChonLam Lao, Shuo Yang, Pravein Govindan Kannan, Jiaqi Gao, Yilong Zhao, Yongji Wu, Kaichao You, et al. An exten- sible software transport layer for gpu networking.arXiv preprint arXiv:2504.17307, 2025. 14

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.