Pith. sign in

REVIEW 1 major objections 5 minor 86 references

This paper claims that CUDA API remoting over LANs can be made practical by pairing hash-based weight caching with asynchronous API forwarding and a contention-aware scheduler, cutting cached model loading from about 150 seconds to 9 second

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:34 UTC pith:3JG7VV63

load-bearing objection Gleam's weight caching and async remoting are real wins; the scheduler's undisclosed calibration term makes the 1.79x throughput claim only partly trustworthy. the 1 major comments →

arxiv 2607.23115 v1 pith:3JG7VV63 submitted 2026-07-25 cs.DC cs.LG

Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs

Zhihao Xu , Hao Zhong , Zeting Zhou , Yuhang Xu , Haoyu Tong , Wei Wang , Jinshan Chen , Keqiang He
show 4 more authors
Chong Zhu Shengzhong Liu Fan Wu Guihai Chen
This is my paper
classification cs.DC cs.LG
keywords CUDA API remotingGPU sharinglocal area networkmodel weight cachingasynchronous API executioncontention-aware schedulingedge AI inferenceCUDA context consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Gleam tries to establish that CUDA API remoting—running GPU calls from a client on a remote server's GPU—can be made practical over ordinary Wi-Fi/Ethernet LANs, not just datacenter interconnects. Its claim is that three coordinated mechanisms remove the communication bottleneck: caching verified model-weight chunks so they are transmitted once, converting frequent short CUDA calls into asynchronous or locally simulated operations, and scheduling tasks with a latency model that predicts both network and GPU contention. On a heterogeneous four-GPU testbed with seven AI workloads, Gleam reports 1.4x–24.2x faster API remoting than prior systems and up to 1.79x higher throughput, with cached model loading shrinking from roughly 150 seconds to about 9 seconds. If true, this makes underutilized nearby GPUs a practical resource for latency-sensitive edge AI inference on personal devices.

Core claim

Gleam's central discovery is that API-remoting latency over bandwidth-limited LANs comes mostly from two addressable sources: redundant transmission of static model weights and synchronous round-trips for short-lived CUDA calls. To remove the first, Gleam fingerprints every host-to-device memory chunk with a hash, verifies at cudaFree time that the chunk never changed, and caches verified weight blocks on the server, so a later invocation of the same task sends only small hash identifiers instead of gigabytes. To remove the second, it classifies APIs into three async-friendly cases—error-code-only calls that can proceed optimistically, state-maintenance calls that the client can simulate loc

What carries the argument

The load-bearing mechanism is the API remoting path manager, which turns every intercepted CUDA call into one of three communication patterns—basic async for error-code-only calls, local simulation for state-maintenance calls, and batch prefetch for resource-handle creation. That pipeline is fed by a model weight manager that identifies static weight chunks via a cudaFree-time consistency check and reuses them across connections through hash-based retrieval. Over both runs a two-phase contention-aware scheduler whose latency predictor uses the linear congestion law alpha = max(1, gamma * sum u_i), where u_i is a task's standalone resource occupancy, to decide which client-server pairs to for

Load-bearing premise

The load-bearing premise is that contention on a GPU or a network link grows linearly with the sum of standalone-measured occupancies, scaled by a single penalty factor gamma (alpha = max(1, gamma * sum u_i)); if this congestion law is wrong, the scheduler's dispatch decisions and the up-to-1.79x throughput claim collapse.

What would settle it

Run one, two, and three identical remote llama-8B tasks on the same server, recording per-task latency and total GPU utilization; the model alpha = max(1, gamma * sum u_i) predicts a specific progression from the single-task occupancy (63%, 13.76s), and if no constant gamma fits all three counts—or the same gamma fails when repeated on a different GPU model—the predictor is fitting noise, not modeling contention.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If Gleam holds, LAN GPU sharing through API remoting becomes practical for heterogeneous personal devices: model weights move once, and later loads take seconds instead of minutes.
  • Frequent short CUDA calls no longer block on network round-trips, so Wi-Fi becomes a viable interconnection for GPU offloading, not just Ethernet.
  • A contention-aware dispatcher can co-locate tasks to raise GPU utilization while controlling per-task latency, improving throughput on heterogeneous GPU pools.
  • Long-lived remoting sessions survive network failures and CUDA-graph capture without context crashes, so the mechanism is usable in production edge deployments.
  • The scheduler's per-invocation overhead stays in the tens of microseconds, making it cheap enough to run periodically at runtime.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the weight-chunk fingerprinting generalizes, the same hash-and-reuse idea could be applied to other semi-static GPU state such as prebuilt CUDA graphs or KV-cache templates to cut repeated transfers in multi-tenant serving.
  • The paper leaves the penalty factor gamma in the contention model unreported; a practical deployment would likely need to calibrate or learn gamma per cluster, since the demonstrated throughput depends on it.
  • The three-case async classification is configuration-driven, so the framework's benefit is expected to grow as more CUDA libraries' APIs are categorized; an immediate test is measuring how much additional speedup appears when cuFFT and NVML calls receive the same treatment.
  • A direct consequence of the two-phase scheduler is that homogeneous task grouping compounds cache hits, so an operator could bias workload admission to batch same-model requests and amplify the 150s-to-9s loading gain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes Gleam, a CUDA API remoting framework for GPU sharing over LANs. It combines (i) a model-weight manager that identifies, hashes, and caches static weight chunks to avoid repeated large transfers; (ii) an API remoting path manager that classifies CUDA APIs into three asynchronous patterns (error-code-only, locally simulatable state, batched resource handles); (iii) a two-phase contention-aware scheduler with an analytical latency predictor; and (iv) a CUDA-context consistency guardian with reconciliation and cross-stream protection. Experiments on a 1–4 GPU testbed with seven AI workloads compare against GVirtuS, cricket+SR, DGSF, FGD, and Mudi, reporting 1.4×–24.2× API remoting efficiency gains and up to 1.79× throughput improvement.

Significance. If the results hold, the paper demonstrates a practical path to edge GPU sharing using only API remoting, avoiding heavy environment setup. The weight-caching and asynchronous-execution mechanisms are mechanistically sound: they attack the dominant ~150s model-transfer cost and the accumulated RPC round-trip overhead, and the ablations in Figures 25–26 attribute the gains to those components. The implementation is substantial (6K C++ + 10K interception code, 1,000+ APIs covered), and the paper is unusually honest about corner cases (e.g., in-place weight blocks, hash collisions). The main weakness is the scheduler: its latency predictor is not independently validated and depends on an unreported penalty factor, which undermines confidence in the throughput claim that rests on it. The detailed tables of API coverage and workload characteristics are also useful for follow-up work.

major comments (1)
  1. [§5.1.1/§5.1.2, Eq. (4), Appendix A.3] The penalty factor γ ≥ 1 in Eq. (4) is never reported. The only model validation (A.3) reuses the same two-loading-task example from Table 3; the computation-contention case in Table 4 is never checked. With γ=1 and u=0.63 from Table 4, Eq. (4) predicts 1.26×13.76=17.34 s vs. measured 18.28 s; fitting that data requires γ≈1.05, but then the Table 3 loading prediction changes from 59.76 s to ~63 s. The paper neither discloses γ nor tests the model on held-out scenarios. Because Phase-1 grouping (Eq. 6) and Phase-2 dispatch use this predictor, the throughput claim in §7.3 is not fully supported. Please report γ, its calibration procedure, and validation on held-out contention mixes.
minor comments (5)
  1. [Abstract and §7.2] The abstract states 1.4×–24.2× API remoting speedup, but §7.2 reports “nearly 54× speedup over GVirtuS” for sd3 under Wi-Fi. Please reconcile these numbers (e.g., clarify whether the quoted range is relative to the best SOTA baseline rather than each individual baseline, or correct the numbers).
  2. [Figure 17] The numeric labels above the bars (e.g., “3.1× 1.6×”) are not defined in the caption or legend. Please state explicitly what ratio each label represents (e.g., Gleam vs. server local, or baseline vs. Gleam).
  3. [Table 3 vs. A.3] Table 3 reports 58.34 s as the concurrent loading latency for Connection 1, while A.3 states “measured latency of 58.32 s.” Use a single consistent value.
  4. [§4.2, Table 11] The “Basic Async” case includes cudaMemcpy_htod and cudaMemcpy_dtod (IDs 26–27). Please state how client-side source buffers are kept valid between the immediate local success response and the actual network transmission; otherwise, an application could overwrite source data while the asynchronous send is still in flight.
  5. [§5.1.2] The remaining-time estimate for inference uses C_invoked/C_total and assumes periodic cudaStreamSynchronize calls. Consider reporting prediction errors for tasks with irregular sync patterns (e.g., llava-7B) or explicitly discussing this limitation.

Circularity Check

0 steps flagged

No load-bearing circularity: API-remoting speedups are empirical against external baselines; the scheduler's in-sample validation is a weakness, not a circular reduction.

full rationale

The paper's headline results (1.4x-24.2x API-remoting speedups, up to 1.79x throughput) are measured end-to-end against external baselines (GVirtuS, cricket+SR, DGSF, FGD, Mudi), not derived from a fitted predictor. The weight-caching and async-path mechanisms are implemented and ablated directly (Figures 25-26), so those claims do not reduce to their inputs. The contention-aware scheduler does have a validation gap: Eq. (4) contains an unreported penalty factor gamma, and Appendix A.3 validates the model on the same Table 3 two-loading-task measurement that motivated it. However, this is an in-sample consistency check and a missing-support/correctness issue, not a circularity: the concurrent latency being 'predicted' is not fed into Eq. (5) as an input, and the throughput claim in Fig. 18 is an independent system measurement rather than a model output. No load-bearing self-citations or uniqueness-importation chains appear. Therefore, the derivation chain is not circular, though the contention model would benefit from held-out validation and a disclosed gamma.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The paper's contribution is mostly engineering. The caching mechanism rests on a loading-pattern heuristic and a hash-collision assumption; the scheduler rests on an ad hoc congestion model with a tuned penalty factor (gamma, undisclosed) and profiled occupancies; the reconciliation module has a hand-chosen buffer bound. None of these is derived from first principles or externally benchmarked, so the reader must treat the end-to-end experiments as the only evidence for the model.

free parameters (4)
  • gamma (latency increase penalty factor) = not reported in paper (must be >1 to match Table 4 data; ~1.2 reconciles 13.76s to 18.28s per inference)
    Appears in Eq. (4) alpha=max(1,gamma*sum u_i) for every scheduler latency prediction. The paper states only gamma>=1 and never reports the value used in Section 7 experiments; Table 4's computation-contention slowdown requires gamma>1, indicating the model was matched to the data it later 'validates' (A.3).
  • Per-task resource occupancies u_i = 67-80% (communication), 13-90% (computation), per task/GPU/stage (Table 17, Figure 27)
    Each task's occupancy is measured via offline standalone profiling and fed into Eq. (5); the model assumes these single-task measurements add linearly under concurrency, which Table 4 contradicts for computation.
  • max_pending_batch (reconciliation buffer bound) = 8 (default from Table 8 sweep 2/4/8/16)
    Bounds in-flight requests for the reconciliation module; the 4.16% degradation figure is tied to this hand-chosen value.
  • Scheduling interval = 1 s
    Scheduler invocation period chosen in Section 7.6; affects how quickly contention is detected and is not justified in the text.
axioms (5)
  • ad hoc to paper Latency under contention scales linearly with the sum of standalone occupancies: alpha=max(1,gamma*sum u_i)
    Section 5.1.1, Eq. (4). The congestion law is asserted, not derived; validated only in-sample against the two-task measurement in A.3, and under-predicts Table 4 without a tuned gamma.
  • domain assumption A cudaMalloc block is model weights iff all HtoD chunks written into it are unchanged at cudaFree
    Section 4.1.1. Heuristic from observed loading pattern (one malloc plus several HtoD copies); fails for in-place activations or inputs (acknowledged in A.4.1, 93%+ identification claimed).
  • domain assumption MD5 over the first 1024 bits of a chunk is a collision-free cache key for chunks >1024 B
    Section A.4.1. 'We do not observe hash collisions' is an empirical claim, not a guarantee; a collision would silently serve wrong weights.
  • domain assumption cudaStreamSynchronize invocation counts track inference progress with a periodic, linearly approximable pattern
    Section 5.1.2 and Figure 14. Remaining-time estimate r_infer relies on this periodicity; breaks for tasks with irregular sync patterns.
  • domain assumption GPU utilization and bandwidth occupancy, measured standalone, are additive across concurrent tasks
    Section 5.1.1. Underlies Eq. (5)'s sum of occupancies; Table 4 indicates two tasks' utilizations do not add linearly (63% plus second task yields 92%).
invented entities (2)
  • Latency-increase factor alpha (congestion factor) no independent evidence
    purpose: Multiplies predicted stage latency to account for concurrent contention in the scheduler (Eq. 4)
    A modeling construct with no external falsifiable handle; its validity is assessed only against the paper's own measurements (Table 4, A.3).
  • Virtual memory layer (1 TB client-side address space, red-black-tree mapping) no independent evidence
    purpose: Maps client-side device pointers to shareable server blocks so cached weights can be reused across mallocs (Figure 6)
    Internal mechanism; observable only through the system itself, with no independent test or released implementation.

pith-pipeline@v1.3.0-alltime-deepseek · 26088 in / 19790 out tokens · 178137 ms · 2026-08-01T03:34:05.915776+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs." pith.science (2026). https://pith.science/paper/3JG7VV63

@misc{pith2026260723115,
  author       = {Pith},
  title        = {Pith review of: Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3JG7VV63}},
  note         = {Machine review of arXiv:2607.23115}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This paper aims to enable computation- and communication-efficient GPU sharing across devices within local area networks (LANs), facilitating ubiquitous AI inference on heterogeneous personal devices. We achieve distributed task offloading via CUDA API remoting. However, beyond raw computation, network constraints emerge as the primary bottleneck: limited bandwidth, high-frequency API invocations, and cross-task contention significantly hinder performance. To address these challenges, we propose Gleam, a novel and network-efficient framework for task-generic GPU sharing across local-area CUDA devices, with three key contributions. First, we reduce bandwidth overhead in CUDA API remoting through automatic model weight caching, and mitigate accumulated latency from frequent API calls by asynchronous execution. Second, we design a runtime task scheduler that dynamically determines API remoting pairs between LAN clients and servers, explicitly accounting for both network conditions and GPU resource contention under parallel workloads. Finally, we introduce dedicated mechanisms to ensure CUDA context consistency across distributed executions. Extensive experiments on heterogeneous NVIDIA GPUs and diverse AI workloads show Gleam consistently outperforms state-of-the-art baselines, achieving 1.4-24.2 times improvements in API remoting efficiency and up to 1.79 times higher system throughput.

Figures

Figures reproduced from arXiv: 2607.23115 by Chong Zhu, Fan Wu, Guihai Chen, Haoyu Tong, Hao Zhong, Jinshan Chen, Keqiang He, Shengzhong Liu, Wei Wang, Yuhang Xu, Zeting Zhou, Zhihao Xu.

Figure 1
Figure 1. Figure 1: Scenario of GPU sharing among local devices. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Typical GPU CUDA API invocation patterns and [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: API remoting analysis, here ‘htod’ refers to mem￾cpy from client (host) to server (device). without experiencing the round-trip paths, the throughput of this API will increase significantly. As shown in the right of [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: The tree of chunk transmission requests. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: Illustration of asynchronous API execution. [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Workflow of API remoting path manager. it can directly produce the correct return values from local state, thereby enabling asynchronous execution. • APIs applying resource handles → Batched Prefetch: These APIs return resource handles that are often con￾sumed immediately by subsequent calls, thereby creat￾ing blocking dependencies. Gleam removes them through batched pre-creation and local caching: the se… view at source ↗
Figure 11
Figure 11. Figure 11: Resource contention illustration. 0 25 50 75 100 125 150 175 200 Time (s) 0 256 512 768 1024 Bandwidth (Mbps) Bandwidth over Time llava-7B wo cache llava-7B with cache Inference Start [PITH_FULL_IMAGE:figures/full_fig_p008_11.png] view at source ↗
Figure 15
Figure 15. Figure 15: Illustration of the two-phase scheduling algorithm, [PITH_FULL_IMAGE:figures/full_fig_p009_15.png] view at source ↗
Figure 17
Figure 17. Figure 17: Comparison on efficiency of API remoting with SOTA baselines. [PITH_FULL_IMAGE:figures/full_fig_p010_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Average end-to-end latency and request throughput [PITH_FULL_IMAGE:figures/full_fig_p010_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Overall queuing delay and makespan. 0 20 40 60 2 GPUs inference latency (s) 0.00 0.25 0.50 0.75 1.00 0 20 40 60 4 GPUs inference latency (s) 0.00 0.25 0.50 0.75 1.00 Distribution ratio Gleam FGD Mudi [PITH_FULL_IMAGE:figures/full_fig_p011_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Cumulative distribution (CDF) of per-task infer [PITH_FULL_IMAGE:figures/full_fig_p011_20.png] view at source ↗
Figure 25
Figure 25. Figure 25: Ablation results on API path manager and task scheduler in Gleam. llama-8B llava-7B 10 0 10 1 10 2 10 3 Performance (s) 0:Naive 1:+Async 2:+Batch 3:+Simulate 4:+W. Cache [PITH_FULL_IMAGE:figures/full_fig_p012_25.png] view at source ↗
Figure 28
Figure 28. Figure 28: Distribution of cache chunks of the remaining 6 [PITH_FULL_IMAGE:figures/full_fig_p019_28.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

86 extracted references · 5 linked inside Pith

  1. [1]

    Cuda toolkit 12.4, 2024

    2024

  2. [2]

    ggml: Tensor library for machine learning, 2025

    2025

  3. [3]

    grpc: A high performance, open source universal rpc framework, 2025

    2025

  4. [4]

    llama.cpp: Llm inference in c/c++, 2025

    2025

  5. [5]

    llava-v1.6-mistral-7b-hf, 2025

    2025

  6. [6]

    Nvidia physicsnemo: open-source deep-learning frame- work for building, training, fine-tuning, and inferring physics ai models, 2025

    2025

  7. [7]

    Protocol buffers, 2025

    2025

  8. [8]

    stable-diffusion.cpp: Diffusion model(sd,flux,wan,...) inference in pure c/c++, 2025

    2025

  9. [9]

    whisper.cpp: High-performance inference of openai’s whisper automatic speech recognition (asr) model, 2025

    2025

  10. [10]

    Scissionlite: Accelerating distributed deep learning with lightweight data compression for iiot.IEEE Transac- tions on Industrial Informatics, 20(10):11950–11960, 2024

    Hyunho Ahn, Munkyu Lee, Sihoon Seong, Gap-Joo Na, In-Geol Chun, Blesson Varghese, and Cheol-Ho Hong. Scissionlite: Accelerating distributed deep learning with lightweight data compression for iiot.IEEE Transac- tions on Industrial Informatics, 20(10):11950–11960, 2024

    2024

  11. [11]

    Tooth: Toward optimal balance of video {QoE} and redundancy cost by {Fine- Grained}{FEC} in cloud gaming streaming

    Congkai An, Huanhuan Zhang, Shibo Wang, Jingyang Kang, Anfu Zhou, Liang Liu, Huadong Ma, Zili Meng, Delei Ma, Yusheng Dong, et al. Tooth: Toward optimal balance of video {QoE} and redundancy cost by {Fine- Grained}{FEC} in cloud gaming streaming. In22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 635–651, 2025

    2025

  12. [12]

    Crux: Gpu-efficient communication scheduling for deep learning training

    Jiamin Cao, Yu Guan, Kun Qian, Jiaqi Gao, Wencong Xiao, Jianbo Dong, Binzhang Fu, Dennis Cai, and Ennan Zhai. Crux: Gpu-efficient communication scheduling for deep learning training. InProceedings of the ACM SIGCOMM 2024 Conference, pages 1–15, 2024

    2024

  13. [13]

    Eva: Cost- efficient cloud-based cluster scheduling

    Tzu-Tao Chang and Shivaram Venkataraman. Eva: Cost- efficient cloud-based cluster scheduling. InProceedings of the Twentieth European Conference on Computer Systems, pages 1399–1416, 2025

    2025

  14. [14]

    Kernel oper- ations on the gpu, with autodiff, without memory over- flows.Journal of Machine Learning Research, 22(74):1– 6, 2021

    Benjamin Charlier, Jean Feydy, Joan Alexis Glaunes, François-David Collin, and Ghislain Durif. Kernel oper- ations on the gpu, with autodiff, without memory over- flows.Journal of Machine Learning Research, 22(74):1– 6, 2021

    2021

  15. [15]

    Remote procedure call as a managed system service

    Jingrong Chen, Yongji Wu, Shihan Lin, Yechen Xu, Xin- hao Kong, Thomas Anderson, Matthew Lentz, Xiaowei Yang, and Danyang Zhuo. Remote procedure call as a managed system service. In20th USENIX Sympo- sium on Networked Systems Design and Implementation (NSDI 23), pages 141–159, 2023

    2023

  16. [16]

    Multiplexing dynamic deep learn- ing workloads with slo-awareness in gpu clusters

    Wenyan Chen, Chengzhi Lu, Huanle Xu, Kejiang Ye, and Chengzhong Xu. Multiplexing dynamic deep learn- ing workloads with slo-awareness in gpu clusters. In Proceedings of the Twentieth European Conference on Computer Systems, pages 589–604, 2025

    2025

  17. [17]

    {GRACE}:{Loss- Resilient}{Real-Time} video through neural codecs

    Yihua Cheng, Ziyi Zhang, Hanchen Li, Anton Arapin, Yue Zhang, Qizheng Zhang, Yuhan Liu, Kuntai Du, Xu Zhang, Francis Y Yan, et al. {GRACE}:{Loss- Resilient}{Real-Time} video through neural codecs. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 509–531, 2024

    2024

  18. [18]

    Flashattention: Fast and memory- efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344– 16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344– 16359, 2022

    2022

  19. [19]

    Oneadapt: Fast adapta- tion for deep learning applications via backpropagation

    Kuntai Du, Yuhan Liu, Yitian Hao, Qizheng Zhang, Haodong Wang, Yuyang Huang, Ganesh Anantha- narayanan, and Junchen Jiang. Oneadapt: Fast adapta- tion for deep learning applications via backpropagation. InProc. ACM SoCC 2023

    2023

  20. [20]

    Niklas Eiling, Jonas Baude, Stefan Lankes, and An- tonello Monti. Cricket: A virtualization layer for dis- tributed execution of cuda applications with check- point/restart support.Concurrency and Computation: Practice and Experience, 34(14):e6474, 2022

    2022

  21. [21]

    Shiwei Fang, Jin Huang, Colin Samplawski, Deepak Ganesan, Benjamin Marlin, Tarek Abdelzaher, and Mag- gie B. Wigness. Optimizing intelligent edge-clouds with partitioning, compression and speculative inference. In MILCOM 2021 - 2021 IEEE Military Communications Conference (MILCOM), pages 892–896, 2021

    2021

  22. [22]

    Dgsf: Disaggregated gpus for serverless functions

    Henrique Fingler, Zhiting Zhu, Esther Yoon, Zhipeng Jia, Emmett Witchel, and Christopher J Rossbach. Dgsf: Disaggregated gpus for serverless functions. In2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 739–750. IEEE, 2022

    2022

  23. [23]

    Rdma over ethernet for distributed training at meta scale

    Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, et al. Rdma over ethernet for distributed training at meta scale. InProceedings of the ACM SIGCOMM 2024 Conference, pages 57–70, 2024. 13

    2024

  24. [24]

    Guin Gilman and Robert J. Walls. Refine: A reactive and fine-grained scheduling framework for concurrency on general purpose gpus. InProceedings of the 37th ACM Symposium on Parallelism in Algorithms and Ar- chitectures, SPAA ’25, page 193–209, New York, NY , USA, 2025. Association for Computing Machinery

    2025

  25. [25]

    A gpgpu transparent virtualization component for high performance computing clouds

    Giulio Giunta, Raffaele Montella, Giuseppe Agrillo, and Giuseppe Coviello. A gpgpu transparent virtualization component for high performance computing clouds. In European Conference on Parallel Processing, pages 379–

  26. [26]

    Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo

    Juncheng Gu, Mosharaf Chowdhury, Kang G. Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. Tiresias: A GPU cluster manager for distributed deep learning. In16th USENIX Sympo- sium on Networked Systems Design and Implementation (NSDI 19), pages 485–500, Boston, MA, February 2019. USENIX Association

    2019

  27. [27]

    Kace: Kernel-aware colocation for effi- cient gpu spatial sharing

    Bing-Shiun Han, Tathagata Paul, Zhenhua Liu, and An- shul Gandhi. Kace: Kernel-aware colocation for effi- cient gpu spatial sharing. InProceedings of the 2024 ACM Symposium on Cloud Computing, SoCC ’24, page 460–469, New York, NY , USA, 2024. Association for Computing Machinery

    2024

  28. [28]

    Microsecond-scale preemption for concurrent GPU-accelerated DNN inferences

    Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. Microsecond-scale preemption for concurrent GPU-accelerated DNN inferences. In16th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 22), pages 539–558, Carlsbad, CA, July 2022. USENIX Association

    2022

  29. [29]

    Multi-agent collaborative infer- ence via dnn decoupling: Intermediate feature compres- sion and edge learning.IEEE Transactions on Mobile Computing, 22(10):6041–6055, 2023

    Zhiwei Hao, Guanyu Xu, Yong Luo, Han Hu, Jianping An, and Shiwen Mao. Multi-agent collaborative infer- ence via dnn decoupling: Intermediate feature compres- sion and edge learning.IEEE Transactions on Mobile Computing, 22(10):6041–6055, 2023

    2023

  30. [30]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  31. [31]

    In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 87–101, 2023

    Changho Hwang, KyoungSoo Park, Ran Shu, Xinyuan Qu, Peng Cheng, and Yongqiang Xiong.{ARK}:{GPU- driven} code execution for distributed deep learning. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 87–101, 2023

    2023

  32. [32]

    Charlie Hu, Xiaojun Lin, and Nan Deng

    Akshay Jajoo, Y . Charlie Hu, Xiaojun Lin, and Nan Deng. A case for task sampling based learning for cluster job scheduling. In19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 19–33, Renton, WA, April 2022. USENIX Association

    2022

  33. [33]

    A house united within itself: Slo-awareness for on-premises containerized ml inference clusters via faro

    Beomyeol Jeon, Chen Wang, Diana Arroyo, Alaa Youssef, and Indranil Gupta. A house united within itself: Slo-awareness for on-premises containerized ml inference clusters via faro. InProceedings of the Twen- tieth European Conference on Computer Systems, pages 524–540, 2025

    2025

  34. [34]

    Mor- ley Mao

    Shuowei Jin, Xueshen Liu, Qingzhao Zhang, and Z. Mor- ley Mao. Compute or load kv cache? why not both?, 2025

    2025

  35. [35]

    Deepum: Tensor migration and prefetching in unified memory

    Jaehoon Jung, Jinpyo Kim, and Jaejin Lee. Deepum: Tensor migration and prefetching in unified memory. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2023, page 207–221, New York, NY , USA, 2023. Association for Computing Machinery

    2023

  36. [36]

    A neural-network- based realization of in-network computation for the in- ternet of things

    Nichoas Kaminski, Irene Macaluso, Emanuele Di Pas- cale, Avishek Nag, John Brady, Mark Kelly, Keith Nolan, Wael Guibene, and Linda Doyle. A neural-network- based realization of in-network computation for the in- ternet of things. In2017 IEEE International Conference on Communications (ICC), pages 1–6, 2017

    2017

  37. [37]

    {SuperServe}:{Fine-Grained} inference serving for unpredictable workloads

    Alind Khare, Dhruv Garg, Sukrit Kalra, Snigdha Grandhi, Ion Stoica, and Alexey Tumanov. {SuperServe}:{Fine-Grained} inference serving for unpredictable workloads. In22nd USENIX Sympo- sium on Networked Systems Design and Implementation (NSDI 25), pages 739–758, 2025

    2025

  38. [38]

    A survey on in-network computing: Programmable data plane and technology specific applications.IEEE Communications Surveys & Tutorials, 25(1):701–761, 2023

    Somayeh Kianpisheh and Tarik Taleb. A survey on in-network computing: Programmable data plane and technology specific applications.IEEE Communications Surveys & Tutorials, 25(1):701–761, 2023

    2023

  39. [39]

    Navigator: Dynamic multi-kernel scheduling to improve gpu per- formance

    Jiho Kim, John Kim, and Yongjun Park. Navigator: Dynamic multi-kernel scheduling to improve gpu per- formance. In2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1–6, 2020

    2020

  40. [40]

    Efficient memory manage- ment for large language model serving with pagedatten- tion

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory manage- ment for large language model serving with pagedatten- tion. InProceedings of the 29th symposium on operating systems principles, pages 611–626, 2023

    2023

  41. [41]

    Forecasting gpu performance for deep learning train- ing and inference

    Seonho Lee, Amar Phanishayee, and Divya Mahajan. Forecasting gpu performance for deep learning train- ing and inference. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vol- ume 1, pages 493–508, 2025. 14

    2025

  42. [42]

    A survey on large language model acceleration based on kv cache management, 2025

    Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. A survey on large language model acceleration based on kv cache management, 2025

    2025

  43. [43]

    THC: Accelerating distributed deep learning using ten- sor homomorphic compression

    Minghao Li, Ran Ben Basat, Shay Vargaftik, ChonLam Lao, Kevin Xu, Michael Mitzenmacher, and Minlan Yu. THC: Accelerating distributed deep learning using ten- sor homomorphic compression. In21st USENIX Sympo- sium on Networked Systems Design and Implementation (NSDI 24), pages 1191–1211, Santa Clara, CA, April

  44. [44]

    Personal llm agents: Insights and survey about the capability, efficiency and security

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Incbricks: To- ward in-network computation with an in-network cache

    Ming Liu, Liang Luo, Jacob Nelson, Luis Ceze, Arvind Krishnamurthy, and Kishore Atreya. Incbricks: To- ward in-network computation with an in-network cache. SIGARCH Comput. Archit. News, 45(1):795–809, April 2017

    2017

  46. [46]

    Cachegen: Kv cache compression and streaming for fast large lan- guage model serving

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. Cachegen: Kv cache compression and streaming for fast large lan- guage model serving. InProceedings of the ACM SIG- COMM 2024 Conference, pages 38–56, 2024

    2024

  47. [47]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Fe- ichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 11976–11986, 2022

    2022

  48. [48]

    Dynamic llm-agent network: An llm-agent collab- oration framework with agent team optimization.arXiv preprint arXiv:2310.02170, 2023

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An llm-agent collab- oration framework with agent team optimization.arXiv preprint arXiv:2310.02170, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    A survey of storage systems in the rdma era.IEEE Transac- tions on Parallel and Distributed Systems, 33(12):4395– 4409, 2022

    Shaonan Ma, Teng Ma, Kang Chen, and Yongwei Wu. A survey of storage systems in the rdma era.IEEE Transac- tions on Parallel and Distributed Systems, 33(12):4395– 4409, 2022

    2022

  50. [50]

    Skyserve: Serving ai mod- els across regions and clouds with spot instances

    Ziming Mao, Tian Xia, Zhanghao Wu, Wei-Lin Chiang, Tyler Griggs, Romil Bhardwaj, Zongheng Yang, Scott Shenker, and Ion Stoica. Skyserve: Serving ai mod- els across regions and clouds with spot instances. In Proceedings of the Twentieth European Conference on Computer Systems, pages 159–175, 2025

    2025

  51. [51]

    Efficient scheduling policies for Microsecond-Scale tasks

    Sarah McClure, Amy Ousterhout, Scott Shenker, and Sylvia Ratnasamy. Efficient scheduling policies for Microsecond-Scale tasks. In19th USENIX Sympo- sium on Networked Systems Design and Implementa- tion (NSDI 22), pages 1–18, Renton, WA, April 2022. USENIX Association

    2022

  52. [52]

    To- ward performance-portable petsc for gpu-based exascale systems.Parallel Computing, 108:102831, 2021

    Richard Tran Mills, Mark F Adams, Satish Balay, Jed Brown, Alp Dener, Matthew Knepley, Scott E Kruger, Hannah Morgan, Todd Munson, Karl Rupp, et al. To- ward performance-portable petsc for gpu-based exascale systems.Parallel Computing, 108:102831, 2021

    2021

  53. [53]

    Porting warpx to gpu- accelerated platforms.Parallel Computing, 108:102833, 2021

    Andrew Myers, A Almgren, Ligia Diana Amorim, J Bell, Luca Fedeli, Lixin Ge, Kevin Gott, David P Grote, M Hogan, Axel Huebl, et al. Porting warpx to gpu- accelerated platforms.Parallel Computing, 108:102833, 2021

    2021

  54. [54]

    Jellyfish: Timely inference serving for dynamic edge networks

    Vinod Nigade, Pablo Bauszat, Henri Bal, and Lin Wang. Jellyfish: Timely inference serving for dynamic edge networks. In2022 IEEE Real-Time Systems Symposium (RTSS), pages 277–290, 2022

    2022

  55. [55]

    Bringing umap closer to the speed of light with gpu acceleration

    Corey J Nolet, Victor Lafargue, Edward Raff, Thejaswi Nanditale, Tim Oates, John Zedlewski, and Joshua Pat- terson. Bringing umap closer to the speed of light with gpu acceleration. InProceedings of the AAAI Confer- ence on Artificial Intelligence, volume 35, pages 418– 426, 2021

    2021

  56. [56]

    Nvidia nvswitch: The world’s highest- bandwidth on-node switch

    WP NVIDIA. Nvidia nvswitch: The world’s highest- bandwidth on-node switch. 2018

    2018

  57. [57]

    Gemel: Model merging for memory-efficient,real-time video analytics at the edge

    Arthi Padmanabhan, Neil Agarwal, Anand Iyer, Anan- thanarayanan, and et.al. Gemel: Model merging for memory-efficient,real-time video analytics at the edge. In20th USENIX Symposium on Networked Systems De- sign and Implementation (NSDI 23), pages 973–994, 2023

    2023

  58. [58]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Pith/arXiv arXiv 2023

  59. [59]

    {CASSINI}:{Network-Aware} job scheduling in machine learning clusters

    Sudarsanan Rajasekaran, Manya Ghobadi, and Aditya Akella. {CASSINI}:{Network-Aware} job scheduling in machine learning clusters. In21st USENIX Sympo- sium on Networked Systems Design and Implementation (NSDI 24), pages 1403–1420, 2024

    2024

  60. [60]

    Sentinel: Efficient tensor migration and allocation on heterogeneous memory systems for deep learning

    Jie Ren, Jiaolin Luo, Kai Wu, Minjia Zhang, Hyeran Jeon, and Dong Li. Sentinel: Efficient tensor migration and allocation on heterogeneous memory systems for deep learning. In2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 598–611. IEEE, 2021. 15

    2021

  61. [61]

    Enabling large dynamic neural net- work training with learning-based memory manage- ment

    Jie Ren, Dong Xu, Shuangyan Yang, Jiacheng Zhao, Zhicheng Li, Christian Navasca, Chenxi Wang, Harry Xu, and Dong Li. Enabling large dynamic neural net- work training with learning-based memory manage- ment. In2024 IEEE International Symposium on High- Performance Computer Architecture (HPCA), pages 788–802, 2024

    2024

  62. [62]

    {Cloud-LoRa}: Enabling cloud radio access {LoRa} networks using reinforce- ment learning based {Bandwidth-Adaptive} compres- sion

    Muhammad Osama Shahid, Daniel Koch, Jayaram Raghuram, Bhuvana Krishnaswamy, Krishna Chinta- lapudi, and Suman Banerjee. {Cloud-LoRa}: Enabling cloud radio access {LoRa} networks using reinforce- ment learning based {Bandwidth-Adaptive} compres- sion. In21st USENIX Symposium on Networked Sys- tems Design and Implementation (NSDI 24), pages 1959– 1976, 2024

    1959

  63. [63]

    Exploiting simultaneous communications to accelerate data parallel distributed deep learning

    Shaohuai Shi, Xiaowen Chu, and Bo Li. Exploiting simultaneous communications to accelerate data parallel distributed deep learning. InProc. IEEE INFOCOM 2021

    2021

  64. [64]

    Orion: Interference-aware, fine-grained gpu sharing for ml ap- plications

    Foteini Strati, Xianzhe Ma, and Ana Klimovic. Orion: Interference-aware, fine-grained gpu sharing for ml ap- plications. InProceedings of the Nineteenth European Conference on Computer Systems, pages 1075–1092, 2024

    2024

  65. [65]

    gremote: Cloud ren- dering on gpu resource pool based on api-forwarding

    Dongjie Tang, Linsheng Li, Jiacheng Ma, Xue Liu, Zhengwei Qi, and Haibing Guan. gremote: Cloud ren- dering on gpu resource pool based on api-forwarding. Journal of Systems Architecture, 116:102055, 2021

    2021

  66. [66]

    Lammps-a flexible simu- lation tool for particle-based materials modeling at the atomic, meso, and continuum scales.Computer physics communications, 271:108171, 2022

    Aidan P Thompson, H Metin Aktulga, Richard Berger, Dan S Bolintineanu, W Michael Brown, Paul S Crozier, Pieter J In’t Veld, Axel Kohlmeyer, Stan G Moore, Trung Dac Nguyen, et al. Lammps-a flexible simu- lation tool for particle-based materials modeling at the atomic, meso, and continuum scales.Computer physics communications, 271:108171, 2022

    2022

  67. [67]

    Plssvm—parallel least squares support vector machine

    Alexander Van Craen, Marcel Breyer, and Dirk Pflüger. Plssvm—parallel least squares support vector machine. Software Impacts, 14:100343, 2022

    2022

  68. [68]

    Aqua: Network-accelerated memory offloading for llms in scale-up gpu domains

    Abhishek Vijaya Kumar, Gianni Antichi, and Rachee Singh. Aqua: Network-accelerated memory offloading for llms in scale-up gpu domains. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 48–62, 2025

    2025

  69. [69]

    Coflow scheduling for llm training

    Xinchen Wan, Xinyu Yang, Kaiqiang Xu, Xudong Liao, Yilun Jin, Yijun Sun, Zhenghang Ren, Han Tian, and Kai Chen. Coflow scheduling for llm training. InProceed- ings of the ACM SIGCOMM 2025 Conference, pages 1232–1234, 2025

    2025

  70. [70]

    Character- izing network requirements for gpu api remoting in ai applications.arXiv preprint arXiv:2401.13354, 2024

    Tianxia Wang, Zhuofu Chen, Wei, and et.al. Character- izing network requirements for gpu api remoting in ai applications.arXiv preprint arXiv:2401.13354, 2024

    Pith/arXiv arXiv 2024

  71. [71]

    Xiaoyang Wang, Yongkun Li, Fan Guo, Yinlong Xu, and John C. S. Lui. Dynamic gpu scheduling with multi- resource awareness and live migration support.IEEE Transactions on Cloud Computing, 11(3):3153–3167, 2023

    2023

  72. [72]

    Beware of fragmentation: Scheduling {GPU-Sharing} workloads with fragmentation gradient descent

    Qizhen Weng, Lingyun Yang, Yinghao Yu, Wei Wang, Xiaochuan Tang, Guodong Yang, and Liping Zhang. Beware of fragmentation: Scheduling {GPU-Sharing} workloads with fragmentation gradient descent. In2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 995–1008, 2023

    2023

  73. [73]

    Transparent {GPU} sharing in container clouds for deep learning workloads

    Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. Transparent {GPU} sharing in container clouds for deep learning workloads. In20th USENIX Symposium on Networked Systems Design and Imple- mentation (NSDI 23), pages 69–85, 2023

    2023

  74. [74]

    Yan, and Junchen Jiang

    Zhengxu Xia, Yajie Zhou, Francis Y . Yan, and Junchen Jiang. Genet: automatic curriculum generation for learn- ing adaptation in networking. SIGCOMM ’22, page 397–413, New York, NY , USA, 2022. Association for Computing Machinery

    2022

  75. [75]

    Efficient tensor offloading for large deep-learning model training based on compute express link

    Dong Xu, Yuan Feng, Kwangsik Shin, Daewoo Kim, Hyeran Jeon, and Dong Li. Efficient tensor offloading for large deep-learning model training based on compute express link. InSC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–18, 2024

    2024

  76. [76]

    Infless: a native serverless system for low-latency, high- throughput inference

    Yanan Yang, Laiping Zhao, Yiming Li, Huanyu Zhang, Jie Li, Mingyang Zhao, Xingzhen Chen, and Keqiu Li. Infless: a native serverless system for low-latency, high- throughput inference. InProceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pages 768–781, 2022

    2022

  77. [77]

    Deep compressive offloading: Speeding up neural net- work inference by trading edge computation for network latency

    Shuochao Yao, Jinyang Li, Dongxin Liu, Tianshi Wang, Shengzhong Liu, Huajie Shao, and Tarek Abdelzaher. Deep compressive offloading: Speeding up neural net- work inference by trading edge computation for network latency. InProceedings of the 18th conference on em- bedded networked sensor systems, pages 476–488, 2020

    2020

  78. [78]

    Horus: granular in-network task sched- uler for cloud datacenters

    Parham Yassini, Khaled Diab, Saeed Zangeneh, and Mo- hamed Hefeeda. Horus: granular in-network task sched- uler for cloud datacenters. InProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation, NSDI’24, USA, 2024. USENIX Asso- ciation. 16

    2024

  79. [79]

    Faaswap: Slo-aware, gpu-efficient serverless inference via model swapping

    Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xi- aonan Luo, Zhuohao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, and Haoran Yang. Faaswap: Slo-aware, gpu-efficient serverless inference via model swapping. arXiv preprint arXiv:2306.03622, 2023

    Pith/arXiv arXiv 2023

  80. [80]

    Ben Zhang, Xin Jin, Sylvia Ratnasamy, John Wawrzynek, and Edward A. Lee. Awstream: adaptive wide-area streaming analytics. InProceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, SIGCOMM ’18, page 236–252, New York, NY , USA, 2018. Association for Computing Machinery

    2018

Showing first 80 references.