Pith. sign in

REVIEW 4 major objections 4 minor 83 references

A single-GPU serving system that coordinates which omni requests enter the shared stage and how many SMs that stage gets when downstream generators co-run preserves class-specific first-response SLOs across text, speech, and image traffic.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 20:53 UTC pith:FABQALSQ

load-bearing objection Real problem, coherent design, plausible gains — but the evaluation needs variance bars, stronger baselines, and a non-circular validation before I'd trust the headline numbers. the 4 major comments →

arxiv 2608.01785 v1 pith:FABQALSQ submitted 2026-08-03 cs.DC

HorizonServe: Coordinating Request Scheduling with GPU Sharing for Omni-Model Serving

Yuning Zhang , Dong Yuan This is my paper
classification cs.DC
keywords omni-model servingheterogeneous SLOsGPU sharingrequest schedulingSM throttlingmemory-bandwidth contentionSLO-satisfied goodputmultimodal inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Omni models — one model that can answer with text, speech, or images — create a serving problem that text-only LLM serving avoids: requests with very different first-response deadlines share the same initial backbone, and longer requests then activate output-specific generators on the same GPU. HorizonServe claims this coupling can be tamed on a single GPU by scheduling admission into the shared backbone and the GPU parallelism given to that backbone as one feedback loop. The loop protects near-deadline short requests, rotates shared-stage opportunities between short and long paths, caps long-path admissions, and throttles the shared stage's SM count when downstream generators are active. On two GPU platforms and three omni-model pipelines, the paper reports up to 4.9x higher SLO attainment under rising load, up to 7.0x higher under downstream-heavy traffic, and 38.4–63.7% lower per-class first-response latency, with under 1% throughput overhead. If this holds, single-GPU and edge omni serving can promise class-specific response targets instead of accepting that speech or image generation silently destroys the text experience.

Core claim

The paper's central claim is that SLO-satisfied goodput on one GPU collapses under omni traffic not because of raw load alone, but because temporal sharing of the shared stage and spatial sharing with downstream generators reinforce each other: a long-path request consumes shared-stage slots, then adds memory-bandwidth pressure that inflates short-path TTFT, and the tightest-SLO class degrades first. HorizonServe establishes that both sides must be controlled together. It admits requests based on remaining slack, hides non-phase requests from the scheduler view for one step to rotate paths, caps ordinary long-path admissions, and maps a smoothed memory-pressure signal to a bounded shared-sta

What carries the argument

The load-bearing object is the omni-specific feedback loop connecting four signals: output-path admission into the shared stage, deadline slack, downstream activation from admitted long-path requests, and the bounded shared-stage SM cap. It is realized by two coupled schedulers: a temporal scheduler that shows the base runtime a filtered queue view (keeping protected and decode work visible, stashing ordinary non-phase prefills, rotating short/long phases, and capping long-path prefills), and a spatial controller that selects an SM cap from an offline ladder based on a normalized, EWMA-smoothed memory-activity signal. The cap is the crucial mechanism: because fixed-SM profiling shows decode

Load-bearing premise

The system assumes that the offline calibration — decode throughput stops improving past roughly 64–80 SMs and full allocation during co-running sharply inflates short-reply latency — remains true under the live workload; if real traffic changes that scaling behavior, the SM-throttling mechanism could hurt instead of help.

What would settle it

Run the same three workloads and arrival rates on a GPU or model where shared-stage decode keeps getting faster well past the calibrated SM cap, with no early throughput saturation and no memory-bandwidth cliff. If bounding the shared stage to the cap then yields lower SLO attainment than allowing full allocation, the central throttling claim fails. A cheaper check: in a long-path sweep, verify that the controller's memory-activity pressure signal tracks measured achieved memory bandwidth; if they diverge during co-running, the controller is steering by a stale proxy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Short-path requests keep their first-response SLO even when a burst of speech or image requests is queued; the tightest-SLO class stops being the first casualty.
  • SLO attainment gains appear on two GPU families with different SM counts and memory bandwidths, so the mechanism is not an artifact of one hardware profile.
  • Shared-stage token throughput is essentially preserved (p50 TPOT changes at most about 0.5 ms/token, p95 at most about 1 ms/token), so the latency gains are not bought by throttling decode.
  • The policy stacks on any base serving runtime that exposes scheduling-step boundaries and a per-stage SM cap; the control loop costs under 1% throughput.
  • The design generalizes across three downstream generation patterns — speech synthesis, audio-code/waveform generation, and image diffusion — covering the spectrum of current omni outputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the temporal half — slack-aware path rotation and long-path admission caps — to disaggregated or multi-GPU omni serving, where the spatial SM split becomes less relevant but path admission still shapes downstream device load.
  • The fixed number of long-path phases and the offline SM ladder are calibration choices; one could make both adaptive online, for instance inferring saturation from runtime TPOT-versus-SM observations instead of a one-time sweep.
  • The memory-pressure proxy in the prototype is tied to one vendor's activity counters; the same ladder logic should port to other hardware via equivalent counters, which is testable on non-NVIDIA GPUs.
  • The 5x SLO multiplier is a policy knob; under tighter multipliers the same feedback loop may need more aggressive path rotation, a regime the paper does not report.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. HorizonServe targets single-GPU serving of omni-models whose requests share an initial multimodal backbone and then diverge into short text-output paths or long speech/image-generation paths with heterogeneous first-response SLOs. The paper makes three observations (§3): heterogeneous latency regimes make shared-stage delay asymmetric across classes, long-path traffic collapses short-path goodput, and full shared-stage SM allocation under co-running downstream stages degrades short-request TTFT. It then presents a serving system that combines (i) offline profiling of per-class solo p95 latency and fixed-SM decode TPOT, (ii) a temporal scheduler with path rotation, slack protection, and a long-path admission cap (§4.3), and (iii) a spatial controller that maps a smoothed GPU memory-activity signal to a shared-stage SM cap via Eq. (1) (§4.4). The evaluation (§6) covers three omni-model pipelines (Qwen2.5-Omni, MiMo-Audio, BAGEL) on two GPUs, comparing against Vanilla, EDF, MPS, and EDF-MPS baselines. The paper claims up to 4.9× SLO-attainment improvement in arrival-rate sweeps and 7.0× under downstream-heavy traffic, with per-class p95 first-response latency reductions of 38.4–63.7%, while preserving shared-stage TPOT and adding <1% throughput overhead.

Significance. If validated, HorizonServe addresses a genuinely new scheduling problem: the feedback path from output-path admission to downstream GPU pressure and back to shared-stage SM allocation. The paper identifies a real workload pattern in recent omni-model architectures and proposes a coherent joint temporal–spatial control loop. Strengths include: implementation on top of vLLM-Omni, evaluation across three distinct downstream-generation patterns and two GPU platforms, ablation of the two main mechanisms (§6.5), direct measurement of scheduling overhead (Table 4), and an honest statement in §6.4 that the hardware-signal trace is not a standalone performance comparison. The empirical claims are, however, supported by representative single runs without error bars, and the spatial controller’s key calibration is validated only by a correlation that is partly circular. These issues do not disprove the central idea, but they mean the quantitative contribution is not yet established at the level expected for a systems venue.

major comments (4)
  1. [§4.4, Eq. (1), §3.3, Fig. 4] The spatial controller is calibrated entirely from an isolated fixed-SM sweep: §4.2 chooses S_cap as the smallest SM count with near-best decode TPOT from Fig. 4(b), and Eq. (1) never lets the runtime cap exceed S_cap. Fig. 4(b) is measured with a fixed batch and a specific downstream stage, outside the mixed workloads of Fig. 8. Nothing in the paper shows that the TPOT saturation knee or the TTFT degradation curve is invariant to batch composition, KV-cache state, downstream generator type, or arrival rate. If the knee shifts, the fixed cap either over-throttles shared-stage throughput (adding queueing delay) or under-throttles (reproducing full-allocation contention). This is load-bearing because the paper’s stated contribution is the feedback loop itself; if S_cap is miscalibrated under the actual workload, SM throttling can actively reduce SLO attainment. I request a concrete robustn
  2. [§6.4] The 0.926 correlation between the smoothed memory signal and the selected SM cap is presented as validating the bandwidth proxy. This is circular: s*_shared is a deterministic monotone function of the smoothed pressure signal through Eq. (1), so a high correlation follows by construction and says nothing about whether the proxy or the cap is correct. The paragraph does acknowledge the figure is not a standalone performance comparison, but the correlation statement still overinterprets the trace. To validate the proxy, compare the PyNVML signal against an independently measured memory-bandwidth counter (e.g., GPU hardware performance counters) under the same traffic, or show that a controller driven by the proxy outperforms one driven by an oracle bandwidth signal. As written, the runtime evidence for the pressure-to-cap mapping is lacking.
  3. [§6.2 and §6.1] The main quantitative claims rest on single representative runs without error bars or confidence intervals. The paper says 'We observed consistent trends across repeated runs and report representative runs' but does not report the variance. In a systems evaluation, the 4.9× and 7.0× improvements (Figs. 8, and the per-class latency claims in §6.3) should be accompanied by repeated-run statistics, especially under Poisson arrivals where stochastic variability can be substantial. I request at least mean ± spread over a small number of seeds for the key stressed points, or a statement of why single runs are sufficient on this platform. This does not invalidate the qualitative direction, but it prevents the reader from assessing reliability of the specific improvement factors.
  4. [§6.1, baselines] The comparison set (Vanilla, EDF, MPS, EDF-MPS) is quite simple. EDF and MPS are primitives, not state-of-the-art serving policies. The paper argues that existing VLM/LMM and GPU-sharing systems do not handle omni output paths, which is fair, but a more informative baseline would combine a modern scheduling policy (e.g., priority-aware or shortest-remaining-time admission within vLLM-Omni) with a modern spatial-sharing mechanism (e.g., MIG if available, or a fixed-SM split chosen by an offline optimizer). At minimum, an ablation that separates path rotation from SM throttling is present (§6.5), but the absence of any competitive serving baseline makes the claimed gains relative only to a weak default. I would like to see at least one stronger baseline or a discussion of why such baselines cannot be adapted without already adopting HorizonServe’s feedback structure.
minor comments (4)
  1. [§6.1, Fig. 8] Inconsistent naming: the figure legend says 'EDF+MPS' while the text and §6.2 use 'EDF-MPS'. Please unify.
  2. [§4.2] The definition of S_cap is split between §4.2 and §4.4; Eq. (1) uses S_cap, S_min, and S_aggr, but the reader must infer the relationship S_aggr > S_cap > S_min from the text. A formal definition or a small table of the ladder would improve readability.
  3. [§6.4, Fig. 10] Figure 10(c) y-axis label 'Stage0 SM cap (%)' and the referenced values (63, 100) are not clearly tied to the SM ladder levels. Please clarify whether the percentage is of total SMs and how ladder levels are converted.
  4. [§6.1] The SLO multiplier is set to 5× for all classes, which is a reasonable and explicitly stated choice. However, the paper should state whether results are sensitive to this multiplier; a single value leaves the reader unsure whether the observed gains are specific to the chosen SLO tightness.

Circularity Check

1 steps flagged

Headline SLO/latency results are empirical and independent; only the Section 6.4 correlation is self-referential, and the paper itself disclaims it as standalone.

specific steps
  1. self definitional [Section 6.4 (Figure 10) vs. Section 4.4 Eq. (1)]
    "Across this run, the correlation between the smoothed memory signal and the selected cap is0.926, showing that the controller tracks the pressure signal while staying within the cap range motivated by Figure 4."

    Eq. (1) defines the selected cap s*_shared(t) as snap_S(S_min + rho(t)(S_cap - S_min)) when downstream is active, where rho(t) is the clamped normalized EWMA of the same B_hat(t) signal. Hence the 'smoothed memory signal' and the 'selected cap' are input and deterministic monotone transform of input by construction; a high correlation between them is baked into the control law and cannot independently validate the PyNVML pressure proxy. The paper explicitly says this is not a standalone bandwidth measurement and it is not used to derive the main SLO-improvement claims, so the circularity is minor and non-load-bearing.

full rationale

HorizonServe's central contributions are supported by direct measurements: SLO attainment (Fig. 8), per-class p95 latency (Fig. 9), TPOT (Table 3), and ablations (Fig. 11) compare HorizonServe against Vanilla/EDF/MPS/EDF-MPS in the same runtime, so the headline 4.9x/7.0x and 38.4-63.7% numbers are empirical, not derived from fitted parameters. The offline S_cap and SM-ladder calibration (Sections 3.3, 4.2) is standard isolated profiling, not fitting to the test workload, and the SM-throttling equation bounds the cap rather than encoding the outcome. The paper's self-citation ([74]) appears only in a related-work enumeration and is not load-bearing. The one genuinely self-referential passage is the 0.926 correlation in Section 6.4, which is between the controller's input and a deterministic function of that input by Eq. (1); the paper itself cautions that it is not a standalone bandwidth measurement. Because the main results do not rest on this correlation, the overall circularity is minor.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The central claim rests on the assumed validity of offline calibration (SLO table, SM ladder) under runtime co-location, and on the premise that SM capping controls memory-bandwidth contention. The scheduler's hand-chosen parameters (N range, M, u_guard) are free parameters not derived from theory.

free parameters (7)
  • S_cap (co-running SM cap)
    Chosen offline as the smallest SM count that reaches near-best decode TPOT without full-allocation latency degradation (Section 4.2). Controls shared-stage parallelism during co-running.
  • SM ladder [S_min, S_cap]
    Discrete SM allocation levels used by the spatial controller, calibrated by fixed-SM sweeps (Section 4.2).
  • N range [2,8] and M=2 = N in [2,8], M=2
    Rotation phase counts; N is mapped linearly from the fraction of protected short-path requests; M is fixed. These are hand-chosen heuristics (Section 4.3).
  • SLO multiplier m_c = 5
    Each class SLO target is set to 5x its solo p95 first-response latency (Section 6.1), a policy choice.
  • EWMA weight gamma = 0.4
    Smoothing factor for the memory-pressure signal (Section 6.4).
  • Saturation pressure B_sat
    Normalization threshold for the bandwidth-pressure signal; value not specified in the text.
  • Slack guard u_guard
    Threshold below which a request is marked protected; value not specified.
axioms (4)
  • domain assumption The shared-stage/downstream-stage abstraction captures the execution structure of representative omni models.
    Section 2.2 generalizes across Thinker-Talker, MoE, and fused audio LMs; if a model does not have a clean shared stage or if downstream activation is not separable, the scheduler's signals break down.
  • domain assumption Offline solo-latency profiling and fixed-SM scaling results transfer to runtime co-located traffic.
    Section 4.2 uses offline p95 and SM sweeps to set SLOs and S_cap; the paper acknowledges these are calibrated values but assumes they remain valid under mixed online load.
  • domain assumption GPU memory-bandwidth contention is the dominant cross-stage bottleneck, and SM count is an effective control knob for it.
    Section 3.3 (Figure 4) shows TTFT spikes when bandwidth utilization rises and TPOT saturates at 64-80 SMs; the entire spatial controller relies on this.
  • domain assumption The PyNVML memory-activity signal is a valid online proxy for achieved bandwidth.
    Section 4.4 uses this normalized signal as the controller input; the paper does not calibrate it against a precise bandwidth counter.

pith-pipeline@v1.3.0-daily-deepseek · 24386 in / 11735 out tokens · 108182 ms · 2026-08-04T20:53:02.000213+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of HorizonServe: Coordinating Request Scheduling with GPU Sharing for Omni-Model Serving." pith.science (2026). https://pith.science/paper/FABQALSQ

@misc{pith2026260801785,
  author       = {Pith},
  title        = {Pith review of: HorizonServe: Coordinating Request Scheduling with GPU Sharing for Omni-Model Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FABQALSQ}},
  note         = {Machine review of arXiv:2608.01785}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Omni models unify text, speech, image, and multimodal reasoning in a single serving backend, but this unified deployment exposes a new scheduling problem. Requests with different output modalities may share an initial multimodal backbone and then diverge into downstream generation stages, creating heterogeneous first-response metrics and service-level objective (SLO) targets on the same GPU. Existing large language model (LLM) and multimodal serving systems mainly optimize token progress or input-side processing, and they do not jointly control temporal sharing in the shared stage and spatial sharing among co-running stages. This paper presents HorizonServe, a single-GPU omni-model serving system that coordinates request admission and GPU allocation under heterogeneous SLOs. HorizonServe profiles per-class first-response latency, protects requests with limited slack, rotates shared-stage opportunities across execution paths, and throttles the shared-stage streaming multiprocessor (SM) allocation when downstream stages are active. Across three omni-model workloads and two GPU platforms, HorizonServe improves SLO attainment by up to 4.9$\times$ in arrival-rate sweeps and 7.0$\times$ under downstream-heavy traffic, and reduces per-class first-response latency by 38.4--63.7\%.

Figures

Figures reproduced from arXiv: 2608.01785 by Dong Yuan, Yuning Zhang.

Figure 1
Figure 1. Figure 1: Single-GPU omni-model serving couples hetero￾geneous request paths and GPU resource sharing. images, and video can be sensitive, bandwidth-heavy, and latency-sensitive, motivating inference close to where data is produced [11, 29, 30, 49, 60]. Large services can amortize omni-model execution across GPU clusters or disaggregate stages across devices [1, 25, 67], but site-level deployments often have a small… view at source ↗
Figure 2
Figure 2. Figure 2: Solo p95 first-response latency across request types on one GPU. Short-path and long-path requests occupy dis￾tinct latency regimes across Qwen-Omni, MiMo-Audio, and BAGEL. 3 Motivation 3.1 SLO Heterogeneity Omni-model serving multiplexes requests that live in fun￾damentally different latency regimes. A text-output request may complete its first response after only the shared stage, while a speech or image… view at source ↗
Figure 4
Figure 4. Figure 4: Cross-stage bandwidth contention in single-GPU omni-model serving [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: System overview of HorizonServe. bursts, and maps observed memory-bandwidth pressure to a bounded shared-stage SM cap during co-running. The joint scheduler applies the queue-view and SM-cap decisions on the single GPU. The queue view determines which paths may consume shared-stage slots in this step, and the SM cap determines how the resulting batch shares the GPU with any active downstream stage. After e… view at source ↗
Figure 7
Figure 7. Figure 7: Bandwidth-guided SM throttling. HorizonServe uses an aggressive shared-stage cap when downstream stages are inactive and throttles the shared stage to a bounded ladder during co-running. Together, path rotation, slack protection, and the long￾path cap shape both current queueing delay and future down￾stream activity. The next section describes the complemen￾tary decision: how HorizonServe sets the shared-s… view at source ↗
Figure 8
Figure 8. Figure 8: End-to-end SLO attainment under increasing arrival rates and long-path request ratios. p95 response latency, shared-stage TPOT, GPU memory￾bandwidth utilization, and scheduling overhead. 6.2 Main Results [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Hardware signal and SM response in a represen￾tative Qwen2.5-Omni run. 0.5 ms/token on RTX 6000 Ada and 0.3 ms/token on RTX PRO 6000. The p95 TPOT is also essentially unchanged for MiMo￾Audio and BAGEL, and changes by only about 1 ms/token for Qwen2.5-Omni. MPS sometimes lowers Qwen2.5-Omni p95 TPOT, but [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 9
Figure 9. Figure 9: Per-class p95 response latency including queueing delay [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Ablation of HorizonServe mechanisms under representative mixed workloads [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

83 extracted references · 43 canonical work pages · 1 internal anchor

  1. [1]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ram- jee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 117–134.https://www.usenix...

    2024

  2. [2]

    Inclusion AI, :, Bowen Ma, Cheng Zou, ChengKun Du, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Chenyu Lian, Chengxiang Fan, Dan- dan Zheng, Fudong Wang, Furong Xu, Guangming Yao, Haohao Liu, Han Peng, Jun Zhou, Junluan Xia, Jingdong Chen, Jianing Li, Jianxin Sun, Jianjiang Zhu, Jianping Jiang, Jinpeng Ou, Jun Peng, Jin Peng, Kaixiang Ji, Li Tang, Libin Wan...

    arXiv 2026

  3. [3]

    Inclusion AI, Biao Gong, Cheng Zou, Chuanyang Zheng, Chunluan Zhou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Dandan Zheng, Fudong Wang, Furong Xu, GuangMing Yao, Jun Zhou, Jingdong Chen, Jianxin Sun, Jiajia Liu, Jianjiang Zhu, Jun Peng, Kaixiang Ji, Kaiyou Song, Kaimeng Ren, Libin Wang, Lixiang Ru, Lele Xie, Longhua Tan, Lyuxin Xue, Lan Wang, Mochen Bai...

    Pith/arXiv arXiv 2025

  4. [4]

    Fan Bai, Pai Peng, Zhengzhi Tang, Zhe Wang, Gong Chen, Xiang Lu, Yinuo Li, Huan Lin, Weizhe Lin, Yaoyuan Wang, et al. 2026. EPD-Serve: A Flexible Multimodal EPD Disaggregation Inference Serving System On Ascend.arXiv preprint arXiv:2601.11590(2026)

    arXiv 2026

  5. [5]

    Anderson

    Joshua Bakita and James H. Anderson. 2023. Hardware Compute Partitioning on NVIDIA GPUs. InProceedings of the 29th IEEE Real- Time and Embedded Technology and Applications Symposium. IEEE, Piscataway, NJ, USA, 54–66. doi:10.1109/RTAS58335.2023.00012

    arXiv 2023

  6. [6]

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food- 101: Mining Discriminative Components with Random Forests. In Proceedings of the European Conference on Computer Vision Workshops. Springer, 446–461

    2014

  7. [7]

    Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2025. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers.IEEE Transac- tions on Audio, Speech and Language Processing33 (2025), 705–718. doi:10.1109/TASLPRO.2025.3530270

    arXiv 2025

  8. [8]

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scal- ing. arXiv:2501.17811 [cs.AI]https://arxiv.org/abs/2501.17811

    Pith/arXiv arXiv 2025

  9. [9]

    Yukang Chen, Weihao Cui, Han Zhao, Ziyi Xu, Xiaoze Fan, Xusheng Chen, Yangjie Zhou, Shixuan Sun, Bingsheng He, and Quan Chen. 2026. Towards High-Goodput LLM Serving with Prefill-decode Multiplexing. InProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 2030–2047

    2026

  10. [10]

    Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. 2022. Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Shar- ing. In2022 USENIX Annual Technical Conference (USENIX ATC 22). USENIX Association, Carlsbad, CA, 199–216.https://www.usenix.org/ conference/atc22/presentation/choi-seungbeom

    work page 2022

  11. [11]

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, and Chunhua Shen. 2023. MobileVLM: A Fast, Strong and Open Vision Language Assistant for Mobile Devices. arXiv:2312.16886 [cs.CV] https://arxiv.org/abs/2312.16886

    Pith/arXiv arXiv 2023

  12. [12]

    Franklin, Joseph E

    Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A Low-Latency Online Prediction Serving System. In14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX As- sociation, Boston, MA, 613–627.https://www.usenix.org/conference/ nsdi17/technical-sessions/presentatio...

    work page 2017

  13. [13]

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi

  14. [14]

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. 2025. Emerging Properties in Unified Multimodal Pretraining.arXiv preprint arXiv:2505.14683(2025)

    Pith/arXiv arXiv 2025

  15. [15]

    Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. 2024. MuxServe: flexible spatial-temporal multiplexing for multiple LLM serving. InProceed- ings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 473, 13 pages

    work page 2024

  16. [16]

    Chaoyou Fu, Haojia Lin, Xiong Wang, YiFan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, Long MA, Xiawu Zheng, Rongrong Ji, Xing Sun, Caifeng Shan, and Ran He. 2026. VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Inter- action. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems.https://ope...

    work page 2026

  17. [17]

    Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: Low- Latency Serverless Inference for Large Language Models. In18th 13 USENIX Symposium on Operating Systems Design and Implementa- tion (OSDI 24). USENIX Association, Santa Clara, CA, 135–153.https: //www.usenix.org/conference/osdi24...

    work page 2024

  18. [18]

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kauf- mann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. In 14th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 20). USENIX Association, Virtual Event, 443–462. https://www.usenix.org/conference/osdi...

    work page 2020

  19. [19]

    Tianyu Guo, Tianming Xu, Xianjie Chen, Junru Chen, Nong Xiao, and Xianwei Zhang. 2025. Rserve: Overlapping encoding and prefill for efficient lmm inference.arXiv preprint arXiv:2509.24381(2025)

    work page arXiv 2025

  20. [20]

    Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. 2022. Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 539– 558.https://www.usenix.org/conference/osdi22/presentation/han

    2022

  21. [21]

    Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Es- peholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. InProceedings of the 29th Inter- national Conference on Neural Information Processing Systems - Vol- ume 1(Montreal, Canada)(NIPS’15). MIT Press, Cambridge, MA, USA, 1693–1701

    work page 2015

  22. [22]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffu- sion Probabilistic Models. InAdvances in Neural Information Pro- cessing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 6840– 6851.https://proceedings.neurips.cc/paper_files/paper/2020/file/ 4c5bcfec8584af0d967f1ab10179ca4...

    work page 2020

  23. [23]

    Yibo Jin, Tao Wang, Huimin Lin, Mingyang Song, Peiyang Li, Yipeng Ma, Yicheng Shan, Zhengfan Yuan, Cailong Li, Yajing Sun, Tiandeng Wu, Xing Chu, Ruizhi Huan, Li Ma, Xiao You, Wenting Zhou, Yun- peng Ye, Wen Liu, Xiangkun Xu, Yongsheng Zhang, Tiantian Dong, Jiawei Zhu, Zhe Wang, Xijian Ju, Jianxun Song, Haoliang Cheng, Xi- aojing Li, Jiandong Ding, Hefei ...

    Pith/arXiv arXiv 2024

  24. [24]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: generative adversarial networks for efficient and high fidelity speech synthesis. InProceedings of the 34th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’20). Curran Associates Inc., Red Hook, NY, USA, Article 1428, 12 pages

    work page 2020

  25. [25]

    Gonzalez, Hao Zhang, and Ion Sto- ica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Sto- ica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles. Association for Computing Machinery, New York, NY, USA, 611–626. doi:10...

    arXiv 2023

  26. [26]

    Zikun Li, Zhuofu Chen, Remi Delacourt, Gabriele Oliaro, Zeyu Wang, Qinghan Chen, Shuhuai Lin, April Yang, Zhihao Zhang, Zhuom- ing Chen, Yi-Hsiang Lai, Xinhao Cheng, Xupeng Miao, and Zhi- hao Jia. 2026. AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding. InProceedings of the 21st European Conference on Computer Systems(M...

    work page arXiv 2026

  27. [27]

    Jianxiong Liao, Quanxing Dong, Yunkai Liang, Zhi Zhou, and Xu Chen

    work page

  28. [28]

    Zejia Lin, Hongxin Xu, Guanyi Chen, Zhiguang Chen, Yutong Lu, and Xianwei Zhang. 2026. Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration. InProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2(USA)(ASP- LOS ’26). Association for Com...

    arXiv 2026

  29. [29]

    2025.Choose Between Cloud-Based and Local AI Models

    Microsoft. 2025.Choose Between Cloud-Based and Local AI Models. https://learn.microsoft.com/en-us/windows/ai/cloud-ai

    work page 2025

  30. [30]

    NVIDIA. 2022. What Is Edge AI and How Does It Work?https: //blogs.nvidia.com/blog/what-is-edge-ai/

    work page 2022

  31. [31]

    2026.CUDA C++ Programming Guide.https://docs.nvidia

    NVIDIA. 2026.CUDA C++ Programming Guide.https://docs.nvidia. com/cuda/cuda-c-programming-guide/

    work page 2026

  32. [32]

    2026.CUDA Driver API: Green Contexts.https: //docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__GREEN_ _CONTEXTS.html

    NVIDIA. 2026.CUDA Driver API: Green Contexts.https: //docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__GREEN_ _CONTEXTS.html

    work page 2026

  33. [33]

    2026.Multi-Process Service User Guide.https://docs.nvidia

    NVIDIA. 2026.Multi-Process Service User Guide.https://docs.nvidia. com/deploy/mps/

    work page 2026

  34. [34]

    2026.NVIDIA Multi-Instance GPU User Guide.https://docs

    NVIDIA. 2026.NVIDIA Multi-Instance GPU User Guide.https://docs. nvidia.com/datacenter/tesla/mig-user-guide/

    work page 2026

  35. [35]

    2026.NVIDIA RTX 6000 Ada Generation Graphics Card.https: //www.nvidia.com/en-us/products/workstations/rtx-6000/

    NVIDIA. 2026.NVIDIA RTX 6000 Ada Generation Graphics Card.https: //www.nvidia.com/en-us/products/workstations/rtx-6000/

    work page 2026

  36. [36]

    2026.NVIDIA RTX PRO 6000 Blackwell Workstation Edition.https://www.nvidia.com/en-us/products/workstations/ professional-desktop-gpus/rtx-pro-6000/

    NVIDIA. 2026.NVIDIA RTX PRO 6000 Blackwell Workstation Edition.https://www.nvidia.com/en-us/products/workstations/ professional-desktop-gpus/rtx-pro-6000/

    work page 2026

  37. [37]

    2026.NVIDIA RTX PRO AI Workstation Solutions.https: //www.nvidia.com/en-us/products/workstations/ai-workstations/

    NVIDIA. 2026.NVIDIA RTX PRO AI Workstation Solutions.https: //www.nvidia.com/en-us/products/workstations/ai-workstations/

    work page 2026

  38. [38]

    OpenAI. 2024. Hello GPT-4o.https://openai.com/index/hello-gpt-4o/

    work page 2024

  39. [39]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur

    work page

  40. [40]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2025. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. InProceedings of the 51st Annual International Symposium on Computer Architecture (Buenos Aires, Argentina)(ISCA ’24). IEEE Press, 118–132. doi:10. 1109/ISCA59077.2024.00019

    arXiv 2025

  41. [41]

    William Peebles and Saining Xie. 2023. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision. 4195–4205

    work page 2023

  42. [42]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Moon- cake: Trading More Storage for Less Computation — A KVCache- centric Architecture for Serving LLM Chatbot. In23rd USENIX Confer- ence on File and Storage Technologies (FAST 25). USENIX Association, Santa Clara, CA, 155–170.https://w...

    2025

  43. [43]

    Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, Íñigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, Ram Ramjee, and Rodrigo Fonseca. 2026. ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving. InProceedings of the 2025 ACM Symposium on Cloud Comput- ing (SoCC ’25). Associatio...

    work page arXiv 2026

  44. [44]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machin...

    work page 2021

  45. [45]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via 14 large-scale weak supervision. InInternational conference on machine learning. PMLR, 28492–28518

    work page 2023

  46. [46]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al

  47. [47]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10684–10695

    2022

  48. [48]

    Yadwadkar, and Christos Kozyrakis

    Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. INFaaS: Automated Model-less Inference Serv- ing. In2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, Virtual Event, 397–411.https://www.usenix. org/conference/atc21/presentation/romero

    work page 2021

  49. [49]

    Ruan, Yucheng Qin, Xun Zhou, Ruihang Lai, Hongyi Jin, Yixin Dong, Bohan Hou, Meng-Shiun Yu, Yiyan Zhai, Sudeep Agarwal, Hangrui Cao, Siyuan Feng, and Tianqi Chen

    Charlie F. Ruan, Yucheng Qin, Xun Zhou, Ruihang Lai, Hongyi Jin, Yixin Dong, Bohan Hou, Meng-Shiun Yu, Yiyan Zhai, Sudeep Agarwal, Hangrui Cao, Siyuan Feng, and Tianqi Chen. 2024. We- bLLM: A High-Performance In-Browser LLM Inference Engine. arXiv:2412.15803 [cs.LG]https://arxiv.org/abs/2412.15803

    Pith/arXiv arXiv 2024

  50. [50]

    Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019. Nexus: A GPU Cluster Engine for Accelerating DNN-Based Video Analysis. InProceedings of the 27th ACM Symposium on Operating Systems Principles. Association for Computing Machinery, New York, NY, USA, 322–337. doi:10.1145/33413...

    arXiv 2019

  51. [51]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang

  52. [52]

    Sudipta Saha Shubha, Haiying Shen, and Anand Iyer. 2024. USHER: Holistic Interference Avoidance for Resource Optimized ML Inference. In18th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 24). USENIX Association, Santa Clara, CA, 947–964. https://www.usenix.org/conference/osdi24/presentation/shubha

    work page 2024

  53. [54]

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. UCF101: A Dataset of 101 Human Actions Classes From Videos in the Wild. arXiv:1212.0402 [cs.CV]

    Pith/arXiv arXiv 2012

  54. [55]

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic Scheduling for Large Language Model Serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 173–191.https://www.usenix.org/conference/osdi24/ presentation/sun-biao

    2024

  55. [56]

    InProceedings of the 40th International Conference on Machine Learning(Honolulu, Hawaii, USA)(ICML’23)

    FlexGen: high-throughput generative inference of large language models with a single GPU. InProceedings of the 40th International Conference on Machine Learning(Honolulu, Hawaii, USA)(ICML’23). JMLR.org, Article 1288, 23 pages

    work page

  56. [57]

    Meituan LongCat Team, Bairui Wang, Bayan, Bin Xiao, Bo Zhang, Bolin Rong, Borun Chen, Chang Wan, Chao Zhang, Chen Huang, Chen Chen, Chen Chen, Chengxu Yang, Chengzuo Yang, Cong Han, Dandan Peng, Delian Ruan, Detai Xin, Disong Wang, Dongchao Yang, Fanfan Liu, Fengjiao Chen, Fengyu Yang, Gan Dong, Gang Huang, Gang Xu, Guanglu Wan, Guoqiang Tan, Guoqiao Yu, ...

    arXiv 2025

  57. [58]

    Wenwen Tong, Hewei Guo, Dongchuan Ran, Jiangnan Chen, Jiefan Lu, Kaibin Wang, Keqiang Li, Xiaoxu Zhu, Jiakui Li, Kehan Li, Xueheng Li, Lumin Li, Chenxu Guo, Jiasheng Zhou, Jiandong Chen, Xianye Wu, Jiahao Wang, Silei Wu, Lei Chen, Hanming Deng, Yuxuan Song, Dinghao Zhou, Guiping Zhong, Ken Zheng, Shiyin Kang, and Lewei Lu

    work page

  58. [59]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. InAdvances in Neural Information Pro- cessing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran As- sociates, Inc.https://p...

    work page 2017

  59. [60]

    Haoxin Wang, Xiaolong Tu, Hongyu Ke, Huirong Chai, Dawei Chen, and Kyungtae Han. 2025. lm-Meter: Unveiling Runtime Inference La- tency for On-Device Language Models. InProceedings of the Tenth ACM/IEEE Symposium on Edge Computing(the Hilton Arlington National Landing, Arlington, VA, USA)(SEC ’25). Association for Computing Machinery, New York, NY, USA, Ar...

    work page arXiv 2025

  60. [61]

    Core Team, Dong Zhang, Gang Wang, Jinlong Xue, Kai Fang, Liang Zhao, Rui Ma, Shuhuai Ren, Shuo Liu, Tao Guo, Weiji Zhuang, Xin Zhang, Xingchen Song, Yihan Yan, Yongzhe He, Cici, Bowen Shen, Chengxuan Zhu, Chong Ma, Chun Chen, Heyu Chen, Jiawei Li, Lei Li, Menghang Zhu, Peidian Li, Qiying Wang, Sirui Deng, Weimin Xiong, Wenshan Huang, Wenyu Yang, Yilin Jia...

    arXiv 2025

  61. [62]

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. 2024. Janus: Decoupling Visual Encoding for Unified 15 Multimodal Understanding and Generation. arXiv:2410.13848 [cs.CV] https://arxiv.org/abs/2410.13848

    Pith/arXiv arXiv 2024

  62. [63]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-Omni Technical Report. arXiv:2503.20215 [cs.CL]https://arxiv.org/abs/2503.20215

    Pith/arXiv arXiv 2025

  63. [64]

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo ...

    Pith/arXiv arXiv 2025

  64. [65]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al . 2024. MiniCPM-V: A GPT-4V Level MLLM on Your Phone.arXiv preprint arXiv:2408.01800(2024)

    Pith/arXiv arXiv 2024

  65. [66]

    Fanjiang Ye, Zhangke Li, Xinrui Zhong, Ethan Ma, Russell Chen, Kai- jian Wang, Jingwei Zuo, Desen Sun, Ye Cao, Triston Cao, Myungjin Lee, Arvind Krishnamurthy, and Yuke Wang. 2026. GENSERVE: Efficient Co-Serving of Heterogeneous Diffusion Model Workloads. arXiv:2604.04335 [cs.DC]https://arxiv.org/abs/2604.04335

    Pith/arXiv arXiv 2026

  66. [67]

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism. InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (Austin, TX, USA)(SOSP ’24). Association for Computing Machinery, New York, NY, USA, 640–654. ...

    arXiv 2024

  67. [68]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Asso- ciation, Carlsbad, CA, 521–538.https://www.usenix.org/conference/ osdi22/presentation/yu

    2022

  68. [69]

    Peifeng Yu and Mosharaf Chowdhury. 2020. Salus: Fine- Grained GPU Sharing Primitives for Deep Learning Applications. arXiv:1902.04610 [cs.DC]https://arxiv.org/abs/1902.04610

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  69. [70]

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. Soundstream: An end-to-end neural au- dio codec.IEEE/ACM Transactions on Audio, Speech, and Language Processing30 (2021), 495–507

    work page 2021

  70. [71]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer

    work page

  71. [72]

    Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. 2019. MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving. In2019 USENIX Annual Technical Con- ference (USENIX ATC 19). USENIX Association, Renton, WA, 1049– 1062.https://www.usenix.org/conference/atc19/presentation/zhang- chengliang

    work page 2019

  72. [73]

    Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, and Hongsheng Liu. 2026. vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models. arXiv:2602.02204 [cs.DC]https: //arxiv.org/abs/2602.02204

    arXiv 2026

  73. [74]

    Yuning Zhang, Nan Yang, Chen Pan, and Dong Yuan. 2026. Joint Optimization of Resource Allocation and Request Batching for Multi- Tenant Inference Serving on GPU.IEEE Transactions on Parallel and Dis- tributed Systems37, 1 (2026), 287–303. doi:10.1109/TPDS.2025.3627574

    work page arXiv 2026

  74. [75]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: efficient execution of structured language model programs. InPro- ceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC,...

    2024

  75. [76]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 193– 210.https://www.usenix.org/co...

    2024

  76. [77]

    Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2025. NanoFlow: Towards Optimal Large Language Model Serving Through- put. In19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). USENIX Ass...

    work page 2025

  77. [78]

    InProceedings of the IEEE/CVF International Conference on Computer Vision

    Sigmoid Loss for Language Image Pre-Training. InProceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, Paris, France, 11975–11986.https://openaccess.thecvf.com/content/ ICCV2023/html/Zhai_Sigmoid_Loss_for_Language_Image_Pre- Training_ICCV_2023_paper.html

    work page

  78. [80]

    Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. SHEPHERD: Serving DNNs in the Wild. In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX As- sociation, Boston, MA, 787–808.https://www.usenix.org/conference/ nsdi23/presentation/zhang-hong

    work page 2023

  79. [2015]

    InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing

    LibriSpeech: An ASR Corpus Based on Public Domain Audio Books. InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 5206–5210

    work page

  80. [2018]

    Improving language understanding by generative pre-training. (2018)

    2018

Showing first 80 references.