Pith. sign in

REVIEW 1 minor 45 references

LUMEN: Coordinated Failure Recovery for Distributed LLM Serving

T0 review · 0 major / 1 minor · reviewed 2026-06-26 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LUMEN improves recovery and serving times in distributed LLM clusters by coordinating load-aware decisions at three points in the failure process.

desk verdict LUMEN shows coordinated load-aware decisions on checkpoints, request rerouting, and reload capacity can cut recovery and serving times in LLM clusters, with prototype and simulation results supporting the gains. read the letter →

arxiv 2606.17787 v1 pith:P45DUGOQ submitted 2026-06-16 cs.DC

Zhang Cao , Shujie Han , Juncheng Zhang , Yuanming Ren , Yongkun Li , Patrick P. C. Lee This is my paper · ORCID
classification cs.DC
keywords LLMservingfaulttolerancedistributedsystemsfailurerecoverycheckpointingloadbalancingKVcache
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In large LLM serving clusters, worker failures cause loss of GPU-resident KV caches and serving capacity, forcing surviving workers to restart requests from scratch while handling extra traffic. LUMEN treats recovery as a load-aware coordination problem by deciding checkpoint placement before failures, how to distribute interrupted requests when a failure occurs, and how to restore serving capacity while the model reloads. The system shows significant improvements in serving and recovery times through experiments and simulations. If correct, this means clusters can maintain higher performance and lower latency for users even when failures happen frequently at scale.

What carries the argument

Load-aware coordination across the three decision points of checkpoint placement, interrupted-request distribution, and serving capacity restoration during reload.

What would settle it

A controlled experiment on a multi-GPU cluster that injects worker failures, measures end-to-end serving latency and recovery duration with and without LUMEN, and checks whether the coordinated version yields shorter times or higher throughput; absence of improvement or net slowdown from coordination overhead would falsify the claim.

Watch

Extended reading notes

Core claim

LUMEN is a fault-tolerant LLM serving system that treats recovery as a load-aware coordination problem across three decision points: checkpoint placement before failures, interrupted-request distribution at failure time, and serving capacity restoration during model reload. Prototype experiments and large-scale simulations demonstrate significant improvements in serving and recovery times over existing approaches that restart requests or use fixed checkpoint locations without considering load.

Load-bearing premise

The overhead of making and executing the three coordinated decisions remains low enough that the net effect improves recovery and serving times.

Editorial extensions

If this is right

  • Interrupted requests get routed to workers according to current load instead of fixed neighboring workers, spreading the restart work.
  • Checkpoints are placed with awareness of expected load so that recovery effort does not concentrate on already busy nodes.
  • Serving capacity begins restoring while the model reloads rather than after, shortening the period the cluster runs at reduced capacity.
  • Overall post-failure throughput stays closer to normal levels because recovery actions are aligned with observed load.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same three-point coordination pattern could be applied to other stateful distributed services that lose both data and capacity on node failure.
  • Large-scale simulations indicate the method scales, yet production traces with bursty or correlated failures might expose new bottlenecks in decision timing.
  • Dynamic addition or removal of workers could be folded into the same load-aware logic to handle elastic serving environments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 1 minor

Summary. LUMEN is presented as a fault-tolerant LLM serving system that treats recovery as a load-aware coordination problem across three decision points: checkpoint placement before failures, interrupted-request distribution at failure time, and serving capacity restoration during model reload. The paper evaluates the system using prototype experiments and large-scale simulations, claiming significant improvements in serving and recovery times.

Significance. If the results hold, the work is significant because it addresses a practical challenge in scaling distributed LLM inference systems where failures are common. By coordinating recovery decisions based on load, it offers a more efficient alternative to existing methods that either restart requests or use fixed checkpoints. The dual evaluation approach with prototypes and simulations is a strength, providing both practical validation and scalability insights.

minor comments (1)
  1. [Abstract] Abstract: The claim of 'significant improvements in serving and recovery times' is stated without any quantitative metrics, error bars, baseline comparisons (e.g., to restart-from-scratch or fixed-neighbor checkpointing), or description of the coordination overhead. Adding one or two key results from the evaluation section would make the central claim more concrete and proportionate to the evidence presented later in the manuscript.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive summary and significance assessment of LUMEN, as well as the recommendation for minor revision. We appreciate the recognition that coordinated, load-aware recovery addresses a practical challenge in distributed LLM serving.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity identified

full rationale

The paper is a systems design contribution describing LUMEN's load-aware coordination for failure recovery across three decision points, evaluated via prototype experiments and large-scale simulations. No equations, derivations, fitted parameters, predictions, or self-citations appear in the provided text that reduce any claim to its own inputs by construction. The central premise is supported by external empirical measurements rather than internal self-definition or renaming of known results.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract supplies no information on free parameters, axioms, or invented entities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LUMEN: Coordinated Failure Recovery for Distributed LLM Serving." pith.science (2026). https://pith.science/paper/P45DUGOQ

@misc{pith2026260617787,
  author       = {Pith},
  title        = {Pith review of: LUMEN: Coordinated Failure Recovery for Distributed LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P45DUGOQ}},
  note         = {Machine review of arXiv:2606.17787}
}
read the original abstract

Modern large language model (LLM) serving clusters distribute inference requests across multiple worker processes on different GPUs, but failures are prevalent at scale. When a worker fails, the cluster simultaneously loses the failed worker's GPU-resident key-value (KV) caches and serving capacity, leaving surviving workers to absorb the redirected traffic while re-running interrupted requests from scratch. Existing fault-tolerant systems either restart interrupted requests from scratch or restore KV caches from checkpoints stored on a fixed neighboring worker, but both approaches route recovery work without considering current cluster load and leave the recovering worker idle during model reload. We present LUMEN, a fault-tolerant LLM serving system that treats recovery as a load-aware coordination problem across three decision points: checkpoint placement before failures, interrupted-request distribution at failure time, and serving capacity restoration during model reload. We evaluate LUMEN using both prototype experiments and large-scale simulations and demonstrate significant improvements in serving and recovery times.

Figures

Figures reproduced from arXiv: 2606.17787 by the authors.

Figure 1
Figure 1. Single-worker failure in a simulated four-worker cluster. 3.1 Impact of Worker Failures Observation 1: A single-worker failure degrades both TTFT and TPOT cluster-wide. We use our simulator to run Llama-3-70B [24] on a cluster of four workers driven by the Splitwise-Conv trace [26]. We evaluate Stop-and-Restart by failing a single worker after the cluster reaches steady state and compare it with the No-Failure case … view at source ↗
Figure 3
Figure 3. LUMEN overview. 4 LUMEN Design LUMEN addresses the challenges in §3.3 via three coordi￾nated mechanisms: load-aware KV checkpointing, locality￾aware recovery scheduling, and speculation-assisted progres￾sive recovery. All three mechanisms share a common princi￾ple that recovery decisions follow current load observations and direct extra work toward less-loaded workers. 4.1 Design Overview Architecture [PITH_FULL_IM… view at source ↗
Figure 4
Figure 4. Load-aware KV checkpointing. • Low steady-state overhead, which keeps the always-on checkpoint path off the GPU decode path and bounded in host memory so that failure-free TTFT and TPOT match those of a serving system without LUMEN. • Scalability, which preserves the recovery benefit across cluster sizes and across concurrent multi-worker failures. 4.2 Load-aware KV Checkpointing To address Challenge 1, LUMEN introd… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Speculation-assisted progressive recovery. matching KV pages into GPU memory. If the persisted prefix is incomplete relative to the current token history, LUMEN bypasses full recomputation by performing a partial prefill on the uncheckpointed suffix. The restored KV pa…
Figure 6
Figure 6. Figure 6: Experiment A.1: (Prototype) End-to-end recovery under a single-worker failure for four-worker (Qwen3-32B) and eight-worker (Qwen3-14B) deployments. We show the failure-impact window (in grey box) of Stop-and-Restart. Note that outside the failure-impact window (i.e., n…
Figure 7
Figure 7. Figure 7: Experiment A.2 (Prototype): Recovery-path breakdown. Stop-and-Restart Fixed-Checkpointing LUMEN 8 9 10 11 12 Request rate (QPS) 0.0 7.5 15.0 22.5 30.0 37.5 Mean TTFT (s) 8 9 10 11 12 Request rate (QPS) 0 25 50 75 100 125 Mean TPOT (ms) (a) Mean TTFT (b) Mean TPOT [PIT…
Figure 8
Figure 8. Figure 8: Experiment A.3 (Prototype): Impact of request rate. Stop-and-Restart Fixed-Checkpointing LUMEN 1 2 4 Failed workers 0.0 2.5 5.0 7.5 10.0 12.5 15.0 Mean TTFT (s) 1 2 4 Failed workers 0 20 40 60 80 100 120 Mean TPOT (ms) (a) Mean TTFT (b) Mean TPOT [PITH_FULL_IMAGE:figu…
Figure 9
Figure 9. Figure 9: Experiment A.4 (Prototype): Impact of number of failures. remains the lowest through the speculation path as queues grow, reducing mean TPOT by 10.9% at 8 QPS and by 7.1% at the default, while Stop-and-Restart’s TPOT increases sharply. Experiment A.4 (Impact of number …
Figure 10
Figure 10. Figure 10: Experiment B.2 (Simulator): Impact of request rate. S&R F-Ckpt +Scheduling +Progressive LUMEN 1 2 3 4 5 Failed workers 0.0 2.5 5.0 7.5 10.0 12.5 15.0 Mean TTFT (s) 1 2 3 4 5 Failed workers 0 50 100 150 200 250 300 Mean TPOT (ms) (a) Mean TTFT (b) Mean TPOT S&R F-Ckpt …
Figure 12
Figure 12. Figure 12: scales the cluster from 4 to 64 workers with a fixed per-worker load of 1.4 QPS and a fixed 25% failure rate (i.e., 1–16 simultaneous failures). LUMEN’s gains are stable across all scales. Compared to Stop-and-Restart, mean TTFT stays 46.8–51.2% lower at 4–64 workers …
Figure 11
Figure 11. Figure 11: Experiment B.3 (Simulator): Impact of number of failed workers on TTFT, TPOT, and recovery time. lower, and recovery time is 46.9% and 33.3% shorter, respec￾tively. With a single failure, the schemes remain close as inter￾rupted requests account for a small fraction o…
Figure 13
Figure 13. Figure 13: Experiment B.5 (Simulator): Scalability with a single failure [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 5 canonical work pages

  1. [1]

    https:// docs.vllm.ai/en/stable/deployment/k8s/, 2026

    Deploying vLLM on Kubernetes. https:// docs.vllm.ai/en/stable/deployment/k8s/, 2026

    2026

  2. [2]

    https: //github.com/flashinfer-ai/flashinfer, 2026

    FlashInfer: Kernel library for LLM serving. https: //github.com/flashinfer-ai/flashinfer, 2026

    2026

  3. [3]

    gRPC: A high-performance, open-source universal RPC framework.https://grpc.io, 2026

    2026

  4. [4]

    https: //huggingface.co/docs/text-generation- inference, 2026

    Hugging Face text generation inference. https: //huggingface.co/docs/text-generation- inference, 2026

    2026

  5. [5]

    https://github.com/kserve/kserve, 2026

    KServe: Standardized model inference platform on Kubernetes. https://github.com/kserve/kserve, 2026

    2026

  6. [6]

    https: //github.com/triton-inference-server/server, 2026

    NVIDIA Triton inference server. https: //github.com/triton-inference-server/server, 2026

    2026

  7. [7]

    https://pytorch.org, 2026

    PyTorch: An open-source machine learning framework. https://pytorch.org, 2026

    2026

  8. [8]

    https://zeromq.org, 2026

    ZeroMQ: An open-source universal messaging library. https://zeromq.org, 2026

    2026

Show all 45 references
  1. [9]

    Vidur: A large-scale simulation framework for LLM inference

    Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav S Gulavani, Ramachan- dran Ramjee, and Alexey Tumanov. Vidur: A large-scale simulation framework for LLM inference. InConfer- ence on Machine Learning and Systems (MLSys), 2024

    2024

  2. [10]

    Taming throughput- latency tradeoff in LLM inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tu- manov, and Ramachandran Ramjee. Taming throughput- latency tradeoff in LLM inference with Sarathi-Serve. InUSENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024

    2024

  3. [11]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff...

    2020

  4. [12]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. https://arxiv.org/abs/ 2302.01318, 2023

    2023 arXiv

  5. [13]

    Recycle: Resilient training of large DNNs using pipeline adaptation

    Swapnil Gandhi, Mark Zhao, Athinagoras Skiadopoulos, and Christos Kozyrakis. Recycle: Resilient training of large DNNs using pipeline adaptation. InACM Sympo- sium on Operating Systems Principles (SOSP), 2024

    2024

  6. [14]

    Cost-efficient large language model serving for multi-turn conversations with CachedAtten- tion

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Y ang, Zhou Y u, and Pengfei Zuo. Cost-efficient large language model serving for multi-turn conversations with CachedAtten- tion. InUSENIX Annual Technical Conference (ATC), 2024

    2024

  7. [15]

    Characterization of large language model devel- opment in the datacenter

    Qinghao Hu, Zhisheng Y e, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xi- aolin Wang, Yingwei Luo, Y onggang Wen, and Tianwei Liu. Characterization of large language model devel- opment in the datacenter. InUSENIX Symposium on Networked Systems De...

    2024

  8. [16]

    Oobleck: Resilient distributed training of large models using pipeline templates

    Insu Jang, Zhenning Y ang, Zhen Zhang, Xin Jin, and Mosharaf Chowdhury. Oobleck: Resilient distributed training of large models using pipeline templates. In ACM Symposium on Operating Systems Principles (SOSP), 2023. 13

    2023

  9. [17]

    GhostServe: A lightweight checkpointing system in the shadow for fault-tolerant LLM serving

    Shakya Jayakody, Y oupeng Zhao, Chinmay Nehate, and Jun Wang. GhostServe: A lightweight checkpointing system in the shadow for fault-tolerant LLM serving. In Conference on Machine Learning and Systems (MLSys), 2026

    2026

  10. [18]

    MegaScale: Scaling large language model training to more than 10,000 GPUs

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Y angrui Chen, Zhi Zhang, Y anghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Y ulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Y an, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, ...

    2024

  11. [19]

    Revisiting reliability in large-scale machine learning research clusters

    Apostolos Kokolis, Michael Kuchnik, John Hoffman, Adithya Kumar, Parth Malani, Faye Ma, Zachary DeVito, Shubho Sengupta, Kalyan Saladi, and Carole-Jean Wu. Revisiting reliability in large-scale machine learning research clusters. InIEEE International Symposium on High-Performa...

    2025

  12. [20]

    Efficient memory man- agement for large language model serving with Page- dAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Y u, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with Page- dAttention. InACM Symposium on Operating Systems Principles (SOSP), 2023

    2023

  13. [21]

    Fast inference from transformers via speculative decoding

    Y aniv Leviathan, Matan Kalman, and Y ossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning (ICML), 2023

    2023

  14. [22]

    PEARL: Parallel speculative decoding with adaptive draft length

    Tianyu Liu, Y un Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. PEARL: Parallel speculative decoding with adaptive draft length. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  15. [23]

    CacheGen: KV cache compression and streaming for fast large language model serving

    Y uhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Y uyang Huang, Qizheng Zhang, Kuntai Liu, Shan Lu He, Ganesh Ananthanarayanan, and Junchen Jiang. CacheGen: KV cache compression and streaming for fast large language model serving. InACM SIGCOMM Conference (SIGCOMM), 2024

    2024

  16. [24]

    The Llama 3 herd of models

    Llama Team. The Llama 3 herd of models. https: //arxiv.org/abs/2407.21783, 2024

    2024 arXiv

  17. [25]

    AMUSD: Asynchronous multi- device speculative decoding for LLM acceleration

    Bradley McDanel. AMUSD: Asynchronous multi- device speculative decoding for LLM acceleration. In IEEE International Symposium on Circuits and Systems (ISCAS), 2025

    2025

  18. [26]

    Splitwise: Efficient generative LLM inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Inigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bian- chini. Splitwise: Efficient generative LLM inference using phase splitting. InIEEE International Symposium on Computer Architecture (ISCA), 2024

    2024

  19. [27]

    ECCheck: Enhancing in-memory check- point with erasure coding in distributed DNN training

    Guicheng Qi, Zongpeng Li, Chuan Wu, Zhuwei Peng, and Yi Zheng. ECCheck: Enhancing in-memory check- point with erasure coding in distributed DNN training. InIEEE International Conference on Distributed Com- puting Systems (ICDCS), 2025

    2025

  20. [28]

    Towards resiliency in large language model serving with KevlarFlow

    Shangshu Qian, Kipling Liu, PC Sruthi, Lin Tan, and Y ongle Zhang. Towards resiliency in large language model serving with KevlarFlow. https://arxiv.org/ abs/2601.22438, 2026

    2026

  21. [29]

    Mooncake: Trading more storage for less computation – a KVCache-centric architecture for serving LLM chat- bot

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Wang, Jiaao Zeng, Y ongchao Zhang, Jian Shi, Chengruidong Bian, Lian Y ang, Fan Lv, Y ang Luo, and Zhongzhi Y ang. Mooncake: Trading more storage for less computation – a KVCache-centric architecture for serving LLM chat- bot. InUSENI...

    2025

  22. [30]

    ShareGPT conversation dataset

    ShareGPT. ShareGPT conversation dataset. https: //sharegpt.com/

  23. [31]

    FlexGen: High- throughput generative inference of large language mod- els with a single GPU

    Ying Sheng, Lianmin Zheng, Binhang Y uan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christo- pher Ré, Ion Stoica, and Ce Zhang. FlexGen: High- throughput generative inference of large language mod- els with a single GPU. InInternational Conference on Machine Learning ...

    2023

  24. [32]

    DéjàVu: KV-cache streaming for fast, fault-tolerant generative LLM serv- ing

    Foteini Strati, Sara McAllister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. DéjàVu: KV-cache streaming for fast, fault-tolerant generative LLM serv- ing. InInternational Conference on Machine Learning (ICML), 2024

    2024

  25. [33]

    Llumnix: Dy- namic scheduling for large language model serving

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Y ong Li, and Wei Lin. Llumnix: Dy- namic scheduling for large language model serving. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024

    2024

  26. [34]

    Bamboo: Making preemptible in- stances resilient for affordable training of large DNNs

    John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yi- fan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. Bamboo: Making preemptible in- stances resilient for affordable training of large DNNs. InUSENIX Symposium on Networked Systems Design and Implementatio...

    2023

  27. [35]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Y asmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, 14 Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernan- des, Je...

    2023 arXiv

  28. [36]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish V aswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  29. [37]

    ByteCheckpoint: A unified checkpointing system for large foundation model development

    Borui Wan, Mingji Han, Yiyao Sheng, Y anghua Peng, Haibin Lin, Mofan Zhang, Zhichao Lai, Menghan Y u, Junda Zhang, Zuquan Song, Xin Liu, and Chuan Wu. ByteCheckpoint: A unified checkpointing system for large foundation model development. InUSENIX Sym- posium on Networked Syste...

    2025

  30. [38]

    Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xin- wei Fu, T. S. Eugene Ng, and Yida Wang. Gemini: Fast failure recovery in distributed training with in-memory checkpoints. InACM Symposium on Operating Systems Principles (SOSP), 2023

    2023

  31. [39]

    Fast distributed inference serving for large language models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Y uanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. InUSENIX Symposium on Networked Systems Design and Implementation (NSDI), 2026

    2026

  32. [40]

    FailSafe: High-performance resilient serv- ing

    Ziyi Xu, Zhiqiang Xie, Swapnil Gandhi, and Christos Kozyrakis. FailSafe: High-performance resilient serv- ing. InConference on Machine Learning and Systems (MLSys), 2026

    2026

  33. [41]

    Qwen3 technical report

    An Y ang, Anfeng Li, Baosong Y ang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Y u, Chang Gao, Chen- gen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Y ang, Jianhong Tu, Jian- wei Zhang, Jianxin Y...

    2025 arXiv

  34. [42]

    Orca: A distributed serving system for transformer-based generative models

    Gyeong-In Y u, Joo Seong Jeong, Geon-Woo Kim, Soo- jeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. InUSENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022

    2022

  35. [43]

    Gonzalez, Clark Bar- rett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Y u, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Bar- rett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. InAdvances in Neural Informatio...

    2024

  36. [44]

    Dist- Serve: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- Serve: Disaggregating prefill and decoding for goodput- optimized large language model serving. InUSENIX Symposium on Operating Systems Design and Imple- mentation (OSDI), 2024

    2024

  37. [45]

    Resiliency at scale: Managing Google’s TPUv4 machine learning supercomputer

    Y azhou Zu, Alireza Ghaffarkhah, Hoang-Vu Dang, Brian Towles, Steven Hand, Safeen Huda, Adekunle Bello, Alexander Kolbasov, Arash Rezaei, Dayou Du, Steve Lacy, Hang Wang, Aaron Wisner, Chris Lewis, and Henri Bahini. Resiliency at scale: Managing Google’s TPUv4 machine learning...

    2024

Pith tools

Reviewed June 26, 2026 · model on record in the stance chip above.