Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Bounding each session's resident KV cache turns a silent serving cliff into a trustworthy latency slope for real-time interaction models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 08:08 UTC pith:KOPZRQVT

load-bearing objection Clean systems diagnosis of a silent metastable KV cliff under periodic residency, fixed by a minimal in-engine bound that also restores a usable admission signal. the 3 major comments →

arxiv 2607.02640 v1 pith:KOPZRQVT submitted 2026-07-02 cs.SD cs.AIcs.NI

Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving

Jiaying Meng , Bojie Li This is my paper
classification cs.SD cs.AIcs.NI
keywords real-time interaction modelsKV cache boundingsliding window attentionattention sinksadmission controlLLM servingperiodic real-time tasksfull-duplex audio
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Real-time interaction models turn serving into a periodic deadline task: every frame a session takes new audio and must answer by a wall-clock deadline, while its key-value cache grows and stays pinned for the whole conversation. Under sustained load the existing path does not slow down gracefully; it falls off a cliff when the shared block pool saturates, and the stall is silent because empty frames still return on time. The paper shows that one change restores both stability and observability: cap each session's resident state to a sliding window plus a few pinned attention-sink tokens. That bound eliminates the collapse, makes per-frame latency a monotone signal of load, and lets a simple admission controller discover the concurrency that still meets the deadline. Without the bound the same controller over-admits into the wall. A first-order fill model predicts when the cliff hits, and ablations confirm the window alone is quality-free in turn-based decode while the sinks keep free-running generation healthy.

Core claim

Real-time interaction serving is periodic serving with unbounded pinned per-session state, and that combination fails by a memory-triggered, metastable, silent latency cliff. Bounding each session's resident KV to a fixed window plus a few attention-sink tokens removes the cliff and restores a monotone latency signal; an online admission controller that acts on that signal then discovers the schedulable concurrency, while the identical controller over-admits into the wall without the bound.

What carries the argument

Metronome's in-engine sink-anchored KV window: each session retains only its last W tokens plus a few pinned early tokens, so per-frame work and resident memory stay bounded while free-running generation stays healthy.

Load-bearing premise

That the growing cache must stay fully resident every frame, because recompute or host-memory swap would cost more bandwidth and time than the recurring deadline can afford.

What would settle it

Run the same open-loop multi-session audio load with an in-engine window and with unmodified unbounded resident KV: if the windowed path still hits a hard stall at the same rate, or if the identical admission loop converges without the window, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that real-time interaction models (Moshi, MiniCPM-o, Qwen-Omni) create a new serving regime: periodic real-time tasks with unbounded, pinned per-session KV that grows every frame. On a real full-duplex stack, this produces a memory-triggered, metastable, silent latency cliff when the KV block pool saturates. Metronome bounds each session to a sliding window of W recent tokens plus a few pinned attention-sink tokens, eliminating the cliff (0/20 vs 14/20 collapses) and restoring a monotone per-frame latency signal on which an AIMD admission controller discovers the schedulable concurrency N⋆; without the bound the same controller over-admits into the wall. A first-order linear-fill model predicts stall time, and quality ablations show the window is free in turn-based decoding while sinks keep free-running generation healthy. Results are end-to-end on real audio across four models on one GPU.

Significance. If the result holds, this is a timely and practically important systems contribution. Interaction/full-duplex models are already shipping, and the paper shows that the currently exposed resumable-KV path fails in a way that standard latency and deadline-miss dashboards cannot see. The combination of end-to-end measurement on a real WebSocket/Go/GPU stack, two independent twenty-run batches, in-engine pool-occupancy traces, a falsifiable first-order fill model that predicts t_sat within a few percent on the 30B, and a controlled quality ablation of window vs sinks is stronger evidence than is typical for serving papers. The principle—bound resident state so that latency becomes a trustworthy control signal—generalizes beyond voice and is stated cleanly. Code and artifact are provided.

major comments (3)
  1. [§5.3 / Figure 7] §5.3 and Figure 7: the admission result is load-bearing for the claim that the bound restores observability, yet the unbounded-KV arm of the identical AIMD controller is only summarized in prose (“over-admits into the wall”) without a counterpart figure or table of admitted N, latency trajectory, and stall time. A single paired plot (or short table) would make the dependency fully checkable rather than asserted.
  2. [§5.4 / Table 2] §5.4 / Table 2: free-running quality (sink necessity) is measured only on Qwen3-Omni-30B at N=32. The paper already notes this limitation, but because the sink half is presented as part of the bound’s design, a second backbone (or an explicit statement that the serving-side claims do not depend on sinks) would strengthen the quality claim that currently rests on one model.
  3. [§7 / §5.5] §7 and §5.5: wall statistics and admission are reported for one engine (vLLM-realtime) and one GPU; MiniCPM-o wall is a single fresh run. The mechanism is well-diagnosed, but the headline 14/20 vs 0/20 comparison and the discovered N⋆≈209 should be framed more carefully as path- and hardware-specific until a second engine or GPU class is shown.
minor comments (5)
  1. [Figure 1 / Figure 4b] Figure 1 and Figure 4b use log-scale latency; a short note in the caption that the ~1.6 s wall is the worker’s 0.8 B return cap (not the 2 s budget) would prevent misreading.
  2. [§3.1 / Eq. (1)] Eq. (1) introduces r as “fraction of pool per second”; stating units and how r is fitted (early-trace slope) in one sentence would make the prediction claim fully self-contained.
  3. [§5.1 / Appendix B] Appendix B’s engine bug-fixes are valuable; a one-line pointer in §5.1 that the artifact includes them would help reproducibility readers.
  4. [§8] Related work could briefly contrast with existing streaming-attention / attention-sink papers (Xiao et al., Beltagy et al.) to clarify that the contribution is the serving-side residency bound under a frame deadline, not a new attention mechanism.
  5. [Abstract / §1] Typo/consistency: “itfalls” / “itfalls off a cliff” in the abstract; a few similar spacing issues appear in the introduction.

Circularity Check

0 steps flagged

No significant circularity: central claims are empirical measurements and ablations, not results forced by definition or self-citation.

full rationale

The paper’s load-bearing chain is systems-empirical, not a closed derivation. Collapse rates (14/20 vs 0/20), latency cliffs, pool-occupancy traces, and AIMD admission behavior are measured end-to-end on a real full-duplex stack under controlled ablations; none of these outcomes is defined by the quantities used to produce them. The first-order fill model ρ(t)=ρ₀+N r t fits r on the early occupancy trace and then checks predicted t_sat against the later measured stall (within a few percent on the 30B). That is ordinary temporal validation of a linear model, not a tautology that renames the fit as the result, and the headline stability/admission claims do not depend on it. Attention-sink anchoring is taken from independent prior work (Xiao et al., 2024) and re-validated by sink-ablated vs sink-restored free-running probes; the window half is likewise ablated in turn-based decoding. There is no self-citation uniqueness theorem, no ansatz smuggled from the authors’ own prior papers, and no equation that reduces by construction to its inputs. Classical AIMD and Liu–Layland framing are used as known tools, not renamed as novel first-principles results. Score 0 is therefore the correct finding.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper rests on classical real-time scheduling, the empirical attention-sink phenomenon, and the engineering claim that residency is forced by the absence of idle gaps. Free parameters are ordinary systems knobs (window size, sink count, AIMD target) chosen for quality or headroom rather than fitted to force the central claim. No new physical entities are postulated.

free parameters (4)
  • window size W = 1024–2048 tokens
    Chosen by quality sweep (W=1024 or 2048 tokens); latency is flat across a wide range, so the value is not load-bearing for the cliff-removal claim but is required for the quality half.
  • sink token count S = 16–32 tokens
    Chosen by ablation (S=16 best; S=32 also recovers); cost is 1–2 KV blocks per session. Not fitted to collapse data.
  • per-session fill rate r = model- and N-dependent fraction of pool per second
    Fitted on early unbounded pool-occupancy traces to predict tsat; used only for the predictability claim, not for the cliff-removal result.
  • AIMD latency target = 600 ms
    Hand-set fraction of frame budget (600 ms of 2000 ms) that drives the admission loop; ordinary control parameter.
axioms (4)
  • standard math A periodic real-time task is schedulable iff Tk(N) ≤ B for every frame k (Liu & Layland 1973).
    Used in Section 2 to define the interaction-session task model.
  • domain assumption Full-attention backbones concentrate softmax mass on the earliest positions (attention sinks); evicting them corrupts free-running decode (Xiao et al. 2024).
    Invoked in §4.1 and validated by ablation in §5.4; the paper does not re-derive the sink phenomenon.
  • domain assumption An interaction session has no idle gap, so recompute or host-swap of growing KV every frame exceeds the frame budget at operating concurrencies; residency is therefore required.
    Stated in Section 2 with a bandwidth-order argument; load-bearing for why the memory cliff is forced rather than optional.
  • domain assumption vLLM's resumable-request path and SGLang streaming sessions keep per-session KV resident across frames without bounding it.
    Empirical substrate of the baseline; documented in §2 and used throughout evaluation.

pith-pipeline@v1.1.0-grok45 · 22200 in / 3177 out tokens · 35092 ms · 2026-07-12T08:08:59.802232+00:00 · methodology

0 comments
read the original abstract

Real-time interaction models -- Moshi, MiniCPM-o, Qwen-Omni -- turn serving into a periodic real-time task: on every frame a session ingests streaming audio and must respond by a recurring wall-clock deadline, while its KV cache grows monotonically and stays pinned for the whole conversation. This regime hides a dangerous failure mode. On a real full-duplex stack, sustained load does not degrade serving gracefully: it falls off a cliff, jumping in one step from milliseconds per frame to a stalled engine when accumulated session state exhausts the KV pool. The collapse is metastable -- identical five-minute runs collapse or survive on run-to-run variance -- and silent: latency and deadline-miss metrics read healthy throughout. We show one move restores both stability and observability: bound each session's resident state, and latency starts telling the truth. Metronome's in-engine KV window eliminates the collapse (0/20 vs. 14/20 runs across two batches) and turns per-frame latency into a monotone load signal, on which an online admission controller discovers the schedulable concurrency; without the window, the identical controller over-admits into the wall. A first-order model predicts the collapse time within a few percent on the headline model, and a quality probe validates the bound's design by ablation: the window alone is quality-free in turn-based decoding, and its few pinned attention-sink tokens are what keep free-running generation healthy. Everything is measured end-to-end on real audio, across four interaction models on one GPU.

Figures

Figures reproduced from arXiv: 2607.02640 by Bojie Li, Jiaying Meng.

Figure 1
Figure 1. Figure 1: Existing serving falls off a latency cliff; Metronome stays on beat. One five-minute call at 128 sessions (Qwen3-Omni-30B): unbounded resident KV jumps in one step from a few ms to a ∼1.6 s ceiling where the stalled engine stops producing tokens; windowed KV stays flat for the whole call. 1 arXiv:2607.02640v1 [cs.SD] 2 Jul 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Metronome keeps the beat. A metronome tick fires once per frame budget B. On each tick the system serves one batch of all due sessions — prefill each session’s new audio chunk, then decode a few tokens (continuous batching) — and must finish within B; the timeline at the bottom shows one tick’s anatomy, with the slack as schedulability headroom. Two interdependent mechanisms keep the beat steady: (a) an ad… view at source ↗
Figure 3
Figure 3. Figure 3: The workload engines were not built for. (a) A chatbot or agent request is turn-based: bursts of prefill and decode separated by idle gaps. Its context accumulates, so between turns the engine parks the KV out of GPU memory — swapped to host, or dropped and recomputed — and pays a reactivation toll that grows with the conversation; the idle gaps hide the movement, and a slow reactivation delays only one re… view at source ↗
Figure 4
Figure 4. Figure 4: Bursts are fine; minutes are a gamble. (a) Fresh single-N runs over 90 s: per-frame latency is flat for both policies to N=160, hundreds of times under the 2 s budget — there is no short-burst capacity problem. (b) The same concurrencies over 300 s (log scale, 10 s bucket medians; both twenty-run batches pooled — ten fresh runs per policy at each N∈{96, 128}; thin lines individual runs, bold the median). W… view at source ↗
Figure 5
Figure 5. Figure 5: The trigger, caught in the act (N=128, 300 s, in-engine stat logger). Top: KV-pool occupancy. Unbounded resident KV climbs monotonically to pool capacity; in-engine windowed KV frees blocks behind the window and plateaus at roughly a quarter of the pool. Bottom: at the instant of saturation the scheduler can no longer allocate blocks and moves all N sessions to the waiting queue (shaded), a hard stall that… view at source ↗
Figure 6
Figure 6. Figure 6: The collapse obeys a first-order model. (a) Unbounded pool occupancy rises linearly (Eq. 1); a straight-line fit on the early trace (dotted) predicts the measured stall (×) within a few percent on Qwen3-Omni-30B (145 s predicted vs. 148 s) and within ∼13% on MiniCPM-o (99 s vs. 114 s). Windowed occupancy plateaus far below capacity. (b) The windowed plateau is linear in N (∼0.2% of pool per session, W=1024… view at source ↗
Figure 7
Figure 7. Figure 7: Online admission discovers the schedulable concurrency — on the bounded-KV signal (open-system ramp: 512 sessions offered at 8/s, 600 ms latency target, windowed worker). Top: admitted concurrency and the controller’s cap rise with arrivals and settle at N⋆≈209; the surplus is shed cleanly (cumulative count, same axis). Bottom: per-frame latency — the only signal the controller uses — probes the target onc… view at source ↗
Figure 8
Figure 8. Figure 8: The bound needs both halves: ablating the sinks fails free-running generation (continuous forced decoding, N=32, 300 s, Qwen3-Omni-30B; rotating spoken questions, one per 30 s segment). (a) Sessions answering the currently playing question, by session age: unbounded KV holds a steady low plateau (continuous greedy decoding is degenerate for all policies); every sink-ablated window declines toward zero as i… view at source ↗
Figure 9
Figure 9. Figure 9: Fresh single-N streaming capacity across four interaction models (90 s burst, one freshly started worker per point). A ≥ arrow means the run was still flat at the largest N tested, so the value is a lower bound. Short-burst capacity is set by architecture (audio-encoder weight, MoE sparsity, frame budget); the long-duration cliff of §3 is a property of the serving path and reproduces across models. 6 Discu… view at source ↗
Figure 10
Figure 10. Figure 10: In-engine windowing beats application-level recycling. Three policies with the same memory horizon at N=96 over 300 s (per-frame p50, 10 s buckets). Unbounded resident KV hits the wall at ∼240 s. The application-level recycling proxy avoids the wall but its periodic re-encode grows over the call (p50 to ∼14–17 ms, p90 to 36 ms). The in-engine window bounds the horizon without ever re-encoding. 512 (20 s) … view at source ↗
Figure 11
Figure 11. Figure 11: A generous window is free. Window-size ablation (in-engine windowed KV, N=128, 60 s runs): p50/p90 stay ∼5 ms up to a 2048-token (∼80 s) window. Turn-based quality detail. In the turn-based probe of Section 5.4 (N=96, 75 s sessions, EOS￾terminated turns), answer-stated counts are 96/96 sessions for both vanilla and windowed policies, with per-frame correctness ∼70% vs. ∼68% (statistically indistinguishabl… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 14 linked inside Pith

  1. [1]

    Gulavani, Alexey Tumanov, and Ramachandran Ramjee

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve . In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024

    2024

  2. [2]

    GQA : Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. GQA : Training generalized multi-query transformer models from multi-head checkpoints. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  3. [3]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    Pith/arXiv arXiv 2004

  4. [4]

    AudioLM : a language modeling approach to audio generation

    Zal \'a n Borsos, Rapha \"e l Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, et al. AudioLM : a language modeling approach to audio generation. arXiv preprint arXiv:2209.03143, 2022

    Pith/arXiv arXiv 2022

  5. [5]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    Pith/arXiv arXiv 1904

  6. [6]

    Analysis of the increase and decrease algorithms for congestion avoidance in computer networks

    Dah-Ming Chiu and Raj Jain. Analysis of the increase and decrease algorithms for congestion avoidance in computer networks. Computer Networks and ISDN Systems, 17 0 (1): 0 1--14, 1989

    1989

  7. [7]

    Qwen2-Audio technical report

    Yunfei Chu et al. Qwen2-Audio technical report. arXiv preprint arXiv:2407.10759, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Franklin, Joseph E

    Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. Clipper: A low-latency online prediction serving system. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2017

    2017

  9. [9]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-XL : Attentive language models beyond a fixed-length context. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019

    2019

  10. [10]

    FlashAttention-2 : Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2 : Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  11. [11]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . FlashAttention : Fast and memory-efficient exact attention with IO -awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  12. [12]

    High fidelity neural audio compression

    Alexandre D \'e fossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022

    Pith/arXiv arXiv 2022

  13. [13]

    Moshi: a speech-text foundation model for real-time dialogue

    Alexandre D \'e fossez, Laurent Mazar \'e , Manu Orsini, et al. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    LLaMA-Omni : Seamless speech interaction with large language models

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. LLaMA-Omni : Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Serving DNNs like Clockwork : Performance predictability from the bottom up

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serving DNNs like Clockwork : Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2020

    2020

  16. [16]

    Congestion avoidance and control

    Van Jacobson. Congestion avoidance and control. In Symposium Proceedings on Communications Architectures and Protocols (SIGCOMM), 1988

    1988

  17. [17]

    Jiang et al

    Albert Q. Jiang et al. Mistral 7B . arXiv preprint arXiv:2310.06825, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Efficient memory management for large language model serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. Efficient memory management for large language model serving with PagedAttention . In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023

    2023

  19. [19]

    SnapKV : LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, et al. SnapKV : LLM knows what you are looking for before generation. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  20. [20]

    AlpaServe : Statistical multiplexing with model parallelism for deep learning serving

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, et al. AlpaServe : Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2023

    2023

  21. [21]

    C. L. Liu and James W. Layland. Scheduling algorithms for multiprogramming in a hard-real-time environment. Journal of the ACM, 20 0 (1): 0 46--61, 1973

    1973

  22. [22]

    Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, et al. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  23. [23]

    KIVI : A tuning-free asymmetric 2bit quantization for KV cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, et al. KIVI : A tuning-free asymmetric 2bit quantization for KV cache. In International Conference on Machine Learning (ICML), 2024

    2024

  24. [24]

    MiniCPM-o 4.5: Towards real-time full-duplex omni-modal interaction

    OpenBMB . MiniCPM-o 4.5: Towards real-time full-duplex omni-modal interaction. arXiv preprint arXiv:2604.27393, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Splitwise: Efficient generative LLM inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, \'I \ n igo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative LLM inference using phase splitting. In ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024

    2024

  26. [26]

    Pine AI : The most natural human-computer interface is your voice

    Pine AI . Pine AI : The most natural human-computer interface is your voice. Blog post, 2026. URL https://www.19pine.ai/blog/pine-ai-the-most-natural-human-computer-interface-is-your-voice. Accessed 2026-07-02

    2026

  27. [27]

    Qwen3-Omni technical report

    Qwen Team . Qwen3-Omni technical report. arXiv preprint arXiv:2509.17765, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (ICML), 2023

    2023

  29. [29]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    Pith/arXiv arXiv 1911

  30. [30]

    FlexGen : High-throughput generative inference of large language models with a single GPU

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, et al. FlexGen : High-throughput generative inference of large language models with a single GPU . In International Conference on Machine Learning (ICML), 2023

    2023

  31. [31]

    SALMONN : Towards generic hearing abilities for large language models

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, et al. SALMONN : Towards generic hearing abilities for large language models. In International Conference on Learning Representations (ICLR), 2024

    2024

  32. [32]

    Interaction models: A scalable approach to human- AI collaboration

    Thinking Machines Lab . Interaction models: A scalable approach to human- AI collaboration. https://thinkingmachines.ai/blog/interaction-models/, 2026. Blog post, May 11, 2026

    2026

  33. [33]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  34. [34]

    vLLM v0.23: Resumable requests and the realtime API

    vLLM Team . vLLM v0.23: Resumable requests and the realtime API . https://docs.vllm.ai/, 2026. Release documentation

    2026

  35. [35]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024

    2024

  37. [37]

    Mini-Omni : Language models can hear, talk while thinking in streaming

    Zhifei Xie and Changqiao Wu. Mini-Omni : Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408.16725, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Qwen2.5-Omni technical report

    Jin Xu et al. Qwen2.5-Omni technical report. arXiv preprint arXiv:2503.20215, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    FlashInfer : Efficient and customizable attention engine for LLM inference serving

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, et al. FlashInfer : Efficient and customizable attention engine for LLM inference serving. In Proceedings of Machine Learning and Systems (MLSys), 2025

    2025

  40. [40]

    Orca: A distributed serving system for transformer-based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022

    2022

  41. [41]

    Big Bird : Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, et al. Big Bird : Transformers for longer sequences. In Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  42. [42]

    H2O : Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, et al. H2O : Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  43. [43]

    SGLang : Efficient execution of structured language model programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, et al. SGLang : Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  44. [44]

    DistServe : Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe : Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024

    2024