Pith. sign in

REVIEW 4 major objections 7 minor 68 references

NELSSA claims that routing short LLM requests to GPUs and long-context requests to near-memory sparse-attention hardware, with runtime migration for growing contexts, lifts decode throughput by up to 5.5x and cuts P99 latency by up to 15x a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:52 UTC pith:R6GACQN7

load-bearing objection A real GPU-PNM serving prototype with a novel length-based placement mechanism, but the headline speedup is a 512 GB projection; the measured system is credible but roughly performance-neutral versus GPU-only outside OOM cases. the 4 major comments →

arxiv 2607.26633 v1 pith:R6GACQN7 submitted 2026-07-29 cs.AR

NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement

Sookyung Choi , Seungyong Lee , Kangkyu Park , Yunseo Chun , Junseok Lee , Hyeongseok Gwak , Myunghyun Rhee , Euiseok Kim
show 6 more authors
Donguk Moon Kwangsik Shin Guseul Heo Youngpyo Joo Hoshik Kim Jongse Park
This is my paper
classification cs.AR
keywords LLM servingmixed-length workloadsProcessing-near-Memory (PNM)length-based request placementsparse attentionKV cache migrationCXL memory disaggregationdecode throughput
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that LLM serving workloads now mix requests from a few hundred to hundreds of thousands of tokens, and GPU-only systems handle this mix poorly because long-context requests exhaust GPU memory and stall short requests. NELSSA is proposed as a division of labour: GPUs run dense attention for short requests plus all the non-attention compute, while Processing-near-Memory (PNM) devices run sparse attention for long requests inside large-capacity memory. The point of division is set by a hardware-derived crossover length rather than by heuristic policy, and a background migration mechanism moves requests that grow during decoding onto the PNM tier without recomputation. If the argument holds, mixed-length serving can keep GPU batches large and tail latency low without sacrificing accuracy, which matters as agentic and reasoning workloads push context lengths upward.

Core claim

NELSSA's central claim is that decode attention should be executed on the tier whose memory is best matched to the request's current length, and that the boundary between tiers can be computed from hardware measurements. The paper equates a memory-bandwidth latency model for GPU dense attention with a model for distributed PNM sparse attention; solving the equation yields a crossover sequence length that becomes the routing threshold. Below it, requests run on the GPU; above it, attention runs entirely on PNM, while QKV projection and feed-forward networks are still batched globally on the GPU. Requests that start short and grow are handed off in the background after their KV cache is transf

What carries the argument

The load-bearing object is a hardware-derived crossover threshold T_input: the sequence length at which GPU dense attention latency equals distributed PNM sparse attention latency. The paper models GPU latency as L*M_kv/BW_GPU and PNM latency as centroid-search cost plus L*S*M_kv/(N*BW_eff_PNM) plus communication cost; equating them gives L_crossover(N), which grows smaller as more PNM devices are added. Around this threshold, NELSSA builds split-batch hybrid routing (the GPU runs all requests' QKV/MLP together and only branches at attention), seamless background migration with incremental chunked clustering of newly generated tokens, head-wise KV partitioning across PNM modules for load bal

Load-bearing premise

The headline gains depend on a hardware projection—the prototype has 32 GB per PNM device and the paper reports projected throughput at 512 GB per device, holding measured effective bandwidth constant across that range (Sections 7 and 8.1)—so if full-capacity PNM modules do not sustain the same bandwidth at larger KV-cache volumes, the 5.5x claim collapses; within the directly measured range, the paper concedes throughput only comparable to GPU-only baselines.

What would settle it

Build or measure a full-capacity 512 GB per-device PNM module and rerun the Section 8.1 projection at the batch sizes where the 5.5x advantage is claimed (e.g., large batches at 128K-512K contexts). If the effective random-access bandwidth at those data volumes falls below the constant value the paper assumes, the projected throughput advantage disappears; if it holds, the claim stands. A cheaper check is to measure actual tokens/sec with several long requests resident on the current 32 GB prototype at batch sizes beyond the directly measured range and compare against the GPU-only baseline.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • GPU batches stay large under mixed traffic instead of collapsing to near one when long contexts arrive, so decode throughput tracks short-request density rather than the longest resident context.
  • Long-context requests no longer block short ones: tail time-per-output-token improves by roughly 6x at P95 and up to 15x at P99 in the paper's mixed-length experiments.
  • Requests whose context grows mid-generation can migrate to the PNM tier in the background, avoiding both recomputation and PCIe-bound swap recovery.
  • Because only attention branches by length, the GPU still performs all QKV and feed-forward work in one global batch, so tensor-core utilization is not sacrificed by the split.
  • The benefit widens as the long-request fraction grows: at high long-request ratios the GPU-only baseline loses throughput steeply while the split system degrades much more slowly.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the crossover-threshold method is generic; any two compute/memory tiers with measurable latency-vs-length curves could be placed on the same kind of curve-crossing argument, so the design could extend to CPU-attached memory or storage-class tiers, not just PNM.
  • Editorial inference: the projection to full-capacity 512 GB PNM boards is the natural place to test the paper's central numbers; if effective random-access bandwidth falls as KV-cache volume grows, the 5.5x figure would shrink, though the batch-size restoration effect could persist.
  • Editorial inference: the static crossover threshold could be made adaptive at the orchestrator level—something the paper lists as future work—which would let the system shift traffic between tiers as queueing delays build, turning NELSSA into a closed-loop controller.
  • Editorial inference: the accuracy operating point (4% selection ratio) is established on specific long-context benchmark tasks; workloads with different token relevance distributions may need their own sparsity calibration, so the exact fixed ratio should not be assumed universal.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. NELSSA is a GPU-PNM heterogeneous serving system for mixed-length LLM decode workloads. It routes requests below a hardware-derived crossover threshold to GPU FlashAttention and longer requests to sparse attention executed on CXL-attached PNM devices, batches all QKV/MLP work on the GPU, and migrates dynamically growing KV caches to the PNM tier in the background without recomputation. The paper contributes an end-to-end prototype (SK hynix CXL-PNM modules with DDR5-6400, ARM Neoverse V2 cores, RoCEv2 RDMA), a decode-latency model for the crossover threshold (Eqs. 1-3), head-wise multi-PNM KV partitioning, and an evaluation against 1-GPU and 2-GPU vLLM baselines and RetroInfer. The abstract claims up to 5.5x decode throughput and 15x P99 TPOT reduction. The directly measured micro-metrics (memory headroom, running batch size, P95/P99 TPOT, migration jitter) support the mechanism; the 5.5x throughput figure, however, is projected to a 512 GB per-device configuration that is not yet built, and the mixed-workload throughput results are in part emulated by injecting measured GPU occupancy traces into unmodified vLLM rather than measured end-to-end.

Significance. If the claims hold, this is a meaningful systems contribution: a credible, full-stack instantiation of length-based execution placement across GPU and PNM tiers, built on real hardware. The strongest parts are the directly measured micro-evidence — GPU memory headroom and running batch size under long-request injection (Fig. 15), migration jitter (Fig. 19), the measured-vs-estimated attention-latency agreement within ±5.6% (Fig. 11), and the accuracy gate at a 4% selection ratio (Fig. 13). The disclosure of the 512 GB projection (Section 7) is transparent, and the migration design is a clean solution to dynamic context growth. The principal risk is that the headline throughput number is not an experimental result: it depends on untested capacity-scaling assumptions, and the paper itself concedes in Section 8.1 that within the directly measured range NELSSA only sustains throughput 'comparable to the GPU-only baselines.' The stress-test concern therefore lands, and the central claim needs re-scoping or additional validation before the paper can be accepted as-is.

major comments (4)
  1. [§7, §8.1, Fig. 10] The abstract's central 5.5× throughput claim is not an experimental result: it is projected to a 512 GB per-device PNM configuration that the authors state they are 'currently developing' (§7), and §8.1 concedes that within the directly measured range 'NELSSA... sustains throughput comparable to the GPU-only baselines.' Every configuration with large gains (e.g., 5.5× at 1M tokens, batch 8) lies outside the measured range. The projection holds BWeff=200 GB/s, PNM-overhead, Transfer, and Others fixed (§8.1 and Fig. 11 caption); the ±5.6% validation bounds PNM-attn estimation only within the measured data-volume range, not 16× capacity scaling or all-four-device contention at larger working sets. Please report projected throughput under a BWeff sensitivity sweep (e.g., 100–200 GB/s) with per-step overheads scaled, and mark measured vs. projected explicitly wherever 5.5× appears (abstract,
  2. [§4.1, Eqs. (2)-(3)] The routing threshold and the projected throughput share the same analytic model and the same prototype-derived parameters (C, S, BWeff, Tcomm), so the threshold is not an independent external benchmark. In particular, C is never defined beyond 'centroid compression factor' and its measured value is not reported, and the crossover from Eq. (3) is not validated against the measured decode-latency crossover (Fig. 3 is schematic). Since a wrong Tinput would either route short requests to PNM (latency regression) or leave long requests on the GPU (OoM), please (a) report C and Tcomm as measured on the prototype, (b) overlay measured decode latency for GPU and PNM paths vs. L against Eqs. (1)-(2), and (c) show the sensitivity of the Fig. 10 speedups to a ±20–30% perturbation of Tinput and to the selection ratio S.
  3. [§8.3, §8.4] The end-to-end mixed-workload throughput results (Figs. 14, 17) are not measured end-to-end: they are produced by injecting measured GPU occupancy traces into an unmodified vLLM engine. The validation of this emulation (4.12% error) is performed at approximately 100K tokens only, and the projected configurations in Fig. 10 (1M tokens, batch ≥ 8) are validated neither directly nor by emulation. This compounds the projection risk identified above. Please add a table classifying each reported number as directly measured / emulated / projected, state the region of applicability of the 4.12% error bound, and demonstrate (or bound) the emulation error in the projected batch-length range rather than asserting it.
  4. [§8.2, Fig. 13, Abstract] The abstract claims NELSSA maintains 'the same level of accuracy as the GPU-only baseline,' but §8.2 reports only two RULER tasks (niah_multikey_3, qa_2) and shows no dense-attention baseline accuracy curve; 'stable accuracy' at a selection ratio ≥4% is a weaker claim than parity with the GPU-only dense path. Furthermore, S=4% is a free parameter of Eq. (2), and no sensitivity of the headline throughput to S is reported — Fig. 13's throughput data are shown at unspecified batch/context conditions. Please add a per-task accuracy table vs. the dense baseline over the evaluated 128K–1024K range (with context lengths stated) and a throughput-vs-S curve at the headline configuration.
minor comments (7)
  1. [§5.1] The sentence 'This section presents the system architecture that makes these mechanisms practically realizable' is duplicated verbatim in consecutive paragraphs.
  2. [Fig. 10] The measured/projected boundary is not visually marked. Add shading or a divider so the reader can tell which bars are measured, and explain the 'X' OoM markers in the caption.
  3. [Figs. 13, 18] Several axis labels and legends contain garbled glyph/unicode artifacts (e.g., the 'uni0000...' sequences in Fig. 18), rendering part of the figures unreadable. The figures need to be regenerated cleanly.
  4. [Eq. (2)] Specify how C is measured for the prototype (algorithm and measured value), and report M_kv per token for Llama3-8B so that the measured-range boundaries stated in §8.1 (e.g., batch 8 at 128K) are reproducible from the hardware specifications.
  5. [Fig. 12] The communication-overhead percentages are produced by an analytical model, not measured. The text and caption should label this clearly as model-based extrapolation rather than presenting it alongside directly measured results.
  6. [§8.3] The burst-size values (BS=1 vs BS=2) and RPS for Fig. 14(a) appear only in the caption; state them in the text, and give the long-request fraction used for Fig. 14(a) explicitly.
  7. [§7, §9] The 2×H100 baseline is called 'cost-comparable,' but no cost or power data are given for the PNM node (host CPU, NIC, four DDR5 modules). Either provide cost/power numbers or soften the term to 'cost-comparable by assumption.'

Circularity Check

0 steps flagged

No significant circularity; the 5.5x headline is a disclosed extrapolation, not a fitted prediction.

full rationale

The load-bearing derivation chain is not circular. The routing threshold T_input is obtained by equating two explicitly stated latency models (Eqs. 1-3). Its parameters are measured or chosen on the same prototype (e.g., BWeff_PNM is 'measured on our prototype under sparse random access'), so the threshold is a calibrated design decision rather than an independent first-principles prediction. However, the paper's evaluation does not use this model to produce the headline speedup from the same fitted values: decode throughput is compared against GPU-only and RetroInfer baselines, and the directly measured range is explicitly reported as 'sustaining throughput comparable to the GPU-only baselines.' The 5.5x figure is an extrapolation to an unbuilt 512 GB configuration, disclosed in Section 7 ('we accordingly report projected throughput at a 512 GB per-device configuration') and Section 8.1 ('the effective bandwidth observed within the measured range is held constant across the full projected range'). Holding a measured value constant is an assumption and a correctness/extrapolation risk, not a circular fit of the target result. The only self-citation with any load-bearing appearance is [8] (same first author and SK hynix team), invoked to motivate sparse PNM attention; but the same discussion cites external systems (LongSight [42], ScalablePNM [18]) and the paper's own Figure 11 measurements independently support the sparsity-offsets-bandwidth claim, so this citation is not load-bearing. The migration/emulation methodology is validated against direct hardware execution (4.12% error at 100K), not against the target result. No uniqueness theorem or ansatz-via-citation is used. Overall, the central claims have independent empirical content; the main weakness is unverified hardware projection, which belongs under correctness risk rather than circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central design rests on measured hardware parameters and two borrowed components (RetroInfer-style clustered sparse attention and the authors' earlier PNM sparse-attention result), plus two extrapolation assumptions (512 GB projection and vLLM emulation) that carry the headline numbers. No new physical entity is introduced; the PNM modules are real SK hynix hardware.

free parameters (4)
  • Sparse selection ratio S = 4% of KV clusters (Top-K)
    Chosen in Section 8.2 as the minimum ratio that 'maintains stable accuracy' on two RULER tasks; used in Eq. 2 and for all throughput results.
  • Effective PNM bandwidth BWeff = not stated numerically; measured on prototype under sparse random access
    Input to Eq. 2/3 for the latency model and crossover; held constant in the 512 GB projection (Section 8.1).
  • Communication latency Tcomm(N) = not stated numerically; measured
    Computed from measured network round-trip, per-device transfer, and host dispatch/aggregation; determines the crossover in Eq. 3.
  • Centroid compression factor C = not stated; from clustering structure
    Used in Eq. 2 to model GPU-side similarity-search cost; not independently measured in the paper.
axioms (5)
  • domain assumption Sparse cluster selection at 4% preserves accuracy for long-context attention
    Central to the accuracy claim; Section 8.2 validates only on two RULER tasks (niah_multikey_3, qa_2) for Llama3-8B-1048K.
  • domain assumption Decode attention latency is dominated by KV-cache bandwidth on both tiers
    Eq. 1 and Eq. 2 omit compute, scheduling, and software overheads; the model is used to derive the routing threshold.
  • ad hoc to paper The 512 GB projection is purely a DRAM capacity expansion with unchanged effective bandwidth and overheads
    Sections 7 and 8.1; the headline 5.5x throughput depends on this extrapolation.
  • ad hoc to paper Injecting measured GPU occupancy traces into unmodified vLLM faithfully reproduces NELSSA end-to-end throughput
    Section 8.3 emulation methodology; validated at 4.12% error at about 100K tokens but not over the full claimed range.
  • domain assumption Clustering overhead is under approximately 2% of prefill time (from RetroInfer)
    Adopted in Section 5.1 without re-measurement in this paper.

pith-pipeline@v1.3.0-daily-deepseek · 22327 in / 12840 out tokens · 133405 ms · 2026-08-01T11:52:54.637418+00:00 · methodology

0 comments
read the original abstract

Modern LLMs and their agentic applications are broadening the range of serving workloads, spanning context lengths from a few hundred tokens to hundreds of thousands. As these requests frequently interleave within the same serving window, LLM serving systems must handle highly heterogeneous mixed-length workloads. Such mixed-length workloads expose fundamental inefficiencies in GPU-centric serving architectures, whose throughput depends on large, memory-constrained batches. In this paper, we present NELSSA, an LLM serving system that integrates GPUs with real-world Processing-near-Memory (PNM) accelerator devices to efficiently support mixed-length workloads. NELSSA employs length-based request placement to route short-context requests to GPUs and long-context requests to the PNM tier, incorporating runtime migration to accommodate dynamic context growth without recomputation. We prototype NELSSA as an end-to-end system, implementing device-level sparse attention on PNM, GPU decode kernels, and a host-side runtime that orchestrates scheduling and cross-tier memory movement over a CXL-enabled infrastructure with RPC and RDMA support. Across mixed-length LLM workloads, NELSSA improves decode throughput by up to 5.5x in tokens/sec and reduces P99 latency by up to 15x compared to GPU-only baselines. Our end-to-end prototype and experimental results suggest that integrated GPU-PNM serving, enabled by CXL-based disaggregation, is a promising system paradigm for scalable and flexible LLM infrastructures that support evolving workloads.

Figures

Figures reproduced from arXiv: 2607.26633 by Donguk Moon, Euiseok Kim, Guseul Heo, Hoshik Kim, Hyeongseok Gwak, Jongse Park, Junseok Lee, Kangkyu Park, Kwangsik Shin, Myunghyun Rhee, Seungyong Lee, Sookyung Choi, Youngpyo Joo, Yunseo Chun.

Figure 1
Figure 1. Figure 1: (a) GPU-only systems share mixed-length requests [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: GPU memory saturation, batch size collapse, and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Decode attention latency across sequence lengths. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overall architecture of NELSSA. specialization is governed by two design principles. First, execu￾tion placement must be determined by request length. GPU-only execution is susceptible to HoL blocking from long requests, while routing all requests to PNM degrades performance for short ones, and no single strategy handles the full spectrum efficiently. Second, once execution transfers to the PNM tier, it mu… view at source ↗
Figure 6
Figure 6. Figure 6: Seamless background migration. Historical KV [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: KV cache partitioning strategies for multi-PNM [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Overall NELSSA SW stack. NelssaHost PNM User Library HW User Attention API add( ) ⅹ N (Parallel Dispatch) op() ⅹ N (Parallel Dispatch) Data[ ], Command[ ] &Command PNM N Data Path Control Path PNM 1 CXL Endpoint CXL.mem PNM Device Manager CMD Executor CMD Executor CMD Executor Attention Device Runtime PNM 0 … KV Partitioner Memory API Job Dispatcher Output Aggregator Command API [PITH_FULL_IMAGE:figures/f… view at source ↗
Figure 9
Figure 9. Figure 9: NELSSA PNM software stack. 6 Implementation The architecture in Section 4 is a reference design targeting a ded￾icated compute pipeline. Our prototype realizes it on real PNM hardware with DDR5 DRAM and ARM Neoverse V2 cores over CXL. Using these general-purpose cores in place of the dedicated compute units, the prototype provides a conservative characteriza￾tion of end-to-end behavior and enables direct m… view at source ↗
Figure 10
Figure 10. Figure 10: Decode throughput comparison across varying sequence lengths and batch sizes (measured/projected). [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Component-level breakdown of decode attention [PITH_FULL_IMAGE:figures/full_fig_p009_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: GPU-PNM cross-node communication overhead [PITH_FULL_IMAGE:figures/full_fig_p009_12.png] view at source ↗
Figure 14
Figure 14. Figure 14: System-wide decode throughput and pending [PITH_FULL_IMAGE:figures/full_fig_p010_14.png] view at source ↗
Figure 17
Figure 17. Figure 17: System-wide decode throughput and preemption [PITH_FULL_IMAGE:figures/full_fig_p011_17.png] view at source ↗
Figure 16
Figure 16. Figure 16: Decode throughput under varying long request [PITH_FULL_IMAGE:figures/full_fig_p011_16.png] view at source ↗
Figure 18
Figure 18. Figure 18: (a) Throughput degradation at preemption events [PITH_FULL_IMAGE:figures/full_fig_p012_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Decode latency jitter during concurrent KV cache [PITH_FULL_IMAGE:figures/full_fig_p012_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 9 linked inside Pith

  1. [1]

    Gulavani, Alexey Tumanov, and Ramachandran Ramjee

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. InUSENIX Symposium on Operating Systems Design and Implementation

    2024

  2. [2]

    Amey Agrawal, Haoran Qiu, Junda Chen, Íñigo Goiri, Chaojie Zhang, Rayyan Shahid, Ramachandran Ramjee, Alexey Tumanov, and Esha Choukse. 2024. No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha.arXiv preprint arXiv:2409.17264(2024)

    arXiv 2024

  3. [3]

    Anthropic. 2024. Claude Code. https://www.anthropic.com/claude-code. Ac- cessed: 2024-04-07

    2024

  4. [4]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, et al . 2024. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069(2024)

    Pith/arXiv arXiv 2024

  5. [5]

    Renze Chen, Zhuofeng Wang, Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. 2024. ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction. InThe Thirty- eighth Annual Conference on Neural Information Processing Systems

    2024

  6. [6]

    Yaoqi Chen, Jinkai Zhang, Baotong Lu, Qianxi Zhang, Chengruidong Zhang, Jingjia Luo, Di Liu, Huiqiang Jiang, Qi Chen, Jing Liu, Bailu Ding, Xiao Yan, Jiawei Jiang, Chen Chen, Mingxing Zhang, Yuqing Yang, Fan Yang, and Mao Yang. 2025. RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference. arXiv:2505.02922 [cs.LG] https://arxiv.org/a...

    Pith/arXiv arXiv 2025

  7. [7]

    Seungbeom Choi, Jeonghoe Goo, Eunjoo Jeon, Mingyu Yang, and Minsung Jang

  8. [8]

    Sookyung Choi, Myunghyun Rhee, Euiseok Kim, Kwangsik Shin, Youngpyo Joo, and Hoshik Kim. 2025. PNM Meets Sparse Attention: Enabling Multi-Million Tokens Inference at Scale.IEEE Computer Architecture Letters24, 2, 353–356. https://doi.org/10.1109/LCA.2025.3624272

    arXiv 2025

  9. [9]

    Andy Chung, Yichi Zhang, Kaixiang Lin, Aditya Rawal, Qiaozi Gao, and Joyce Chai. [n. d.]. Evaluating Long-Context Reasoning in LLM-Based WebAgents. InNeurIPS 2025 Workshop on Bridging Language, Agent, and World Models for Reasoning and Planning

    2025

  10. [10]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems35 (2022), 16344–16359

    2022

  11. [11]

    Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. 2024. Efficient llm scheduling by learning to rank.Advances in Neural Information Processing Systems37 (2024), 59006–59029

    2024

  12. [12]

    Ruihao Gong, Shihao Bai, Siyu Wu, Yunqian Fan, Zaijun Wang, Xiuhong Li, Hailong Yang, and Xianglong Liu. 2025. Past-future scheduler for llm serving under sla guarantees. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume

    2025

  13. [13]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems37 (2024), 1270–1303

    2024

  14. [14]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654 (2024)

    Pith/arXiv arXiv 2024

  15. [15]

    Kerui Huang, Shuhan Liu, Xing Hu, Tongtong Xu, Lingfeng Bao, and Xin Xia

  16. [16]

    Carlos E Jimenez, John Yang, et al . 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InInternational Conference on Learning Rep- resentations (ICLR)

    2024

  17. [17]

    Reasoning efficiently through adaptive chain-of-thought compression: A self-optimizing framework.arXiv preprint arXiv:2509.14093(2025)

    arXiv 2025

  18. [18]

    Dowon Kim, MinJae Lee, Janghyeon Kim, HyuckSung Kwon, Hyeonggyu Jeong, Sang-Soo Park, Minyong Yoon, Si-Dong Roh, Yongsuk Kwon, Jinin So, et al. 2025. Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits. In2025 34th International Conference on Parallel Architectures and Compilation Techniques (PACT)...

    2025

  19. [19]

    2023.𝑆 3: Increasing GPU Utilization during Generative Inference for Higher Throughput.Advances in Neural Information Processing Systems36 (2023), 18015–18027

    Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. 2023.𝑆 3: Increasing GPU Utilization during Generative Inference for Higher Throughput.Advances in Neural Information Processing Systems36 (2023), 18015–18027

    2023

  20. [20]

    Minsu Kim, Seongmin Hong, RyeoWook Ko, Soongyu Choi, Hunjong Lee, Junsoo Kim, Joo-Young Kim, and Jongse Park. 2025. Oaken: Fast and efficient llm serving with online-offline hybrid kv cache quantization. InProceedings of the 52nd Annual International Symposium on Computer Architecture. 482–497

    2025

  21. [21]

    Lee, Sangdoo Yun, and Hyun Oh Song

    Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, and Hyun Oh Song. 2025. KVzip: Query-Agnostic KV Cache Compression with Con- text Reconstruction. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS)

    2025

  22. [22]

    Gonzalez, Haotong Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Haotong Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of the 29th Symposium on Operating Systems Principles(2023)

    2023

  23. [23]

    Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang. 2024. Infinipot: Infinite context processing on memory-constrained llms. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 16046–16060

    2024

  24. [24]

    Yinmin Li et al. 2024. Preble: Efficient Distributed Prompt Scheduling for LLM Serving. InInternational Conference on Learning Representations (ICLR)

    2024

  25. [25]

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. InUSENIX Symposium on Operating Systems Design and Implementation

    2024

  26. [26]

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang, Yuqing Yang, and Lili Qiu. 2025. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

    2025

  27. [27]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. [n. d.]. SnapKV: LLM Knows What You are Looking for Before Generation. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems

  28. [28]

    Tengxiao Liu, Zifeng Wang, Jin Miao, I Hsu, Jun Yan, Jiefeng Chen, Rujun Han, Fangyuan Xu, Yanfei Chen, Ke Jiang, et al. 2025. Budget-aware tool-use enables effective agent scaling.arXiv preprint arXiv:2511.17006(2025)

    arXiv 2025

  29. [29]

    Lian Liu, Shixin Zhao, Bing Li, Haimeng Ren, Zhaohui Xu, Mengdi Wang, Xi- aowei Li, Yinhe Han, and Ying Wang. 2025. Make LLM inference affordable to everyone: Augmenting GPU memory with NDP-DIMM. In2025 IEEE Inter- national Symposium on High Performance Computer Architecture (HPCA). IEEE, 1751–1765

    2025

  30. [30]

    Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, et al . 2025. Lmcache: An efficient KV cache layer for enterprise-scale LLM inference.arXiv preprint arXiv:2510.09665(2025)

    arXiv 2025

  31. [31]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. AgentBench: Evaluating LLMs as Agents. InInternational Conference on Learning Represe...

    2024

  32. [32]

    Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. 2024. Concise thoughts: Impact of output length on llm reasoning and cost.arXiv preprint arXiv:2407.19825(2024)

    Pith/arXiv arXiv 2024

  33. [33]

    Xiaoyu Ma and David Patterson. 2026. Challenges and Research Directions for Large Language Model Inference Hardware. https://api.semanticscholar.org/ CorpusID:284543821

    2026

  34. [34]

    OpenAI. 2026. Codex. https://openai.com/index/introducing-gpt-5-3-codex/. Accessed: 2026-04-07

    2026

  35. [35]

    OpenAI. 2022. ChatGPT. https://openai.com/blog/chatgpt. Accessed: 2026-04-07

    2022

  36. [36]

    Sang-Soo Park, KyungSoo Kim, Jinin So, Jin Jung, Jonggeon Lee, Kyoungwan Woo, Nayeon Kim, Younghyun Lee, Hyungyo Kim, Yongsuk Kwon, Jinhyun Kim, Jieun Lee, YeonGon Cho, Yongmin Tai, Jeonghyeon Cho, Hoyoung Song, Jung Ho 13 , , Sookyung Choi, Seungyong Lee, Kangkyu Park, Yunseo Chun, Junseok Lee, Hyeongseok Gwak, Myunghyun Rhee, Euiseok Kim, Donguk Moon, K...

    arXiv 2024

  37. [37]

    Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. 2024. Instinfer: In-storage attention offloading for cost-effective long-context llm inference.arXiv preprint arXiv:2409.04992 (2024)

    Pith/arXiv arXiv 2024

  38. [38]

    Leonid Pekelis, Michael Feil, Forrest Moret, Mark Huang, and Tiffany Peng. 2024. Llama 3 Gradient: A series of long context models. https://doi.org/10.57967/hf/ 3372

    doi:10.57967/hf/ 2024

  39. [39]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 118–132

    2024

  40. [40]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, Santa Clara, CA, 155–170. https://www.u...

    2025

  41. [41]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Brad- bury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference.Proceedings of machine learning and systems5 (2023), 606–624

    2023

  42. [42]

    Ezgi Yücel, Jinkwon Kim, José F

    Derrick Quinn, E. Ezgi Yücel, Jinkwon Kim, José F. Martínez, and Mohammad Alian. 2025. LongSight: Compute-Enabled Memory to Accelerate Large-Context LLMs via Sparse Attention. InProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, MICRO 2025. ACM, 34–48. https://doi.org/10. 1145/3725843.3756062

    arXiv 2025

  43. [43]

    Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T Kalbarczyk, Tamer Basar, and Ravishankar K Iyer

  44. [44]

    Myunghyun Rhee, Joonseop Sim, Taeyoung Ahn, Seungyong Lee, Daegun Yoon, Euiseok Kim, Kyoung Park, Youngpyo Joo, and Hoshik Kim. 2025. HPU: High- bandwidth processing unit for scalable, cost-effective LLM inference via GPU co-processing.arXiv preprint arXiv:2504.16112(2025)

    arXiv 2025

  45. [45]

    Rana Shahout, Chunwei Liu, Weifan Jiang, Minlan Yu, Michael Mitzenmacher, et al. [n. d.]. DON’T STOP ME NOW: EMBEDDING BASED SCHEDULING FOR LLMS. InThe Thirteenth International Conference on Learning Representations

  46. [46]

    Pol G Recasens, Ferran Agullo, Yue Zhu, Chen Wang, Eun Kyung Lee, Olivier Tardieu, Jordi Torres, and Josep Ll Berral. 2025. Mind the memory gap: Unveiling gpu bottlenecks in large-batch llm inference. In2025 IEEE 18th International Conference on Cloud Computing (CLOUD). IEEE, 277–287

    2025

  47. [47]

    Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, et al . 2025. Burstgpt: A real-world workload dataset to optimize llm serving systems. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 5831–5841

    2025

  48. [48]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  49. [49]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. InInternational Conference on Machine Learning. PMLR, 31094–31116

    2023

  50. [50]

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, junxian guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2025. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. InInternational Conference on Learning Representations (ICLR)

    2025

  51. [51]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. InInternational Conference on Learning Representations (ICLR)

    2024

  52. [52]

    Yuxing Xiang, Xue Li, Kun Qian, Wenyuan Yu, Ennan Zhai, and Xin Jin. 2025. Servegen: Workload characterization and generation of large language model serving in production.arXiv preprint arXiv:2505.09999(2025)

    Pith/arXiv arXiv 2025

  53. [53]

    Huanyi Xie, Yubin Chen, Liangyu Wang, Lijie Hu, and Di Wang. 2026. Pre- dicting LLM output length via entropy-guided representations.arXiv preprint arXiv:2602.11812(2026)

    arXiv 2026

  54. [54]

    Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Zhiruo Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Melroy Maben, Raj Mehta, Wayne Chi, Lawrence Keunho Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. 2025. TheAgentCompany: Benchmarking LLM Agents on Consequential...

    2025

  55. [55]

    Meiman Xiao, Ante Wang, Qingguo Hu, Zhongjian Miao, Huangjun Shen, Longyue Wang, Weihua Luo, and Jinsong Su. 2026. Can LLMs Track Their Output Length? A Dynamic Feedback Mechanism for Precise Length Regulation. arXiv preprint arXiv:2601.01768(2026)

    arXiv 2026

  56. [56]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations (ICLR)

    2023

  57. [57]

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, et al. 2024. Llm inference unveiled: Survey and roofline model insights.arXiv preprint arXiv:2402.16363 (2024)

    Pith/arXiv arXiv 2024

  58. [58]

    John Yang, Carlos E Jimenez, et al . 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering. InNeural Information Processing Systems (NeurIPS)

    2024

  59. [59]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems36 (2023), 34661–34710

    2023

  60. [60]

    Yiren Zhao and Junyi Liu. 2026. Heterogeneous Computing: The Key to Powering the Future of AI Agent Inference.arXiv preprint arXiv:2601.22001(2026)

    arXiv 2026

  61. [61]

    Hengrui Zhang, Pratyush Patel, August Ning, and David Wentzlaff. 2025. SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference.arXiv preprint arXiv:2510.08544(2025)

    arXiv 2025

  62. [62]

    2024.{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024.{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193–210

    2024

  63. [63]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. InInternational Conference on Learning Representations (ICLR)

    2024

  64. [64]

    Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You

  65. [68]

    Adam Zweiger, Xinghong Fu, Han Guo, and Yoon Kim. 2026. Fast KV Compaction via Attention Matching.arXiv preprint arXiv:2602.16284(2026). 14

    Pith/arXiv arXiv 2026

  66. [2023]

    Response length perception and sequence scheduling: An llm-empowered llm inference pipeline.Advances in Neural Information Processing Systems36 (2023), 65517–65530

    2023

  67. [2024]

    InInternational Conference on Architectural Support for Programming Languages and Operating Systems

    Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction. InInternational Conference on Architectural Support for Programming Languages and Operating Systems

  68. [2025]

    ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor.arXiv preprint arXiv:2505.09142(2025)

    Pith/arXiv arXiv 2025