Pith. sign in

REVIEW 3 major objections 5 minor 49 references

LLM serving optimization should start from resource floors, not grid search, so layout choice becomes a computable function of operating point.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.

T0 review reviewed 2026-07-11 challenge →

load-bearing objection Solid analytical triage for LLM serving with a real production layout judgment; the residual-threshold product claim is still unmeasured, and the draft says so. the 3 major comments →

arxiv 2607.05876 v2 pith:IJGGTCBP submitted 2026-07-07 cs.PF cs.AIcs.DC

Think Before You Grid-Search: Floor-First Triage for LLM Serving

Yihua Liu This is my paper
classification cs.PF cs.AIcs.DC
keywords LLM servingperformance triageresource floorswall orderingMLA MoEtensor parallelismexpert parallelismH20 ridge point
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Production teams optimizing large language model serving usually try many layouts and open heavy profilers whenever latency targets are missed. This paper argues the reverse: first build a five-resource analytical floor for each decode step, then treat any benchmark as a residual against that floor, and open profilers only when the residual exceeds a stated threshold. Summing costs that share a resource and maximizing across independent resources yields an optimistic floor; the plain sum is a pessimistic floor. Where a measurement sits inside that interval is already a free diagnosis of how well the engines are overlapping. Deployments are then compared by wall ordering—which resource limit binds first as concurrency grows—rather than by same-batch point benchmarks. On a 671B MoE model with latent attention running on 16 export-grade H20 GPUs, the floors show tensor-parallel attention is capacity-limited to roughly 70 concurrent 8K requests, while expert-parallel plus data-parallel attention raises that wall by an order of magnitude, yet single-stream latency favors the tensor-parallel layout by more than two times. The layout judgment therefore flips with the target concurrency, which explains why identical hardware has shipped opposite choices.

Core claim

A residual-driven triage workflow called Floor First models each decode step as a five-dimensional resource vector and produces a two-sided floor interval. Measurements inside the interval diagnose overlap quality without a profiler; residuals above a stated threshold alone justify escalation. Comparing layouts by which resource wall binds first as load grows turns the TP-versus-EP+DP-attention choice for MLA MoE decode into a computable function of the operating point, not a matter of folklore or point benchmarks.

What carries the argument

The two-sided floor: for the five-dimensional resource vector of each decode step, terms that share a hardware resource add while independent engines maximize, producing an optimistic floor (max) and a no-overlap floor (sum). Position of a measurement inside [max, sum] is a zero-cost overlap diagnostic that gates profiler use and, together with wall ordering, decides which layout survives.

Load-bearing premise

The stop-versus-escalate cutoffs—roughly 1.3 times the floor residual and the MBU utilization bands—are practitioner defaults whose grounding measurement sweep has not yet been run, so wrong cutoffs would mis-fire triage even when the arithmetic is correct.

What would settle it

An open-loop residual sweep across the three layouts, several batch sizes, and context lengths that either confirms residuals below 1.3 times the optimistic floor need no profiler action while residuals above it always yield an identifiable root cause, or shows the cutoffs must be revised.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Floor First, a residual-driven triage workflow for LLM serving optimization. Decode is modeled as a five-dimensional resource vector (HBM bytes, FLOPs, network bytes, network messages, KV capacity); within-resource sums and across-resource max yield a two-sided [max, sum] floor interval whose position diagnoses overlap before profilers are opened. Deployments are compared by wall ordering rather than point benchmarks. A case study of a DeepSeek-V3.2-style 671B MoE/MLA model on 16×H20 derives that TP16 is KV-capacity-limited to ~70 concurrent 8K requests while EP16+DP attention raises the capacity wall to ~644, yet single-stream latency favors TP by 2.4×—so the layout judgment is a function of operating point. The account is compositional and ships as a zero-dependency calculator plus an agent skill. The draft states that case-study numbers are analytical floors and that the residual-calibration sweep is planned, not performed.

Significance. If the wall-ordering arithmetic and residual-triage discipline hold under measurement, the paper supplies a practical decision procedure that production teams currently lack: when to stop optimizing, when to change layout, and when to open a profiler. The H20 ridge-point portrait (~74 FLOP/byte) and the capacity-first derivation of the MLA TP-versus-DP judgment are concrete, transferable contributions; they explain an observed production disagreement without requiring new hardware or kernels. Strengths that should be credited include: closed-form MoE expert-union and rank-touch expectations; explicit two-sided floors that keep overlap observable rather than assumed; a zero-dependency, command-reproducible artifact (Appendix A); and honest scoping of blind spots (continuous batching, expert imbalance, host-bound regimes). The agent-skill packaging is a useful systems contribution for agentic optimization loops.

major comments (3)
  1. [§2.4, Status, §8] §2.4, Status of this draft, and §8-(2): The 1.3× residual escalation threshold and the decode MBU bands (>70% / 40–70% / <40%) are stated as practitioner defaults whose grounding residual sweep “is planned, not yet performed.” These cutoffs are load-bearing for the product claim that residual position should gate profiler escalation (Figure 1, steps 4–6). Without the open-loop steady-state residual distribution over {TP16, TP16+DSA, EP16+DPA}×B×S, the stop-vs-escalate procedure remains unvalidated even if the resource arithmetic is correct. Completing §8-(2) and folding measured-versus-floor [max,sum] plots into §5 is required for the workflow claim.
  2. [§5, §5.7, §8] §5 and Status note: All case-study numbers in Tables 2–3 and Figures 2–3 are analytical floors; no measured P50 TPOT/TTFT reconciliation is reported. The claim that interval position is a “zero-cost overlap diagnostic” (§3.1, §5.7) is therefore illustrated only with hypothetical measurements (25 ms and 45 ms). At least one near-floor and one high-residual configuration with Nsight Systems traces (§8-(3)) is needed to show that the three-question reading list and budget-table indictment work as advertised, and to confirm that the capacity-wall ordering survives real service-time residuals.
  3. [§3.3, §5.3, §7] §3.3–§3.4 and §7: The MoE weight-byte account and EP rank-touch expectation assume uniform routing. The paper correctly flags load imbalance as a failure mode, but the case-study walls (B_max ≈70 vs ≈644; weight terms in Table 2) are reported only under the uniform/closed-form model. A sensitivity bound—e.g., worst-case hot-expert inflation of the weight-bandwidth wall, or the batch range where the union expectation remains within 10% of measured expert occupancy—would make the deployment judgment robust rather than conditional on ideal routing. This is especially important because the paper’s own calibration note shows communication constants moved substantially while HBM/capacity carried the verdicts; expert imbalance could move the HBM account itself.
minor comments (5)
  1. [Figure 1] Figure 1 is dense; the ≤1.3× / >1.3× branch labels and the two execution arms would be clearer if the escalation threshold were annotated as “calibration target (default 1.3×)” rather than a fixed constant.
  2. [§5.1, Table 1] §5.1 / Table 1: State explicitly whether H20 FP8 peak (296 T) and HBM BW (4.0 TB/s) are datasheet or measured; §4 later reports 81% HBM and 89% FP8 under fallback tools, which slightly shifts the ridge and should be cross-referenced in the portrait.
  3. [§3.1, §5.6] §3.1: The single-stream caveat (honest floor is t_sum at B=1) is important; consider elevating it into the floor definition rather than a parenthetical, since §5.6’s 2.4× inversion depends on it.
  4. [§6] Related work: LIMINAL [9] and AIC++ [38] are well positioned; a one-sentence contrast table (closed form vs kernel DB; overlap fixed vs [max,sum]; hardware transfer) would help readers place Floor First in the pipeline.
  5. [Abstract, §1, §3.1] Typos / polish: “Floor Firstmodels” spacing (Abstract); “adecision procedure” (§1); “topt” / “t sum” notation inconsistency (§3.1); ensure B_max and B^*_GEMM use consistent subscript style throughout Tables 2–3.

Circularity Check

0 steps flagged

No significant circularity: floors and wall-ordering judgments are closed-form arithmetic from model dimensions and hardware rates, not fitted to the outcomes they bound.

full rationale

The paper’s load-bearing claims (TP16 B_max ≈70 vs EP16+DP ≈644; single-stream TP 2.4× better; layout as a function of operating point) are produced by explicit resource-vector arithmetic: HBM/FLOP/network terms from published model dimensions and datasheet or cluster-calibrated rates, capacity via the B_max formula of §3.5, MoE weight traffic via the closed-form expert-union expectation under uniform routing (§3.3), and two-sided floors by the max/sum aggregation rules of §3.1. These inputs do not include the measured latencies or production layout choices the floors are meant to bound or explain; the residual sweep that would ground the 1.3× escalation threshold and MBU bands is openly stated as planned, not performed (§2.4, Status, §8), so those cutoffs are labeled practitioner defaults rather than presented as first-principles predictions. No parameter is fitted to a subset of data and then re-sold as a prediction; no uniqueness theorem or load-bearing premise is imported via self-citation; the single-author draft cites external deployment reports and prior analytical models as context, not as circular support. The derivation is therefore self-contained closed-form accounting, not circular.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 3 invented entities

The central layout and triage claims rest on standard resource-accounting axioms plus a small set of practitioner cutoffs and routing idealizations. No new physical entities are postulated. The free parameters that most affect the workflow’s stop/escalate behavior (1.3× residual, MBU bands) are explicitly uncalibrated in this draft; the arithmetic walls themselves depend mainly on model dimensions, HBM capacity/bandwidth, and the MLA unshardability declaration.

free parameters (4)
  • residual_escalation_threshold = 1.3×
    Default 1.3× optimistic floor used to decide whether to open Nsight Systems; paper states it is a practice default pending residual-sweep calibration (§2.3–2.4, §8).
  • decode_MBU_triage_bands = 70% / 40% (decode MBU); 50% / 25% (MoE prefill MFU)
    Bands >70% / 40–70% / <40% (and lowered MoE prefill MFU 50%/25%) systematize practitioner experience, not first-principles constants (§2.4).
  • cluster_collective_constants = AR ~43 GB/s; ℓ~33 µs; DeepEP ~60 µs
    Effective all-reduce ~43 GB/s, small-message latency ~33 µs, DeepEP dispatch ~60 µs are measured on the target cluster and enter network floors (§4).
  • expert_union_routing_model = uniform routing; E=256, k=8
    Uniform top-k routing expectation f_expert(B)=1-(1-k/E)^B (or min bound) sets weight-byte accounts; imbalance is a known failure mode (§3.3, §7).
axioms (6)
  • domain assumption Terms contending for the same hardware resource add in time; independent engines (HBM, SMs, NIC) may fully overlap, so optimistic step time is the max of resource times.
    Stated as following from hardware concurrency in §3.1; at batch-1 the paper itself switches to sum because layer-serial dependence removes overlap.
  • domain assumption MLA latent KV is unshardable across attention heads under head-parallel TP, so every TP rank stores and reads the full cache.
    Taken from engineering and architecture literature cited in §3.3; drives the capacity-wall gap between TP and DP attention.
  • domain assumption Decode goodput under a TPOT SLO is B/t_step over the feasible region bounded by B_max from residual HBM after weights and overhead.
    §3.5 capacity wall formula; treats capacity as first-class rather than post-hoc OOM check.
  • standard math MFU/MBU = I_work / I_ridge holds identically, so the utilization whose ceiling is 100% for the binding resource should be reported.
    Roofline identity restated in §2.2 from Williams et al. and practitioner MBU usage.
  • domain assumption Datasheet GPU rates plus cluster-calibrated collectives are acceptable optimistic floors; implementation gaps appear as residuals above the interval.
    §4 calibration philosophy; paper notes fully calibrated floors are 15–24% looser than datasheet.
  • domain assumption Open-loop, steady-state, tail-aware benchmarks are required for capacity claims; closed-loop load understates overload.
    §2.1 citing Schroeder et al.; shapes what measurements may be reconciled against floors.
invented entities (3)
  • Floor First residual-driven triage workflow no independent evidence
    purpose: Controller that sequences SLO definition, analytical floors, survivor benchmarks, residual gating, and limited profiler questions before kernel work.
    The paper’s primary methodological object (Figure 1, §2.3); independent evidence would be the planned residual-calibration campaign showing fewer wasted Nsight sessions.
  • Five-dimensional decode resource vector (HBM bytes, FLOPs, network bytes, network messages, KV capacity) no independent evidence
    purpose: Uniform account from which [max,sum] floors and wall ordering are computed, including capacity as a first-class dimension.
    Compositional interface in §3.1 and §3.6; related work uses similar decompositions but not this exact five-tuple with capacity in the vector.
  • Wall ordering as primary deployment comparator no independent evidence
    purpose: Rank layouts by which resource wall binds first as load grows rather than by same-batch point latency.
    §5.3–5.6 case-study method that yields the operating-point-dependent TP vs EP+DP judgment.

reviewed 2026-07-11 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Think Before You Grid-Search: Floor-First Triage for LLM Serving." pith.science (2026). https://pith.science/paper/IJGGTCBP

@misc{pith2026260705876,
  author       = {Pith},
  title        = {Pith review of: Think Before You Grid-Search: Floor-First Triage for LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJGGTCBP}},
  note         = {Machine review of arXiv:2607.05876}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

LLM serving optimization typically benchmarks many configurations and reaches for heavy profilers when latency targets are missed. We argue for the reverse discipline: estimation is the analytical layer of profiling -- without it, optimization degenerates to grid search. Floor First is a residual-driven triage workflow. Each decode step is modeled as a five-dimensional resource vector (HBM bytes, FLOPs, network bytes, network messages, KV capacity); summing within a resource and maximizing across resources gives an optimistic floor, the plain sum a pessimistic one. Where a measurement lands inside this [max, sum] interval reads out overlap quality before any profiler is opened, and profilers escalate only on residuals above a stated threshold. Deployment alternatives are compared by wall ordering -- which resource wall binds first as load grows -- rather than by point benchmarks. The account is compositional: new attention or state-space variants enter by declaring one module, and the workflow ships as a zero-dependency calculator plus an agent skill that enforces the discipline in agentic optimization loops. As a case study we analyze a DeepSeek-V3.2-style 671B MoE/MLA model on 16 NVIDIA H20 GPUs, whose ridge point of ~74 FLOP/byte (vs ~590 for H100) makes it an extreme decode-oriented part. The floors show TP16 decoding is KV-capacity-limited to ~70 concurrent 8K requests; sparse attention removes the KV-bandwidth term but not the capacity wall; an EP16+DP-attention layout accepts slightly worse same-batch weight traffic for an order-of-magnitude higher capacity wall (~644) -- while single-stream latency favors TP by 2.4x. The layout judgment is thus a computable function of the operating point, explaining why production deployments on identical hardware have shipped opposite attention layouts.

Figures

Figures reproduced from arXiv: 2607.05876 by Yihua Liu.

Figure 1
Figure 1. Figure 1: The serving-optimization loop. The floor model decides whether the next action should be lay [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FP8 rooflines of H20 and H100 with the case study’s operating points. Decode at [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Analytical TPOT floors (top) and goodput ceilings (bottom) versus concurrency at 8K context, [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 49 canonical work pages · 28 internal anchors

  1. [1]

    Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in llm inference with sarathi-serve.arXiv preprint arXiv:2403.02310, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    Vidur: A Large-Scale Simulation Framework For LLM Inference

    Amey Agrawal et al. Vidur: A large-scale simulation framework for llm inference.arXiv preprint arXiv:2405.05465, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles

    Moiz Arif, Avinash Maurya, Sudharshan Vazhkudai, and Bogdan Nicolae. Understanding inference scaling for llms: Bottlenecks, trade-offs, and performance principles.arXiv preprint arXiv:2605.19775, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  4. [4]

    Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models

    Abhimanyu Bambhaniya, Ritik Raj, Geonhwa Jeong, Souvik Kundu, Sudarshan Srinivasan, Suvinay Subramanian, Midhilesh Elavazhagan, Madhu Kumar, and Tushar Krishna. Demystifying ai platform design for distributed inference of next-generation llm models.arXiv preprint arXiv:2406.01698, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Rooflinebench: A benchmarking framework for on-device llms via roofline analysis.arXiv preprint arXiv:2602.11506, 2026

    Zhen Bi, Xueshu Chen, Luoyang Sun, Yuhang Yao, Qing Shen, Jungang Lou, and Cheng Deng. Rooflinebench: A benchmarking framework for on-device llms via roofline analysis.arXiv preprint arXiv:2602.11506, 2026

    work page arXiv 2026

  6. [6]

    Inference performance evaluation for LLMs on edge devices with a novel benchmarking framework and metric

    Hao Chen, Cong Tian, Zixuan He, Bin Yu, Yepang Liu, and Jialun Cao. Inference performance eval- uation for llms on edge devices with a novel benchmarking framework and metric.arXiv preprint arXiv:2508.11269, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode

    Josef Chen. Memory-bound but not bandwidth-limited: The physical ai inference gap in batch-1 llm decode.arXiv preprint arXiv:2605.30571, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  8. [8]

    Llm inference performance engineering: Best practices.https://www

    Databricks Engineering. Llm inference performance engineering: Best practices.https://www. databricks.com/blog/llm-inference-performance-engineering-best-practices, 2023. Intro- duces the model bandwidth utilization (MBU) metric

    work page 2023

  9. [9]

    Liminal: Exploring the frontiers of llm decode performance.arXiv preprint arXiv:2507.14397, 2025

    Michael Davies, Neal Crago, Karthikeyan Sankaralingam, and Christos Kozyrakis. Liminal: Exploring the frontiers of llm decode performance.arXiv preprint arXiv:2507.14397, 2025

    work page arXiv 2025

  10. [10]

    DeepSeek-V3 Technical Report

    DeepSeek-AI. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024. 13

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Deepseek-v3/r1 inference system overview and profiling data.https://github.com/ deepseek-ai/profile-data, 2025

    DeepSeek-AI. Deepseek-v3/r1 inference system overview and profiling data.https://github.com/ deepseek-ai/profile-data, 2025. Production deployment: prefill EP32, decode EP144 with data- parallel attention on H800

    work page 2025

  12. [12]

    DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models

    DeepSeek-AI. Deepseek-v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUs

    Pengcuo Dege, Qiuming Luo, Rui Mao, and Chang Kong. Flashmla-etap: Efficient transpose attention pipeline for accelerating mla inference on nvidia h20 gpus.arXiv preprint arXiv:2506.01969, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  14. [14]

    Asap: an agentic solution to auto- optimize performance of large-scale llm training.arXiv preprint arXiv:2511.03844, 2025

    Yuran Ding, Xinwei Chen, Xiaofan Zhang, and Zongwei Zhou. Asap: an agentic solution to auto- optimize performance of large-scale llm training.arXiv preprint arXiv:2511.03844, 2025

    work page arXiv 2025

  15. [15]

    Inference economics of language models

    Ege Erdil. Inference economics of language models.arXiv preprint arXiv:2506.04645, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    Deepseek v3/r1 inference efficiency analysis (in chinese)

    Jiarui Fang. Deepseek v3/r1 inference efficiency analysis (in chinese). https://zhuanlan.zhihu.com/p/16445683081, 2025. Reverse-engineered roofline account of the official H800 deployment; companion tool LLMRoofline

    work page arXiv 2025

  17. [17]

    Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization

    Jiading Gai, Shuai Zhang, Kaj Bostrom, Jin Huang, Vihang Patil, Haoyang Fang, Bernie Wang, Huzefa Rangwala, and George Karypis. Optimizing cuda like a human: Micro-profiling tools as expert surro- gates for llm-based gpu kernel optimization.arXiv preprint arXiv:2606.26453, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  18. [18]

    Moesd: Unveil speculative decoding’s potential for accelerating sparse moe.arXiv preprint arXiv:2505.19645, 2025

    Zongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, and Tianyu Zhang. Moesd: Unveil speculative decoding’s potential for accelerating sparse moe.arXiv preprint arXiv:2505.19645, 2025

    work page arXiv 2025

  19. [19]

    Moe-cap: Benchmarking cost, accuracy and performance of sparse mixture- of-experts systems.arXiv preprint arXiv:2412.07067, 2024

    Yinsicheng Jiang, Yao Fu, Yeqi Huang, Ping Nie, Zhan Lu, Leyang Xue, Congjie He, Man-Kit Sit, Jilong Xue, Li Dong, et al. Moe-cap: Benchmarking cost, accuracy and performance of sparse mixture- of-experts systems.arXiv preprint arXiv:2412.07067, 2024

    work page arXiv 2024

  20. [20]

    Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference

    Joyjit Kundu, Wenzhe Guo, Ali BanaGozar, Udari De Alwis, Sourav Sengupta, Puneet Gupta, and Arindam Mallik. Performance modeling and workload analysis of distributed large language model training and inference.arXiv preprint arXiv:2407.14645, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services

    Ma lgorzata Lazuka, Andreea Anghel, and Thomas Parnell. Llm-pilot: Characterize and optimize per- formance of your llm inference services.arXiv preprint arXiv:2410.02425, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    From Human Guidance to Autonomy: Agent Skill System for End-to-End LLM Deployment on Spatial NPUs

    Jiajie Li, Erwei Wang, Zhiru Zhang, and Samuel Bayliss. From human guidance to autonomy: Agent skill system for end-to-end llm deployment on spatial npus.arXiv preprint arXiv:2606.07586, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  23. [23]

    Epoch: An agentic protocol for multi-round system optimization.arXiv preprint arXiv:2603.09049, 2026

    Zhanlin Liu, Yitao Li, and Munirathnam Srikanth. Epoch: An agentic protocol for multi-round system optimization.arXiv preprint arXiv:2603.09049, 2026

    work page arXiv 2026

  24. [24]

    Serving deepseek-r1 on h20-96g.https://www.lmsys.org/blog/ 2025-09-26-sglang-ant-group/, 2025

    LMSYS Org and Ant Group. Serving deepseek-r1 on h20-96g.https://www.lmsys.org/blog/ 2025-09-26-sglang-ant-group/, 2025. Decode uses TP16 attention with EP16 MoE on 2×8 H20-96G

    work page 2025

  25. [25]

    GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding

    Fanxu Meng. Gqla: Group-query latent attention for hardware-adaptive large language model decoding. arXiv preprint arXiv:2605.15250, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  26. [26]

    Splitwise: Efficient generative LLM inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Inigo Goiri, Saeed Maleki, and Ri- cardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting.arXiv preprint arXiv:2311.18677, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  27. [27]

    Efficiently Scaling Transformer Inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  28. [28]

    Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xin- ran Xu. Mooncake: A kvcache-centric disaggregated architecture for llm serving.arXiv preprint arXiv:2407.00079, 2024. 14

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  29. [29]

    Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference

    Pol G. Recasens, Ferran Agullo, Yue Zhu, Chen Wang, Eun Kyung Lee, Olivier Tardieu, Jordi Torres, and Josep Ll. Berral. Mind the memory gap: Unveiling gpu bottlenecks in large-batch llm inference. arXiv preprint arXiv:2503.08311, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  30. [30]

    Open versus closed: A cautionary tale

    Bianca Schroeder, Adam Wierman, and Mor Harchol-Balter. Open versus closed: A cautionary tale. In 3rd Symposium on Networked Systems Design and Implementation (NSDI 06), 2006

    work page 2006

  31. [31]

    Sglang v0.4: Zero-overhead batch scheduler, cache-aware load balancer, faster struc- tured outputs.https://lmsys.org/blog/2024-12-04-sglang-v0-4/, 2024

    SGLang Team. Sglang v0.4: Zero-overhead batch scheduler, cache-aware load balancer, faster struc- tured outputs.https://lmsys.org/blog/2024-12-04-sglang-v0-4/, 2024. Introduces data-parallel attention for MLA: head-parallel TP duplicates the latent KV cache

    work page 2024

  32. [32]

    Step-3 is large yet affordable: Model-system co-design for cost-effective decoding.arXiv preprint arXiv:2507.19427, 2025

    StepFun. Step-3 is large yet affordable: Model-system co-design for cost-effective decoding.arXiv preprint arXiv:2507.19427, 2025

    work page arXiv 2025

  33. [33]

    Kernelskill: A multi-agent framework for gpu kernel optimization.arXiv preprint arXiv:2603.10085, 2026

    Qitong Sun, Jun Han, Tianlin Li, Zhe Tang, Sheng Chen, Fei Yang, Aishan Liu, Xianglong Liu, and Yang Liu. Kernelskill: A multi-agent framework for gpu kernel optimization.arXiv preprint arXiv:2603.10085, 2026

    work page arXiv 2026

  34. [34]

    TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    Xiaojuan Tang, Fanxu Meng, Pingzhi Tang, Yuxuan Wang, Di Yin, Xing Sun, and Muhan Zhang. Tpla: Tensor parallel latent attention for efficient disaggregated prefill and decode inference.arXiv preprint arXiv:2508.15881, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    Xshare: Collaborative in-batch expert sharing for faster moe inference.arXiv preprint arXiv:2602.07265, 2026

    Daniil Vankov, Nikita Ivkin, Kyle Ulrich, Xiang Song, Ashish Khetan, and George Karypis. Xshare: Collaborative in-batch expert sharing for faster moe inference.arXiv preprint arXiv:2602.07265, 2026

    work page arXiv 2026

  36. [36]

    RFC: Data parallel attention and expert parallel moes.https://github.com/ vllm-project/vllm/issues/16037, 2025

    vLLM Project. RFC: Data parallel attention and expert parallel moes.https://github.com/ vllm-project/vllm/issues/16037, 2025

    work page 2025

  37. [37]

    Roofline: An insightful visual performance model for multicore architectures.Communications of the ACM, 52(4):65–76, 2009

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline: An insightful visual performance model for multicore architectures.Communications of the ACM, 52(4):65–76, 2009

    work page 2009

  38. [38]

    How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving

    Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee, Tuhin Khare, Sudarshan Srinivasan, Suvinay Subramanian, Souvik Kundu, Madhu Kumar, Midhilesh Elavazhagan, William Won, Amir Yazdanbakhsh, and Tushar Krishna. How far can disaggregation go? a design-space explo- ration of attention-ffn disaggregation for efficient moe llm serving.arXiv prep...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  39. [39]

    Aiconfigurator: Lightning-fast configuration optimization for multi- framework llm serving.arXiv preprint arXiv:2601.06288, 2026

    Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yiyi Chen, Yi Shen, Qin Zhou, Xumeng Chen, et al. Aiconfigurator: Lightning-fast configuration optimization for multi- framework llm serving.arXiv preprint arXiv:2601.06288, 2026

    work page arXiv 2026

  40. [40]

    KernelSight-LM: A Kernel-Level LLM Inference Simulator

    Xiteng Yao, Taeho Kim, Hengzhi Pei, Xinle Liu, Kyle Ulrich, Leonard Lausen, Ashish Khetan, Xiang Song, George Karypis, and Martin Herbordt. Kernelsight-lm: A kernel-level llm inference simulator. arXiv preprint arXiv:2606.28565, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  41. [41]

    LLM Inference Unveiled: Survey and Roofline Model Insights

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, et al. Llm inference unveiled: Survey and roofline model insights.arXiv preprint arXiv:2402.16363, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    Rethinking llm inference bottlenecks: Insights from latent attention and mixture-of-experts.arXiv preprint arXiv:2507.15465, 2025

    Sungmin Yun, Seonyong Park, Hwayong Nam, Younjoo Lee, Gunjun Lee, Kwanhee Kyung, Sangpyo Kim, Nam Sung Kim, Jongmin Kim, Hyungyo Kim, et al. Rethinking llm inference bottlenecks: Insights from latent attention and mixture-of-experts.arXiv preprint arXiv:2507.15465, 2025

    work page arXiv 2025

  43. [43]

    Hardware-Efficient Attention for Fast Decoding

    Ted Zadouri, Hubert Strauss, and Tri Dao. Hardware-efficient attention for fast decoding.arXiv preprint arXiv:2505.21487, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  44. [44]

    A Hardware Evaluation Framework for Large Language Model Inference

    Hengrui Zhang, August Ning, Rohan Prabhakar, and David Wentzlaff. A hardware evaluation framework for large language model inference.arXiv preprint arXiv:2312.03134, 2023. 15

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  45. [45]

    Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures.arXiv preprint arXiv:2505.09343, 2025

    Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, et al. Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures.arXiv preprint arXiv:2505.09343, 2025

    work page arXiv 2025

  46. [46]

    DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving.arXiv preprint arXiv:2401.09670, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  47. [47]

    Mixserve: An automatic distributed serving system for moe models with hybrid parallelism based on fused communication algorithm.arXiv preprint arXiv:2601.08800, 2026

    Bowen Zhou, Jinrui Jia, Wenhao He, Yong Zhang, and Fang Dong. Mixserve: An automatic distributed serving system for moe models with hybrid parallelism based on fused communication algorithm.arXiv preprint arXiv:2601.08800, 2026

    work page arXiv 2026

  48. [48]

    MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference

    Ruijie Zhou, Fanxu Meng, Yufei Xu, Tongxuan Liu, Guangming Lu, Muhan Zhang, and Wen- jie Pei. Misa: Mixture of indexer sparse attention for long-context llm inference.arXiv preprint arXiv:2605.07363, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  49. [49]

    MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, et al. Megascale-infer: Serving mixture-of-experts at scale with disaggregated expert parallelism.arXiv preprint arXiv:2504.02263, 2025. A Artifact Commands The following commands reproduce Table 2 (add--full-expertsfor the co...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

This paper was first reviewed by grok-4.5 on July 11, 2026.