Think Before You Grid-Search: Floor-First Triage for LLM Serving

Yihua Liu

arxiv: 2607.05876 · v2 · pith:IJGGTCBP · submitted 2026-07-07 · cs.PF · cs.AI· cs.DC

Think Before You Grid-Search: Floor-First Triage for LLM Serving

Yihua Liu This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 21:44 UTCglm-5.2pith:IJGGTCBP record.json open to challenge →

classification cs.PF cs.AIcs.DC

keywords wallattentioncapacityoptimizationpointresourcebeforebenchmarks

0 comments

The pith

Compute the floor before you benchmark: a triage workflow for LLM serving

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that LLM serving optimization should begin with a five-resource analytical floor (HBM bytes, FLOPs, network bytes, network messages, KV capacity) computed from model dimensions and hardware rates, not with grid search or profiler sessions. The workflow produces two bounds per decode step: an optimistic floor (max across independent resources, assuming perfect overlap) and a pessimistic floor (plain sum, assuming no overlap). Where a benchmark measurement lands inside this [max, sum] interval is a zero-cost diagnostic of overlap quality. If the measurement sits within a 1.3x escalation threshold of the optimistic floor, profiling stops; if it exceeds the interval entirely, profilers escalate with a specific three-question reading list. Deployment alternatives are compared by wall ordering, which resource limit binds first as concurrency grows, rather than by point benchmarks. The case study applies this to a 671B MoE/MLA model on 16 H20 GPUs, showing that the choice between tensor-parallel (TP16) and expert-parallel-plus-data-parallel-attention (EP16+DP) layouts is a computable function of the target operating point: TP16 is KV-capacity-limited to about 70 concurrent 8K requests while EP16+DP raises that wall to about 644, yet at batch size 1 TP wins single-stream latency by 2.4x because its all-reduce latency is lower than the all-to-all latency EP requires. This explains why production teams on identical hardware shipped opposite attention layouts.

Core claim

The central object is the two-sided floor: for each decode step, resources contending for the same hardware add, while independent engines (HBM, compute, network) take the max under perfect overlap, giving an optimistic bound; the plain sum gives a pessimistic bound. The interval between them is an overlap diagnostic that costs nothing to compute. The second key object is wall ordering: rather than comparing layouts by same-batch latency, compare which resource wall (KV capacity, KV bandwidth, weight bandwidth, compute) binds first as load grows. For the H20 case study, the H20's ridge point of approximately 74 FLOP/byte (versus 590 for H100) makes it an extreme decode-oriented part, and the

What carries the argument

Two-sided floor: [max, sum] interval from a five-dimensional resource vector (HBM bytes, FLOPs, network bytes, network messages, KV capacity). Wall ordering: which resource wall binds first as concurrency increases. Roofline identity: MFU/MBU = I_work/I_ridge, dictating which utilization metric is meaningful per phase. MoE union correction: weight bytes follow the union of experts activated across the batch, not the per-token activated count. MLA replication cost: latent KV cannot shard across TP heads, making DP attention structurally favored at high concurrency.

If this is right

Teams deploying MLA-based MoE models can determine whether TP or EP+DP attention is correct for their concurrency target using a page of arithmetic before spending GPU time on benchmarks.
The H20 ridge point characterization (74 FLOP/byte vs 590 for H100) implies that export-compliant GPUs are naturally suited to decode workloads and weak for prefill, with direct consequences for prefill/decode disaggregation decisions.
Sparse attention methods that reduce KV reads (like DSA) do not raise the KV capacity wall, meaning they extend context length without extending concurrency, a distinction invisible to same-batch latency benchmarks.
The workflow's compositional module interface means new attention variants or state-space layers can be evaluated for deployment fitness by declaring five quantities, without re-deriving the framework.

Load-bearing premise

The triage thresholds (the 1.3x escalation factor and the MBU bands at 70%/40%) are stated as practitioner calibration targets, not derived constants. The paper explicitly lists the residual calibration sweep as a required experiment that has not yet been performed. If these thresholds are wrong, the workflow either escalates to profiling when it should stop, or misses actionable residuals that warrant investigation.

What would settle it

Measure steady-state TPOT for TP16 and EP16+DP configurations across a sweep of batch sizes and context lengths on 16xH20. If the measured goodput frontier does not show TP16 dominating at low concurrency and EP16+DP dominating at high concurrency, with a crossover point predictable from the capacity wall arithmetic, the wall-ordering methodology does not correctly capture the deployment tradeoff.

Figures

Figures reproduced from arXiv: 2607.05876 by Yihua Liu.

**Figure 2.** Figure 2: FP8 rooflines of H20 and H100 with the case study’s operating points. Decode at [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Figure 3: Analytical TPOT floors (top) and goodput ceilings (bottom) versus concurrency at 8K context, [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

read the original abstract

LLM serving optimization typically benchmarks many configurations and reaches for heavy profilers when latency targets are missed. We argue for the reverse discipline: estimation is the analytical layer of profiling -- without it, optimization degenerates to grid search. Floor First is a residual-driven triage workflow. Each decode step is modeled as a five-dimensional resource vector (HBM bytes, FLOPs, network bytes, network messages, KV capacity); summing within a resource and maximizing across resources gives an optimistic floor, the plain sum a pessimistic one. Where a measurement lands inside this [max, sum] interval reads out overlap quality before any profiler is opened, and profilers escalate only on residuals above a stated threshold. Deployment alternatives are compared by wall ordering -- which resource wall binds first as load grows -- rather than by point benchmarks. The account is compositional: new attention or state-space variants enter by declaring one module, and the workflow ships as a zero-dependency calculator plus an agent skill that enforces the discipline in agentic optimization loops. As a case study we analyze a DeepSeek-V3.2-style 671B MoE/MLA model on 16 NVIDIA H20 GPUs, whose ridge point of ~74 FLOP/byte (vs ~590 for H100) makes it an extreme decode-oriented part. The floors show TP16 decoding is KV-capacity-limited to ~70 concurrent 8K requests; sparse attention removes the KV-bandwidth term but not the capacity wall; an EP16+DP-attention layout accepts slightly worse same-batch weight traffic for an order-of-magnitude higher capacity wall (~644) -- while single-stream latency favors TP by 2.4x. The layout judgment is thus a computable function of the operating point, explaining why production deployments on identical hardware have shipped opposite attention layouts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The analytical framework for LLM serving layout decisions is sound and reproducible; the workflow validation is not yet done but the core contributions stand independently.

read the letter

The main thing to know: the analytical core of this paper is solid and reproducible. The capacity wall arithmetic checks out — with 96GB HBM, ~42GB weights under TP16, ~70KB KV/token, and reasonable overhead, the B_max values of 70 and 644 reproduce from the stated constants. The wall ordering (KV capacity binds at B≈70, compute knee at B≈225) is robust because it depends on the sparsity ratio 671/37≈18× and the H20 ridge point of 74 FLOP/byte, both well-established. The single-stream inversion (TP wins by 2.4× at B=1) also holds up: TP16's 122 all-reduces at 33μs give 4.0ms, EP16+DP's 116 all-to-alls at 60μs give 7.0ms, and the 8× weight traffic difference adds the rest. Even with optimistic all-to-all latency at the 30μs transport floor, TP still wins at B=1 because EP16+DP's per-GPU weight traffic is 8× higher. The qualitative inversion is robust to communication constant perturbation — I checked this and it holds up under reasonable perturbation of the communication constants. The stress-test concern about the explanatory claim (that the framework explains why LMSYS-Ant shipped TP16 while DeepSeek shipped DP) is narrower than it first appears. The paper does not verify that the two deployments actually target different operating points — it infers this from the layout choice itself. If both ran at similar concurrency, the explanatory claim weakens. But the analytical framework doesn't depend on this; the layout judgment as a function of operating point stands regardless. The real soft spot is the workflow claim. The triage thresholds (1.3× escalation, MBU bands at 70%/40%) are practitioner defaults, and Section 8 explicitly lists the residual calibration sweep as planned but not performed. The paper is transparent about this — it's a draft that says what's done and what isn't. The artifact is not yet public, and the HBM calibration used a torch fallback rather than nvbandwidth. These are real gaps for the workflow's practical utility, but they don't touch the analytical contributions. The two-sided [max, sum] floor model that keeps overlap observable is a genuine improvement over prior work (LIMINAL [9]) that fixes overlap assumptions. The wall-ordering framework and the capacity-first DP-attention derivation for unmodified MLA are new and correct. The MoE union correction (Section 3.3) is a clean application of an existing closed form to a new problem. The H20 ridge-point characterization is a useful data point that I haven't seen elsewhere. This paper is for practitioners and researchers working on LLM serving deployment decisions who need analytical grounding before benchmarking. It deserves a serious referee who can verify the arithmetic and push on the workflow validation plan. The analytical contributions are ready for review; the workflow validation is honestly flagged as incomplete.

Referee Report

1 major / 7 minor

Summary. The paper presents Floor First, a residual-driven triage workflow for LLM serving optimization. The core methodology models each decode step as a five-dimensional resource vector (HBM bytes, FLOPs, network bytes, network messages, KV capacity), computes two-sided floors [max, sum], and uses wall ordering to compare deployment alternatives. A case study on a DeepSeek-V3.2-style 671B MoE/MLA model on 16×H20 GPUs derives that TP16 is KV-capacity-limited to ~70 concurrent 8K requests while EP16+DP raises the capacity wall to ~644, yet TP wins single-stream latency by 2.4×, making the layout choice a computable function of the operating point. The paper ships a zero-dependency Python artifact and an agent-readable skill document.

Significance. The paper makes several genuine contributions. The two-sided floor [max, sum] that preserves overlap as an observable quantity (rather than fixing an assumption a priori) is a clean methodological improvement over existing analytical models. The H20 ridge-point characterization (~74 FLOP/byte vs. ~590 for H100) and its deployment consequences appear novel. The wall-ordering framework, treating capacity as a first-class resource dimension rather than a post-hoc OOM check, is well-motivated. The MoE union correction (§3.3) is properly attributed to prior work [18, 35] and correctly applied to deployment sizing. The capacity wall arithmetic in Table 2 is internally consistent and reproducible from the artifact. The zero-dependency calculator and the agent skill artifact are concrete deliverables. The single-stream inversion result (2.4× in favor of TP at B=1) is robust to communication constant perturbation, as the weight traffic difference alone (2.4 GB vs. 19.3 GB) is decisive.

major comments (1)

§5.6 and §1: The paper claims the framework 'explains why production deployments on identical hardware have shipped opposite attention layouts,' citing the LMSYS–Ant deployment (TP16 attention) and the official DeepSeek deployment (DP attention). However, the paper never verifies that these two deployments actually operate at different points on the concurrency frontier. The analytical result shows that TP wins at low concurrency and DP wins at high concurrency; the explanatory claim requires evidence that LMSYS–Ant targets low concurrency while DeepSeek targets high concurrency. Without this verification, the framework is consistent with the production disagreement but does not yet explain it. This is load-bearing for the paper's framing in §1 and the abstract ('explaining why production deployments on identical hardware have shipped opposite attention layouts'). At minimum, the paper应该

minor comments (7)

§3.4, Eq. for t_AR: The all-reduce traffic formula uses B·H·b_act, but the notation b_act is not defined. It presumably refers to the activated hidden dimension per token, but this should be stated explicitly.
§5.3, Table 3: The compute wall knee is stated as 'B*_GEMM ≈ 670' and 'incl. 8K MLA ~225.' The derivation of these specific values is not shown in the text or the artifact commands. A brief derivation or a reference to the specific artifact output would help reproducibility.
§4: The calibration fallback (torch-level device copy for HBM, production DeepGEMM kernels for FP8) is mentioned, but the paper states 'the HBM point should be re-taken with nvbandwidth in place of the torch fallback.' Since this is a known measurement gap, the paper should note the expected direction of the bias (is the torch fallback likely over- or under-estimating HBM bandwidth?) so readers can assess the sensitivity of the floors.
§5.7: The reconciliation example uses 25 ms and 45 ms as hypothetical measurements. While illustrative, at least one real measured data point (even from the calibration campaign that was executed) would strengthen the workflow demonstration. The calibration data mentioned in §4 is said to exist but is not shown.
Figure 3: The y-axis label 'goodput ceiling (tok/s)' in the bottom panel could be confused with the TPOT floors in the top panel. A brief caption note clarifying that the bottom panel is B_max / t_step would help.
§7, point (ii): The paper notes that expert imbalance makes the union expectation optimistic, but does not quantify the sensitivity. A brief statement of how much hot-expert skew would shift the weight-bandwidth wall (e.g., '10% of experts receiving 2× traffic shifts the wall by X') would help practitioners assess risk.
References: Several arXiv identifiers use future-dated IDs (e.g., [3] arXiv:2605.19775, [7] arXiv:2605.30571). If these are genuine preprints, the dates should be verified; if placeholder, they should be corrected before publication.

Simulated Author's Rebuttal

1 responses · 0 unresolved

The referee identifies a genuine gap: the paper claims to 'explain' the LMSYS–Ant vs. official DeepSeek production disagreement but does not verify that the two deployments actually operate at different concurrency points. We agree this is a load-bearing framing issue and will revise.

read point-by-point responses

Referee: §5.6 and §1: The paper claims the framework 'explains why production deployments on identical hardware have shipped opposite attention layouts,' citing the LMSYS–Ant deployment (TP16 attention) and the official DeepSeek deployment (DP attention). However, the paper never verifies that these two deployments actually operate at different points on the concurrency frontier. The analytical result shows that TP wins at low concurrency and DP wins at high concurrency; the explanatory claim requires evidence that LMSYS–Ant targets low concurrency while DeepSeek targets high concurrency. Without this verification, the framework is consistent with the production disagreement but does not yet explain it.

Authors: The referee is correct. The paper derives an analytical frontier showing that TP16 wins at low concurrency and EP16+DP wins at high concurrency, and then claims this 'explains' the production disagreement between LMSYS–Ant (TP16) and official DeepSeek (DP). But we never verify that the two deployments actually sit at different operating points on that frontier. The analytical result shows the framework is *consistent* with the production disagreement; it does not yet show the deployments *are* at the points the framework predicts. This is a real gap between what the analysis proves and what the framing claims. We will revise in two ways. First, we will soften the claim throughout: the abstract and §1 will say the framework is 'consistent with' or 'provides a candidate explanation for' the production disagreement, not that it 'explains' it. Second, we will add a subsection (or expand §5.6) that assembles the available public evidence about the two deployments' operating points. Specifically: (a) the official DeepSeek deployment uses EP144 with DP attention on H800, which is a large-scale serving system designed for high concurrency — the profile data they publish [11] shows batch sizes well beyond the TP16 capacity wall of ~70, placing them in the DP-favored regime by construction. (b) The LMSYS–Ant deployment [24] is a public-facing chatbot service (DeepSeek-R1 on lmsys.org) where single-stream and low-concurrency latency is the primary SLO target, placing it in the TP-favored regime. We acknowledge that this evidence is circumstantial rather than a direct measurement of both deployments' concurrency distributions, and we will state that limitation explicitly. The fully convincing version — measuring both deployments' operating points on the same frontier — is listed as revision: no

Circularity Check

0 steps flagged

No circularity found: the derivation chain is self-contained with no self-citations

full rationale

I walked the paper's full derivation chain and found no circular steps. The floor calculations (§3.1–3.5) are parameter-free arithmetic from model dimensions, hardware specs, and calibrated communication rates — the outputs (B_max, TPOT floors, wall ordering) do not reduce to their inputs by construction in any hidden way. The MoE union correction (§3.3) uses a standard probability formula f_expert(B) = 1-(1-k/E)^B cited from [18, 35], which are external authors (Huang et al., Vankov et al.) — not self-citations. The roofline identity MFU/MBU = I_work/I_ridge (§2.2) is an algebraic identity, but the paper presents it as such and uses it for metric selection guidance, not as a prediction. The triage thresholds (1.3× escalation, 70%/40% MBU bands) are explicitly stated as 'calibration targets, not constants of nature' (§2.4) with a stated calibration protocol — they are not fitted to data and then presented as derived results. The deployment explanation (§5.6) is an interpretive claim that rests on an unverified assumption about production operating points, but that is a correctness/completeness gap, not circularity. The author Yihua Liu has no self-citations in the reference list. The case study numbers reproduce from a zero-dependency artifact with stated commands (Appendix A). This is a self-contained analytical paper with honest epistemic labeling of its empirical components.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical entities, particles, forces, or dimensions. The five-dimensional resource vector is a modeling framework, not a postulated physical object. The agent skill (SKILL.md) is a software artifact, not an invented entity. All axioms are domain assumptions grounded in hardware behavior or standard roofline theory. The free parameters are calibration constants and practitioner thresholds, all explicitly flagged as such.

free parameters (6)

Escalation threshold (1.3x) = 1.3
Stated in Section 2.4 as a practitioner default; determines when profiling is warranted. Not derived from first principles.
MBU triage bands = 70% / 40%
Section 2.4 systematizes practitioner experience into bands (near-floor / overlap-recovery / system-defect). Stated as calibration targets, not constants of nature.
MoE prefill MFU bands = 50% / 25%
Section 2.4 lowers prefill bands for MoE due to structural all-to-all and expert imbalance. Practitioner-derived.
Calibrated all-reduce effective bandwidth = 43 GB/s
Section 4: cluster-calibrated against 100 GB/s line rate. Used in network time calculations for Table 2.
Small-message latency = 33 microseconds
Section 4: measured per-op latency for all-reduce, identified as NCCL launch overhead. Used in single-stream floor (Section 5.6).
DeepEP dispatch latency = 60 microseconds
Section 4: measured op-level dispatch latency for all-to-all. Used in EP16+DP single-stream bound (Section 5.6).

axioms (5)

domain assumption Decode steps can be modeled as a five-dimensional resource vector (HBM bytes, FLOPs, network bytes, network messages, KV capacity) with additive contention within a resource and maximization across independent resources.
Section 3.1. The aggregation rules (add within resource, max across) assume perfect overlap for the optimistic floor and zero overlap for the pessimistic floor. This is the structural basis of the two-sided floor model.
domain assumption MoE expert routing is uniform, enabling the closed-form union expectation f_expert(B) = 1 - (1 - k/E)^B.
Section 3.3. The paper acknowledges this is optimistic under load imbalance (Section 7, item ii). The union correction affects weight-bandwidth and compute-wall calculations.
domain assumption MLA latent KV is unshardable across attention heads under head-parallel TP, requiring full replication per rank.
Section 3.3. Attributed to engineering channels [31, 36] and architecture work [25, 34, 43]. This is the structural fact that makes DP attention favored for high-concurrency MLA serving.
standard math The MFU/MBU = I_work/I_ridge identity holds and dictates which utilization metric is meaningful per phase.
Section 2.2. This is a standard roofline identity [37] applied to metric selection. Used to justify reporting MBU for bandwidth-bound decode and MFU for compute-bound prefill.
domain assumption At batch 1, the layer-serial dependency chain removes overlap opportunity, so the honest single-stream floor is t_sum not t_opt.
Section 3.1 caveat and Section 5.6. This assumption drives the single-stream bounds and the TP-vs-EP judgment inversion at low concurrency.

pith-pipeline@v1.1.0-glm · 19162 in / 3328 out tokens · 533140 ms · 2026-07-08T21:44:58.367019+00:00 · methodology

Think Before You Grid-Search: Floor-First Triage for LLM Serving

Core claim

What carries the argument

If this is right

Load-bearing premise

What would settle it

discussion (0)