arxiv: 2601.21351 · v3 · submitted 2026-01-29 · 💻 cs.LG · cs.AI

Recognition: 2 theorem links

· Lean Theorem

Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads

Chendong Song , Meixuan Wang , Hang Zhou , Hong Liang , Yuan Lyu , Zixi Chen , Yuwei Fan , Zijie Zhou

Authors on Pith no claims yet

Pith reviewed 2026-05-16 09:50 UTC · model grok-4.3

classification 💻 cs.LG cs.AI

keywords LLM servingdisaggregationattentionFFNprovisioningstochastic workloadsmean-field analysisrenewal theory

0 comments

The pith

A single workload statistic determines the optimal Attention-to-FFN provisioning ratio for disaggregated LLM serving via a closed-form mean-field rule.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes an analytical framework to provision Attention and FFN resources separately in LLM serving systems that disaggregate these computations. It models stochastic workloads using renewal-reward theory to find a governing statistic theta that works across different prompt and decode length distributions. This yields a simple rule for the best ratio of Attention to FFN workers, split into cases where attention, communication, or FFN is the limiting factor, plus a correction for synchronization delays between workers. If accurate, operators can set resource ratios from trace data without running full simulations for every workload. The approach addresses why disaggregated serving is sensitive to provisioning choices and how to get them right under randomness.

Core claim

Under stochastic workloads, the per-slot stationary token load in an Attention-FFN disaggregated setup is characterized by a renewal-reward process that depends on a single nonparametric statistic θ. This statistic admits an estimator from request traces and leads to a closed-form mean-field expression for the optimal A/F ratio, which decomposes into Attention-bottleneck, communication-bottleneck, and FFN-bottleneck regimes. A Gaussian approximation then refines the rule to account for the overhead of barriers imposed by the slowest Attention worker. The resulting predictions lie within 10% of the ratios that minimize latency in trace-calibrated simulations.

What carries the argument

The renewal-reward characterization of per-slot stationary token load, which reduces the entire provisioning problem to a single workload statistic θ.

If this is right

The optimal A/F ratio is given by a closed-form expression that changes depending on whether the bottleneck is Attention computation, inter-worker communication, or FFN computation.
A Gaussian barrier-aware refinement quantifies the additional overhead from synchronizing multiple Attention workers.
The framework applies to arbitrary prefill-decode length distributions through the nonparametric θ statistic.
The predicted ratio matches the simulation-optimal value within 10% across different workloads.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This provisioning rule could be used to dynamically adjust resource allocation in real-time serving clusters as workload statistics are observed.
The mean-field approach may generalize to other forms of model disaggregation, such as separating prefill from decode phases.
Trace-based estimation of θ enables operators to provision hardware without access to the full workload distribution.

Load-bearing premise

That a single nonparametric workload statistic fully governs the optimal provisioning ratio under any prefill and decode length distributions while the renewal-reward model of token load stays accurate as KV caches expand.

What would settle it

A trace-driven simulation in which the analytically predicted optimal A/F ratio differs from the ratio that actually minimizes average latency by more than 10 percent under a new workload distribution.

Figures

Figures reproduced from arXiv: 2601.21351 by Chendong Song, Hang Zhou, Hong Liang, Meixuan Wang, Yuan Lyu, Yuwei Fan, Zijie Zhou, Zixi Chen.

**Figure 1.** Figure 1: Architecture of AFD. Stateful Attention (A) layers manage the KV cache, while Feed-Forward Network (F) layers are stateless. During each decode step, every continuing request generates one output token whose key-value is appended to the KV cache (red blocks); when a request completes, its slot is immediately refilled with a new prefill request (green block). (a) Ideal microbatch (b) Issue raised after o… view at source ↗

**Figure 2.** Figure 2: Microbatch Pipelining and Masking. (a) An ideal schedule where Attention, Communication, and FFN are perfectly overlapped across microbatches, fully hiding data transfer latency. (b) After one decode step, Attention execution time increases (due to longer KV cache reads) while Communication and FFN remain unchanged, breaking the balanced overlap and introducing pipeline bubbles. practitioners typically c… view at source ↗

**Figure 3.** Figure 3: Empirical distributions of decode lengths from production LLM traces (Wang et al., 2023; 2024; Zheng et al., 2023; Zhao et al., 2024). Decode lengths exhibit a geometric (discreteexponential) pattern. 3.4. Optimization Objective Cycle time. The per-step cycle time is determined by the slowest component: τ (B; r) = max tA(T), tC (B), tF (rB) [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Figure 4: Per-instance throughput as a function of A/F ratio r with B = 256, µD = 500, µP = 100. Throughput increases with r until reaching the optimal point r ∗ ≈ 9.3, after which FFN becomes saturated and throughput per instance decreases. We notice a systematic gap emerges at large r: simulated throughput falls increasingly below the theoretical prediction, with the discrepancy reaching approximately 15% at r = … view at source ↗

**Figure 6.** Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

**Figure 5.** Figure 5: Attention idle ratio ηA (blue) and FFN idle ratio ηF (orange) as functions of A/F ratio r. The crossover point where ηA ≈ ηF indicates the balanced configuration. 5.4. Ablation Studies on Workload and Configuration Parameters We now investigate how the optimal A/F ratio r ∗ varies with key system parameters: microbatch size B, decode length distribution parameter p, and prefill length distribution paramet… view at source ↗

**Figure 7.** Figure 7: Impact of prefill, decode length µP , µD on per-instance throughput and optimal A/F ratio r ∗ . Solid lines denote simulation results; dashed lines denote theoretical predictions. Vertical dotted lines indicate the theoretical optimal r ∗ for each configuration. figurations with larger µP + µD achieve lower maximum throughput at their respective optimal r ∗ . This is because the per-token Attention cost gr… view at source ↗

**Figure 8.** Figure 8: Visualization of the latency models. Left: Attention latency tA(T) scales linearly with total token load T. Right: FFN latency tF (B) and communication latency tC (B) as functions of batch size B. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

read the original abstract

Attentio-FFN disaggregation (AFD) is an emerging architecture for LLM decoding that separates state-heavy, KV-cache-dominated Attention computation from stateless, compute-intensive FFN computation, connected by per-step communication. While AFD enables independent scaling of memory and compute resources, its performance is highly sensitive to the Attention/FFN provisioning ratio: mis-sizing induces step-level blocking and costly device idle time. We develop an analytical provisioning framework for AFD bundles in an $r$A--$1$F topology under stochastic workloads. Two sources of randomness shape the problem: per-slot Attention workload evolves as KV caches grow and completed requests are replenished with random prompt and decode lengths, and synchronized execution across Attention workers introduces a barrier governed by the slowest worker. We address both via a renewal-reward characterization of the per-slot stationary token load, identifying a single workload statistic $\theta$ that governs provisioning under arbitrary prefill-decode distributions and admits a nonparametric estimator from request traces. The analysis yields a closed-form mean-field rule for the optimal A/F ratio decomposing into Attention-, communication-, and FFN-bottleneck regimes, together with a Gaussian barrier-aware refinement that quantifies cross-worker synchronization overhead. A trace-calibrated AFD simulator supports the framework across workloads: the predicted optimal ratio matches the simulation-optimal within 10%. Together, these results provide a compact, calibratable account of how stochastic workload structure determines provisioning in disaggregated LLM serving.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gives a closed-form renewal-reward rule for A/F ratios in disaggregated LLM serving that matches simulation within 10%, but the stationarity assumption under KV growth needs explicit checking.

read the letter

This paper derives a closed-form mean-field rule for the optimal Attention/FFN provisioning ratio in disaggregated LLM serving. It models per-slot token load via renewal-reward processes, reduces the problem to a single nonparametric statistic theta estimated from traces, and splits the rule into Attention, communication, and FFN bottleneck regimes plus a Gaussian correction for worker barriers. The predicted ratio matches their simulator within 10% across tested workloads. That analytical step is new relative to the simulation-heavy literature on LLM serving systems. The nonparametric estimator is practical because it avoids fitting to the target ratio itself. The decomposition into regimes is a clean way to see how workload structure drives sizing decisions. The main soft spot is the stationarity claim. The renewal-reward characterization assumes the per-slot load distribution is fully captured by theta and remains independent across workers once the process settles. When KV caches grow monotonically inside a request and new requests arrive with random lengths, the slowest worker's state can correlate with barrier time and feed back into the FFN load. The abstract does not list the ergodicity conditions or show a counter-example where this coupling appears, so the 10% match may hold only in the lower-variance traces they used. A broader validation set with explicit error analysis would strengthen the result. The paper is aimed at systems researchers who size large inference clusters and want an analytical shortcut instead of repeated full simulations. Readers who already use renewal theory or mean-field approximations in queuing models will follow it easily. It deserves peer review. The derivation is a genuine addition and the practical estimator is worth referee scrutiny even if the assumptions require tightening.

Referee Report

2 major / 2 minor

Summary. The paper develops an analytical provisioning framework for Attention-FFN disaggregated (AFD) LLM serving in an rA–1F topology under stochastic workloads. It characterizes per-slot stationary token load via renewal-reward processes, identifies a single nonparametric workload statistic θ (estimable from traces) that governs the optimal A/F ratio under arbitrary prefill-decode distributions, derives a closed-form mean-field rule decomposing into Attention-, communication-, and FFN-bottleneck regimes, and adds a Gaussian barrier-aware refinement for cross-worker synchronization overhead. A trace-calibrated simulator validates that the predicted optimal ratio matches the simulation-optimal within 10%.

Significance. If the central claims hold, the work supplies a compact, calibratable analytical account of how stochastic workload structure determines provisioning ratios in disaggregated LLM serving, with the nonparametric θ estimator and closed-form mean-field decomposition offering practical value for reducing device idle time without heavy simulation. The low free-parameter count and trace-based calibration are notable strengths for deployment relevance.

major comments (2)

[renewal-reward characterization and mean-field rule derivation] The renewal-reward characterization of per-slot stationary token load (which underpins the claim that a single θ fully governs provisioning) assumes ergodicity conditions on inter-replenishment times and length distributions that may fail to hold once KV-cache growth couples monotonically to stochastic request replenishment; this risks introducing unaccounted cross-terms between Attention-side state and FFN load in the mean-field bottleneck decomposition.
[validation and simulator experiments] The reported 10% match between predicted and simulation-optimal A/F ratios does not yet rule out the coupling concern, because the trace-calibrated simulator may have been exercised only in regimes where slowest-worker KV size and barrier time remain weakly correlated; an explicit sensitivity analysis or counter-example under strong synchronization would be needed to confirm the decomposition remains accurate.

minor comments (2)

[abstract] Abstract contains a typo: 'Attentio-FFN' should read 'Attention-FFN'.
[introduction and model section] Notation for the rA–1F topology and the precise definition of the barrier time should be introduced earlier with an accompanying diagram to aid readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on the renewal-reward analysis and validation. We address each major comment below and indicate planned revisions.

read point-by-point responses

Referee: The renewal-reward characterization of per-slot stationary token load (which underpins the claim that a single θ fully governs provisioning) assumes ergodicity conditions on inter-replenishment times and length distributions that may fail to hold once KV-cache growth couples monotonically to stochastic request replenishment; this risks introducing unaccounted cross-terms between Attention-side state and FFN load in the mean-field bottleneck decomposition.

Authors: We acknowledge the potential for monotonic KV-cache growth to challenge strict ergodicity. Our renewal-reward model defines cycles at request completion events, with θ as the long-run average token load per slot derived from the stationary distribution. The mean-field rule follows from comparing expected loads across regimes, and we maintain that cross-terms average to zero in the long-run expectation used for provisioning. To address the concern directly, we will add a subsection clarifying the ergodicity conditions and providing a brief argument that the single nonparametric θ remains sufficient for the bottleneck decomposition under stable operation. revision: partial
Referee: The reported 10% match between predicted and simulation-optimal A/F ratios does not yet rule out the coupling concern, because the trace-calibrated simulator may have been exercised only in regimes where slowest-worker KV size and barrier time remain weakly correlated; an explicit sensitivity analysis or counter-example under strong synchronization would be needed to confirm the decomposition remains accurate.

Authors: The simulator was driven by real traces exhibiting natural variability in KV sizes and barrier synchronization. The 10% agreement held across tested load points and worker counts. We agree that regimes with strong correlation between slowest-worker KV size and barrier time warrant explicit testing. In the revision we will add a sensitivity study that modulates decode-length variance to induce stronger synchronization coupling and report the resulting prediction error of the analytical rule. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained from renewal-reward analysis and nonparametric estimation

full rationale

The central result is a closed-form mean-field rule for the optimal A/F ratio obtained by characterizing the per-slot stationary token load via renewal-reward theory, identifying a single workload statistic θ that admits a nonparametric estimator directly from request traces. This θ is not fitted to the target provisioning ratio or simulation outcomes. The Gaussian barrier-aware refinement follows from the same stationary characterization and quantifies synchronization overhead without reducing to fitted inputs or self-citations. The 10% simulation match is presented as external validation rather than part of the derivation. No load-bearing step equates a prediction to its own inputs by construction, and the framework remains independent of any self-citation chain.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 0 invented entities

Framework rests on renewal-reward theorem for stationary workload and mean-field approximation for large worker groups; theta is treated as an observable statistic rather than a fitted parameter.

free parameters (1)

theta
Single workload statistic extracted nonparametrically from traces that governs the optimal ratio.

axioms (2)

standard math Renewal-reward theorem applies to the per-slot stationary token load under growing KV caches and random request replenishment
Invoked to reduce stochastic workload to the governing statistic theta.
domain assumption Mean-field limit holds for synchronized execution across Attention workers
Used to obtain the closed-form rule and bottleneck regimes.

pith-pipeline@v0.9.0 · 5584 in / 1430 out tokens · 51047 ms · 2026-05-16T09:50:40.915448+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

renewal-reward characterization of the per-slot stationary token load, identifying a single workload statistic θ
IndisputableMonolith/Foundation/BranchSelection.lean branch_selection unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

closed-form mean-field rule for the optimal A/F ratio decomposing into Attention-, communication-, and FFN-bottleneck regimes

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages · 4 internal anchors

[1]

D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

work page 1901
[2]

Chen, Z., Bu, T., Song, C., Lu, X., Ye, Y ., and Zhou, Z

URL https://le.qun.ch/en/b log/2023/05/13/transformer-batching/. Chen, Z., Bu, T., Song, C., Lu, X., Ye, Y ., and Zhou, Z. A universal load balancing principle and its application to large language model serving,

work page 2023
[3]

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H

URL https: //arxiv.org/abs/2601.17855. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113,

work page arXiv
[4]

Scaling Laws for Neural Language Models

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

work page internal anchor Pith review Pith/arXiv arXiv 2001
[5]

Flash communication: Reducing tensor parallelization bottleneck for fast large language model inference.arXiv preprint arXiv:2412.04964,

Li, Q., Zhang, B., Ye, L., Zhang, Y ., Wu, W., Sun, Y ., Ma, L., and Xie, Y . Flash communication: Reducing tensor parallelization bottleneck for fast large language model inference.arXiv preprint arXiv:2412.04964,

work page arXiv
[6]

URL https://developer.nvidia.com/blog/ma stering-llm-techniques-inference-opt imization/. OpenAI. GPT-4 technical report. arxiv 2303.08774.View in Article, 2(5),

work page internal anchor Pith review Pith/arXiv arXiv
[7]

Step-3 is large yet affordable: Model-system co-design for cost-effective decoding

Wang, B., Wang, B., Wan, C., Huang, G., Hu, H., Jia, H., Nie, H., Li, M., Chen, N., Chen, S., et al. Step-3 is large yet affordable: Model-system co-design for cost-effective decoding.arXiv preprint arXiv:2507.19427,

work page arXiv
[8]

Openchat: Advancing open-source language models with mixed-quality data.arXiv preprint arXiv:2309.11235,

Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., and Liu, Y . Openchat: Advancing open-source language models with mixed-quality data.arXiv preprint arXiv:2309.11235,

work page arXiv
[9]

C., et al

Wang, Y ., Chen, Y ., Li, Z., Kang, X., Tang, Z., He, X., Guo, R., Wang, X., Wang, Q., Zhou, A. C., et al. Burstgpt: A real-world workload dataset to optimize llm serving systems.arXiv preprint arXiv:2401.17644,

work page arXiv
[10]

J., et al

Yuan, Z., Shang, Y ., Zhou, Y ., Dong, Z., Zhou, Z., Xue, C., Wu, B., Li, Z., Gu, Q., Lee, Y . J., et al. Llm inference unveiled: Survey and roofline model insights.arXiv preprint arXiv:2402.16363,

work page arXiv
[11]

Janus: Disaggregating Attention and Experts for Scalable MoE Inference

Zhang, Z., Wang, Y ., Wang, X., Zhao, Y ., Jiang, J., Weng, Q., Shi, S., Chen, Y ., and Yu, M. Janus: Disaggregating attention and experts for scalable moe inference.arXiv preprint arXiv:2512.13525,

work page internal anchor Pith review Pith/arXiv arXiv
[12]

WildChat: 1M ChatGPT Interaction Logs in the Wild

Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y ., and Deng, Y . Wildchat: 1m chatgpt interaction logs in the wild.arXiv preprint arXiv:2405.01470,

work page internal anchor Pith review arXiv
[13]

P., et al

Zheng, L., Chiang, W.-L., Sheng, Y ., Li, T., Zhuang, S., Wu, Z., Zhuang, Y ., Li, Z., Lin, Z., Xing, E. P., et al. Lmsys-chat-1m: A large-scale real-world llm conversa- tion dataset.arXiv preprint arXiv:2309.11998,

work page arXiv
[14]

A., Wang, D., Zhang, X., Zhou, H., Wei, H., Cheng, Y ., et al

Zhu, R., Jiang, Z., Jin, C., Wu, P., Stuardo, C. A., Wang, D., Zhang, X., Zhou, H., Wei, H., Cheng, Y ., et al. Megascale-infer: Serving mixture-of-experts at scale with disaggregated expert parallelism.arXiv preprint arXiv:2504.02263,

work page arXiv
[15]

Serving large lan- guage models on huawei cloudmatrix384.arXiv preprint arXiv:2506.12708,

Zuo, P., Lin, H., Deng, J., Zou, N., Yang, X., Diao, Y ., Gao, W., Xu, K., Chen, Z., Lu, S., et al. Serving large lan- guage models on huawei cloudmatrix384.arXiv preprint arXiv:2506.12708,

work page arXiv
[16]

We use Multi-Token Prediction (MTP) with depth

For the MoE layers, the expert intermediate dimension is dexpert = 2048, with a total of Nexpert = 256 experts across the system, where each token is routed tok= 8experts. We use Multi-Token Prediction (MTP) with depth

work page 2048
[17]

MLA Attention (Memory-Bound) During decoding, Attention computation ismemory-bound, dominated by reading the compressed KV cache from HBM

B.2. MLA Attention (Memory-Bound) During decoding, Attention computation ismemory-bound, dominated by reading the compressed KV cache from HBM. The compute cost scales with total token loadT= PB b=1(sb +i b). B.2.1. DERIVATION Data volume per token.With KV compression dimension (dc +d rope) = 576 and BF16 precision (2 bytes per element): Vtoken = (dc +d r...

work page 2048