Pith. sign in

REVIEW 3 major objections 6 minor 33 references

When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Four-way disaggregation of LLM serving into stage-specialized hardware can deliver up to 2.06x throughput on agentic workloads, but only when the workload is prefill-heavy and each stage can get genuinely different hardware.

desk verdict First credible simulation study of four-way prefill/decode/attention/FFN disaggregation; the conditional results are plausible, but the headline multipliers rest on an unvalidated scheduling layer, so treat them as predictions. read the letter →

arxiv 2608.03741 v1 pith:QD3PX54S submitted 2026-08-04 cs.DC

Przemyslaw Forys , Haoran Wu , Can Xiao , Jiayi Nie , Tony Liu , Rika Antonova , Timothy Jones , Robert Mullins
show 3 more authors
Wayne Luk Aaron Zhao George A. Constantinides
This is my paper
classification cs.DC
keywords disaggregatedLLMservingprefill-decodedisaggregationattention-FFNspecializationheterogeneousNPUdesignspacerooflinesimulationagenticinferencemixed-precisionquantizationthroughput
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Agentic LLM workloads — multi-turn sessions with tool calls whose context grows to tens of thousands of tokens — strain the assumption that one homogeneous GPU can serve an entire inference pass. This paper asks when it pays to disaggregate: splitting prefill from decode (PD), and going further to split attention from FFN within each phase, giving each of the four stages its own hardware, parallelism, and numeric precision. Because real clusters are too costly to sweep, the authors build HeteroPanacea, a roofline-based event-driven simulator, and search the joint design space under a power or cost budget. The answer is conditional: disaggregation only clears parity once the workload is prefill-heavy, with the crossover sitting between a 1:1 and a 10:1 prefill/output ratio, and the four-way PDAF split only beats plain PD when the design space lets attention and FFN receive genuinely different devices — something commercial GPU catalogs do not allow. Inside that regime, PDAF is the most consistent winner, reaching up to 2.06x throughput on agentic workloads.

What carries the argument

The mechanism is a per-stage roofline timing model: each disaggregation stage is decomposed into a FLOP count and a byte count, and its execution time is the larger of the compute-limited time Φ/Feff and the bandwidth-limited time β/Beff, with effective rates adjusted for per-pool tensor, pipeline, data, and expert parallelism. A device is a point in a synthetic NPU design space pairing peak TFLOPS with a memory technology (SRAM, HBM, DDR, LPDDR, GDDR) at representative capacity/bandwidth operating points; power is a sub-linear power law on compute plus a physics-based memory term calibrated to datasheet TDPs. Two connectivity domains are modeled separately: device-to-device (D2D) for collec

What would settle it

Build the same models and workload ratios on a real disaggregated serving stack, with PD and PDAF pools provisioned under one power or cost budget, and measure end-to-end throughput at prefill/output ratios 1 and 10. The paper's sharpest quantitative claim is that PDAF crosses from roughly 0.5x to over 2x co-located throughput within that single decade when attention and FFN can receive genuinely different devices; a deployment that shows no such crossover, or that shows plain PD already capturing the full gain, would falsify the conditional. A cheaper check: sweep kv_lora_rank (KV latent widt

Watch

Extended reading notes

Core claim

The central result is a conditional characterization of when disaggregated inference pays. Confirming the established prefill–decode split, the simulator shows PD disaggregation delivering up to 1.82x throughput over co-located serving on agentic workloads (1.29x on chatbot traffic), and up to 75% over traditional serving on current GPUs. The new four-way PDAF split — prefill-attention, prefill-FFN, decode-attention, and decode-FFN provisioned as independent pools — reaches up to 2.06x, but only inside a specific regime: averaged across models, PDAF stands at 0.48x of co-located serving at a 1:1 prefill/output ratio and jumps to 2.10x at 10:1, plateaus between 1.5x and 1.8x, then fades as pr

Load-bearing premise

The simulator's serving behavior — its batching, queueing, and KV-transfer scheduling — was never validated end-to-end against a real disaggregated serving system; only the component models (compute roofline, collectives, power) were checked on a single 8x B200 node, so the relative throughput gains and the location of the crossover could shift if real scheduling behaves differently.

Editorial extensions

If this is right

  • An operator can decide whether to disaggregate at all from one workload number — the prefill/output token ratio: below roughly 1:1 co-location wins, the payoff appears between 10:1 and 100:1, and it fades past 500:1 as prefill stops being the bottleneck.
  • For custom-NPU designers, the four-way split is worth its complexity only if devices can decouple compute from memory bandwidth; on GPU-class hardware, PDAF should be expected to match rather than beat plain PD.
  • Which stage tolerates 4-bit quantization is a property of the workload, not the model: on tool-calling traffic the FFN stages take low precision and attention collapses, on reasoning traffic the pattern inverts — so serving systems handling both need workload-aware precision choices.
  • Because PDAF's benefit grows as decode-attention KV traffic grows, the same trend that is lengthening agentic contexts (larger KV per token) is the trend that widens the case for separating decode-attention from decode-FFN.
  • If the ablation's relationship holds across more architectures, the four-way split could be ruled in or out from a model's configuration alone — from decode-attention's KV/weight arithmetic intensity and active FFN compute — without running a hardware search at all.
  • A cheap pre-screening rule follows from the ablation: compute decode-attention's arithmetic intensity (KV bytes versus weight bytes per step) and the active-FFN workload; PDAF's edge should track the ratio between them, giving a no-simulation filter for whether a four-way hardware search is worth funding.
  • The unvalidated part of the simulator — its batching, queueing, and KV-handoff scheduler — is the most likely place for the crossover ratio to shift under real memory pressure; at the decode-attention pool's KV-capacity boundary, a real system could decay faster than the model predicts, moving the crossover above 10:1.
  • The quantization asymmetry implies a serving design the paper does not evaluate: route requests by task type (tool-calling versus reasoning) to differently quantized stage pools, since a single global precision is shown to sacrifice one workload class.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. HeteroPanacea is an event-driven simulator for disaggregated LLM serving that combines roofline device models, interconnect models, tensor/pipeline/data/expert parallelism, memory-aware continuous batching, and a combined hardware/quantization search to explore up to four-way Prefill-Decode-Attention-FFN (PDAF) specialization. The paper validates component-level timing, collectives, and power against an 8×B200 node (simulated-to-real ratios 0.67–1.21 in Tables III and IV), then sweeps eight dense and MoE models over a synthetic NPU design space and over commercial GPU instances. It reports up to 2.06× throughput over non-disaggregated serving on prefill-heavy agentic workloads, with a sharp crossover between I/O=1 and I/O=10, and finds that PDAF only outranks plain PD when the hardware design space permits attention and FFN devices to differ substantially. A controlled ablation identifies decode-attention KV traffic and active FFN compute as the main drivers of PDAF benefit, and a quantization study shows that the stage most tolerant of low precision is workload-dependent.

Significance. If the simulation is credible, this is a valuable design-space exploration tool for an emerging class of heterogeneous serving systems (e.g., Vera-Rubin-style GPU+LPU platforms). The paper combines axes that prior simulators treat in isolation: per-stage quantization, disaggregated parallelism, and PDAF architectural heterogeneity. The component-level calibration against real measurements is a genuine strength, and the paper is unusually transparent about what is and is not validated. The conditional conclusion that disaggregation pays only for prefill-heavy workloads, and that PDAF pays only when the hardware space is rich enough, is interesting and falsifiable. The main limitation is that the headline end-to-end numbers rest on an unvalidated scheduling/batching layer, so the quantitative claims should currently be read as well-motivated hypotheses rather than measured serving behavior. The commitment to open-source the simulator is important and should be honored for the claims to be reproducible.

major comments (3)
  1. [§IV-E, §VII] The central quantitative claims—up to 2.06× throughput, the I/O=1 vs I/O=10 parity crossover in Fig. 5, and the PD-vs-PDAF rankings in Figs. 6–7—are outputs of the discrete-event scheduler described in only one paragraph in §IV-E. The scheduler's memory-aware continuous batching, queueing, KV-transfer, and micro-batch pipeline behavior are not specified in enough detail to reproduce, and no end-to-end validation against a real serving stack is provided. The Conclusions state: 'Validation covers the simulator's components rather than its end-to-end serving behavior, leaving scheduling and batching effects unverified.' Since batching efficiency directly multiplies throughput, an unmodeled scheduling artifact would scale the reported gains directly. This is load-bearing and needs work: either validate the end-to-end simulator against a real disaggregated or co-located serving system (e.g.,
  2. [§II, §IV-E] The paper argues in §II that AF disaggregation 'requires accurate modelling of per-layer inter-group traffic, together with the pipeline balance it depends on,' yet §IV-E does not describe the micro-batch pipeline model used for AF/PDAF, nor provide equations for pipeline bubble/overlap, nor any validation of the pipeline balance. The conclusion that AF never beats ND (Figs. 6 and 7) and the claim that PDAF beats PD only on NPUs depend directly on this pipeline model. Without a detailed description and sensitivity analysis, the AF/PDAF comparisons cannot be assessed. Please add a formal model of the micro-batch pipeline, its steady-state throughput, and its sensitivity to per-stage latency imbalance.
  3. [§IV-B(c), Table II] The power model P_compute = P_ref (F/F_ref)^α, with α fit to three published GPUs, is extrapolated to synthetic NPUs up to 20,000 TFLOPS (Table II)—a more than 10× extrapolation beyond the fitted data. The power budget is the optimization constraint in Eqs. (2)–(4), so errors in the power model directly change which hardware the search selects and hence the NPU-vs-GPU conclusion in §VI-D (e.g., the claim that 31 of 32 GPU PDAF stage assignments are H100 while the NPU search produces a wide spread). Please report the fitted α and the three anchor points, and provide a sensitivity analysis showing whether the ranking of PD/AF/PDAF is stable to α within a plausible range, or validate the power law on additional accelerators (e.g., B200, MI300X).
minor comments (6)
  1. [Fig. 1] The label 'HeteroPenacea Serving' in Fig. 1(b) is inconsistent with the name 'HeteroPanacea' used throughout the text.
  2. [§VI-A] Each configuration simulates 500 requests with lengths drawn from a normal distribution, but no seeds, repetitions, or confidence intervals are reported. A single stochastic run makes it difficult to know whether the reported differences (e.g., the 0.95× vs 1.05× boundaries in Fig. 6) are significant. Please report variance over multiple seeds or provide a statistical test.
  3. [Table VII] The 'Baseline' row shows BFCL 18%, while the uniform 8/8/8/8 configuration shows 21%, meaning the 8-bit quantized model appears more accurate than the stated baseline. This is confusing and should be explained, especially because the quantization study uses only a single run with no repeated trials.
  4. [Abstract, §VI-B] The abstract reports 'up to 2.06× throughput gain' while Fig. 5 shows a maximum of 2.10× and the text in §VI-B mentions values up to 1.92× at I/O=100. Please reconcile these numbers or state explicitly that the abstract figure is a per-model value from a different sweep point.
  5. [§IV-B(a)–(b)] The device parameterization says peak compute is 'precision-agnostic peak,' but quantization changes effective FLOPs and bytes. Please clarify how F_eff and B_eff in Eq. (1) are adjusted when a stage is quantized to a lower precision, since this is central to the quantization search.
  6. [References] Several references (e.g., [23], [24], [28]) are preprints without full publication details. Please add venues or version identifiers where available, and ensure the MASE reference [28] includes a year or archive identifier.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; results are simulator outputs with disclosed end-to-end validation gap.

full rationale

Central findings (2.06x throughput, PD 75%, I/O=10 crossover, PDAF-vs-PD on NPUs vs GPUs) are outputs of the HeteroPanacea simulator, not derivations that assume their own conclusion. Execution time follows the standard roofline bound t=max(Phi/Feff, beta/Beff) (Eq. 1), validated against B200 measurements (Fig. 2); power model alpha is fit to published TDP (Sec. IV-B) and compared to datasheet TDP (Fig. 3); communication/parallelism models are calibrated and checked against measured latencies (Tables III-IV). The hardware search (Sec. V-B) uses a roofline capacity score only to pre-filter, then simulates top-K and chooses the highest simulated throughput (Eq. 4), so the final ranking is not the ranking score itself. The conditional results are explicitly conditional: disaggregation only clears parity for prefill-heavy workloads, and PDAF only beats PD when the design space permits genuinely different attention/FFN devices — a property of the defined space, not a hidden input-output equivalence. Self-citations (PLENA [24] as compute substrate, MemExplorer [23] for power) are adopted transparently as parameterizable models; the conclusions do not reduce to those citations. The paper itself flags the real weakness in Conclusions: 'Validation covers the simulator's components rather than its end-to-end serving behavior, leaving scheduling and batching effects unverified.' This is a disclosed validation gap, not a circular reduction. No step exhibits Eq. X = Eq. Y by construction or a fitted parameter renamed as a prediction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a small set of modeling choices: a roofline execution model, a discrete-event scheduler that is not end-to-end validated, a power law fit to three GPUs, the choice of PLENA as the compute substrate, and a stylized workload generator. No new physical entities are introduced. The free parameters are the fitted power exponent and an unspecified workload length spread.

free parameters (2)
  • Power-law exponent alpha for P_compute = not reported
    Section IV-B(c): P_compute = P_ref * (F/F_ref)^alpha, with alpha fit to published TDP minus memory power across three accelerators. The fitted value and its uncertainty are not reported, and the formula is used to set power for all synthetic NPU design points.
  • Standard deviation of per-request input/output lengths = not reported
    Section VI-A: per-request input and output lengths are drawn from normal distributions centered on target values, but the spread is unspecified. This choice affects batching pressure and therefore the simulated throughput comparisons.
assumptions (5)
  • domain assumption Roofline model t = max(Phi/F_eff, beta/B_eff), with no queuing, kernel-launch, or occupancy effects below per-stage granularity.
    Section IV-B(b). All stage timing, and therefore all throughput comparisons, reduce to this bound.
  • domain assumption The event-driven scheduler's batching and queueing behavior faithfully represents real disaggregated serving, despite no end-to-end validation.
    Section IV-E builds the simulator; the Conclusions admit that validation covers components rather than end-to-end serving behavior.
  • ad hoc to paper P_compute = P_ref * (F/F_ref)^alpha, with alpha fit to three published GPUs, extrapolates to every synthetic NPU in the design space.
    Section IV-B(c). This fit anchors hardware allocations under the power budget; alpha's value and error bars are not reported.
  • domain assumption PLENA is a representative NPU compute architecture for all disaggregation stages.
    Section II-B adopts PLENA as the baseline compute architecture; no independent architectural baseline is compared.
  • domain assumption A workload with output length fixed at 1000 tokens, normally distributed I/O lengths, 500 requests at 125 rps represents agentic serving.
    Section VI-A fixes these values; the paper varies the prefill/output ratio through input length but does not check sensitivity to request rate or output length.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference." pith.science (2026). https://pith.science/paper/QD3PX54S

@misc{pith2026260803741,
  author       = {Pith},
  title        = {Pith review of: When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QD3PX54S}},
  note         = {Machine review of arXiv:2608.03741}
}
read the original abstract

Agentic inference now dominates the LLM inference landscape, requiring LLMs to actively engage in multi-turn interactions with tool-calling capabilities. This introduces a more complex workload for the underlying inference system: serving stages such as prefill and decode exhibit substantially different behaviors and demand distinct compute and memory-bandwidth capabilities. As a result, a single homogeneous GPU system now struggles to support agentic inference, motivating an industry shift toward heterogeneous systems with disaggregated serving capabilities, such as the emerging Vera-Rubin platform with GPUs and Groq LPUs. However, the question of what the optimal hardware should look like for each component in a heterogeneous system remains underexplored. To this end, we propose a novel simulation framework for disaggregated serving, termed \textbf{HeteroPanacea}, that enables system-level simulation across three dimensions: 1) disaggregated quantization, 2) automated intra- and inter-device parallelization scheduling, and 3) PDAF (prefill-decode-attention-FFN) NPU architectural heterogeneity. By combining these three axes, we provide a cross-stack simulation framework for future heterogeneous agentic serving systems. We confirm the benefit of Prefill Decode disaggregation, simulating increased serving throughput by up to 75\% compared to traditional serving with current GPUs and demonstrate 4 way Prefill Decode Attention FFN disaggregation is the most consistent for increasing throughput across different models, assuming custom NPUs. We also investigate the relationship between model architecture and gain from disaggregation by running a set of ablation studies.

Figures

Figures reproduced from arXiv: 2608.03741 by the authors.

Figure 1
Figure 1. Motivation of HeteroPanacea, a heterogeneous NPU [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of achieved TFLOPS of the roofline model compared with experimental data [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison of simulated power of GPUs with [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Overview of HeteroPanacea. TP - Tensor Parallelism, PP - Pipeline Parallelism, DP - Data parallelism, D2D - Device [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: PDAF throughput relative to ND, averaged across [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: NPU throughput relative to No Disaggregation ND ( [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: GPU throughput relative to No Disaggregation ND ( [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages

  1. [1]

    Inside NVIDIA Groq 3 LPX: The Low- Latency Inference Accelerator for the NVIDIA Vera Rubin Platform,

    K. Aubrey and F. Ghodsian, “Inside NVIDIA Groq 3 LPX: The Low- Latency Inference Accelerator for the NVIDIA Vera Rubin Platform,” https://developer.nvidia.com/blog/inside-nvidia-groq-3-lpx-the-low- latency-inference-accelerator-for-the-nvidia-vera-rubin-platform/, Mar. 2026, nVIDIA Technical Blog. Accessed: 2026-05-13

    work page 2026

  2. [2]

    Web agents with world models: Learning and leveraging environment dynamics in web navigation,

    H. Chae, N. Kim, K. T.-i. Ong, M. Gwak, G. Song, J. Kim, S. Kim, D. Lee, and J. Yeo, “Web agents with world models: Learning and leveraging environment dynamics in web navigation,” inICLR, 2025

    work page 2025

  3. [3]

    Llmservingsim 2.0: A unified simulator for heterogeneous and disaggregated llm serving infrastructure,

    J. Cho, H. Choi, G. Heo, and J. Park, “Llmservingsim 2.0: A unified simulator for heterogeneous and disaggregated llm serving infrastructure,” 2026. [Online]. Available: https://arxiv.org/abs/2602. 23036

    work page 2026

  4. [4]

    Deepseek-v3 technical report,

    DeepSeek-AI, “Deepseek-v3 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2412.19437

    arXiv 2024

  5. [5]

    Deepseek-v4 technical report,

    DeepSeek-AI, “Deepseek-v4 technical report,” https://huggingface.co/ collections/deepseek-ai/deepseek-v4, 2026, preview release; includes DeepSeek-V4-Pro and DeepSeek-V4-Flash

    work page 2026

  6. [6]

    Coral npu: A full-stack platform for edge ai,

    Google Research, “Coral npu: A full-stack platform for edge ai,” 2025

    work page 2025

  7. [7]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al- Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Ma...

    work page

  8. [8]

    Webvoyager: Building an end-to-end web agent with large multimodal models,

    H. He, W. Yao, K. Ma, W. Yu, Y . Dai, H. Zhang, Z. Lan, and D. Yu, “Webvoyager: Building an end-to-end web agent with large multimodal models,” inACL, 2024

    work page 2024

Show all 33 references
  1. [9]

    Not all prefills are equal: Ppd disaggregation for multi-turn llm serving,

    Z. Li, J. Liu, Z. Xu, Y . Zhang, T. Rabbani, and C. Zhang, “Not all prefills are equal: Ppd disaggregation for multi-turn llm serving,” 2026. [Online]. Available: https://arxiv.org/abs/2603.13358

    2026 arXiv

  2. [10]

    The llama 4 herd: The beginning of a new era of natively mul- timodal ai innovation,

    Meta AI, “The llama 4 herd: The beginning of a new era of natively mul- timodal ai innovation,” https://ai.meta.com/blog/llama-4-multimodal- intelligence/, 2025, llama 4 Scout and Llama 4 Maverick

    2025

  3. [11]

    Kernelcraft: Benchmarking for agentic close-to-metal kernel generation on emerging hardware,

    J. Nie, H. Wu, Y . Lai, Z. Cao, C. Zhang, B. Lou, E. Wang, J. Cheng, T. M. Jones, R. Mullins, R. Antonova, and Y . Zhao, “Kernelcraft: Benchmarking for agentic close-to-metal kernel generation on emerging hardware,”arXiv preprint arXiv:2603.08721, 2026

    2026 arXiv

  4. [12]

    NVLink and NVLink switch,

    NVIDIA Corporation, “NVLink and NVLink switch,” https://www. nvidia.com/en-us/data-center/nvlink/, 2026, accessed: 2026-07-30

    2026

  5. [13]

    gpt-oss-120b & gpt-oss-20b model card,

    OpenAI, “gpt-oss-120b & gpt-oss-20b model card,” 2025. [Online]. Available: https://arxiv.org/abs/2508.10925

    2025 arXiv

  6. [14]

    Splitwise: Efficient generative LLM inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, ´I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative LLM inference using phase splitting,” inProceedings of the 51st Annual International Symposium on Computer Architecture (ISCA), 2024. [Online]. Available: https://...

    2024 arXiv

  7. [15]

    The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models,

    S. G. Patil, H. Mao, C. Cheng-Jie Ji, F. Yan, V . Suresh, I. Stoica, and J. E. Gonzalez, “The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models,” inICML, 2025

    2025

  8. [16]

    Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot,

    R. Qin, Z. Li, W. He, J. Cui, F. Ren, M. Zhang, Y . Wu, W. Zheng, and X. Xu, “Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot,” inFAST, 2025

    2025

  9. [17]

    Qwen3 technical report,

    Qwen Team, “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388

    2025 arXiv

  10. [18]

    Microscopiq: Acceler- ating foundational models through outlier-aware microscaling quantiza- 12 tion,

    A. Ramachandran, S. Kundu, and T. Krishna, “Microscopiq: Acceler- ating foundational models through outlier-aware microscaling quantiza- 12 tion,” inProceedings of the 52nd Annual International Symposium on Computer Architecture, 2025, pp. 1193–1209

    2025

  11. [19]

    Longcodebench: Evaluating coding LLMs at 1m context windows,

    S. Rando, L. Romani, A. Sampieri, L. Franco, J. Yang, Y . Kyuragi, F. Galasso, and T. Hashimoto, “Longcodebench: Evaluating coding LLMs at 1m context windows,” inCOLM, 2025

    2025

  12. [20]

    Microscaling data formats for deep learning,

    B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, S. Dusan, V . Elango, M. Golub, A. Heinecke, P. James-Roxby, D. Jani, G. Kolhe, M. Langhammer, A. Li, L. Melnick, M. Mesmakhosroshahi, A. Rodriguez, M. Schulte,...

    2023 arXiv

  13. [21]

    Step-3 is large yet affordable: Model-system co-design for cost-effective decoding,

    StepFun, “Step-3 is large yet affordable: Model-system co-design for cost-effective decoding,” 2025. [Online]. Available: https://arxiv.org/ abs/2507.19427

    2025

  14. [22]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems, vol. 30, 2017, pp. 5998–6008

    2017

  15. [23]

    Memexplorer: Navigating the heterogeneous memory design space for agentic inference npus,

    H. Wu, Z. Cao, Y . Lai, B. Lou, J. Nie, C. Xiao, T. Adeniran, P. Forys, K. Johar, C. Wright, J. Liu, K. Shi, N. D. Lane, R. Antonova, J. Cheng, T. Jones, A. Zhao, and R. Mullins, “Memexplorer: Navigating the heterogeneous memory design space for agentic inference npus,” 2026. ...

    2026 arXiv

  16. [24]

    Combating the memory walls: Optimization pathways for long-context agentic llm inference,

    H. Wu, C. Xiao, J. Nie, X. Guo, B. Lou, J. T. H. Wong, Z. Mo, C. Zhang, P. Forys, C. Ai, T. Adeniran, W. Luk, H. Fan, J. Cheng, T. M. Jones, R. Antonova, R. Mullins, and A. Zhao, “Combating the memory walls: Optimization pathways for long-context agentic llm inference,”arXiv p...

    2026 arXiv

  17. [25]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments,

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y . Liu, Y . Xu, S. Zhou, S. Savarese, C. Xiong, V . Zhong, and T. Yu, “Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” inNeurIPS, 2024

    2024

  18. [26]

    Flightllm: Efficient large language model inference with a complete mapping flow on fpgas,

    S. Zeng, J. Liu, G. Dai, X. Yang, T. Fu, H. Wang, W. Ma, H. Sun, S. Li, Z. Huang, Y . Dai, J. Li, Z. Wang, R. Zhang, K. Wen, X. Ning, and Y . Wang, “Flightllm: Efficient large language model inference with a complete mapping flow on fpgas,” 2024. [Online]. Available: https://a...

    2024 arXiv

  19. [27]

    MR-GSM8K: A meta- reasoning benchmark for large language model evaluation,

    Z. Zeng, P. Chen, S. Liu, H. Jiang, and J. Jia, “MR-GSM8K: A meta- reasoning benchmark for large language model evaluation,” inICLR, 2025

    2025

  20. [28]

    Mase: An efficient represen- tation for software-defined ml hardware system exploration

    C. Zhang, J. Cheng, Z. Yu, and Y . Zhao, “Mase: An efficient represen- tation for software-defined ml hardware system exploration.”

  21. [29]

    Llmcompass: Enabling efficient hardware design for large language model inference,

    H. Zhang, A. Ning, R. B. Prabhakar, and D. Wentzlaff, “Llmcompass: Enabling efficient hardware design for large language model inference,” inProceedings of the 51st Annual International Symposium on Computer Architecture, ser. ISCA ’24. IEEE Press, 2025, p. 1080–1096. [Online]...

    2025

  22. [30]

    Glm-4.6,

    Zhipu AI, “Glm-4.6,” https://z.ai/blog/glm-4.6, 2025

    2025

  23. [31]

    Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving,” inProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’...

    2024

  24. [32]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,” inOSDI, 2024. 13

    2024

  25. [2024]

    Available: https://arxiv.org/abs/2407.21783

    [Online]. Available: https://arxiv.org/abs/2407.21783

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.