The reviewed record of science sign in
Pith

arxiv: 2605.07985 · v2 · pith:XC46HSFB · submitted 2026-05-08 · cs.DC · cs.AI

Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

Joon Ha Kim , Geon-Woo Kim , Anoop Rachakonda , Daehyeok Kim This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-05-22 10:17 UTCgrok-4.3pith:XC46HSFBrecord.jsonopen to challenge →

classification cs.DC cs.AI
keywords LLM inferenceprofilingsimulationlatency modelingredundancy-awareconfiguration-agnosticGPU measurementattention backends
0
0 comments X

The pith

Dooly profiles LLM operations once across configurations by tracking dimension origins, cutting profiling GPU hours by 56 percent while keeping simulation errors under 8 percent.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Dooly as a profiling method for latency models that power simulators of LLM inference. Current tools repeat full profiling for every model configuration, hardware platform, and backend, which quickly becomes too expensive. Dooly observes that each operation input is either fixed by the model or set by the request, and that fixed values often repeat across models. It runs a single inference pass, marks the origin of every dimension with taint tracking, and profiles only the new request-dependent cases while pulling prior results for the rest. The resulting regression models plug into existing simulators and deliver time predictions within 5 percent MAPE for first-token latency and 8 percent for per-token latency across tested platforms and architectures.

Core claim

Dooly performs one inference pass, labels every input dimension of each operation with its source via taint propagation, isolates stateful kernels such as attention by reusing the engine's own initialization code, and selectively profiles only operations missing from its latency database; the database then supplies regression models that replace the profiling step inside any existing simulator.

What carries the argument

Taint propagation during a single inference pass to classify fixed versus request-dependent dimensions, feeding a selective latency database that skips redundant profiles.

If this is right

  • Existing simulators gain a drop-in backend that supports rapid sweeps over hardware, engines, and backends.
  • Profiling cost grows with the number of distinct request sizes rather than the number of full model configurations.
  • The same database serves multiple attention backends and GPU platforms without per-combination restarts.
  • Stateful operations like attention require no custom instrumentation beyond the engine's standard setup.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Public release of the latency database could let new models start from partial results instead of zero.
  • The dimension-labeling step might transfer to simulators for other layered models that share similar input structures.
  • Runtime systems could consult the same models to pick configurations on the fly as request patterns change.

Load-bearing premise

That model-configuration values such as head size and layer count recur across architectures so one sweep over request-dependent dimensions covers all of them.

What would settle it

A new model whose configuration values are entirely distinct from prior ones, forcing a complete re-profiling run instead of database reuse.

Figures

Figures reproduced from arXiv: 2605.07985 by Anoop Rachakonda, Daehyeok Kim, Geon-Woo Kim, Joon Ha Kim.

Figure 1
Figure 1. Figure 1: Performance variation across configurations. TTFT and TPOT are reported for prefill and decode workloads. Red lines mark winner inversions; ties are within 0.5% of the best (FA: FlashAttention, FI: FlashInfer, TRI: TritonAttention). No single configuration performs opti￾mally across the full H ×S ×M ×W space. We demonstrate this empiri￾cally using two similar-sized models, Llama-3.1-8B [25] and Command￾R7B… view at source ↗
Figure 2
Figure 2. Figure 2: Dooly architecture. Workflow [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: DoolySim accuracy on four model architectures with vLLM and FlashInfer on an A100 GPU, using ShareGPT4 at 0.5 requests per second. End-to-end performance and scheduling. Dooly predicts vLLM’s measured latency with MAPE below 5% for TTFT (Figure 3a) and below 8% for TPOT (Figure 3b) across all reported percentiles. The higher TPOT error reflects per-token decode latency’s small absolute magnitude: sub-milli… view at source ↗
Figure 4
Figure 4. Figure 4: Per-batch latency predictions for three attention backends on A100 and H100, using Llama-3.1-8B. Configuration coverage. To validate that DoolySim expresses the configuration space of §2.1, we simulate the prefill-heavy workload on Llama-3.1-8B across three attention backends on both A100 (Figure 4a) and H100 (Figure 4b) GPUs. Dooly predicts TTFT within 0.5–8.0% error across all configurations, with the la… view at source ↗
Figure 5
Figure 5. Figure 5: Deduplication in Dooly amortizes profiling costs by 56%, sharing latency mea￾surements across configurations. Model labels are defined in Appendix I. Group Variant N R Profile (h) Saved (h) Red. (%) Attention (aggregate) 42 27 10.86 14.24 56.7 32/8/128 24 21 2.83 11.62 80.4 28/4/128 6 3 2.24 2.24 50.0 32/32/128 6 3 0.35 0.38 52.2 window=4K 3 0 2.24 0.00 0.0 window=32K 3 0 3.20 0.00 0.0 Linear (aten::linear… view at source ↗
Figure 6
Figure 6. Figure 6: Bottom-up resolution process. Sufficiency of a single trace. A natural concern is whether a single dummy-prompt trace can cover both prefill and decode call paths. Phase-dependent branching is driven entirely by token-count fields in the attention metadata that the serving engine passes through the forward context, and only context-dependent modules consume this metadata; all other operations in the runnab… view at source ↗
Figure 7
Figure 7. Figure 7: Latency database schema. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The MAPE of TTFT and TPOT predictions are 2% and 5% respectively. The simulated [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 8
Figure 8. Figure 8: DoolySim accuracy on five model architectures with vLLM and FlashInfer on an H100 GPU, using ShareGPT4 at 1.0 requests per second. G Per-Batch Prediction Accuracy Experiment Setup. Instead of streaming requests, we run a single batch with a fixed size and measure its latency. This isolates per-batch latency prediction accuracy from scheduling effects, which is critical for accurately modeling the serving e… view at source ↗
Figure 9
Figure 9. Figure 9: DoolySim accurately predicts the per-batch latency on Command-R7B with three attention backends on both A100 and H100 GPUs for a prefill-heavy workload. FLASHINFER FLASH_ATTN TRITON_ATTN measured predicted 512 1K 2K 4K 8K 16K 512 1K 2K 4K 8K 16K Token Count 0 0.01 0.02 Latency [s] 5.9% 7.6% 4.4% 5.6% 4.6% 4.9% 3.2% 4.0% 3.1% 3.5% 3.5% 3.4% 7.6% 5.0% 4.7% 3.6% 3.9% 1.6% 6.6% 7.1% 4.9% 5.0% 5.1% 3.5% 3.7% 3.… view at source ↗
Figure 10
Figure 10. Figure 10: DoolySim accurately predicts the per-batch latency across different model and attention backend selections on A100 and H100 GPUs for a decode-heavy workload. See [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: DoolySim correctly predicts the inversion points across different model and attention backend selections on an A100 GPU. See [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: DoolySim correctly predicts the inversion points across different model and attention backend selections on an H100 GPU. DoolySim’s high per-batch accuracy enables it to correctly predict the inversion points across model and attention backend selections on both A100 ( [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: SGLang counterpart to [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
read the original abstract

Selecting the optimal LLM inference configuration requires evaluation across hardware, serving engines, attention backends, and model architectures, since no single choice performs best across all workloads. Profile-based simulators are the standard tool, yet they hardcode their operation set to a specific configuration and re-profile every operation from scratch, making exploration prohibitively expensive. This cost stems from a missing structural understanding: every input dimension of each operation is fixed by the model configuration or determined by the incoming request. Many model-configuration values (e.g., head size, layer count) recur across models, so the same operation runs in many configurations; a single sweep over the request-dependent dimensions can serve them all. We present Dooly, which exploits this structure to achieve configuration-agnostic, redundancy-aware profiling. Dooly performs a single inference pass, labels each input dimension with its origin via taint propagation, and selectively profiles only operations absent from its latency database; stateful operations such as attention are isolated by reusing the serving engine's own initialization code, eliminating manual instrumentation. It builds latency regression models based on the database, which becomes a drop-in backend for existing simulators. Across two GPU platforms, three attention backends, and diverse model architectures, Dooly achieves simulation accuracy within 5% MAPE for TTFT and 8% for TPOT while reducing profiling GPU-hours by 56.4% across 12 models compared to the existing profiling approach. We have open-sourced Dooly at https://github.com/dooly-project.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript presents Dooly, a profiling system for LLM inference simulation. It performs a single inference pass with taint propagation to label each operation's input dimensions as either fixed by the model configuration or determined by the incoming request. This enables selective profiling of only absent operations (with stateful attention isolated via the serving engine's initialization code), construction of a latency database, and regression models that serve as a drop-in backend for existing simulators. The central empirical claim is that, across two GPU platforms, three attention backends, and 12 models, the approach yields simulation accuracy within 5% MAPE for TTFT and 8% for TPOT while cutting profiling GPU-hours by 56.4% relative to conventional per-configuration profiling.

Significance. If the accuracy and cost-reduction claims hold under independent verification, the work would meaningfully lower the expense of configuration exploration for LLM serving, particularly when sweeping across hardware, engines, and backends. The open-sourcing of the implementation is a concrete strength that supports reproducibility and adoption.

major comments (2)
  1. [Abstract and §5] Abstract and §5: the reported 56.4% GPU-hour reduction and 5%/8% MAPE figures are presented without explicit description of experimental controls, baseline implementation details, data exclusion criteria, or how the 12-model corpus was selected. These omissions are load-bearing for the central performance claim because post-hoc choices could materially affect the measured savings.
  2. [§3.2] §3.2 (Taint Propagation): the mechanism for labeling dimensions is described at a high level but does not address potential failure modes in stateful or custom kernels (e.g., dynamic KV-cache shapes, padding logic, or tensor views inside FlashAttention/xFormers). If taint tracking misses indirect or runtime-determined dimensions, the resulting latency database would contain gaps or duplicates, directly undermining both the reported reduction and the regression accuracy.
minor comments (2)
  1. [§4.1] §4.1: the regression model formulation would benefit from an explicit equation showing how the latency database is mapped to predicted TTFT/TPOT values.
  2. [Table 2] Table 2: the per-model breakdown of profiled operations would be clearer with an additional column indicating the fraction of operations that were reused from prior configurations.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments. We address each major comment below and have revised the manuscript to provide the requested details on experimental controls and to expand the discussion of taint propagation limitations.

read point-by-point responses

  1. Referee: [Abstract and §5] Abstract and §5: the reported 56.4% GPU-hour reduction and 5%/8% MAPE figures are presented without explicit description of experimental controls, baseline implementation details, data exclusion criteria, or how the 12-model corpus was selected. These omissions are load-bearing for the central performance claim because post-hoc choices could materially affect the measured savings.

    Authors: We agree that the manuscript would benefit from greater transparency on these points. In the revised version we have inserted a new subsection §5.1 that explicitly states: the baseline is the conventional per-configuration profiler that re-profiles every operation for each (model, hardware, backend) tuple; no profiling runs were excluded from the reported averages; GPU-hours were measured directly with nvidia-smi on the two evaluated platforms; and the 12-model corpus was assembled to span decoder-only and encoder-decoder families, parameter counts from 1 B to 70 B, and the three attention backends under test. These additions make the 56.4 % reduction and accuracy numbers directly reproducible and remove any ambiguity about post-hoc selection. revision: yes

  2. Referee: [§3.2] §3.2 (Taint Propagation): the mechanism for labeling dimensions is described at a high level but does not address potential failure modes in stateful or custom kernels (e.g., dynamic KV-cache shapes, padding logic, or tensor views inside FlashAttention/xFormers). If taint tracking misses indirect or runtime-determined dimensions, the resulting latency database would contain gaps or duplicates, directly undermining both the reported reduction and the regression accuracy.

    Authors: We appreciate the referee’s focus on this potential weakness. The taint tracker operates on PyTorch tensor metadata and therefore observes the concrete runtime shapes that reach each kernel, including those produced by padding and view operations. Stateful attention is deliberately excluded from the profiled path by reusing the serving engine’s own initialization code, as already noted in §3.2. Nevertheless, we acknowledge that highly opaque third-party kernels could in principle hide indirect dependencies. The revised §3.2 now contains an explicit limitations paragraph that describes this edge case and states our mitigation: any operation whose input dimensions cannot be fully classified falls back to conventional per-configuration profiling. Our evaluation across three distinct attention backends produced no such fallbacks, supporting the reported savings and accuracy. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical claims rest on external measurements

full rationale

The paper describes an empirical profiling system that uses taint propagation on a single inference pass to classify operation input dimensions as model-configuration-fixed or request-determined, then selectively profiles only missing operations and fits latency regression models. Central claims of 5% MAPE TTFT / 8% TPOT accuracy and 56.4% GPU-hour reduction are obtained by running the resulting simulator against actual inference timings across 12 models, two GPU platforms, and three attention backends. No equation or derivation reduces to a fitted parameter renamed as a prediction, no self-citation supplies a load-bearing uniqueness theorem, and the validation data are independent of the profiling database used to train the regressions. The approach is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The approach rests on the structural claim that operation input dimensions are either model-fixed or request-dependent and that model-fixed values recur across architectures; no free parameters, invented entities, or additional axioms are stated in the abstract.

axioms (1)
  • domain assumption Every input dimension of each operation is fixed by the model configuration or determined by the incoming request.
    Stated directly in the abstract as the missing structural understanding that enables the single-sweep approach.

pith-pipeline@v0.9.0 · 5826 in / 1331 out tokens · 27115 ms · 2026-05-22T10:17:12.595151+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

44 extracted references · 44 canonical work pages · 9 internal anchors

  1. [1]

    URLhttps://developer.nvidia.com/cupti

    Nvidia cuda profiling tools interface, 2025. URLhttps://developer.nvidia.com/cupti

    work page 2025

  2. [2]

    URL https://docs.pytorch.org/tutorials/ recipes/recipes/profiler_recipe.html

    Pytorch profiler documentation, Jul 2026. URL https://docs.pytorch.org/tutorials/ recipes/recipes/profiler_recipe.html

    work page 2026

  3. [3]

    Vidur: A Large-Scale Simulation Framework For LLM Inference

    Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gula- vani, Ramachandran Ramjee, and Alexey Tumanov. Vidur: A large-scale simulation framework for llm inference, 2024. URLhttps://arxiv.org/abs/2405.05465

    work page Pith review arXiv 2024

  4. [4]

    Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in llm inference with sarathi-serve, 2024. URLhttps://arxiv.org/abs/2403.02310

    work page Pith review arXiv 2024

  5. [5]

    Revati: Transparent gpu-free time-warp emulation for llm serving, 2026

    Amey Agrawal, Mayank Yadav, Sukrit Kumar, Anirudha Agrawal, Garv Ghai, Souradeep Bera, Elton Pinto, Sirish Gambhira, Mohammad Adain, Kasra Sohrab, Chus Antonanzas, and Alexey Tumanov. Revati: Transparent gpu-free time-warp emulation for llm serving, 2026. URL https://arxiv.org/abs/2601.00397

    work page arXiv 2026

  6. [6]

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. URLhttps://arxiv.org/abs/2305.13245

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    vllm v1 performance optimization, 2026

    AMD. vllm v1 performance optimization, 2026. URL https://rocm.docs.amd.com/en/ latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html

    work page 2026

  8. [8]

    Flowdroid: precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps

    Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bartel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick McDaniel. Flowdroid: precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps. InProceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementatio...

    work page arXiv 2014

  9. [9]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer,

    work page

  10. [10]

    URLhttps://arxiv.org/abs/2004.05150

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  11. [11]

    Siyuan Chen, Zhipeng Jia, Samira Khan, Arvind Krishnamurthy, and Phillip B. Gibbons. Slos-serve: Optimized serving of multi-slo llms, 2025. URL https://arxiv.org/abs/ 2504.08784

    work page arXiv 2025

  12. [12]

    C., & Talati, N

    Jaehong Cho, Minsu Kim, Hyunmin Choi, Guseul Heo, and Jongse Park. Llmservingsim: A hw/sw co-simulation infrastructure for llm inference serving at scale. In2024 IEEE Inter- national Symposium on Workload Characterization (IISWC), page 15–29. IEEE, Septem- ber 2024. doi: 10 .1109/iiswc63097.2024.00012. URL http://dx.doi.org/10.1109/ IISWC63097.2024.00012

    work page arXiv 2024

  13. [13]

    Llmservingsim2.0: A unified simulator for het- erogeneous hardware and serving techniques in llm infrastructure.IEEE Computer Architecture Letters, 24(2):361–364, July 2025

    Jaehong Cho, Hyunmin Choi, and Jongse Park. Llmservingsim2.0: A unified simulator for het- erogeneous hardware and serving techniques in llm infrastructure.IEEE Computer Architecture Letters, 24(2):361–364, July 2025. ISSN 2473-2575. doi: 10 .1109/lca.2025.3628325. URL http://dx.doi.org/10.1109/LCA.2025.3628325

    work page doi:10.1109/lca.2025.3628325 2025

  14. [14]

    Coherelabs/c4ai-command-r7b-12-2024

    Cohere. Coherelabs/c4ai-command-r7b-12-2024. https://huggingface.co/CohereLabs/ c4ai-command-r7b-12-2024/

    work page 2024

  15. [15]

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.org/abs/ 2205.14135

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  16. [16]

    Adapting vidur to vllm and profiling cpu overhead

    duanzhaol. Adapting vidur to vllm and profiling cpu overhead. https://github.com/ microsoft/vidur/issues/51, 2025. 10

    work page 2025

  17. [17]

    Cox, Jaeyeon Jung, Patrick McDaniel, and Anmol N

    William Enck, Peter Gilbert, Seungyeop Han, Vasant Tendulkar, Byung-Gon Chun, Landon P. Cox, Jaeyeon Jung, Patrick McDaniel, and Anmol N. Sheth. Taintdroid: An information-flow tracking system for realtime privacy monitoring on smartphones.ACM Trans. Comput. Syst., 32 (2), June 2014. ISSN 0734-2071. doi: 10 .1145/2619091. URL https://doi.org/10.1145/ 2619091

    work page 2014

  18. [18]

    Frontier: Simulating the next generation of llm inference systems, 2025

    Yicheng Feng, Xin Tan, Kin Hang Sew, Yimin Jiang, Yibo Zhu, and Hong Xu. Frontier: Simulating the next generation of llm inference systems, 2025. URL https://arxiv.org/ abs/2508.03148

    work page arXiv 2025

  19. [19]

    How to get the profile csv of vllm instead of tensorrt-llm? https://github.com/ casys-kaist/LLMServingSim/issues/10, 2025

    fwyc0573. How to get the profile csv of vllm instead of tensorrt-llm? https://github.com/ casys-kaist/LLMServingSim/issues/10, 2025

    work page 2025

  20. [20]

    Perfetto trace viewer

    Google. Perfetto trace viewer. URLhttps://perfetto.dev

    work page

  21. [21]

    Questions about simulation fidelity under vllm version differences, profiling utiliza- tion, and throughput estimation

    hariag. Questions about simulation fidelity under vllm version differences, profiling utiliza- tion, and throughput estimation. https://github.com/microsoft/apex_plus/issues/8, 2025

    work page 2025

  22. [22]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    A deeper look into rowhammer’s sensitivities: Experimental analysis of real dram chips and implications on future attacks and defenses,

    Vijay Kandiah, Scott Peverelle, Mahmoud Khairy, Junrui Pan, Amogh Manjunath, Timothy G. Rogers, Tor M. Aamodt, and Nikos Hardavellas. Accelwattch: A power modeling framework for modern gpus. MICRO ’21, page 738–753, New York, NY , USA, 2021. Association for Computing Machinery. ISBN 9781450385572. doi: 10 .1145/3466752.3480063. URL https://doi.org/10.1145...

    work page doi:10.1145/3466752.3480063 2021

  24. [24]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    work page 2023

  25. [25]

    arXiv preprint arXiv:2411.17651 , year=

    Yi-Chien Lin, Woosuk Kwon, Ronald Pineda, and Fanny Nina Paravecino. Apex: An extensible and dynamism-aware simulator for automated parallel execution in llm serving, 2025. URL https://arxiv.org/abs/2411.17651

    work page arXiv 2025

  26. [26]

    meta-llama/llama-3.1-8b.https://huggingface.co/meta-llama/Llama-3.1-8B

    Meta. meta-llama/llama-3.1-8b.https://huggingface.co/meta-llama/Llama-3.1-8B

    work page

  27. [27]

    Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software

    James Newsome and Dawn Song. Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In12th Annual Network and Distributed System Security Symposium (NDSS), San Diego, California, 2005

    work page 2005

  28. [28]

    openchat_sharegpt4_dataset

    OpenChat. openchat_sharegpt4_dataset. https://huggingface.co/datasets/openchat/ openchat_sharegpt4_dataset

    work page

  29. [29]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-perfo...

    work page 2019

  30. [30]

    Splitwise: Efficient generative LLM inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting, 2024. URLhttps://arxiv.org/abs/2311.18677. 11

    work page Pith review arXiv 2024

  31. [31]

    GitHub - pytorch/kineto: A CPU+GPU Profiling library that provides access to timeline traces and hardware performance counters

    PyTorch. GitHub - pytorch/kineto: A CPU+GPU Profiling library that provides access to timeline traces and hardware performance counters. — github.com. https://github.com/ pytorch/kineto

    work page

  32. [32]

    The Anatomy of a Triton Attention Kernel, 2025

    Burkhard Ringlein, Jan van Lunteren, Radu Stoica, and Thomas Parnell. The anatomy of a triton attention kernel, 2025. URLhttps://arxiv.org/abs/2511.11581

    work page arXiv 2025

  33. [33]

    Running problems with replica scheduler orca/sarathi

    rxz 0420. Running problems with replica scheduler orca/sarathi. https://github.com/ microsoft/vidur/issues/64, 2025

    work page 2025

  34. [34]

    Adding new model

    samarth1612. Adding new model. https://github.com/microsoft/vidur/issues/32, 2024

    work page 2024

  35. [35]

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V . Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer.CoRR, abs/1701.06538, 2017. URLhttp://arxiv.org/abs/1701.06538

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  36. [36]

    Trtion.https://triton-lang.org/main/index.html, 2020

    Philippe Tillet. Trtion.https://triton-lang.org/main/index.html, 2020

    work page 2020

  37. [37]

    Attention Is All You Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  38. [38]

    Optimization and tuning, 2026

    vLLM. Optimization and tuning, 2026. URL https://docs.vllm.ai/en/stable/ configuration/optimization/#attention-backend-selection

    work page 2026

  39. [39]

    Qwen2.5-1M Technical Report

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, and Zipeng Zhang. Qwen2.5-1m technical re...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  40. [40]

    FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for llm inference serving, 2025. URL https://arxiv.org/ abs/2501.01005

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  41. [41]

    SGLang: Efficient Execution of Structured Language Model Programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024. URL https://arxiv.org/abs/2312.07104

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving, 2024. URLhttps://arxiv.org/abs/2401.09670. 12 A Comparison with Existing LLM Simulators Dooly RT [5] VD [3] FT [17] LS[11] AP [24] LS2 [12] Profiler Automatic Mo...

    work page Pith review arXiv 2024

  43. [43]

    Error: missing context

    work page

  44. [44]

    Sufficiency of a single trace.A natural concern is whether a single dummy-prompt trace can cover both prefill and decode call paths

    Absorb: children absorbed by parent Import & Run SuccessFailure Resolved Fallback to parent Runnable Set:Attention Figure 6:Bottom-up resolution process. Sufficiency of a single trace.A natural concern is whether a single dummy-prompt trace can cover both prefill and decode call paths. Phase-dependent branching is driven entirely by token-count fields in ...

    work page 2024