Pith. sign in

REVIEW 4 major objections 5 minor 75 references

This paper claims that per-token generation timing exposed by streaming LLM APIs is enough to detect speculative decoding, recover draft-model context lengths, and rank candidate architectural configurations — layers, hidden size, attention

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:32 UTC pith:L6LQKWGC

load-bearing objection Solid local validation and an honest timing-model methodology, but the abstract and intro overstate what the experiments actually test: attention heads and MLP width are never recovered, and the top-10 >90% number appears nowhere in the body. the 4 major comments →

arxiv 2607.20723 v1 pith:L6LQKWGC submitted 2026-07-22 cs.CR cs.LG

Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing

Sadegh Majidi , Niloofar Mireshghallah , Kazem Taram This is my paper
classification cs.CR cs.LG
keywords timing side-channellarge language model securityspeculative decodingarchitecture extractionruntime modelingper-token latencyremote API attackGPU inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to prove that a remote adversary, with no access beyond a streaming chat interface, can learn proprietary details of a production LLM by measuring how long each token takes to arrive. It introduces two attacks. The first detects speculative decoding by sending prompts whose required context exceeds the draft model's shorter context window, producing a sharp latency spike at a measurable breakpoint that reveals the draft context length. The second builds a polynomial timing model of per-token GPU runtime as a function of layers, hidden size, attention heads, and MLP width, then ranks candidate configurations by matching synthesized timing traces to the observed trace. If true, the streaming interface becomes a practical side channel for model architecture and serving optimizations, challenging the confidentiality of deployed LLMs.

Core claim

The central claim is that the fine-grained per-token latency observable through standard streaming APIs is a reliable side channel. On the deployment side, the paper claims that Google Gemini Flash 1.5, Flash 2.5, and Flash 2.5-Lite all use speculative decoding, with draft-model context lengths of 32K, 128K, and 128K tokens, inferred from the prompt length at which per-token time jumps. On the architecture side, the paper claims that a timing model calibrated on an open Llama 1B model with artificially varied layers and hidden sizes can rank unseen Llama-family configurations: the correct number of layers appears in the top-5 for over 86%, 97%, and 83% of test configurations for eager, Flash

What carries the argument

The load-bearing object is the polynomial time-scaling model: each transformer subcomponent is decomposed into asymptotic cost terms such as T·H^2·L, T^2·H·L, and T^2·A·L, multiplied by empirically fitted coefficients from instrumented measurements. This linear model acts as an oracle that, for a candidate configuration, synthesizes a timing trace; the attack then ranks candidates by RMSE against the observed trace. For speculative decoding, the mechanism is the draft model's shorter context window: prompts that force the draft model beyond its context produce disagreement with the main model, causing a measurable per-token latency spike whose breakpoint estimates the draft context length. A

Load-bearing premise

The architecture attack assumes that timing behavior measured on an instrumented open model of the same family, on a GPU class the attacker knows, stays invariant for the target's exact software stack, kernel implementation, and single-GPU serving setup, so the fitted predictor remains a valid oracle.

What would settle it

Serve a model with publicly known architecture on a GPU of the same class but through a different inference framework and kernel set, then run the ranking attack: if the true configuration falls outside the reported top-5, the invariance assumption is refuted. Separately, deploy a draft model whose context window equals the main model's; if no timing breakpoint appears as prompt length grows, the speculative-decoding detection is specifically tied to the context-length mismatch claimed here.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Providers using speculative decoding with a draft model of shorter context can be detected and have that draft context length estimated through ordinary API interaction, with no privileged access.
  • Architectural leakage works across multiple attention implementations: the correct layer count is found in the top-5 for over 86% of test configurations under eager execution, over 97% under FlashAttention2, and over 83% with KV-cache enabled.
  • The timing predictor generalizes to other open model families such as Qwen2.5, Phi3.5, and Gemma2 without retraining, with normalized RMSE below 0.19 on test data.
  • On a remote Llama 3.1 8B API, the predicted hidden dimension matched the documented value as the top-ranked candidate when layer count was known, and a near-correct configuration reached rank 8 out of 2068 candidates.
  • Effective mitigations, such as constant per-token time or output buffering, directly conflict with low-latency streaming, so closing the leak appears costly in user experience.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If these results hold, timing probes could serve as an independent audit tool: a short streamed interaction could verify whether a served model matches its advertised architecture or whether a provider quietly changes draft-model configurations.
  • The draft-context side channel likely extends beyond standard speculative decoding to other multi-model or cascade decoding schemes wherever a smaller model's context or capability limit creates a similar breakpoint; this is a testable extension.
  • Combining timing fingerprints with logit-based leakage that recovers hidden size could shrink the candidate grid and push joint (L, H, A, I) recovery beyond top-5 ranking into more precise identification.
  • The attack's reliance on a known GPU class suggests an arms race: providers that hide GPU type or serve from heterogeneous hardware pools would degrade the calibrated oracle, making load-balancing artifacts a natural next target for measurement.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents two timing side-channel attacks on LLM APIs: (1) detecting speculative decoding and estimating the draft model's context length by observing per-token generation-time spikes at prompt lengths exceeding the draft model's context, and (2) inferring architectural parameters (L, H, A, I) by fitting a linear-regression runtime predictor to instrumented open-model timings and then ranking candidate configurations from a search grid against an observed timing trace. The speculative-decoding attack is validated locally on TinyLlama (recovering its 2048-token context as ~2018 before rounding) and applied to Gemini APIs, where the authors claim Flash 1.5/2.5/2.5-Lite use speculative decoding with draft contexts 32K/128K/128K. The architecture attack is evaluated on Llama variants, cross-family models (Qwen, Phi, Gemma), and a remote W&B Llama 3.1 8B API, reporting prediction NRMSE as low as 0.12 for eager attention and top-5 architecture retrieval for L and H individually.

Significance. If the claims hold, this is a significant contribution to the security/privacy literature: it shows that a passive network-level observer of streaming token timings can learn proprietary deployment details and coarse architecture information. The strengths are concrete: the local speculative-decoding experiment is tight and includes a control (disabling speculation), the timing predictor is evaluated on held-out configurations with honest NRMSE numbers, cross-family experiments are attempted, and the authors commit to releasing artifacts. However, the flagship claims about recovering attention heads and 'top-10 >90%' accuracy are not supported by the reported experiments.

major comments (4)
  1. [Abstract and §5.2/§5.5 (Table 4)] The abstract and §1 state that the architecture attack recovers 'the number of attention heads' and lists I in the target set (§5.1), but no experimental result for A or I is reported. §5.2 says offline variants are generated 'by altering the number of layers (L) or the hidden dimension (H)', and Table 4 reports only L, H, and (H,L). The success criteria in §5.4 define step sizes for A=4 and I=1024, so these metrics were evidently planned but omitted. As written, the claimed capability to leak attention heads and MLP width is untested. This is a load-bearing mismatch with the paper's central contribution statement.
  2. [Abstract and §5.5 (Table 4, Table 6)] The abstract claims 'the near-correct architectural configuration appears in the top-10 guesses more than 90% of the time.' No top-10 results are reported anywhere. The closest reported numbers are top-5 accuracies for (H,L): 65.38% (Eager), 54.62% (Flash2), 45.27% (KV-Flash2) on test data, and 50%/80%/47.5% on Qwen/Phi/Gemma in Table 6. A top-10 number could be higher, but it is not given, so the headline claim cannot be checked and appears implausible given the top-5 numbers. The abstract should either report top-10 results or be revised to match the body.
  3. [§5.1, §5.3, and §3 (threat model)] The architecture attack depends on knowing the GPU class (§3) and on the predictor's coefficients staying invariant across serving implementations. The paper explicitly assumes invariance 'across input sizes' but the stronger assumption is invariance across frameworks and kernels. There is no procedure in §5.3 to determine which predictor (Eager, Flash2, or KV-Flash2) is appropriate for a closed target, nor any way to detect kernel/library divergence (e.g., TensorRT, vLLM, custom fused kernels). The remote W&B experiment uses a model from the same family and likely a similar stack; it does not test the harder cases. This is a correctness-risk concern for the claimed 'remote API' generality, and should be addressed with a detection heuristic or explicit scoping.
  4. [§5.5, Table 6] The text states that cross-family results show a 'high success rate' and that the framework 'generalizes well across architectures.' But the (H,L) top-5 accuracies are 50.00% for Qwen2.5, 80.00% for Phi3.5, and 47.50% for Gemma2. These are not high for a top-5 criterion and indicate that simultaneous H and L recovery degrades significantly out-of-family. This deserves acknowledgement and a more cautious interpretation, especially since the abstract's 'top-10 >90%' claim is not evaluated in this setting either.
minor comments (5)
  1. [§4.5.1] The estimated draft context length is 2018 before rounding to 2048. The text says 'approximately at Tbreak=2030' and then '2018 tokens', which is consistent. Consider clarifying that the 2018 value is before power-of-two rounding.
  2. [§5.2, Eq. (1) vs Eq. (3)] Notation for coefficients is inconsistent: Eq. (1) uses α, β, C, while Eq. (3) uses a1..a4 and C. Use consistent notation throughout.
  3. [§5.4] The paragraph 'When assuming L is known... When fixing L and inferring H...' is redundant and confusing; the second sentence appears to repeat the first with a different rank. Please clarify.
  4. [§A.2/Table 7] The remote speculative-decoding results for Gemini report draft context lengths but no ground truth is available. The paper should state explicitly that these are estimates validated only by internal consistency, not by independent disclosure.
  5. [§5.3, 'Reverse Engineering'] The success criterion defines one-step tolerance for H (128) and L (1), but the search grid §B.7 includes H values that differ by less than 128 (e.g., 32, 64, 128). Clarify how the step size is applied when grid spacing is finer than the step.

Circularity Check

0 steps flagged

No significant circularity: the timing model is calibrated on held-out data and the target result does not enter the fitting procedure.

full rationale

The paper's two attacks are empirical calibrations rather than circular derivations. The speculative-decoding attack (§4) induces a timing signal by controlling prompt length and estimates the draft context as T_break − prologue_len; the breakpoint is measured from the victim's timing curve and validated on a local white-box system (LLaMA2/TinyLlama, §4.5.1) where the draft context (2048) is known. The architecture attack (§5) derives asymptotic cost terms from the transformer computation graph (Eqs. 1–4, 9–26), fits coefficients to instrumented Llama 1B timing data, and then evaluates on held-out configurations (Llama 3B test sets, cross-family Qwen/Phi/Gemma, and a remote W&B Llama 3.1 8B API). Target labels never enter the regression fit; RMSE-based ranking of candidate configurations is a genuine inversion of a calibrated forward model. There is no self-citation: the reference list contains no prior work by the present authors, and no uniqueness or ansatz is imported from author-controlled sources. The skeptic's concerns about missing A/I results (Table 4 only reports L, H, and (H,L)) and the abstract's unsupported 'top-10 >90%' claim are evidence/completeness problems, not circularity, because absent experiments do not feed back into any fitted constant or derivation step.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new physical or architectural entities are introduced. The ledger captures the fitted parameters of the timing model and the domain assumptions about speculative decoding and GPU transferability. The most burdensome items are the linear-runtime superposition axiom and the same-GPU-class transfer assumption, which together carry the architecture attack.

free parameters (4)
  • Regression coefficients alpha_i and C for each subcomponent predictor (Eager, Flash2, KV-Flash2 prefill/decode) = not individually reported; fit by linear regression on per-component timing data (§5.2, Eq. 3) and on the full-model res
    These coefficients are learned from profiling an open Llama 1B model with varied L and H. They encode hardware- and implementation-dependent costs for each asymptotic term. The entire architecture attack rests on these fitted values transferring to unseen targets.
  • LightGBM cuBLAS kernel classifier and per-kernel random forest regressors = trained on a matmul profiling dataset; no parameters listed
    Used for the Eager(Corrected) predictor to correct non-linearities at unseen dimensions. The fitted kernel-selection model and kernel-runtime models are a form of parameter fitting to local GPU data.
  • Draft context length rounding to nearest power of two = 2018→2048; 131,042→128K; 32,743→32K
    The breakpoint measurement T_break minus prologue length gives a raw estimate; the paper then rounds to the nearest power-of-two. This post-hoc refinement improves the estimate but is a modeling choice (§4.5).
  • Search grid boundaries and pruning criteria = H in multiples of 64/128; L in 1..45; A and I not varied in grid (§B.7)
    The grid is pruned using conventions about modern LLM architecture shapes. This encodes strong prior knowledge about the target and excludes many valid configurations, trading coverage for tractability.
axioms (5)
  • domain assumption A decoder-only transformer's per-token runtime is a linear combination of asymptotic cost terms of the form T^a H^b A^c I^d L, with hardware-dependent coefficients (§5.2 Eq. 1-3).
    This is the foundation of the timing predictor. It ignores non-linear interactions between kernels and assumes asymptotic terms dominate. The paper itself notes cuBLAS kernel selection creates deviations, which it then patches with ML correctors.
  • domain assumption The draft model in a speculative decoding deployment has a shorter context window than the main model (§4.1).
    The entire speculative-decoding attack depends on this premise to make the timing spike occur at the draft context boundary. The paper cites [72] for examples, but the premise is not true for all deployments (e.g., same-context drafts), which the paper acknowledges as false-negative risk.
  • domain assumption The breakpoint in the per-token timing curve corresponds exactly to the draft model's context window, and the spike is caused by draft-model disagreement rather than by other serving behaviors (§4.3).
    This gives the interpretation T_break - prologue_length = draft context. Alternative causes (main-model KV-cache effects, provider-side context chunking, throttling) are not experimentally excluded for the remote APIs.
  • domain assumption The attacker knows the GPU class of the target and the target runs single-GPU inference (§3).
    The timing predictor is calibrated on specific GPUs (RTX 2080 Ti, A10, B200). If the target uses a different accelerator or multi-GPU parallelism, the RMSE ranking in §5.3 has no validity guarantee. The remote W&B experiment is one favorable instance.
  • domain assumption The empirical coefficients learned on an instrumented open model remain valid for the target's inference framework and kernel implementation when the GPU class matches (§5.1).
    The paper trains separate predictors for eager, FlashAttention2, and KV-cache, each on open implementations. A closed target using a proprietary fused kernel or a different serving stack would break this transfer, and the paper gives no test to detect such a mismatch.

pith-pipeline@v1.3.0-alltime-deepseek · 42550 in / 13146 out tokens · 125591 ms · 2026-08-01T09:32:03.753269+00:00 · methodology

0 comments
read the original abstract

This work presents LeakyLMs, a set of attacks that leak proprietary model, architecture, and deployment information from production language models. LeakyLMs is the first to demonstrate that key model and deployment details can be inferred using only token generation timing, even when interacting through remote APIs. LeakyLMs introduces two core attacks. The first attack targets inference optimizations and deployment strategies. For example, our attack detects whether a provider uses speculative decoding, a widely deployed inference-time optimization, and further identifies the context length of the draft model used in the pipeline. Our measurements show that Google Gemini Flash 2.5 uses speculative decoding with a draft context window of approximately 128K tokens. The second attack recovers key architectural properties, including the number of transformer layers, hidden dimension size, and number of attention heads. To achieve this, LeakyLMs builds a detailed and accurate model of token-generation timing on modern NVIDIA GPUs, characterizing how latency scales with model configuration and hardware parameters. The attack then performs a search over the architecture space using this timing model. In experiments with Llama models, the near-correct architectural configuration appears in the top-10 guesses more than 90% of the time.

Figures

Figures reproduced from arXiv: 2607.20723 by Kazem Taram, Niloofar Mireshghallah, Sadegh Majidi.

Figure 1
Figure 1. Figure 1: The architecture of a decoder-only transformer. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Timing pattern for a local implementation of specu [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-token generation time of Gemini models. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Predicted token-generation times vs. ground-truth [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Predicted token-generation times vs. ground-truth [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: shows our method for correcting the nonlinearities that arise from varying CUDA kernel selections described in §5.1. As￾sume a candidate architectural configuration and input length (𝐻, 𝐿, 𝐴, 𝐼, 𝑇 ). When applying the correction pipeline to the original runtime predictor, we take the following steps: ▶ In step ➀, the candidate configuration is expanded into the full set of matmul operand shapes required to… view at source ↗
Figure 8
Figure 8. Figure 8: Per-token generation time of models where specu [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Predicted token-generation times (generated by a [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Predicted token-generation times (generated by a [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Predicted token-generation times (generated by a [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗
Figure 14
Figure 14. Figure 14: Predicted token-generation times vs. ground-truth [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

75 extracted references · 24 linked inside Pith

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    Pith/arXiv arXiv 2024

  2. [3]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr’on, and Sumit K. Sanghai. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.ArXivabs/2305.13245 (2023). https://api.semanticscholar.org/CorpusID:258833177

    Pith/arXiv arXiv 2023

  3. [4]

    Artificial Analysis. [n. d.]. Models. https://artificialanalysis.ai/models. Accessed: 2025-11-11

    2025

  4. [5]

    Baseten. 2025. Inference Platform: Deploy AI models in production. https: //www.baseten.co/. Accessed: 2025-11-14

    2025

  5. [6]

    2019.{CSI}{ NN}: Reverse engineering of neural network architectures through electromagnetic side channel

    Lejla Batina, Shivam Bhasin, Dirmanto Jap, and Stjepan Picek. 2019.{CSI}{ NN}: Reverse engineering of neural network architectures through electromagnetic side channel. In28th USENIX Security Symposium (USENIX Security 19). 515–532

    2019

  6. [7]

    Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Mohammad Rastegari, and Mahyar Najibi. 2024. Speculative streaming: Fast llm inference without auxiliary models.arXiv preprint arXiv:2402.11131(2024)

    Pith/arXiv arXiv 2024

  7. [8]

    Lee, Dem- ing Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Dem- ing Chen, and Tri Dao. 2024. MEDUSA: Simple LLM inference acceleration framework with multiple decoding heads. InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 203, 27 pages

    2024

  8. [9]

    Shiheng Cao, Junmin Wu, Junshi Chen, Hong An, and Zhibin Yu. 2025. AMALI: An Analytical Model for Accurately Modeling LLM Inference on Modern GPUs. In Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25). Association for Computing Machinery, New York, NY, USA, 1495–1508. doi:10.1145/3695053.3731064

    arXiv 2025

  9. [10]

    Nicholas Carlini and Milad Nasr. 2024. Remote timing attacks on efficient language model inference.arXiv preprint arXiv:2410.17175(2024)

    Pith/arXiv arXiv 2024

  10. [11]

    Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick, and Florian Tramèr

    Nicholas Carlini, Daniel Paleka, Krishnamurthy (Dj) Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick, and Florian Tramèr. 2024. Stealing part of a production language model. InProceedings of the 41st International Conference on Machine Learning(Vienna, Aust...

    2024

  11. [12]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Lau- rent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318(2023)

    Pith/arXiv arXiv 2023

  12. [13]

    Jolly Chen, Ana Lucia Varbanescu, and Monica Dessole. 2025. Component-Based Analytical Modeling of GPU Runtime Performance: a Case-Study in Scientific Computing. InProceedings of the 16th ACM/SPEC International Conference on Performance Engineering(Toronto ON, Canada)(ICPE ’25). Association for Com- puting Machinery, New York, NY, USA, 192–203. doi:10.114...

    arXiv 2025

  13. [14]

    Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin Chen-Chuan Chang, and Jie Huang. 2025. Cascade Speculative Drafting for Even Faster LLM Inference. arXiv:2312.11462 [cs.LG] https://arxiv.org/abs/2312.11462

    Pith/arXiv arXiv 2025

  14. [15]

    OpenAI Corporation. [n. d.]. GPT-5 System Card. https://cdn.openai.com/gpt- 5-system-card.pdf Accessed: 2025-11-13

    2025

  15. [16]

    Le, and Rus- lan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Rus- lan Salakhutdinov. 2019. Transformer-XL: Attentive Language Models Be- yond a Fixed-Length Context. arXiv:1901.02860 [cs.LG] https://arxiv .org/abs/ 1901.02860

    Pith/arXiv arXiv 2019

  16. [17]

    Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv:2307.08691 [cs.LG] https://arxiv.org/abs/2307.08691

    Pith/arXiv arXiv 2023

  17. [18]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135 [cs.LG] https://arxiv.org/abs/2205.14135

    Pith/arXiv arXiv 2022

  18. [19]

    China is going to win the AI race

    Deutsche Welle. 2025. “China is going to win the AI race” — Nvidia CEO Jensen Huang decries the price of electricity in the U.S., contrasts it with China’s subsidized pricing. https://www .dw.com/en/china-ai-artificial-intelligence- deepseek-us-chatgpt-semiconductors-graphics-technology-v2/a-74361630. Ac- cessed: 2025-11-14

    2025

  19. [20]

    Vasisht Duddu, Debasis Samanta, D Vijay Rao, and Valentina E. Balas. 2019. Stealing Neural Networks via Timing Side Channels. arXiv:1812.11720 [cs.CR] https://arxiv.org/abs/1812.11720

    Pith/arXiv arXiv 2019

  20. [21]

    feifeibear. [n. d.]. Fast inference from transformers via speculative decoding im- plementation. https://github.com/feifeibear/LLMSpeculativeSampling. Accessed: 2025-11-13

    2025

  21. [22]

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of LLM inference using LOOKAHEAD DECODING. InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 561, 20 pages

    2024

  22. [23]

    Yansong Gao, Huming Qiu, Zhi Zhang, Binghui Wang, Hua Ma, Alsharif Abuadbba, Minhui Xue, Anmin Fu, and Surya Nepal. 2024. Deeptheft: Stealing dnn model architectures through power side channel. In2024 IEEE Symposium on Security and Privacy (SP). IEEE, 3311–3326

    2024

  23. [24]

    Zibo Gao, Junjie Hu, Feng Guo, Yixin Zhang, Yinglong Han, Siyuan Liu, Haiyang Li, and Zhiqiang Lv. 2025. I Know What You Said: Unveiling Hardware Cache Side- Channels in Local Large Language Model Inference. arXiv:2505.06738 [cs.CR] https://arxiv.org/abs/2505.06738

    Pith/arXiv arXiv 2025

  24. [25]

    Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi, Percy Liang, and Tatsunori Hashimoto. 2025. Auditing Prompt Caching in Language Model APIs. arXiv:2502.07776 [cs.CL] https://arxiv.org/abs/2502.07776

    Pith/arXiv arXiv 2025

  25. [26]

    Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024. REST: Retrieval-Based Speculative Decoding. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computatio...

    doi:10.18653/v1/2024.naacl-long.88 2024

  26. [27]

    Hennessy and David A

    John L. Hennessy and David A. Patterson. 2017.Computer Architecture: A Quan- titative Approach(6 ed.). Morgan Kaufmann, San Mateo, CA

    2017

  27. [28]

    Weizhe Hua, Zhiru Zhang, and G Edward Suh. 2018. Reverse engineering convo- lutional neural networks through side-channel information leaks. InProceedings of the 55th Annual Design Automation Conference. 1–6

    2018

  28. [29]

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, An- drew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2018

  29. [30]

    Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: a highly efficient gradient boost- ing decision tree. InProceedings of the 31st International Conference on Neural Information Processing Systems(Long Beach, California, USA)(NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 3149–3157

    2017

  30. [31]

    Aamodt, and Timothy G

    Mahmoud Khairy, Zhesheng Shen, Tor M. Aamodt, and Timothy G. Rogers

  31. [32]

    Mahoney, Amir Gholami, and Kurt Keutzer

    Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. 2023. Speculative decoding with big little decoder. InProceedings of the 37th International Conference on Neural Infor- mation Processing Systems(New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 1705, 21 pages

    2023

  32. [33]

    Paul Kocher, Jann Horn, Anders Fogh, , Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp, Stefan Mangard, Thomas Prescher, Michael Schwarz, and Yuval Yarom. 2019. Spectre Attacks: Exploiting Speculative Execu- tion. In40th IEEE Symposium on Security and Privacy (S&P’19)

    2019

  33. [34]

    Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. 2024. CLLMs: consistency large language models. InProceedings of the 41st International Con- ference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 1018, Majidi et al. 15 pages

    2024

  34. [35]

    Joyjit Kundu, Wenzhe Guo, Ali BanaGozar, Udari De Alwis, Sourav Sen- gupta, Puneet Gupta, and Arindam Mallik. 2024. Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference . In2024 IEEE International Symposium on Workload Characterization (IISWC). IEEE Computer Society, Los Alamitos, CA, USA, 57–67. doi:10.110...

    arXiv 2024

  35. [36]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles(Koblenz, Germany)(SOSP ’23). Association for Computing Machinery, New York, N...

    arXiv 2023

  36. [37]

    Seonho Lee, Amar Phanishayee, and Divya Mahajan. 2025. Forecasting GPU Performance for Deep Learning Training and Inference. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1(Rotterdam, Netherlands)(ASP- LOS ’25). Association for Computing Machinery, New York, NY, USA,...

    arXiv 2025

  37. [38]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. InProceedings of the 40th International Conference on Machine Learning(Honolulu, Hawaii, USA)(ICML’23). JMLR.org, Article 795, 13 pages

    2023

  38. [39]

    Jiachen Liu, Jae-Won Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. 2024. Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services. arXiv:2404.16283 [cs.DC] https://arxiv.org/ abs/2404.16283

    Pith/arXiv arXiv 2024

  39. [40]

    Geoff McDonald and Jonathan Bar Or. 2025. Whisper Leak: a side-channel attack on Large Language Models.arXiv preprint arXiv:2511.03675(2025)

    arXiv 2025

  40. [41]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2024. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. InProceedings of the 29th...

    arXiv 2024

  41. [42]

    National Telecommunications and Information Administration (NTIA). 2024. AI System Disclosures. https://www.ntia.gov/issues/artificial-intelligence/ai- accountability-policy-report/developing-accountability-inputs-a-deeper- dive/information-flow/ai-system-disclosures. Accessed: 2025-11-14

    2024

  42. [43]

    NVIDIA Corporation. 2024. cuBLAS Library User Guide / Reference Manual. https://docs.nvidia.com/cuda/cublas/index.html Version 12.4, Accessed: 2025- 11-13

    2024

  43. [44]

    OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Apple- baum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives,...

    Pith/arXiv arXiv 2025

  44. [45]

    Sihyeong Park, Sungryeol Jeon, Chaelyn Lee, Seokhun Jeon, Byung-Soo Kim, and Jemin Lee. 2025. A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency. arXiv:2505.01658 [cs.CL] https: //arxiv.org/abs/2505.01658

    arXiv 2025

  45. [46]

    Rajeev Patwari, Ashish Sirasao, and Devleena Das. 2025. Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling. arXiv:2508.00904 [cs.PF] https://arxiv.org/abs/2508.00904

    Pith/arXiv arXiv 2025

  46. [47]

    Huwan Peng, Scott Davidson, C.-J Shi, and Michael Taylor. 2025. ReaLLM: A Trace-Driven Framework for Rapid Simulation of Large-Scale LLM Inference. 85–92. doi:10.1109/ASAP65064.2025.00022

    arXiv 2025

  47. [48]

    Alec Radford and Karthik Narasimhan. 2018. Improving Language Understanding by Generative Pre-Training. https://api.semanticscholar.org/CorpusID:49313245

    2018

  48. [49]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019)

    2019

  49. [50]

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. Mixture-of-depths: Dynamically allocating compute in transformer-based language models.arXiv preprint arXiv:2404.02258 (2024)

    Pith/arXiv arXiv 2024

  50. [51]

    Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodolà. 2023. Accelerating Transformer Inference for Translation via Parallel Decoding. 12336–12355. doi:10.18653/v1/ 2023.acl-long.689

    doi:10.18653/v1/ 2023

  51. [52]

    Linke Song, Zixuan Pang, Wenhao Wang, Zihao Wang, XiaoFeng Wang, Hongbo Chen, Wei Song, Yier Jin, Dan Meng, and Rui Hou. 2025. The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems. IEEE Transactions on Information Forensics and Security20 (2025), 11431–11446. doi:10.1109/TIFS.2025.3622954

    arXiv 2025

  52. [53]

    Benjamin Spector and Chris Re. 2023. Accelerating LLM Inference with Staged Speculative Decoding. arXiv:2308.04623 [cs.AI] https://arxiv.org/abs/2308.04623

    Pith/arXiv arXiv 2023

  53. [54]

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. InProceedings of the 32nd International Conference on Neural Information Processing Systems(Montréal, Canada)(NIPS’18). Curran Associates Inc., Red Hook, NY, USA, 10107–10116

    2018

  54. [55]

    Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. 2024. SpecTr: Fast Speculative Decoding via Optimal Transport. arXiv:2310.15141 [cs.LG] https://arxiv.org/abs/2310.15141

    Pith/arXiv arXiv 2024

  55. [56]

    Gemini Team. 2025. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CL] https://arxiv.org/abs/2312.11805

    Pith/arXiv arXiv 2025

  56. [57]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahri- ari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsit- sulin, Nino Vieillard, Piotr Stanczyk, Sertan Girg...

    Pith/arXiv arXiv 2024

  57. [58]

    Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. https:// qwenlm.github.io/blog/qwen2.5/ Accessed: 2026-02-05

    2024

  58. [59]

    Tim Dettmers. [n. d.]. Guanaco-13B (finetuned chatbot models, 4-bit QLoRA). https://huggingface.co/timdettmers/guanaco-13b. Accessed: 2025-11-13

    2025

  59. [60]

    TinyLlama. [n. d.]. TinyLlama-1.1B-Chat-v1.0. https://huggingface .co/ TinyLlama/TinyLlama-1.1B-Chat-v1.0. Accessed: 2025-11-13

    2025

  60. [61]

    Tom’s Hardware. 2025. China is going to win the AI race — Nvidia CEO Jensen Huang decries the price of electricity in the U.S., contrasts Leaky Language Models it with China’s subsidized pricing. https://www .tomshardware.com/tech- industry/artificial-intelligence/china-is-going-to-win-the-ai-race-nvidia-ceo- jensen-huang-decries-the-price-of-electricity-...

    2025

  61. [62]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models.ArXiv abs/2302.13971 (2023). https://api.semanticscholar.org/Corp...

    Pith/arXiv arXiv 2023

  62. [63]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InProceedings of the 31st International Conference on Neural Information Processing Systems(Long Beach, California, USA)(NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 6000–6010

    2017

  63. [64]

    Jiankun Wei, Abdulrahman Abdulrazzag, Tianchen Zhang, Adel Muursepp, and Gururaj Saileshwar. 2025. When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs. arXiv:2411.01076 [cs.CL] https://arxiv .org/abs/ 2411.01076

    arXiv 2025

  64. [65]

    Junyi Wei, Yicheng Zhang, Zhe Zhou, Zhou Li, and Mohammad Abdullah Al Faruque. 2020. Leaky dnn: Stealing deep-learning model secret with gpu context-switching side-channel. In2020 50th Annual IEEE/IFIP International Con- ference on Dependable Systems and Networks (DSN). IEEE, 125–137

    2020

  65. [66]

    Weights & Biases. 2024. Meta Llama 3.1 8B Instruct. https://wandb.ai/inference/ coreweave/cw_meta-llama_Llama-3.1-8B-Instruct. Weights & Biases Inference page, accessed 2026-04-27

    2024

  66. [67]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement De- langue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of...

    2020

  67. [68]

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui

  68. [69]

    Rui Xiao, Sibo Feng, Soundarya Ramesh, Jun Han, and Jinsong Han. 2026. Peering Inside the Black-Box: Long-Range and Scalable Model Architecture Snooping via GPU Electromagnetic Side-Channel. InProceedings of the Network and Distributed System Security Symposium (NDSS). https://www .ndss-symposium.org/ndss- paper/peering-inside-the-black-box-long-range-and...

    2026

  69. [70]

    Haoxuan Xu, Chen Gong, Beijie Liu, Haizhong Zheng, Beidi Chen, and Mengyuan Li. 2026. Wave: Leveraging Architecture Observation for Privacy-Preserving Model Oversight. InProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2(USA)(ASPLOS ’26). Association for Computing Machine...

    arXiv 2026

  70. [71]

    Fletcher, and Josep Torrellas

    Mengjia Yan, Christopher W. Fletcher, and Josep Torrellas. 2020. Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN Architectures. In29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 2003–2020. https: //www.usenix.org/conference/usenixsecurity20/presentation/yan

    2020

  71. [72]

    Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, and Bo An. 2025. LongSpec: Long-Context Speculative Decoding with Efficient Drafting and Verification. doi:10.48550/arXiv.2502.17421

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.17421 2025

  72. [73]

    Tianchen Zhang, Gururaj Saileshwar, and David Lie. 2024. Time Will Tell: Timing Side Channels via Output Token Count in Large Language Models. arXiv:2412.15431 [cs.LG] https://arxiv.org/abs/2412.15431

    Pith/arXiv arXiv 2024

  73. [74]

    MvvVk9X4fUuAw29tlaFjgxHDLlo=

    Xinyao Zheng, Husheng Han, Shangyi Shi, Qiyan Fang, Zidong Du, Xing Hu, and Qi Guo. 2024. InputSnatch: Stealing Input in LLM Services via Timing Side- Channel Attacks. arXiv:2411.18191 [cs.CR] https://arxiv.org/abs/2411.18191 A Additional Details on Leaking Inference Optimizations Attack A.1 Network Effect on Time Measurements Prior to measurement, we pro...

    Pith/arXiv arXiv 2024

  74. [2020]

    InProceedings of the ACM/IEEE 47th Annual International Sympo- sium on Computer Architecture(Virtual Event)(ISCA ’20)

    Accel-sim: an extensible simulation framework for validated GPU modeling. InProceedings of the ACM/IEEE 47th Annual International Sympo- sium on Computer Architecture(Virtual Event)(ISCA ’20). IEEE Press, 473–486. doi:10.1109/ISCA45697.2020.00047

    arXiv 2020

  75. [2023]

    InFindings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.)

    Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation. InFindings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 3909–3925. doi:10.18653/v1/2023.findings- emnlp.257

    doi:10.18653/v1/2023.findings- 2023