Pith. sign in

REVIEW 2 major objections 6 minor 24 references

STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU

T0 review · 2 major / 6 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read STEEL maps fused FlashAttention onto AMD XDNA NPUs so laptop agents can run long-sequence prefill with far less energy than the CPU or GPU.

desk verdict Solid open XDNA FlashAttention mapping with a useful sparsity-aware placement trick and real silicon numbers; the agentic framing overreaches the microbenchmarks, but the operator results hold. read the letter →

arxiv 2607.09385 v1 pith:6EK3DDNM submitted 2026-07-10 cs.DC cs.AIcs.PF

Victor J.B. Jung , Gagandeep Singh , Joseph Melber , Kristof Denolf , Francesco Conti , Luca Benini This is my paper
classification cs.DCcs.AIcs.PF
keywords FlashAttentionNPUXDNAdataflowsparsity-awareplacementenergy-efficientinferencelong-sequenceprefilllaptopSoC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Laptop SoCs now ship neural engines meant for on-device LLM agents, yet attention—especially the long-context prefill stage—is hard to map onto their spatial, data-movement-explicit arrays without waste. STEEL is an open dataflow formulation of FlashAttention that decomposes the algorithm into a three-stage pipeline of processing tiles, keeps intermediate scores and softmax statistics on-chip, and places those pipelines so the causal mask’s sparsity is balanced across workers. On the Ryzen AI 9 HX 370 the resulting kernel cuts average energy by roughly nine times versus optimized multi-core CPU FlashAttention and by almost two times versus the integrated GPU; on earlier XDNA silicon it is nearly ten times faster than the previous published FlashAttention mapping. The point is practical: agentic workloads that once needed cloud GPUs can stay local, private, and within a laptop thermal budget if attention itself is rewritten for the NPU’s spatial fabric.

What carries the argument

STEEL pipeline: FlashAttention decomposed into three cooperating AIE stages (score computation, online softmax, value accumulation) whose Q-tile assignment is interleaved so that causal-mask sparsity is equalized inside each broadcast group, cutting synchronization stalls.

What would settle it

Run the identical Llama-scale attention dimensions inside an end-to-end multi-layer prefill+decode agent workload on the same Ryzen AI SoC, measure total package energy and wall-clock latency, and check whether the NPU still beats the GPU by roughly 1.75× and the CPU by roughly 9×.

Watch

Extended reading notes

Core claim

A carefully balanced three-stage FlashAttention pipeline plus sparsity-aware placement of those pipelines across the XDNA tile array makes prefill attention both faster and substantially more energy-efficient on laptop NPUs than either hand-tuned CPU/GPU kernels or prior NPU mappings, while remaining portable across XDNA generations.

Load-bearing premise

The measured energy and latency gains for isolated attention heads under fixed head dimensions and turbo NPU mode will still dominate when the same kernels sit inside a full multi-layer agent serving loop with decode traffic, host orchestration, and thermal limits.

Editorial extensions

If this is right

  • Long-context agent prefill can stay on-device on XDNA-class laptops without the energy cost of CPU or GPU FlashAttention.
  • Causal-mask imbalance is no longer an automatic pipeline stall; the same interleaving idea can be reused for other sparse attention patterns.
  • Open IRON designs of STEEL give a concrete baseline against which later XDNA operators or competing NPUs can be compared.
  • Porting the same three-stage dataflow already yields nearly 10× latency gains on the previous XDNA generation, suggesting the formulation is not generation-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If host–NPU launch and KV-cache traffic remain cheap, the energy advantage should widen further for multi-layer transformer stacks whose attention dominates total FLOPs.
  • The same sparsity-balancing placement may transfer to other spatial NPUs that broadcast K/V tiles and must wait on the slowest consumer.
  • Once decode-stage kernels receive similar treatment, fully local long-context agents become realistic under laptop power envelopes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. STEEL presents the first open-source FlashAttention mapping for AMD XDNA-like NPUs. It reformulates prefill attention as a three-stage AIE pipeline (score computation, online softmax, output accumulation) with explicit ObjectFIFO data movement, Mem-tile 4-D layout handling, and on-core causal-mask generation. To address broadcast-induced stalls under causal sparsity, the authors introduce a sparsity-aware row-to-pipeline placement that equalizes work within pipeline groups. Evaluated on Ryzen AI 9 HX 370 (XDNA 2) and ported to XDNA 1, STEEL reports average energy reductions of 9.17× vs Zen5 FlashAttention and 1.75× vs RDNA 3.5 FlashAttention (Llama3.1-1B head dims, seq 2k–32k), 9.6× lower latency than DATO on XDNA 1, 22.8× vs layer-by-layer IRON attention on XDNA 2, and a 38% latency gain from sparsity-aware vs uniform placement.

Significance. If the silicon measurements hold, this is a useful systems contribution for edge LLM inference: it is the first open FlashAttention for the XDNA family, ships a concrete IRON/MLIR-AIE mapping, and demonstrates that careful pipeline balancing and causal-mask-aware placement matter on spatial dataflow NPUs with explicit DMA programming. The head-to-head energy comparison across NPU, CPU, and GPU on the same SoC, plus the port to XDNA 1 beating DATO, strengthens the result beyond a single-platform kernel paper. Open-sourcing via the IRON stack is a clear credit. The main limitation is that the agentic/OS-workflow framing rests on isolated prefill-attention microbenchmarks rather than full-model serving; the operator-level claims themselves remain valuable for NPU mapping research.

major comments (2)
  1. [§V-A / §V-D / Abstract] §V-A, §V-D, Abstract, and Introduction: Energy and latency claims are framed around agentic long-sequence LLM inference on laptop SoCs (privacy, reliability, OS agents), yet all reported numbers are isolated prefill-attention microbenchmarks (fixed BERT/Llama head dims, turbo NPU, no multi-layer stack, no decode, no host–NPU orchestration or thermal throttling). This premise is load-bearing for the paper’s motivation and abstract wording. Either add end-to-end or multi-layer measurements, or revise the abstract/intro/conclusion to scope claims explicitly to the attention operator and state the microbenchmark limitation clearly.
  2. [§V-A] §V-A Evaluation Setup: Latency is averaged over 50 iterations and power over 100 iterations via AGT at 50 ms sampling, but no variance, std, or range is reported for any figure (Figs. 5–8). For short kernels and coarse power sampling, this weakens confidence in the 9.17×/1.75× energy averages and the 38% placement gain. Report error bars or at least min/max or std for the main energy and latency tables/figures.
minor comments (6)
  1. [§IV-C] §IV-C and contributions list: The 38% latency reduction from sparsity-aware placement is stated but not shown as a dedicated plot or table with sequence lengths; a small ablation figure would make the claim easier to verify.
  2. [Fig. 6 / §V-B] Fig. 6: The off-chip traffic model is useful; briefly state whether the model was validated against measured DMA counters or is purely analytical.
  3. [§V-C] §V-C: DATO comparison stops at seq 4096 due to compilation-time issues; note this limitation more prominently in the figure caption or text so readers do not over-extrapolate the 9.6× average.
  4. [Algorithm 1] Notation: Algorithm 1 uses m‘, l‘, θ1 inconsistently with the prose (m′, ℓ); unify symbols and define all released tokens.
  5. [Table I] Table I: Peak TOPS numbers for commercial NPUs would benefit from a citation or footnote on precision (INT8 vs BF16) so the comparison is not misread.
  6. [Throughout] Typos/style: “38,%” (comma), “XDNA™-like” hyphenation consistency, and occasional missing spaces before units (e.g., 512 kB).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: STEEL's claims rest on open IRON dataflow mapping, explicit three-stage pipeline, and silicon measurements against external CPU/GPU/DATO baselines, not on self-referential definitions or fitted predictions.

full rationale

The paper is a systems mapping and measurement study. Its load-bearing claims (energy 9.17×/1.75× vs CPU/GPU FlashAttention, 9.6× latency vs DATO, 22.8× vs layer-by-layer IRON, 38 % from sparsity-aware placement) are obtained by implementing Algorithm 1 on the XDNA array, placing pipelines as shown in Figs. 3–4, and reporting wall-clock latency and AGT power averages on the Ryzen AI 9 HX 370 (and a port to XDNA 1). FlashAttention itself is taken from the external literature (Dao 2023); the three-stage decomposition and causal-mask skipping are engineering choices whose correctness is verified by comparison to the same external Torch/ROCm FlashAttention backend, not by algebraic identity with any fitted constant. Self-citations (e.g., SPARTA, IRON stack papers) supply only architectural context and are not used to justify uniqueness or to forbid alternatives. No parameter is fitted to a subset of the reported curves and then re-presented as a prediction; no uniqueness theorem is imported from the authors’ prior work. The derivation chain is therefore self-contained against external benchmarks and exhibits none of the six circularity patterns.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

Central claims rest on standard FlashAttention math, documented XDNA hardware limits, and experimental methodology choices rather than fitted physical constants or new particles. Free parameters are engineering knobs (tile sizes, pipeline count, placement policy, power mode). Domain assumptions include causal prefill dominance and microbenchmark representativeness.

free parameters (3)
  • Q/KV tile sizes Bq, Bkv and number of STEEL pipelines (10 on XDNA2)
    Chosen to match vector layout, Mem-tile ports, and array geometry; they determine balance and utilization but are not fitted to a scientific constant.
  • Sparsity-aware vs uniform row-to-pipeline assignment
    Design choice that yields the reported 38% latency reduction; alternative placements change the result.
  • NPU turbo frequency (1.8 GHz) and AGT power sampling (50 ms, 100 iters)
    Measurement configuration that directly affects energy and latency numbers.
assumptions (4)
  • standard math FlashAttention-2 online softmax with per-row m and ℓ statistics correctly computes attention without materializing full A
    Taken from Dao et al.; Algorithm 1 is a staged reformulation of that algorithm.
  • domain assumption XDNA Mem tiles have six ports and AIE local DMA is 3-D while Mem DMA is 4-D, constraining ObjectFIFOs and swizzle placement
    Hardware limits in §II-C and §IV-B that force the two-Mem-tile, broadcast-K/V design.
  • domain assumption Causal mask sparsity is the dominant load-imbalance source for prefill on this pipeline, and balancing it within broadcast groups reduces sync stalls
    Stated in §IV-C; underpins the 38% placement claim.
  • ad hoc to paper Isolated attention microbenchmarks with BERT/Llama head dims represent the energy story for long-sequence agent inference on the SoC
    Implicit in abstract and §V-D energy conclusions without full-model serving data.
invented entities (1)
  • STEEL three-stage AIE pipeline with sparsity-aware placement independent evidence
    purpose: Map fused FlashAttention onto XDNA spatial array while balancing causal-mask work
    Software/dataflow construct, not a new physical entity; independent evidence is the open implementation and silicon measurements.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU." pith.science (2026). https://pith.science/paper/6EK3DDNM

@misc{pith2026260709385,
  author       = {Pith},
  title        = {Pith review of: STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6EK3DDNM}},
  note         = {Machine review of arXiv:2607.09385}
}
read the original abstract

The growing adoption of large language model-based agents within operating system workflows has increased the importance of energy-efficient inference on laptop-class systems-on-chip (SoCs). While cloud offloading remains common, it introduces reliability and privacy concerns that are particularly problematic for agentic workloads. Recent laptop SoCs, therefore, incorporate neural processing engines (NPUs) optimized for energy efficiency; however, effectively mapping attention mechanisms onto NPUs remains challenging due to architectural diversity and explicit data-movement programming models. In this work, we present STEEL, the first open-source implementation of FlashAttention targeting XDNA-like NPUs. STEEL introduces a dataflow formulation of prefill attention, enabling efficient exploitation of spatial parallelism and on-chip memory. Furthermore, STEEL addresses the load imbalance induced by the causal mask by leveraging a sparsity-aware pipeline placement onto the NPU array, reducing synchronization overhead and improving utilization. We evaluate STEEL on the AMD Ryzen AI 9 HX 370 SoC and compare its performance against optimized CPU and GPU implementations. Experimental results show that STEEL reduces energy consumption by an average of 9.17x and 1.75x relative to CPU and GPU baselines, respectively. On XDNA 1, STEEL achieves an average 9.6x latency reduction over the prior state of the art, and delivers a 22.8x speedup on average compared to a layer-by-layer attention implementation on XDNA 2.

Figures

Figures reproduced from arXiv: 2607.09385 by the authors.

Figure 1
Figure 1. Overview of the software stack used to program the XDNA™ 2 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the XDNA™ 2 NPU. XDNA™ 2 features three types [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the data movement between DRAM and the STEEL [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Description of two pipeline placement strategies over the attention [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Benchmark of the layer-by-layer attention against the STEEL’s fused [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Benchmark of STEEL against DATO [13] on XDNA™ 1 for several [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Benchmark of the attention energy efficiency on AMD Ryzen™ AI [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 6 linked inside Pith

  1. [1]

    AIOS: LLM Agent Operating System,

    K. Meiet al., “AIOS: LLM Agent Operating System,” Aug. 2025

    2025

  2. [2]

    Intelligence per Watt: Measuring Intelligence Efficiency of Local AI,

    J. Saad-Falconet al., “Intelligence per Watt: Measuring Intelligence Efficiency of Local AI,” Nov. 2025, arXiv:2511.07885 [cs]

    arXiv 2025

  3. [3]

    A survey on privacy risks and protection in large language models,

    K. Chenet al., “A survey on privacy risks and protection in large language models,”Journal of King Saud University Computer and Information Sciences, vol. 37, no. 7, p. 163, Aug. 2025

    2025

  4. [4]

    AMD XDNA NPU in Ryzen AI Processors,

    A. Ricoet al., “AMD XDNA NPU in Ryzen AI Processors,”IEEE Micro, vol. 44, no. 6, pp. 73–82, Nov. 2024

    2024

  5. [5]

    FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUs,

    H. Linet al., “FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUs,” Oct. 2024, arXiv:2410.16663 [cs]

    arXiv 2024

  6. [6]

    NITRO: LLM Inference on Intel Laptop NPUs,

    A. Fei and M. S. Abdelfattah, “NITRO: LLM Inference on Intel Laptop NPUs,” Dec. 2024, arXiv:2412.11053 [cs]

    arXiv 2024

  7. [7]

    LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention,

    S. Yanget al., “LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention,” May 2025

    2025

  8. [8]

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,

    T. Dao, “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,” Oct. 2023

    2023

Show all 24 references
  1. [9]

    ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers,

    G. Islamogluet al., “ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers,” in2023 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), Aug. 2023, pp. 1–6

    2023

  2. [10]

    PACE: An Optimal Piecewise Polynomial Approximation Unit for Flexible and Efficient Transformer Non- linearity Acceleration

    A. S. Prasadet al., “PACE: An Optimal Piecewise Polynomial Approximation Unit for Flexible and Efficient Transformer Non- linearity Acceleration.” IEEE Computer Society, Jul. 2025, pp. 1–6

    2025

  3. [11]

    Efficiency, Expressivity, and Extensibility in a Close- to-Metal NPU Programming Interface,

    E. Hunhoffet al., “Efficiency, Expressivity, and Extensibility in a Close- to-Metal NPU Programming Interface,” in2025 IEEE 33rd Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), May 2025, pp. 85–94, iSSN: 2576-2621

    2025

  4. [12]

    Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve,

    A. Agrawalet al., “Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve,” 2024, pp. 117–134

    2024

  5. [13]

    Dato: A Task-Based Programming Model for Dataflow Accelerators,

    S. Fanget al., “Dato: A Task-Based Programming Model for Dataflow Accelerators,” Sep. 2025, arXiv:2509.06794 [cs]

    2025 arXiv

  6. [14]

    The Llama 3 Herd of Models,

    A. Grattafioriet al., “The Llama 3 Herd of Models,” Nov. 2024, arXiv:2407.21783 [cs]

    2024 arXiv

  7. [15]

    Attention is All you Need,

    A. Vaswaniet al., “Attention is All you Need,” inAdvances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., 2017

    2017

  8. [16]

    Online normalizer calculation for softmax,

    M. Milakov and N. Gimelshein, “Online normalizer calculation for softmax,” Jul. 2018, arXiv:1805.02867 [cs]

    2018 arXiv

  9. [17]

    Qualcomm Hexagon DSP: An architecture optimized for mobile multimedia and communications,

    L. Codrescu, “Qualcomm Hexagon DSP: An architecture optimized for mobile multimedia and communications,” in2013 IEEE Hot Chips 25 Symposium (HCS), Aug. 2013, pp. 1–23

    2013

  10. [18]

    ASCEND-CC: Confidential Computing on Heterogeneous NPU for Emerging Generative AI Workloads,

    A. Dharet al., “ASCEND-CC: Confidential Computing on Heterogeneous NPU for Emerging Generative AI Workloads,” Jun. 2024

    2024

  11. [19]

    SPARTA: Spatial Acceleration for Efficient and Scalable Horizontal Diffusion Weather Stencil Computation,

    G. Singhet al., “SPARTA: Spatial Acceleration for Efficient and Scalable Horizontal Diffusion Weather Stencil Computation,” inProceedings of the 37th ACM International Conference on Supercomputing, ser. ICS ’23. New York, NY , USA: Association for Computing Machinery, Jun. 202...

    2023

  12. [20]

    You Only Look Once: Unified, Real-Time Object Detection,

    J. Redmonet al., “You Only Look Once: Unified, Real-Time Object Detection,” 2016, pp. 779–788

    2016

  13. [21]

    14.5 Envision: A 0.26-to-10TOPS/W subword- parallel dynamic-voltage-accuracy-frequency-scalable Convolutional Neural Network processor in 28nm FDSOI,

    B. Moonset al., “14.5 Envision: A 0.26-to-10TOPS/W subword- parallel dynamic-voltage-accuracy-frequency-scalable Convolutional Neural Network processor in 28nm FDSOI,” in2017 IEEE International Solid-State Circuits Conference (ISSCC), Feb. 2017, pp. 246–247, iSSN: 2376-8606

    2017

  14. [22]

    Basic Linear Algebra Subprograms for Fortran Usage,

    C. L. Lawsonet al., “Basic Linear Algebra Subprograms for Fortran Usage,”ACM Trans. Math. Softw., vol. 5, no. 3, pp. 308–323, Sep. 1979

    1979

  15. [23]

    AKG: automatic kernel generation for neural processing units using polyhedral transformations,

    J. Zhaoet al., “AKG: automatic kernel generation for neural processing units using polyhedral transformations,” inProceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, ser. PLDI 2021. New York, NY , USA: Association for...

    2021

  16. [24]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library,

    A. Paszkeet al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” inAdvances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc., 2019

    2019

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.