Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

An FPGA overlay processor that skips unneeded tokens and layers at runtime claims 1.23x–3.83x better bandwidth efficiency than GPUs and other FPGAs for LLM inference.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:09 UTC pith:NYGCJ4DJ

load-bearing objection Real architecture novelty under a comparison that doesn't reconstruct: the KV-buffer and overpacking work deserves review, but the 1.23x–3.83x bandwidth claim is not yet falsifiable. the 5 major comments →

arxiv 2603.14785 v2 pith:NYGCJ4DJ submitted 2026-03-16 cs.AR

SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation

Zicheng He , Anhao Zhao , Xiaoyu Shen , Chen Wu , Lei He This is my paper
classification cs.AR
keywords dynamic computation allocationtoken-wise pruningFPGA acceleratorLLM inferenceKV cache reuseKV invariance buffermixed-precision DSP packingoverlay processor
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that token-wise and layer-wise dynamic pruning—where a lightweight router decides which computations each token actually needs—can be made hardware-efficient rather than remaining a theoretical FLOP saving. SkipOPU is an FPGA overlay processor that hides the latency of nonlinear operations by fusing their reductions with adjacent matrix multiplications, packs mixed-precision (FP16 activation, INT4 weight) multiply-accumulates into DSP blocks using an overpacking technique, and keeps reused key-value cache entries on-chip via a KV-invariance buffer so skipped tokens do not fragment HBM traffic. If the results hold, dynamic pruning shifts from an algorithmic curiosity to a deployable inference method, and the KV-reuse strategy cuts KV storage by up to 25.4% while sustaining near-peak bandwidth on an off-the-shelf FPGA with HBM.

Core claim

SkipOPU demonstrates a complete hardware mapping for token-level dynamic computation: instead of treating routers and nonlinearities as serial overheads, it decouples reductions (RMSNorm/softmax statistics) from element-wise computation, computes them incrementally in parallel with linear layers, and then streams normalized tiles directly into the next matrix multiply. On the memory side, it observes that the key/value entries of skipped tokens are invariant across layers until reactivated, so it pins those reused entries in an on-chip URAM buffer and restricts HBM traffic to newly generated KV entries, converting irregular cross-layer accesses into long, local bursts. On the compute side, i

What carries the argument

The carrier of the argument is a set of three coupled mechanisms: (1) a fused dataflow for router/RMSNorm and self-attention that performs softmax and RMSNorm statistics incrementally on-chip while the PE array is still computing, so nonlinear latency is hidden; (2) a mixed-precision PE array that overpacks two FP16 mantissa multiplications into one DSP via 5-bit truncation and uses one-time FP-to-BFP conversion for fixed-point accumulation, avoiding expensive floating-point accumulation; (3) a token-wise HBM mapping plus a proactive KV-invariance buffer that serves reused KV entries from URAM, eliminating irregular AXI bursts and channel collisions. These are what convert dynamic sparsity i

Load-bearing premise

The claimed 1.23x–3.83x bandwidth-efficiency advantage rests on normalization factors in the comparison table that are stated but not derived, and on runtime figures whose source (measured on the board, simulated, or estimated) is never disclosed; if those factors or figures are not what they appear, the headline comparison is unsupported.

What would settle it

Run the same implementation on the target board, measure raw tokens/s for a 7B model with 4-bit weights at the reported prefill/decode lengths without normalization, and compare against a standard GPU inference engine at the same quantization; if the raw throughput gap does not reproduce, the core bandwidth-efficiency claim collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Dynamic token/layer pruning can be executed on real hardware with bandwidth efficiency close to the HBM ceiling, not just claimed as FLOP reductions.
  • KV-cache reuse across layers cuts storage by up to 25.4%, extending the feasible context length on memory-limited edge FPGAs.
  • The fused, incremental nonlinear dataflow removes the pipeline bubbles typically caused by softmax/RMSNorm row reductions.
  • DSP overpacking with BFP accumulation roughly halves the LUT cost of accumulation for hybrid FP16/INT4 workloads under the same DSP budget.
  • The design sustains its advantage at 1,024-token decode lengths, where prior FPGA accelerators degrade.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The KV-invariance buffer idea transfers beyond FPGAs: any accelerator with a software-managed scratchpad could pin reused KV entries to avoid scattered DRAM fetch, so the claimed memory-system benefit is architectural, not vendor-specific.
  • If the paper's aggregate effective bandwidth (467.8 GB/s) is real, on-chip reuse can push throughput past the physical HBM2 ceiling, suggesting that for this workload the binding constraint shifts from off-chip bandwidth to on-chip buffer capacity and ports.
  • The DSP overpacking/BFP accumulation scheme may generalize to any mixed-precision transformer inference, even without dynamic pruning, making low-bit-weight LLM inference cheaper on FPGAs generally.
  • A natural testable extension is to scale the KV-invariance buffer to batch sizes above 1, where multiple concurrent sequences could share or contend for buffer ports; the paper evaluates only batch 1.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents SkipOPU, an FPGA overlay processor for SkipGPT-style LLM inference with token-wise and layer-wise dynamic computation skipping. The architecture combines a fused dataflow for routing/RMSNorm/softmax, a mixed-precision DSP-overpacked PE array with BFP-style fixed-point accumulation, a tile-based nonlinear processing engine, and a proactive on-chip KV invariance buffer that converts irregular cross-layer KV reuse traffic into localized URAM accesses. Evaluation on an AMD U280 is reported for Llama-2-7B/13B workloads, with the abstract claiming 1.23x–3.83x higher bandwidth efficiency than GPU/FPGA baselines and up to 25.4% KV storage reduction.

Significance. If the performance and memory-efficiency numbers can be substantiated, the work would be a useful contribution to FPGA-based LLM acceleration for dynamic, input-dependent execution. The paper has real strengths: a complete RTL implementation described in detail, explicit resource utilization tables, cycle-level scheduling diagrams, two concrete fused dataflow algorithms, and a novel on-chip KV reuse mechanism. The central comparison, however, is built on a normalization procedure that cannot be reconstructed from the text, and the paper never states how the reported token/s and bandwidth numbers were obtained. The significance of the headline claims is therefore conditional: the architecture may be sound, but the evidence for its claimed advantage is not yet independently checkable.

major comments (5)
  1. [§5.5, Table 3] The headline 1.23x–3.83x bandwidth-efficiency advantage rests on the normalization stated as “to a 7B parameter model with dense 4-bit weight quantization,” but the table cannot be reproduced from that rule. vLLM is scaled 45.3→181.2 (×4.0) despite being HF16; FlightLLM is left at 55 despite W8; the gpt2-345m rows are scaled by ≈0.096–0.098; SkipOPU 7B scales 124.1→143.4 (×1.16) and 13B scales 75.6→144.5 (×1.91). No formula or per-row basis is given. Moreover, the denominator for BW efficiency is “the maximum achievable bandwidth supported by the design’s operating frequency,” which is never defined; for SkipOPU, 88.4% corresponds to 467.8 GB/s aggregate, implying a denominator near 529 GB/s, above the U280’s physical 460 GB/s HBM2 ceiling. Counting on-chip URAM reuse in an HBM bandwidth-efficiency metric is apples-to-oranges unless precisely justified. Please provide a closed-form norma
  2. [§5.1.2, §5.5] The provenance of the performance numbers is missing. The paper states synthesis and implementation details (Vivado 2020.1, 225 MHz core, 450 MHz HBM) but never says whether the reported token/s figures (e.g., 124.1 tokens/s for Llama-2-7B) come from on-board measurement, RTL simulation, or an analytical/compiler model. This distinction is essential for judging the memory-bandwidth claims in §5.4 (408.7 GB/s dense, 360.2 GB/s mapped, 467.8 GB/s aggregate) and the end-to-end table. Please specify the measurement methodology, describe the test harness and how the CPU/FPGA system is invoked, and, if simulation was used, provide at least one on-board/simulation correlation point.
  3. [§4.4.2, §4.4.3] The zero-overhead claim for the KV history buffer is internally inconsistent. §4.4.2 describes the buffer update as “computationally free” and “temporal-free,” but §4.4.3 explicitly restricts scheduling width: Round 1 is limited to two reused entries to match buffer read ports, and Round 2 is capped at three unskipped entries so that proactive updates do not exceed the buffer’s write-port capacity. A capacity-driven cap on the memory schedule is a throughput constraint, not a zero-cost mechanism. Please quantify how often this cap binds in the evaluated prefill/decode traces and include its effect in the aggregate-bandwidth figures, or temper the “free” claim accordingly.
  4. [Abstract, §5.4] The second headline claim—reducing KV storage overhead by up to 25.4% across varying sequence lengths—is not supported by any experiment or derivation. §5.4 only states that KV reuse reduces total data volume by “approximately 25%.” No measurement or formula links the 25.4% figure to the skip statistics or to the prefill:decode configurations. Please provide the storage reduction as a function of sequence length and derive the reported percentage, or remove the precise number.
  5. [§5.1.1] No end-to-end accuracy or perplexity result is reported for the pruned and quantized workloads. The motivation claims that dynamic computation allocation preserves model accuracy, and Table 1 reports only PE-level numerical error. Since the accelerator’s performance is meaningful only if the SkipGPT+GTPQ models remain accurate, please report at least one end-to-end quality metric (perplexity or task accuracy) for Llama-2-7B/13B at the 25% skip rate used in Table 3, or explicitly state that accuracy validation is inherited from prior work with a quantitative citation.
minor comments (5)
  1. [Table 3] The table formatting is broken: the design/device rows list seven columns, the Frequency row lists eight values, and Token/s, Norm Throughput, and BW efficiency rows list nine values. This must be fixed before the table can be interpreted.
  2. [§2.2, §4.1, §4.4.2] Typos and wording: “dynmaic” (§2.2), “Grumbel-softmax” (§2.2), “continous datflow” (§4.1), “slighted pruned LLMs” (§4.4.2), and “reduces the the” (near §4.4.2) should be corrected.
  3. [Figure 8/9] Figures 8 and 9 lack complete axis labels and units. In particular, the x-axis in Figure 8 (“prefill:decode”) should list the exact sequence lengths used, and Figure 9 should state whether the y-axis is GB/s or normalized bandwidth.
  4. [Algorithm 2] The notation in Algorithm 2 is dense: the loop bound H/2 and the two-head packing variables are not clearly defined in the text before the algorithm. A short explanation of the head-packing grouping would improve reproducibility.
  5. [§4.2.3, Table 1] Table 1 reports resource usage “averaged for one column of accumulation,” but the row for IMPL3 lists “32 + 31.5” DSPs. Please clarify the units and why a fractional DSP average is reported.

Circularity Check

2 steps flagged

Partial circularity: the reported 25.4% KV-storage reduction is the configured ~25% skip rate restated as a result, and the bandwidth-efficiency table is normalized by a self-defined 'maximum achievable bandwidth'; the core DSP/dataflow design is independently derived.

specific steps
  1. fitted input called prediction [Section 5.1.1 (Workload); Section 5.4; Abstract/§7 conclusion]
    "We evaluate SkipOPU with Llama2 [40] and apply SkipGPT methodology to prune Llama2 models with around 25% skipping probability. ... As we move toward the more memory-efficient scenarios utilizing KV reuse, the total data volume is reduced by approximately 25% ... can reduce up to 25.4% KV storage overhead across varying sequence lengths."

    The headline storage-reduction number is the experiment's input skip rate renamed as an output result. Under the KV-reuse rule of Eq. (1)-(2), a token skipped at a layer by definition contributes no new KV entry; total KV storage scales with the active token-layer visits. Setting the SkipGPT pruning target to 'around 25% skipping probability' therefore forces a maximum KV-storage reduction of roughly 25%. The 'up to 25.4%' claim is a restatement of this chosen workload parameter, not an independent quantity derived from the hardware.

  2. self definitional [Section 5.5 (end-to-end comparison) and Table 3; Section 5.4 (effective bandwidth)]
    "our bandwidth efficiency metric is calculated relative to the maximum achievable bandwidth supported by the design's operating frequency, which allows us to recognize performance gains resulting from architectural dataflow optimizations. ... an aggregate effective bandwidth of 467.8 GB/s, effectively surpassing the physical ceiling of the HBM2 interface through architectural data reuse."

    The central 1.23x-3.83x bandwidth-efficiency comparison is normalized by a self-chosen denominator that is never defined. For SkipOPU on U280, 467.8 GB/s (which already includes on-chip URAM reuse) is reported as 88.4% efficiency, implying a denominator near 529 GB/s, above the physical 460 GB/s HBM2 ceiling. Efficiency defined relative to the design's own 'maximum achievable bandwidth' is a self-referential ratio; comparing it with vLLM's 31.5% or FlightLLM's 66% (relative to physical peak) does not measure the same quantity, so the headline advantage is by construction partly an artifact of the chosen normalization.

full rationale

The FPGA implementation contributions—the deep-fused router/RMSNorm/softmax dataflow, DSP overpacking with BFP accumulation, and KV-invariance buffer scheduling—are developed from internal reasoning and are not circular. The circularity burden is partial: (1) the KV-storage reduction headline restates the ~25% skip rate selected in §5.1.1, and (2) the bandwidth-efficiency metric in §5.5 is normalized by an undefined, design-specific 'maximum achievable bandwidth', making the headline cross-platform comparison internally normalized rather than independently falsifiable. The paper also uses SkipGPT [55] and cross-layer KV-reuse [16], both from the same group, as the workload and the memory-reuse premise; these are load-bearing for the application-level value proposition but the hardware derivation itself does not reduce to those citations. Per the hard rules, I have not treated the unexplained per-row normalization factors in Table 3 as circularity per se; they are a correctness/reproducibility concern. Overall score 6 reflects one input-derived result plus a self-defined efficiency metric, with the central architecture retaining substantial independent content.

Axiom & Free-Parameter Ledger

6 free parameters · 8 axioms · 4 invented entities

The paper contributes a hardware implementation, so its 'axioms' are the upstream algorithm assumptions it imports plus the hardware-datasheet behaviors it relies on. The most notable feature of the ledger is that nearly every algorithm-level input comes from the authors' own prior publications: the SkipGPT routing paradigm and its accuracy claims ([55], co-authored by present authors), cross-layer KV reuse ([16, 55]), and the 25% skip rate. The engineering degrees of freedom - truncation widths, buffer capacity, packing factor - are tuned against the paper's own control experiments and workload, which is normal for architecture but means the headline numbers are not parameter-free. There are no invented physical entities; the invented structures are hardware components with no external falsifiable handle.

free parameters (6)
  • Skip probability (router skip rate) = ~25%
    Workload parameter from SkipGPT (§5.1.1). It determines the ~25% KV traffic reduction, the 25.4% storage-saving headline, and most of the memory-side speedups.
  • Table 3 normalization multipliers = 4.0 (vLLM), 1.0 (FlightLLM), 0.098 (gpt2-345m rows), 1.156 (SkipOPU 7B), 1.912 (SkipOPU 13B)
    Per-row factors chosen to bring all designs to a '7B/W4 dense' baseline; applied inconsistently with the stated rule, and the SkipOPU factors have no stated basis (§5.5, Table 3). Directly shape the 1.23x-3.83x headline.
  • DSP overpacking truncation/injection widths = Truncate LSB of u0 and MSB of u1; 5-bit auxiliary multiplier; 15-bit product to accumulation tree
    Chosen via Table 1 control experiments to trade numerical error (0.035-0.116%) against LUT savings; documented engineering tuning, specific to this PE.
  • KV invariance buffer capacity = 512 URAMs / 1024 tokens' KV entries
    Matches the maximum evaluated decode length (1024 tokens), so the headline KV-buffer benefit is only shown within its capacity envelope (§5.2, §5.1.1).
  • Multi-head packing factor = 2 heads per pass (H/2 loop)
    Dataflow choice in Algorithm 2 that aggregates KV accesses and bounds on-chip attention storage; not swept in experiments.
  • Core/HBM clock frequencies = 225 MHz core+DDR4; 450 MHz HBM2
    Design choice in §5.1.2; the bandwidth-efficiency denominator is defined relative to a 'maximum achievable bandwidth supported by the design's operating frequency' (§5.5), so the frequency choice directly affects reported efficiency.
axioms (8)
  • domain assumption SkipGPT's token-wise routing at ~25% skip preserves the model's generation accuracy.
    Imported from [55] (co-authored by two present authors); this paper measures no end-to-end accuracy. §5.1.1.
  • domain assumption Replacing a skipped token's KV entries with those from the most recent executed layer preserves attention semantics (Eq. 2).
    Adopted from [16,55]; the paper explicitly relies on this for the KV-invariance buffer. §2.1.
  • domain assumption During decode, routing bitmasks for all preceding tokens at layer i+1 are known one step ahead, so buffer maintenance is 'temporal-free'.
    Argued in prose, not measured; underpins the zero-overhead claim. §4.4.2.
  • standard math DSP48E2 behaves per the Xilinx datasheet (27x18 native multiply, pre-adder, sign inversion, C-port injection, 48-bit adder) and the truncation/sign-inversion arithmetic exactly recovers the two FP16 products.
    The overpacking derivation in §4.2.1 depends on the DSP internal datapath.
  • domain assumption Block-floating-point accumulation with 15-bit truncated fractional products preserves accuracy within LLM dynamic range.
    Validated only on random data and Llama-2-sampled inputs in Table 1; no attention-output or end-to-end error measured. §4.2.2-4.2.3.
  • standard math FlashAttention's online-softmax update rules carry over to the fused QK^T pipeline without significant error accumulation.
    Algorithm 2 uses the m/l rescaling from [8] verbatim. §3.2.
  • ad hoc to paper 'Maximum achievable bandwidth supported by the design's operating frequency' is a valid cross-platform normalization denominator.
    Defined only in §5.5; it produces efficiency values (88.4%) whose absolute basis is design-relative, and lets aggregate on-chip+off-chip throughput exceed the HBM2 physical ceiling.
  • domain assumption GPTQ symmetric 4-bit weight quantization preserves generation quality on the evaluated tasks.
    Taken from [14]; the workload section adopts it without re-evaluation. §5.1.1.
invented entities (4)
  • KV invariance buffer (on-chip URAM history buffer) no independent evidence
    purpose: Stores cross-layer-invariant KV entries of skipped tokens so attention reads them on-chip instead of issuing irregular HBM fetches (§4.4.2, §5.2).
    Internal design component; its benefit is shown only in the paper's own scheduling analysis and self-reported bandwidth figures, and it is capped at 1024 tokens.
  • 'Temporal-free' proactive buffer update no independent evidence
    purpose: Background maintenance of the KV buffer using one-step-ahead bitmasks, claimed to have zero latency overhead (§4.4.2).
    Asserted in prose ('We argue', 'computationally free'); no dedicated measurement isolates buffer-update cost.
  • Truncated-mantissa DSP overpacking with pre-adder sign inversion no independent evidence
    purpose: Maps two FP16 mantissa products onto one DSP48E2 with a 5-bit auxiliary correction multiply, raising DSP throughput with recovery logic folded into the C-port addition (§4.2.1).
    A bit-level arithmetic scheme; validated only by Table 1 control experiments, without formal proof of exactness or bitstream-level power/timing measurements.
  • Token-wise HBM port mapping no independent evidence
    purpose: Pins each token's KV to a single HBM pseudo-channel so reused and fresh KV reads are long contiguous bursts (§4.4.1).
    A memory-layout scheme validated only in the paper's reported bandwidth numbers; no externally reproducible trace.

pith-pipeline@v1.3.0-alltime-deepseek · 20646 in / 32096 out tokens · 286568 ms · 2026-08-02T18:09:42.707447+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have achieved remarkable performance across a wide range of tasks, but their inference efficiency remains a critical bottleneck due to rapidly growing parameters. Recent advances in dynamic computation allocation address this challenge by exploiting the highly uneven contributions of different tokens and layers, enabling selective execution that significantly reduces redundant computation while preserving model accuracy. However, existing hardware platforms and accelerators are primarily optimized for uniform, static execution, limiting their ability to efficiently support such dynamic inference patterns. In this work, we propose SkipOPU, an FPGA-based overlay processor that dynamically allocates computation across tokens and layers with high flexibility through a lightweight routing mechanism. First, we decouple reduction operations from element-wise computation in nonlinear modules and perform reductions incrementally, which enables both stages to be fused with adjacent linear operations (router or matrix multiplication) for effective latency hiding. Second, motivated by asymmetric sensitivity to numerical precision between activation and weight, we design a PE array that efficiently supports float-fixed hybrid execution. A novel DSP overpacking technique is introduced to maximize hardware utilization while minimizing resource overhead. Finally, we develop a proactive on-chip KV history buffer that exploits cross-layer KV invariance of pruned tokens, eliminating irregular HBM accesses during decoding and supplementing off-chip bandwidth through high-locality on-chip reuse. Experimental results demonstrate that SkipOPU on an AMD U280 FPGA outperforms GPU and other FPGA-based accelerators by 1.23x-3.83x in bandwidth efficiency for LLMs inference with dynamic computation allocation and can reduce up to 25.4% KV storage overhead across varying sequence lengths.

Figures

Figures reproduced from arXiv: 2603.14785 by Anhao Zhao, Chen Wu, Lei He, Xiaoyu Shen, Zicheng He.

Figure 1
Figure 1. Figure 1: (b) summarizes the key challenges in developing hardware accelerators for SkipGPT: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: (a) The framework of SkipGPT models. (b) Challenges in SkipGPT computation, and the corresponding solution in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SkipOPU where HBM operates at high clock domain and accelerator operates at low clock domain [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison between differences DSP packing techniques for FP16 fraction multiplication. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a) The detailed design of mixed precision PE. (b) FP16 accumulation tree with BFP-FP convertion [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The micro-architecture of nonlinear processing engine(NPE). [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: (a) Token-wise memory mapping scheme for KV cache (b) Channel interference to load KV cache from previous layer. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) Valid KV cache scheduling. (b) Invalid KV cache scheduling. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Normalized MHA speedup under different dataflow optimization and varying [prefill:decode] [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Effective memory bandwidth for KV cache with different mapping and scheduling options. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

    cs.AI 2026-07 conditional novelty 7.0

    Token-wise dynamic width pruning of attention-head and FFN groups, paired with mask-reordering GPU kernels, retains far more accuracy than dynamic depth pruning at 50% sparsity while delivering practical prefill/decod...

  2. UniRank: Unified Rank Allocation for Low-Rank LLM Compression

    cs.LG 2026-06 unverdicted novelty 7.0

    UniRank introduces dual local-global scoring for rank allocation in LLM low-rank decomposition plus rank-preserving fine-tuning, achieving up to 50% lower perplexity than uniform baselines in one-shot compression.

  3. Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models

    cs.LG 2026-06 conditional novelty 5.0

    On Llama3.1-8b, splitting a fixed sparsity budget as a near-balanced mix of parameter pruning and token-level layer skipping outperforms either mechanism alone and delays the sharp performance-decay point.

Reference graph

Works this paper leans on

55 extracted references · 4 canonical work pages · cited by 3 Pith papers

  1. [2]

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. arXiv:2404.05961 [cs.CL] https://arxiv.org/abs/2404.05961

    Pith/arXiv arXiv 2024

  2. [3]

    Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. 2025. Small language models are the future of agentic ai.arXiv preprint arXiv:2506.02153(2025)

    Pith/arXiv arXiv 2025

  3. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  4. [5]

    Jialin Cao, Xuanda Lin, Manting Zhang, Kejia Shi, Jun Yu, and Kun Wang. 2023. PP-Transformer: Enable Efficient Deployment of Transformers Through Pattern Pruning. In2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). 1–9. doi:10.1109/ICCAD57390.2023.10323836

    arXiv 2023

  5. [6]

    Hongzheng Chen, Jiahao Zhang, Yixiao Du, Shaojie Xiang, Zichao Yue, Niansong Zhang, Yaohui Cai, and Zhiru Zhang. 2024. Understanding the Potential of FPGA-based Spatial Acceleration for Large Language Model Inference.ACM Trans. Reconfigurable Technol. Syst.18, 1, Article 5 (Dec. 20 Zicheng He and Anhao Zhao, et al. 2024), 29 pages. doi:10.1145/3656177

    doi:10.1145/3656177 2024

  6. [7]

    Yuli Chen, Bo Cheng, Jiale Han, Yingying Zhang, Yingting Li, and Shuhao Zhang. 2025. DLP: Dynamic Layerwise Pruning in Large Language Models. arXiv:2505.23807 [cs.CL] https://arxiv.org/abs/2505.23807

    Pith/arXiv arXiv 2025

  7. [8]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135 [cs.LG] https://arxiv.org/abs/2205.14135

    Pith/arXiv arXiv 2022

  8. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL] https://arxiv.org/abs/1810.04805

    Pith/arXiv arXiv 2019

  9. [10]

    Longwei Ding, Anhao Zhao, Fanghua Ye, Ziyang Chen, and Xiaoyu Shen. 2026. From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models.arXiv preprint arXiv:2601.18091(2026)

    arXiv 2026

  10. [11]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929 [cs.CV] https://arxiv.org/abs/2010.11929

    Pith/arXiv arXiv 2021

  11. [12]

    Yingqi Fan, Anhao Zhao, Jinlan Fu, Junlong Tong, Hui Su, Yijie Pan, Wei Zhang, and Xiaoyu Shen. 2025. VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 18896–18913

    2025

  12. [13]

    Chao Fang, Aojun Zhou, and Zhongfeng Wang. 2022. An Algorithm–Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers. IEEE Transactions on Very Large Scale Integration (VLSI) Systems30, 11 (2022), 1573–1586. doi:10.1109/TVLSI.2022.3197282

    arXiv 2022

  13. [14]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323 [cs.LG] https://arxiv.org/abs/2210.17323

    Pith/arXiv arXiv 2023

  14. [15]

    Tae Jun Ham, Yejin Lee, Seong Hoon Seo, Soosung Kim, Hyunji Choi, Sung Jun Jung, and Jae W. Lee. 2021. ELSA: Hardware-Software Co-design for Efficient, Lightweight Self-Attention Mechanism in Neural Networks. In2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA). 692–705. doi:10.1109/ISCA52012.2021.00060

    arXiv 2021

  15. [16]

    Chao Han, Yijuan Liang, Zihao Xuan, Daokuan Wu, Wei Zhang, and Xiaoyu Shen. 2025. Informed Routing in LLMs: Smarter Token-Level Computation for Faster Inference.arXiv preprint arXiv:2510.13831(2025)

    arXiv 2025

  16. [17]

    Zicheng He, Shaoqiang Lu, Tiandong Zhao, Jinlong Yan, Chen Wu, and Lei He. 2025. METAL: A Memory-Efficient Transformer Architecture for Long-Context Inference on FPGA.2025 IEEE 36th International Conference on Application-specific Systems, Architectures and Processors (ASAP)(2025), 93–100. https://api.semanticscholar.org/CorpusID:280696019

    2025

  17. [18]

    Zicheng He, Tiandong Zhao, Siyuan Miao, Chen Wu, and Lei He. 2024. An FPGA-Based Efficient Streaming Vector Processing Engine for Transformer- Based Models. In2024 2nd International Symposium of Electronics Design Automation (ISEDA). 722–727. doi:10.1109/ISEDA62518.2024.10617499

    arXiv 2024

  18. [19]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models.arXiv preprint arXiv:2203.1555610 (2022)

    Pith/arXiv arXiv 2022

  19. [20]

    Seongmin Hong, Seungjae Moon, Junsoo Kim, Sungjae Lee, Minsub Kim, Dongsoo Lee, and Joo-Young Kim. 2023. DFX: A Low-Latency Multi- FPGA Appliance for Accelerating Transformer-Based Text Generation. InProceedings of the 55th Annual IEEE/ACM International Symposium on Microarchitecture(Chicago, Illinois, USA)(MICRO ’22). IEEE Press, 616–630. doi:10.1109/MIC...

    arXiv 2023

  20. [21]

    Suyeon Hur, Seongmin Na, Dongup Kwon, Joonsung Kim, Andrew Boutros, Eriko Nurvitadhi, and Jangwoo Kim. 2023. A Fast and Flexible FPGA-based Accelerator for Natural Language Processing Neural Networks.ACM Trans. Archit. Code Optim.20, 1, Article 11 (Feb. 2023), 24 pages. doi:10.1145/3564606

    work page doi:10.1145/3564606 2023

  21. [22]

    Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical Reparameterization with Gumbel-Softmax. arXiv:1611.01144 [stat.ML] https://arxiv.org/ abs/1611.01144

    Pith/arXiv arXiv 2017

  22. [23]

    Yikun Jiang, Huanyu Wang, Lei Xie, Hanbin Zhao, Chao Zhang, Hui Qian, and John C.S. Lui. 2024. D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, I...

    work page doi:10.52202/079017-0055 2024

  23. [24]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361(2020)

    Pith/arXiv arXiv 2020

  24. [25]

    Hamza Khan, Asma Khan, Zainab Khan, Lun Bin Huang, Kun Wang, and Lei He. 2021. NPE: An FPGA-based Overlay Processor for Natural Language Processing. InThe 2021 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays(Virtual Event, USA)(FPGA ’21). Association for Computing Machinery, New York, NY, USA, 227. doi:10.1145/3431920.3439477

    arXiv 2021

  25. [26]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180 [cs.LG] https://arxiv.org/abs/2309.06180

    Pith/arXiv arXiv 2023

  26. [27]

    Haoran Li, Yulin Chen, Jinglong Luo, Jiecong Wang, Hao Peng, Yan Kang, Xiaojin Zhang, Qi Hu, Chunkit Chan, Zenglin Xu, Bryan Hooi, and Yangqiu Song. 2024. Privacy in Large Language Models: Attacks, Defenses and Future Directions. arXiv:2310.10383 [cs.CL] https://arxiv.org/abs/2310.10383

    Pith/arXiv arXiv 2024

  27. [28]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv:2306.00978 [cs.CL] https://arxiv.org/abs/2306.00978

    Pith/arXiv arXiv 2024

  28. [29]

    Jun Liu, Shulin Zeng, Li Ding, Widyadewi Soedarmadji, Hao Zhou, Zehao Wang, Jinhao Li, Jintao Li, Yadong Dai, Kairui Wen, Shan He, Yaqi Sun, Yu Wang, and Guohao Dai. 2025. FlightVGM: Efficient Video Generation Model Inference with Online Sparsification and Hybrid Precision on FPGAs. InProceedings of the 2025 ACM/SIGDA International Symposium on Field Prog...

    arXiv 2025

  29. [30]

    Wenjie Liu, Hao Wu, Xin Qiu, Yingqi Fan, Yihan Zhang, Anhao Zhao, Yunpu Ma, and Xiaoyu Shen. 2026. ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention.arXiv preprint arXiv:2602.07574(2026)

    Pith/arXiv arXiv 2026

  30. [31]

    Liqiang Lu, Yicheng Jin, Hangrui Bi, Zizhang Luo, Peng Li, Tao Wang, and Yun Liang. 2021. Sanger: A Co-Design Framework for Enabling Sparse Attention using Reconfigurable Architecture. InMICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture(Virtual Event, Greece)(MICRO ’21). Association for Computing Machinery, New York, NY, USA, 977...

    arXiv 2021

  31. [32]

    Shaoqiang Lu, Tiandong Zhao, Ting-Jung Lin, Rumin Zhang, Chen Wu, and Lei He. 2025. MCoreOPU: An FPGA-based Multi-Core Overlay Processor for Transformer-based Models.ACM Trans. Reconfigurable Technol. Syst.18, 3, Article 37 (Aug. 2025), 27 pages. doi:10.1145/3742437

    work page doi:10.1145/3742437 2025

  32. [33]

    Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Sing...

    Pith/arXiv arXiv 2025

  33. [34]

    Yubin Qin, Yang Wang, Dazheng Deng, Zhiren Zhao, Xiaolong Yang, Leibo Liu, Shaojun Wei, Yang Hu, and Shouyi Yin. 2023. FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation Prediction. InProceedings of the 50th Annual International Symposium on Computer Architecture(Orlando, FL, USA)(ISCA ’23). Association for Computing Machiner...

    doi:10.1145/3579371 2023

  34. [35]

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. arXiv:2404.02258 [cs.LG] https://arxiv.org/abs/2404.02258

    Pith/arXiv arXiv 2024

  35. [36]

    Leo Schwinn, David Dobre, Stephan Günnemann, and Gauthier Gidel. 2023. Adversarial Attacks and Defenses in Large Language Models: Old and New Threats. arXiv:2310.19737 [cs.AI] https://arxiv.org/abs/2310.19737

    Pith/arXiv arXiv 2023

  36. [37]

    Yixin Song, Zhenliang Xue, Dongliang Wei, Feiyang Chen, Jianxiang Gao, Junchen Liu, Hangyu Liang, Guangshuo Qin, Chengrong Tian, Bo Wen, et al. 2025. Smallthinker: A family of efficient large language models natively trained for local deployment.arXiv preprint arXiv:2507.20984(2025)

    Pith/arXiv arXiv 2025

  37. [38]

    Hui Su, Zhi Tian, Xiaoyu Shen, and Xunliang Cai. 2024. Unraveling the mystery of scaling laws: Part i.arXiv preprint arXiv:2403.06563(2024)

    Pith/arXiv arXiv 2024

  38. [39]

    Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. InProceedings of the 28th ACM International Conference on Information and Knowledge Management (Beijing, China)(CIKM ’19). Association for Computing Machinery, New York, NY, USA...

    arXiv 2019

  39. [40]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...

    Pith/arXiv arXiv 2023

  40. [41]

    Hanrui Wang, Zhekai Zhang, and Song Han. 2021. SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning. In2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 97–110. doi:10.1109/hpca51647.2021.00018

    arXiv 2021

  41. [42]

    Zhican Wang, Hongxiang Fan, and Guanghui He. 2025. DESA: Dataflow Efficient Systolic Array for Acceleration of Transformers.IEEE Trans. Comput.74, 6 (2025), 2058–2072. doi:10.1109/TC.2025.3549621

    arXiv 2025

  42. [43]

    Wilkinson

    James H. Wilkinson. 1994.Rounding Errors in Algebraic Processes. Dover Publications, Inc., USA

    1994

  43. [44]

    Chen Wu, Mingyu Wang, Xinyuan Chu, Kun Wang, and Lei He. 2021. Low-precision Floating-point Arithmetic for High-performance FPGA-based CNN Acceleration.ACM Trans. Reconfigurable Technol. Syst.15, 1, Article 6 (Nov. 2021), 21 pages. doi:10.1145/3474597

    work page doi:10.1145/3474597 2021

  44. [45]

    Chen Wu, Jinming Zhuang, Kun Wang, and Lei He. 2021. MP-OPU: A Mixed Precision FPGA-based Overlay Processor for Convolutional Neural Networks. In2021 31st International Conference on Field-Programmable Logic and Applications (FPL). 33–37. doi:10.1109/FPL53798.2021.00014

    arXiv 2021

  45. [46]

    Qiong Wu, Zhaoxi Ke, Yiyi Zhou, Xiaoshuai Sun, and Rongrong Ji. 2025. Routing experts: Learning to route dynamic experts in existing multi-modal large language models. InThe Thirteenth International Conference on Learning Representations

    2025

  46. [47]

    Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. 2023. Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity.Proc. VLDB Endow.17, 2 (Oct. 2023), 211–224. doi:10.14778/3626292.3626303 22 Zicheng He and Anhao Zhao, et al

    arXiv 2023

  47. [48]

    Xue Xia, Pong Eksombatchai, Nikil Pancha, Dhruvil Deven Badani, Po-Wei Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang, and Andrew Zhai. 2023. TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining(Long Beach, CA, ...

    arXiv 2023

  48. [49]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, et al. 2025. Native sparse attention: Hardware-aligned and natively trainable sparse attention. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 23078–23097

    2025

  49. [50]

    Zhengqing Yuan, Weixiang Sun, Yixin Liu, Huichi Zhou, Rong Zhou, Yiyang Li, Zheyuan Zhang, Wei Song, Yue Huang, Haolong Jia, Keerthi- ram Murugesan, Yu Wang, Lifang He, Jianfeng Gao, Lichao Sun, and Yanfang Ye. 2025. EfficientLLM: Efficiency in Large Language Models. arXiv:2505.13840 [cs.CL] https://arxiv.org/abs/2505.13840

    Pith/arXiv arXiv 2025

  50. [51]

    Dewen Zeng, Nan Du, Tao Wang, Yuanzhong Xu, Tao Lei, Zhifeng Chen, and Claire Cui. 2023. Learning to Skip for Language Modeling. arXiv:2311.15436 [cs.CL] https://arxiv.org/abs/2311.15436

    Pith/arXiv arXiv 2023

  51. [52]

    Shulin Zeng, Jun Liu, Guohao Dai, Xinhao Yang, Tianyu Fu, Hongyi Wang, Wenheng Ma, Hanbo Sun, Shiyao Li, Zixiao Huang, Yadong Dai, Jintao Li, Zehao Wang, Ruoyu Zhang, Kairui Wen, Xuefei Ning, and Yu Wang. 2024. FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs. InProceedings of the 2024 ACM/SIGDA International Sympo...

    arXiv 2024

  52. [53]

    Chen Zhang, Shijie Cao, Guohao Dai, Chenbo Geng, Zhuliang Yao, Wencong Xiao, Yunxin Liu, Ming Wu, Lintao Zhang, Guangyu Sun, Zhigang Ji, Runsheng Wang, and Ru Huang. 2025. Fine-Grained Structured Sparse Computing for FPGA-Based AI Inference.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems44, 7 (2025), 2544–2557. doi:10.1109/TC...

    arXiv 2025

  53. [54]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01068 [cs.CL]...

    Pith/arXiv arXiv 2022

  54. [55]

    Anhao Zhao, Fanghua Ye, Yingqi Fan, Junlong Tong, Zhiwei Fei, Hui Su, and Xiaoyu Shen. 2025. SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling. arXiv:2506.04179 [cs.CL] https://arxiv.org/abs/2506.04179

    Pith/arXiv arXiv 2025

  55. [56]

    Tiandong Zhao, Shaoqiang Lu, Chen Wu, and Lei He. 2025. ChatOPU: An FPGA-based Overlay Processor for Large Language Models with Unstructured Sparsity. InProceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design(Newark Liberty International Airport Marriott, New York, NY, USA)(ICCAD ’24). Association for Computing Machinery, New Yo...

    arXiv 2025