Pith. sign in

REVIEW 4 major objections 5 minor 70 references

Accelerating Large Language Models through Partially Linear Feed-Forward Network

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TARDIS cuts 80% of feed-forward parameters by folding non-linear activations into a single precomputed matrix.

desk verdict The per-neuron linearization idea is genuinely novel and the speedups are plausible, but the 80% parameter-reduction headline doesn't survive contact with the fallback weights, so the paper needs major revision before it stands. read the letter →

arxiv 2501.10054 v1 pith:GWTQHJBX submitted 2025-01-17 cs.LG cs.AI

Gansen Hu , Zhaoguo Wang , Jinglin Wei , Wei Huang , Haibo Chen This is my paper · ORCID
classification cs.LGcs.AI
keywords TARDISconstantfoldingfeed-forwardnetworkcompressionactivationlinearapproximationspeculativeresultfixingLLMinferenceaccelerationpost-trainingtransformerFFN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the feed-forward block of a transformer LLM can be compressed by treating its non-linear activation as a linear function on the input range where activations actually land, then folding the two weight matrices into one precomputed matrix. Because roughly 65% of each neuron's activation inputs sit inside about 20% of the total input range, a straight-line fit covers most computations with small error; a small predictor catches the outlier inputs and the system recomputes only those neurons exactly. On models from roughly 1.6B to 11.1B parameters, the paper reports removing 80% of feed-forward parameters, with end-to-end generation speeding up about 1.4x in a standard Python implementation and 1.6x in a production serving engine, at a roughly 10.9% accuracy cost on a 7B model. This matters because parameter-loading I/O, not arithmetic, dominates token generation, so shrinking the feed-forward weight matrices is a direct path to faster serving.

What carries the argument

The load-bearing object is the partially linear per-neuron activation $\varphi_n(x)=a_n x+b_n$ for $x\in[\ell_1,\ell_2)$ and $\varphi_n(x)=\sigma(x)$ otherwise, together with the folded matrices $C=\sum_n a_n W_{1:,n}W_{2n,:}$ and $B=\sum_n b_n W_{2n,:}$ that result from constant folding. A single linear range per neuron avoids the exponential blow-up in folded matrices that a multi-range scheme would create. The online predictor is a compressed version of $W_1$ ($2$-bit quantized in the implementation) that decides, from the layer input, which neurons are outside their linear range; a custom kernel then subtracts the folded contribution and recomputes the exact term for exactly those neurons. A two-level adaptive thresholding scheme assigns wider linear ranges to less important layers and neurons, and a greedy centroid-anchored range search with kernel density estimation finds each neuron's range from calibration data.

What would settle it

Run a folded model on held-out tokens and compare the predictor's in-range verdict with the true range membership of every neuron's activation input; any false negative (an out-of-range input the predictor does not flag) violates the correctness premise. A directly checkable version is to count, on a fresh corpus such as the WikiText-2 benchmark used in the paper, how often the predicted fixing set omits a neuron whose actual input lies outside its assigned interval.

Watch

Extended reading notes

Core claim

TARDIS's central discovery is that the two-matrix feed-forward computation $FFN(x)=\sigma(xW_1)W_2$ can be replaced by a single matrix multiplication $xC+B$ whenever $\sigma$ is replaced by a per-neuron line $ax+b$: associativity lets the constants absorb the weights as $C=\sum_n a_n W_{1:,n}W_{2n,:}$ and $B=\sum_n b_n W_{2n,:}$. The paper establishes that this substitution can be made safe in practice by profiling each neuron's activation input distribution on a tiny calibration set, assigning each neuron a 'hot' range and a least-squares linear fit, and using a compressed 2-bit quantized copy of $W_1$ as an online predictor that flags neurons whose current input falls outside their range. For flagged neurons the incorrect linear contribution is subtracted and the true $\sigma$ computation added back. Across five models, the folded layers keep perplexity close to the dense model at moderate compression, and at 80% FFN compression the method keeps far higher downstream accuracy than two leading pruning baselines, with measured 1.4x-1.6x end-to-end speedups.

Load-bearing premise

The argument assumes the compressed online predictor never misses a neuron whose activation input is outside its linear range, because a single missed outlier would leave the folded-matrix value in the output and silently corrupt the layer.

Editorial extensions

If this is right

  • For a feed-forward block with hidden width $h=4d$, the parameter count drops from $2dh$ to $d^2$ in theory, and the paper reports an 80% measured reduction after accounting for the predictor and folded matrix.
  • End-to-end token generation on a 7B model speeds up by about 1.4x in a standard transformer implementation and 1.6x in a production serving engine, at a reported accuracy loss around 10.9% on downstream tasks.
  • At 80% FFN compression, the method claims up to 65% higher downstream accuracy than state-of-the-art pruning baselines, with dramatically lower perplexity (e.g., 13.4 versus 1489-7796 on one benchmark).
  • Calibration needs only about 8 text samples: the actual in-range fraction lands within 1.8% of target, and swapping calibration corpora moves perplexity by less than 0.4.
  • The folding trick works for GELU, SiLU, and ReLU feed-forward networks, but the paper states it does not directly extend to gated (GLU-variant) FFNs, where folded matrices grow exponentially.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 2-bit predictor has any false negatives on deployment data, silently wrong layer outputs are possible; a cheap safeguard would compute a small always-exact control neuron per layer and trigger full fallback when that neuron's own input is out of range.
  • The nearly lossless results on a ReLU-based 6.7B model in the evaluation suggest TARDIS is a natural fit for models that already show activation sparsity, and the folded matrix itself could then be quantized or pruned for further gains.
  • One could replace the greedy centroid-anchored range search with an optimal dynamic-programming interval choice, likely reducing approximation error for the same threshold and shrinking the predictor's workload.
  • The large accuracy gap over pruning at high compression, if it holds, changes the cost-benefit calculus of extreme compression: the relevant baseline for 80% FFN compression may be a folded linear computation rather than sparse pruning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. TARDIS proposes to compress the feed-forward network (FFN) blocks of large language models by replacing each neuron's non-linear activation with a per-neuron linear approximation on a "hot" input range, pre-folding the two FFN matrices into a single d-by-d matrix C plus a bias vector B, and using a quantized predictor to flag out-of-range neurons whose exact contributions are recomputed online. The paper reports up to 80% parameter reduction in FFNs, accuracy substantially better than Wanda and RIA at high compression ratios, and end-to-end speedups of 1.4x on HuggingFace and 1.6x on vLLM for a 7B model. The constant-folding algebra in Section 5.2 is correct for exactly linear activations, and the evaluation is broad, covering five models, three language-generation benchmarks, three zero-shot tasks, and sensitivity analyses. However, the headline parameter-reduction claim is not supported, because the result-fixing mechanism requires retaining the original FFN weights for every neuron that can be flagged, and the compression-ratio accounting in Section 7.1 omits those weights.

Significance. If the parameter-reduction claim were valid, the paper would introduce a genuinely new compression paradigm: turning a two-matrix FFN with a non-linear activation into a single dense matrix while preserving most of the model's behavior. The constant-folding insight is elegant, the derivation in Section 5.2 is internally consistent for linear activations, and the speedup measurements may reflect a real speculative-execution effect worth studying. The paper also deserves credit for reporting calibration-set sensitivity, predictor-size sensitivity, and floating-point reordering effects, which are useful empirical checks. The central problem is that the comparison to pruning baselines is not commensurable and the reported compression ratio excludes the original weights that the fallback path must load. Because the paper's main advertised contribution is parameter reduction, and that claim is not substantiated by the deployment accounting, the significance of the work as presented is substantially lower than claimed.

major comments (4)
  1. [Section 5.4 and Section 7.1] The 80% parameter-reduction claim omits the original W1 and W2 weights required by the result-fixing mechanism. Section 5.4 states that fixing subtracts the linear approximation and adds back the exact computation for flagged neurons; this exact computation requires W1:,n and W2n,: (or equivalent data) for every neuron n that the predictor may flag. Because Section 5.3's predictor can flag any neuron on any token and no neuron is permanently excluded, a correct deployment must retain the full original FFN weight matrices. With the typical h = 4d architecture, retaining 2dh = 8d^2 original parameters already equals the original FFN size, and adding the folded matrix (d^2) and the predictor only increases the total footprint. Thus the abstract's '80% parameter reduction' and the Section 7.1 compression-ratio calculation are not supported as stated.
  2. [Section 7.1] The accuracy comparisons to Wanda and RIA are not apples-to-apples. Wanda and RIA delete pruned weights, so their reported compression ratios correspond to genuinely smaller models; TARDIS keeps the original weights available for online fallback while counting only the folded matrix and the 2-bit predictor in its compression ratio. The 'up to 65% higher accuracy' claim therefore compares a method that can revert to the exact original computation against methods that have permanently removed weights. The paper should recompute the reported ratios using the true deployed footprint, or explicitly reframe the result as a speculative-I/O reduction rather than a parameter reduction.
  3. [Section 5.3 and Section 7.6] Correctness depends on the 2-bit GPTQ predictor flagging every out-of-range neuron, but the paper never measures the predictor's false-negative rate. If the predictor misses an out-of-range neuron, the speculative folded-matrix result is kept without correction, so the layer output is silently wrong in a way that is not bounded by the linear-range approximation error. Figure 15 reports only aggregate perplexity versus predictor size, which does not reveal how often such misses occur or whether they are concentrated in particular layers or input regimes. The authors should measure false-negative rates on held-out data and, if nonzero, analyze their effect on the accuracy claims.
  4. [Section 7.2] For OPT-6.7B, Tables 3 and 4 report identical results at 50%, 70%, and 80% compression because TARDIS assigns the same linear function in all three cases. This is discussed only as a consequence of the model's activation pattern, but it also means that the reported compression ratio does not correspond to a change in the compressed artifact for this model. The parameter-reduction and speedup claims should be reconciled with the observation that the threshold parameter can leave the folded model unchanged.
minor comments (5)
  1. [Section 2.2] 'therotical analysis' should be 'theoretical analysis'.
  2. [Table 3 and Figure 11] The third dataset is labeled 'PDB' in Table 3 and Figure 11a; it should be 'PTB' for consistency with the text.
  3. [Section 7.2] The phrase 'We first quantifies the performance' should be 'We first quantify the performance'.
  4. [Section 7.4] The explanation that long-prompt generation 'reduces the sparsity patterns that TARDIS leverages' is imprecise: TARDIS does not exploit activation sparsity but rather the concentration of activation inputs in linear ranges. Consider rewording.
  5. [Section 7.5] The runtime breakdown in Figure 14 groups 'mask generation and index conversion' under 'Others', but the text does not define this category precisely; please state what operations are included and how their cost was measured.

Circularity Check

1 steps flagged · score 6.0 of 10

Compression-ratio metric is self-defined: reports 80% FFN parameter reduction while the original FFN weights required for result fixing are excluded from the count.

  1. self definitional [Section 7.1 (Settings) and Section 5.4 (Inference Runtime, Memory Footprint)]
    "We also account for the folded matrix and predictor size–around 10% and 4.8% of the model size–when calculating the compression ratio (Section 5.4). ... The memory requirements of TARDIS consist of three main components. 1) The constant folded matrix and its bias terms; 2) The lightweight predictor; 3) Only the original weights of neurons that require exact computation (fixing). ... This correction involves two steps. First, we remove the approximate results for the identified neurons. Then, we compute and add back their actual results using the original FFN computation."

    The reported 80% FFN parameter reduction is computed as if the deployed FFN consists only of the d-by-d folded matrix plus the 2-bit predictor. But TARDIS's own result-fixing design requires the original W1 columns and W2 rows for any neuron the predictor flags, and since the predictor can flag any neuron on any token, the full original FFN weights are part of the required model state. With h=4d those weights are exactly 2dh=8d^2, equal to the original FFN size; adding the folded matrix and predictor cannot yield an 80% reduction in total footprint. Thus the '80% reduction' is an artifact of excluding a required component from the compression-ratio definition, rather than a derived property of the folded computation.

full rationale

The constant-folding identity FFN(x)=xC+B under a linear activation is mathematically correct, and the linear parameters are fit on a small calibration set and evaluated on held-out benchmarks, so the accuracy and perplexity results are not circular. The out-of-range predictor is an actual quantized copy of W1 and is evaluated against downstream tasks, not fitted to those tasks. No load-bearing self-citation or imported uniqueness theorem appears; the only same-group citation (PowerInfer) supports a non-central empirical observation already demonstrated in Figure 6. The circularity is confined to the headline compression metric: the paper counts the folded matrix and predictor but not the original W1/W2 that result-fixing requires, even though any neuron can be flagged out-of-range. Defining the compression ratio this way makes the '80% parameter reduction' true by construction rather than by total model footprint. The comparison with pruning baselines inherits the same incommensurability. This inflates the central parameter-reduction claim but does not infect the algebraic derivation or the held-out accuracy numbers.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The core identity is a standard algebraic reorganization. The empirical parameters, per-neuron slopes and intercepts, thresholds, predictor precision, and calibration sample size, are fitted or chosen by hand. The method's success depends on the stability of activation distributions and on predictor accuracy, both assumed rather than demonstrated with error bounds.

free parameters (5)
  • coverage threshold t = varied from 0.1 to 0.8 in experiments
    User-specified target portion of activation inputs that should fall inside the linear range; directly controls the compression versus accuracy tradeoff.
  • per-neuron linear approximation slope and intercept (a_n, b_n) = not reported per neuron
    Obtained by least-squares regression on calibration samples within each neuron's selected range; these are fitted values, not derived from theory.
  • range expansion step size s = not specified
    Algorithm 1 expands the linear range in steps of size s; the value is never reported, making the search procedure underspecified.
  • predictor quantization precision = 2 bits
    The predictor is a GPTQ 2-bit quantization of W1; the choice affects both predictor size and accuracy, and only perplexity versus size is shown.
  • calibration dataset size and source = 8 samples of 2048 tokens from C4
    The paper reports sensitivity to sample count and source, but the specific choice of 8 samples is a manual setting, not derived from a criterion.
assumptions (5)
  • standard math Matrix multiplication is associative and distributes over linear transformations of the activation.
    Invoked in Section 3.1 to fold a(xW1)W2 into x(aW1W2).
  • domain assumption Activation input distributions measured on calibration data are representative of deployment inputs.
    Section 4.1 and Section 5.1 select per-neuron ranges using a small C4 sample; the whole method assumes these ranges cover the promised fraction of future activations.
  • domain assumption GELU and SiLU are well approximated by a single linear function inside the selected hot range.
    Section 5.1 fits one linear function per neuron; the approximation quality is measured on calibration data but is not guaranteed outside that data.
  • domain assumption The 2-bit GPTQ quantized predictor correctly identifies out-of-range neurons.
    Section 5.3 relies on this prediction to trigger result fixing; no false-negative rate is reported.
  • domain assumption Empirical MSE is an adequate proxy for neuron and layer importance when allocating thresholds.
    Section 5.1.1 minimizes calibration error to assign thresholds; this assumes lower approximation error on calibration data translates to better downstream accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accelerating Large Language Models through Partially Linear Feed-Forward Network." pith.science (2026). https://pith.science/paper/GWTQHJBX

@misc{pith2026250110054,
  author       = {Pith},
  title        = {Pith review of: Accelerating Large Language Models through Partially Linear Feed-Forward Network},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWTQHJBX}},
  note         = {Machine review of arXiv:2501.10054}
}
read the original abstract

Large language models (LLMs) demonstrate remarkable capabilities but face deployment challenges due to their massive parameter counts. While existing compression techniques like pruning can reduce model size, it leads to significant accuracy degradation under high compression ratios. We present a novel perspective inspired by constant folding in compiler optimization. Our approach enables parameter reduction by treating activation functions in LLMs as linear functions. However, recent LLMs use complex non-linear activations like GELU that prevent direct application of this technique. We propose TARDIS, which enables optimization of LLMs with non-linear activations by partially approximating them with linear functions in frequently occurring input ranges. For outlier inputs, TARDIS employs an online predictor to dynamically fall back to original computations. Our experiments demonstrate that TARDIS achieves 80% parameter reduction in feed-forward networks, while significantly outperforming state-of-the-art pruning methods Wanda and RIA with up to 65% higher accuracy. In practical deployments for a 7B model, TARDIS achieves 1.6x end-to-end inference speedup when integrated with the vLLM serving system, and 1.4x speedup with the widely adopted HuggingFace implementation, while incurring only a 10.9% accuracy trade-off.

Figures

Figures reproduced from arXiv: 2501.10054 by the authors.

Figure 1
Figure 1. The inference process of transformer based LLM [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 4
Figure 4. Figure of SiLU and GELU activation functions in a [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 3
Figure 3. Illustration of constant folding in the FFN block [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Density estimation of the activation function input [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 7
Figure 7. Figure 7: Architecture of TARDIS with offline and online [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 9
Figure 9. Figure 9: Example of the exponential growth in the number [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: The speculative approximation and result fixing [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison of Falcon-7B under dif [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Perplexity (left) and actual percentage of tokens [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Inference Speedup of Falcon-7B with TARDIS [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 15
Figure 15. Figure 15: Influence of predictor size on perplexity in Wiki [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 50 canonical work pages

  1. [1]

    org/wiki/Constant_folding

    Constant folding - wikipedia.https://en.wikipedia. org/wiki/Constant_folding

    work page

  2. [2]

    [Online; accessed 2024- 11-01]

    Cuda c++ programming guide. [Online; accessed 2024- 11-01]

    work page 2024

  3. [3]

    [Online; ac- cessed 2025-01-09]

    openai-community/gpt2-xl · hugging face. [Online; ac- cessed 2025-01-09]

    work page 2025

  4. [4]

    https://gcc.gnu.org/onlinedocs/gcc/ Optimize-Options.html

    Optimize options (using the gnu compiler collection (gcc)). https://gcc.gnu.org/onlinedocs/gcc/ Optimize-Options.html. (Accessed on 11/29/2024)

    work page 2024

  5. [5]

    https://github.com/python/cpython?tab= readme-ov-file

    python/cpython: The python programming language. https://github.com/python/cpython?tab= readme-ov-file. (Accessed on 11/29/2024)

    work page 2024

  6. [6]

    [Online; accessed 2024-11-24]

    Release pytorch 2.5.1: bug fix release · pytorch/pytorch. [Online; accessed 2024-11-24]

    work page 2024

  7. [7]

    [Online; accessed 2025-01-14]

    Release v0.6.6 · vllm-project/vllm. [Online; accessed 2025-01-14]

    work page 2025

  8. [8]

    [Online; accessed 2024-12-16]

    Sixth computational and data science school for hep (codas-hep 2024) (22-26 july 2024): Floating point arith- metic is not real· indico. [Online; accessed 2024-12-16]

    work page 2024

Show all 70 references
  1. [9]

    apache.org/sql/

    Spark sql & dataframes | apache spark.https://spark. apache.org/sql/. (Accessed on 11/29/2024)

    2024

  2. [10]

    https://huggingface.co/docs/ transformers/index

    Transformers. https://huggingface.co/docs/ transformers/index. (Accessed on 06/23/2024)

    2024

  3. [11]

    https://docs.rapids.ai/api/ cuml/stable/

    Welcome to cuml’s documentation! — cuml 24.10.00 documentation. https://docs.rapids.ai/api/ cuml/stable/. (Accessed on 12/03/2024)

    2024

  4. [12]

    [Online; accessed 2025-01-09]

    Eleutherai/lambada_openai · datasets at hugging face, 10 2023. [Online; accessed 2025-01-09]

    2023

  5. [13]

    [Online; accessed 2024-11-01]

    Lds.128 loads from shared memory - cuda / cuda pro- gramming and performance - nvidia developer forums, 8 2023. [Online; accessed 2024-11-01]

    2023

  6. [14]

    philschmid/sharegpt-raw · datasets at hugging face, 5

  7. [15]

    [Online; ac- cessed 2025-01-13]

    tiiuae/falcon-11b · hugging face, 6 2023. [Online; ac- cessed 2025-01-13]

    2023

  8. [16]

    [On- line; accessed 2025-01-09]

    allenai/ai2_arc · datasets at hugging face, 8 2024. [On- line; accessed 2025-01-09]

    2024

  9. [17]

    [Online; accessed 2025-01-09]

    facebook/opt-6.7b · hugging face, 11 2024. [Online; accessed 2025-01-09]

    2024

  10. [18]

    [Online; accessed 2025-01-08]

    google/switch-c-2048 · hugging face, 3 2024. [Online; accessed 2025-01-08]

    2024

  11. [19]

    The falcon series of open language models, 2023

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of...

    2023

  12. [20]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  13. [21]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  14. [22]

    Dsformer: Effective compression of text-transformers by dense-sparse weight factorization

    Rahul Chand, Yashoteja Prabhu, and Pratyush Kumar. Dsformer: Effective compression of text-transformers by dense-sparse weight factorization. arXiv preprint arXiv:2312.13211, 2023

    2023 arXiv

  15. [23]

    Fast and accurate deep network learning by exponential linear units (elus)

    Djork-Arné Clevert. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015

    2015 arXiv

  16. [24]

    Language modeling with gated convolutional networks

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learn- ing, pages 933–941. PMLR, 2017

    2017

  17. [25]

    Llm.int8(): 8-bit matrix multiplication for transformers at scale

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th Inter- national Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY , USA, 2024. Curran Associates Inc

    2024

  18. [26]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  19. [27]

    Sigmoid- weighted linear units for neural network function ap- proximation in reinforcement learning

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- weighted linear units for neural network function ap- proximation in reinforcement learning. Neural networks, 107:3–11, 2018

    2018

  20. [28]

    Optq: Accurate quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2022

    2022

  21. [29]

    Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. 13

    2023

  22. [30]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Bider- man, Sid Black, Anthony DiPofi, Charles Foster, Lau- rence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Linta...

    2024

  23. [31]

    Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023

    2023 arXiv

  24. [32]

    How to access global memory efficiently in cuda c/c++ kernels | nvidia technical blog, 4 2014

    Mark Harris. How to access global memory efficiently in cuda c/c++ kernels | nvidia technical blog, 4 2014. [Online; accessed 2024-11-01]

    2014

  25. [33]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  26. [34]

    Finequant: Unlocking effi- ciency with fine-grained weight-only quantization for llms

    Young Jin Kim, Rawn Henry, Raffy Fahim, and Hany Hassan Awadalla. Finequant: Unlocking effi- ciency with fine-grained weight-only quantization for llms. arXiv preprint arXiv:2308.09723, 2023

    2023 arXiv

  27. [35]

    Self-normalizing neural networks

    Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017

    2017

  28. [36]

    Pruning vs quanti- zation: Which is better? Advances in neural information processing systems, 36:62414–62427, 2023

    Andrey Kuzmin, Markus Nagel, Mart Van Baalen, Arash Behboodi, and Tijmen Blankevoort. Pruning vs quanti- zation: Which is better? Advances in neural information processing systems, 36:62414–62427, 2023

    2023

  29. [37]

    Efficient memory man- agement for large language model serving with page- dattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with page- dattention. In Proceedings of the 29th Symposium on Operating Systems Principle...

    2023

  30. [38]

    Bloom: A 176b-parameter open-access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. 2023

    2023

  31. [39]

    Losparse: Structured compression of large language models based on low-rank and sparse approximation

    Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse approximation. In International Conference on Machine Learning, pages 20336–20350. PMLR, 2023

    2023

  32. [40]

    A method for calculating the deriva- tive of activation functions based on piecewise linear approximation

    Xuan Liao, Tong Zhou, Longlong Zhang, Xiang Hu, and Yuanxi Peng. A method for calculating the deriva- tive of activation functions based on piecewise linear approximation. Electronics, 12(2):267, 2023

    2023

  33. [41]

    Awq: Activation- aware weight quantization for on-device llm compres- sion and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei- Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation- aware weight quantization for on-device llm compres- sion and acceleration. Proceedings of Machine Learning and Systems, 6:87–...

    2024

  34. [42]

    Deja vu: Con- textual sparsity for efficient llms at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Con- textual sparsity for efficient llms at inference time. In International Conference on Machine Learning, pages 22137–22176....

    2023

  35. [43]

    Llm- pruner: On the structural pruning of large language mod- els

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm- pruner: On the structural pruning of large language mod- els. Advances in neural information processing systems, 36:21702–21720, 2023

    2023

  36. [44]

    The penn treebank: an- notating predicate argument structure

    Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: an- notating predicate argument structure. In Proceedings of the Workshop on Human Language Technology, HLT ’94, page 114–119, ...

    1994

  37. [45]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  38. [46]

    Relu strikes back: Exploiting activation sparsity in large language models

    Seyed Iman Mirzadeh, Keivan Alizadeh-Vahid, Sachin Mehta, Carlo C del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models. In The Twelfth International Confer- ence on Learni...

    2024

  39. [47]

    Implementation of a digital neuron with nonlinear activation function using piecewise linear approximation technique

    Amit Mishra, Krishna Raj, et al. Implementation of a digital neuron with nonlinear activation function using piecewise linear approximation technique. In2007 Inter- natonal Conference on Microelectronics, pages 69–72. IEEE, 2007

    2007

  40. [48]

    On estimation of a probability den- sity function and mode

    Emanuel Parzen. On estimation of a probability den- sity function and mode. The annals of mathematical statistics, 33(3):1065–1076, 1962

    1962

  41. [49]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. 14

    2020

  42. [50]

    Ma- trix compression via randomized low rank and low pre- cision factorization

    Rajarshi Saha, Varun Srivastava, and Mert Pilanci. Ma- trix compression via randomized low rank and low pre- cision factorization. Advances in Neural Information Processing Systems, 36, 2023

    2023

  43. [51]

    Glu variants improve transformer, 2020

    Noam Shazeer. Glu variants improve transformer, 2020

    2020

  44. [52]

    Powerinfer: Fast large language model serving with a consumer-grade gpu

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with a consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Princi- ples, pages 590–606, 2024

    2024

  45. [53]

    A simple and effective pruning approach for large lan- guage models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large lan- guage models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  46. [54]

    Variable kernel density estimation

    George R Terrell and David W Scott. Variable kernel density estimation. The Annals of Statistics, pages 1236– 1265, 1992

    1992

  47. [55]

    Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty

    Inar Timiryasov and Jean-Loup Tastet. Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty. arXiv preprint arXiv:2308.02019, 2023

    2023 arXiv

  48. [56]

    Norm (mathemat- ics) - wikipedia, 9 2004

    Contributors to Wikimedia projects. Norm (mathemat- ics) - wikipedia, 9 2004. [Online; accessed 2024-11-13]

    2004

  49. [57]

    Llama: Open and efficient foun- dation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foun- dation lang...

    2023

  50. [58]

    Model compression and efficient inference for large language models: A survey

    Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.09748, 2024

    2024 arXiv

  51. [59]

    Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145, 2023

    2023 arXiv

  52. [60]

    A survey of resource- efficient llm and multimodal foundation models

    Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. A survey of resource- efficient llm and multimodal foundation models. arXiv preprint arXiv:2401.08092, 2024

    2024 arXiv

  53. [61]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  54. [62]

    Llmcbench: Benchmarking large language model compression for efficient deployment

    Ge Yang, Changyi He, Jinyang Guo, Jianyu Wu, Yifu Ding, Aishan Liu, Haotong Qin, Pengliang Ji, and Xian- glong Liu. Llmcbench: Benchmarking large language model compression for efficient deployment. In The Thirty-eight Conference on Neural Information Process- ing Systems Data...

    2024

  55. [63]

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Informa- tion Processing Systems, 35:27168–27183, 2022

    2022

  56. [64]

    Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity

    Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Kumar Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsit...

    2024

  57. [65]

    Shiftaddllm: Accelerat- ing pretrained llms via post-training multiplication-less reparameterization

    Haoran You, Yipin Guo, Yichao Fu, Wei Zhou, Hui- hong Shi, Xiaofan Zhang, Souvik Kundu, Amir Yazdan- bakhsh, and Yingyan Celine Lin. Shiftaddllm: Accelerat- ing pretrained llms via post-training multiplication-less reparameterization. arXiv preprint arXiv:2406.05981, 2024

    2024 arXiv

  58. [66]

    A novel sigmoid function approx- imation suitable for neural networks on fpga

    Peter W Zaki, Ahmed M Hashem, Emad A Fahim, Mostafa A Masnour, Sarah M ElGenk, Maggie Mashaly, and Samar M Ismail. A novel sigmoid function approx- imation suitable for neural networks on fpga. In 2019 15th International Computer Engineering Conference (ICENCO), pages 95–99. I...

    2019

  59. [67]

    Inves- tigating layer importance in large language models

    Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. Inves- tigating layer importance in large language models. In Proceedings of the 7th BlackboxNLP Workshop: Analyz- ing and Interpreting Neural Networks for NLP, pages 469–479, 2024

    2024

  60. [68]

    Plug-and-play: An efficient post-training pruning method for large lan- guage models

    Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play: An efficient post-training pruning method for large lan- guage models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  61. [69]

    A survey on efficient inference for large language models

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Ji- aming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhi- hang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models. arXiv preprint arXiv:2404.14294, 2024. 15

    2024 arXiv

  62. [2023]

    [Online; accessed 2025-01-07]

    2025

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.