Pith. sign in

REVIEW 4 major objections 6 minor 43 references

TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Compressing each token embedding with tensor-train decomposition keeps task performance at 2x embedding compression while cutting per-query energy by about half.

desk verdict A plausible training-free embedding-compression method whose compression results are believable, but whose headline energy claim rests on a cost model that omits the decoder and uses an inconsistent memory/compute ratio. read the letter →

arxiv 2506.13514 v1 pith:NIH3TFGA submitted 2025-06-16 cs.CL cs.LGcs.NAmath.NA

Mingxue Xu , Yao Lei Xu , Danilo P. Mandic This is my paper · ORCID
classification cs.CLcs.LGcs.NAmath.NA
keywords tensor-traindecompositionembeddingcompressionsmalllanguagemodelson-deviceinferenceenergyefficiencymatrixproductstatelow-rankfactorizationedgedeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Small language models deployed on phones and single-board computers are held back by their embedding layers, which can account for more than 20 percent of parameters. This paper tries to establish that those layers can be compressed without any training by decomposing each embedding vector into a tensor-train, or matrix-product-state, form. It reports that on GPT-2, CerebrasGPT, and OPT models with under a billion parameters, this gives around 2x embedding-layer compression while language modelling, classification, and zero-shot reasoning performance stay comparable to the original models, and that a single query on a Raspberry Pi 5 consumes roughly half the energy. Because compression is training-free and per-vector, vocabulary updates can be handled on the device itself rather than requiring a server round-trip.

What carries the argument

The central object is the Tensor-Train Decomposition (TTD), applied per token embedding vector rather than to the whole embedding matrix. Each $d$-dimensional vector is reshaped into an order-$N$ tensor with mode sizes $I_1,\dots,I_N$ and decomposed by TT-SVD into a chain of small cores $G^{(k)} \in \mathbb{R}^{r_{k-1} \times I_k \times r_k}$; this chain is a Matrix Product State (MPS). The compression ratio is $d / \sum_{k=1}^N r_{k-1} I_k r_k$ per vector. At inference, the cores are contracted back into a full embedding before entering the transformer, so the decoder is untouched, and the per-vector locality is what lets new tokens be compressed on the edge device in milliseconds.

What would settle it

Recompute the energy ratio in Equations (2) and (3) using the paper's own Table 1 costs (70-260 pJ per float32 memory access versus 1-3 pJ per computation, i.e. a ratio of 23-260 instead of 5) while including decoder weight traffic; if whole-model single-query energy stays above half of the uncompressed model's, the half-energy claim is falsified. The decisive version is a power-meter measurement of end-to-end inference on the same Raspberry Pi.

Watch

Extended reading notes

Core claim

The central claim is that a pretrained token embedding matrix carries per-vector low-rank structure that can be removed without retraining. Each embedding vector $x \in \mathbb{R}^d$ is reshaped into an order-$N$ tensor $X$ and decomposed by TT-SVD into $N$ cores $G^{(k)} \in \mathbb{R}^{r_{k-1} \times I_k \times r_k}$; the cores form a Matrix Product State whose total parameter count is much smaller than $d$. The paper shows that about 2x compression of the embedding layer can be reached with perplexity, sentiment classification, and zero-shot reasoning scores close to the uncompressed model, and that this compression reduces the estimated energy of a single query by about half before the decoder. On the device, a new token is compressed in the same way within milliseconds, which is what makes dynamic vocabulary updates practical.

Load-bearing premise

The half-energy claim rests on an energy model that counts only the embedding stage of a query and assumes a memory-to-computation cost ratio of 5; if the decoder's energy is included, the claimed saving is not established.

Editorial extensions

If this is right

  • Embedding layers of sub-billion models can be reduced to about half their parameter count with negligible loss on language modelling, sentiment classification, and zero-shot reasoning.
  • A single query's estimated energy on a low-end CPU device drops by roughly 50 percent because memory-bound embedding look-ups are replaced by cheaper tensor contractions.
  • Vocabulary adaptation becomes an on-device operation: a new token's embedding is tensorized and decomposed locally, with per-token compression latency under a few milliseconds on Raspberry Pi.
  • At comparable compression ratios, per-vector tensor compression preserves performance on unconstrained reasoning tasks such as ARC-challenge and BoolQ better than matrix SVD compression.
  • Because the decoder is left untouched, this embedding compression can be combined with other techniques that target the transformer layers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The half-energy claim is best read as an upper bound on whole-model savings, since the paper's own equations count only embedding-stage energy and ignore the decoder, which usually dominates inference cost.
  • The same per-vector tensor-train cores could be quantized or pruned, which would push compression beyond the reported 2x embedding ratio.
  • The per-vector decomposition is model-agnostic and could be tested on BERT-style encoders or multilingual models, where embedding tables are also large relative to total parameters.
  • A power-meter measurement of end-to-end inference on the same device would separate embedding-stage savings from decoder energy and settle whether the 50 percent figure holds in practice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes TensorSLM, a training-free method for compressing the token embedding layers of sub-billion-parameter language models via tensor-train decomposition. Each embedding vector is reshaped into a tensor and stored as matrix product state cores obtained by TT-SVD. The authors evaluate the method on GPT-2, CerebrasGPT, and OPT models, reporting compression ratios, perplexity, sentiment classification accuracy, zero-shot reasoning scores, latency on a Raspberry Pi 5, and estimated energy consumption. The abstract claims that at roughly 2.0x embedding compression the method preserves language task performance while halving the energy consumption of a single query.

Significance. The methodological core is simple and reproducible: per-vector tensor-train decomposition of embeddings, with no fine-tuning, is a reasonable candidate for memory-constrained deployment, and the empirical comparison against SVD and Tucker provides useful evidence that the tensor structure retains task performance better than a matrix baseline in some settings. The measured latency on a Raspberry Pi is a concrete, useful artifact, and the paper explicitly addresses the edge deployment scenario. However, the central quantitative claim of the paper — that single-query energy consumption drops by half — is not supported by the evidence as presented, because the energy model in Section 4.1 excludes the decoder, charges the full vocabulary per query, and uses an uncalibrated memory-to-computation energy ratio. If the claim is rescoped to embedding-stage memory traffic under the stated cost model, the compression and task-performance results remain interesting, but the headline claim requires either real power measurements or a substantially revised analysis.

major comments (4)
  1. [Section 4.1, Eqs. (2)-(3)] The energy model is explicitly stated to apply "only before the decoder" and yet the abstract and conclusion claim that "the energy consumption of a single query drops by half" for the whole inference process. For the evaluated models, the decoder (attention, MLP, normalization) dominates both parameter count and per-token memory traffic; for example, GPT-2's embedding layer accounts for roughly 39M of the 124M total parameters listed in Table 6. Since per-token inference reads or writes decoder weights at every layer, omitting the decoder makes the reported ratio an embedding-stage ratio, not a full-query energy ratio. This is load-bearing for the paper's main claim and must be fixed by either measuring total device energy or explicitly restricting the claim to the embedding stage.
  2. [Eq. (2)] The original-model energy E_nu = nu(dV + ld) charges the full vocabulary size V as a per-query memory cost, but an autoregressive query of length l accesses only l embedding rows, so the original per-query embedding-memory traffic is O(ld), not O(Vd). Since V >> l, the dV term dominates the denominator and inflates the apparent energy saving of the compressed model. The paper's energy ratio should compare per-query memory access costs (ld versus l times the compressed-storage parameter count), not storage sizes of the embedding matrix.
  3. [Appendix F.3, Table 1] The estimate uses nu/tau = 5 for the ratio of memory-energy cost to computation-energy cost, but the paper's own Table 1 lists memory cost at 70-260 pJ/float32 and computation cost at roughly 1-3 pJ/float32, which implies nu/tau in the range 23-260. The reported value of 5 is therefore inconsistent with the stated hardware parameters, and since omega_TT depends directly on this ratio, the "half energy" figure is not calibrated. A sensitivity analysis over the plausible range of nu/tau, or direct power measurements, is needed to support the claim.
  4. [Section 5.3 and Appendix G] The energy comparison is reported only for compression cases selected post hoc as having "negligible" task performance drop, defined in Appendix G as perplexity no more than 100.0. For GPT-2-class models on WikiText2 this is a large absolute degradation, and the selection makes it difficult to assess the actual Pareto frontier of compression, performance, and energy. The paper should either report energy for a fixed compression ratio (for example, 2.0x) across all models or provide a scatter of energy ratios for all tested configurations, so the reader can see how the half-energy figure depends on the selection rule.
minor comments (6)
  1. [Figure 2h] The caption states that 100% represents original energy consumption, but the plot shows only estimated ratios and provides no confidence intervals or measured data points; please label the y-axis as "estimated energy cost ratio" and state that these values come from Eqs. (2)-(3), not from direct measurement.
  2. [Appendix F.2] Appendix F.2 mentions a YOJOCK J7-c USB power meter, but no measured power or energy values are reported anywhere in the paper; either provide the measurements or remove the mention to avoid implying that the energy results were measured.
  3. [Section 4.1] In the energy analysis the text says the estimate is "similar with (Yang et al., 2017)" but does not specify how the cited method's parameters map to nu and tau; please give a one-sentence derivation or citation to the specific equation in that work.
  4. [Section 4.2 and Eq. (7)] The reconstruction step is described as following Eq. (7), but the actual reconstruction procedure is defined in Section B.1; either cite that appendix or move the contraction equation forward so the reference is unambiguous.
  5. [Tables 5 and 6] The notation PPL_alpha is used in the table captions but is not defined in the main text; define it at first use in Section 5.1 or in the caption of Table 5.
  6. [Algorithm 1] The truncation parameter delta = epsilon * sqrt(N-1) * ||X||_F is introduced without a reference or justification; please cite the TT-SVD truncation analysis or add a brief explanation of why this choice guarantees the stated error bound.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central performance and compression claims are empirical, and the energy ratio is an explicit cost-model calculation rather than a fitted or definitionally forced prediction.

full rationale

The paper's derivation chain is not circular. TensorSLM compresses each pre-trained token embedding vector with standard TT-SVD (Oseledets, 2011), then reconstructs the embeddings and feeds them through the unchanged decoder; all language-task claims (perplexity, sentiment classification, zero-shot reasoning) come from direct evaluation of the compressed model against the original model, so they are not built into the construction. The compression ratio (Eq. 21) and the energy ratios (Eqs. 2-5) are explicit algebraic definitions and arithmetic computations from stated hyperparameters and the Table 1 cost ranges; the 'energy drops by half' statement is an estimate from that cost model, not a parameter fitted to the claimed outcome. The only author self-citation, (Lu et al., 2024), is used for the background definition of SLMs and is not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The main weaknesses are scoping and calibration, not circularity: the cost model is explicitly limited to processing 'only before the decoder' (Sec. 4.1), while the abstract and conclusion generalize to 'single query' and 'inference energy'; and the choice ν/τ = 5 (App. F) is an assumption rather than a measurement. Likewise, selecting compression cases with PPL below 100 before reporting energy (Secs. 5.3, G) is a reporting choice that does not make the energy ratio equivalent to the selection criterion by construction. These issues are correctness risks, not circular steps.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. It introduces several hand-chosen hyperparameters and one ad hoc energy-model assumption; the energy ratio and PPL threshold directly shape the headline claims.

free parameters (5)
  • Tensor mode dimensions I_k = Model-specific, e.g. 4x2x17x4x2 for CerebrasGPT-256M; not fully tabulated
    The reshape shape determines the TT decomposition; no principled rule is provided and the appendix proof that I = 2 is optimal is incorrect.
  • TT ranks r_k = Set to achieve target compression ratios; r = 1 for maximum compression, other values not fully listed
    Ranks control the compression ratio and approximation error; selection is empirical.
  • Approximation accuracy epsilon in Algorithm 1 = Not reported
    The truncation threshold in TT-SVD is an input to Algorithm 1 but no values are given.
  • Energy ratio nu/tau = 5
    Used in Section F.3 to compute energy ratios; inconsistent with Table 1 which gives 23 to 260 for Raspberry Pi.
  • PPL threshold for 'negligible' drop = 100
    Section G defines negligible performance drop as perplexity at most 100; this loose criterion is used to select the energy comparison cases.
assumptions (4)
  • standard math Standard TT-SVD with error bound (Oseledets 2011)
    Algorithm 1 applies the standard tensor-train SVD; the error bound is assumed.
  • domain assumption Each pretrained token embedding, after reshaping to the chosen tensor shape, is well approximated at low TT rank
    The method's performance relies on this low-rank structure; evidence is empirical only.
  • ad hoc to paper Inference energy of a query can be represented by the embedding-stage memory and compute formulas with nu/tau = 5, ignoring decoder layers
    Introduced in Section 4.1 and used to make the half-energy claim; not measured and inconsistent with Table 1.
  • domain assumption New tokens can be added by compressing a supplied embedding vector on the edge device without retraining
    The workflow in Section 2.1 assumes new token embedding vectors exist; the paper does not explain how they are produced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices." pith.science (2026). https://pith.science/paper/NIH3TFGA

@misc{pith2026250613514,
  author       = {Pith},
  title        = {Pith review of: TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NIH3TFGA}},
  note         = {Machine review of arXiv:2506.13514}
}
abstract

Small Language Models (SLMs, or on-device LMs) have significantly fewer parameters than Large Language Models (LLMs). They are typically deployed on low-end devices, like mobile phones and single-board computers. Unlike LLMs, which rely on increasing model size for better generalisation, SLMs designed for edge applications are expected to have adaptivity to the deployment environments and energy efficiency given the device battery life constraints, which are not addressed in datacenter-deployed LLMs. This paper addresses these two requirements by proposing a training-free token embedding compression approach using Tensor-Train Decomposition (TTD). Each pre-trained token embedding vector is converted into a lower-dimensional Matrix Product State (MPS). We comprehensively evaluate the extracted low-rank structures across compression ratio, language task performance, latency, and energy consumption on a typical low-end device, i.e. Raspberry Pi. Taking the sub-billion parameter versions of GPT-2/Cerebres-GPT and OPT models as examples, our approach achieves a comparable language task performance to the original model with around $2.0\times$ embedding layer compression, while the energy consumption of a single query drops by half.

Figures

Figures reproduced from arXiv: 2506.13514 by the authors.

Figure 1
Figure 1. Typical SLM layer composition and the SLM application requirement of adaptability. patterns. This superiority is more pronounced when no fine-tuning data is available to adjust model parameters for specific deployment environments. The contributions of this paper are summarised as follows: 1. We systematically analyse LLMs on high-end GPU servers and SLMs on low-end edge devices to address the two unique requirement… view at source ↗
Figure 2
Figure 2. Experimental results. (a): Perplexity-compression trade-off across different model sizes. This trade-off is measured by the ratio between perplexity and compression ratio of embedding layers; lower ratio values indicate better trade-offs. (b): Perplexity of the compressed models with different tensor decomposition approaches. (c)-(f): Task performance on sentiment classification with increasing compression ratio. Hi… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 14 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    S mol LM

    H ugging F ace. S mol LM . https://huggingface.co/huggingface/Smol, 2024. [Accessed 20-11-2024]

    work page 2024

  3. [3]

    Tqcompressor: improving tensor decomposition methods in neural networks via permutations

    Abronin, V., Naumov, A., Mazur, D., Bystrov, D., Tsarova, K., Melnikov, A., Oseledets, I., and Brasher, R. Tqcompressor: improving tensor decomposition methods in neural networks via permutations. arXiv preprint arXiv:2401.16367, 2024

    arXiv 2024

  4. [4]

    Online embedding compression for text classification using low rank matrix factorization

    Acharya, A., Goel, R., Metallinou, A., and Dhillon, I. Online embedding compression for text classification using low rank matrix factorization. In Proceedings of the AAAI conference on artificial intelligence , pp.\ 6196--6203, 2019

    work page 2019

  5. [5]

    Direction is what you need: Improving Word Embedding Compression in Large Language Models

    Ba azy, K., Banaei, M., Lebret, R., Tabor, J., and Aberer, K. Direction is what you need: improving word embedding compression in large language models. arXiv preprint arXiv:2106.08181, 2021

    work page Pith review arXiv 2021

  6. [6]

    Piqa: Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pp.\ 7432--7439, 2020

    work page 2020

  7. [7]

    Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  8. [8]

    Efficient GPT model pre-training using tensor train matrix representation

    Chekalina, V., Novikov, G., Gusak, J., Panchenko, A., and Oseledets, I. Efficient GPT model pre-training using tensor train matrix representation. In Huang, C.-R., Harada, Y., Kim, J.-B., Chen, S., Hsu, Y.-Y., Chersoni, E., A, P., Zeng, W. H., Peng, B., Li, Y., and Li, J. (eds.), Proceedings of the 37th Pacific Asia Conference on Language, Information and...

    work page 2023

Show all 43 references
  1. [9]

    S., Gusak, J., Oseledets, I., and Panchenko, A

    Chekalina, V., Novikov, G. S., Gusak, J., Oseledets, I., and Panchenko, A. Efficient gpt model pre-training using tensor train matrix representation. ArXiv, abs/2306.02697, 2023 b

    2023 arXiv

  2. [10]

    One billion word benchmark for measuring progress in statistical language modeling

    Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013

    2013 arXiv

  3. [11]

    Groupreduce: Block-wise low-rank approximation for neural language model shrinking

    Chen, P., Si, S., Li, Y., Chelba, C., and Hsieh, C.-J. Groupreduce: Block-wise low-rank approximation for neural language model shrinking. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processi...

    2018

  4. [12]

    Drone: Data-aware low-rank compression for large nlp models

    Chen, P., Yu, H.-F., Dhillon, I., and Hsieh, C.-J. Drone: Data-aware low-rank compression for large nlp models. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 29321--29334. ...

    2021

  5. [13]

    R., and Sun, Y

    Chen, T., Min, M. R., and Sun, Y. Learning k-way d-dimensional discrete codes for compact embedding representations. In International Conference on Machine Learning, pp.\ 854--863. PMLR, 2018 b

    2018

  6. [14]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  7. [15]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018

    2018 arXiv

  8. [16]

    S., Desai, A., Poli, M., Grogan, J., Liu, A., Rao, A., Rudra, A., and Re, C

    Dao, T., Chen, B., Sohoni, N. S., Desai, A., Poli, M., Grogan, J., Liu, A., Rao, A., Rudra, A., and Re, C. Monarch: Expressive structured matrices for efficient and accurate training. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proc...

    2022

  9. [17]

    Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster

    Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et al. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. arXiv preprint arXiv:2304.03208, 2023

    2023 arXiv

  10. [18]

    Kronecker decomposition for GPT compression

    Edalati, A., Tahaei, M., Rashid, A., Nia, V., Clark, J., and Rezagholizadeh, M. Kronecker decomposition for GPT compression. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume...

    2022 doi

  11. [19]

    Tensorized embedding layers

    Hrinchuk, O., Khrulkov, V., Mirvakhabova, L., Orlova, E., and Oseledets, I. Tensorized embedding layers. In Cohn, T., He, Y., and Liu, Y. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp.\ 4847--4860, Online, November 2020. Association for Comp...

    2020 doi

  12. [20]

    Language model compression with weighted low-rank factorization

    Hsu, Y.-C., Hua, T., Chang, S., Lou, Q., Shen, Y., and Jin, H. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=uPv9Y3gmAI5

    2022

  13. [21]

    Melting point: Mobile evaluation of language transformers

    Laskaridis, S., Kateveas, K., Minto, L., and Haddadi, H. Melting point: Mobile evaluation of language transformers. arXiv preprint arXiv:2403.12844, 2024

    2024 arXiv

  14. [22]

    S., Patel, A., Tuli, S., Shen, Y., Jin, H., and Hsu, Y.-C

    Lin, C.-H., Gao, S., Smith, J. S., Patel, A., Tuli, S., Shen, Y., Jin, H., and Hsu, Y.-C. Modegpt: Modular decomposition for large language model compression. arXiv preprint arXiv:2408.09632, 2024

    2024 arXiv

  15. [23]

    Mcunet: Tiny deep learning on iot devices

    Lin, J., Chen, W.-M., Lin, Y., cohn, j., Gan, C., and Han, S. Mcunet: Tiny deep learning on iot devices. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 11711--11722. Curran Associa...

    2020

  16. [24]

    A., and Rezagholizadeh, M

    Lioutas, V., Rashid, A., Kumar, K., Haidar, M. A., and Rezagholizadeh, M. Improving word embedding factorization for compression using distilled nonlinear neural decomposition. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.\ 2774--2784, 2020

    2020

  17. [25]

    Improved residual vector quantization for high-dimensional approximate nearest neighbor search

    Liu, S., Lu, H., and Shao, J. Improved residual vector quantization for high-dimensional approximate nearest neighbor search. arXiv preprint arXiv:1509.05195, 2015

    2015 arXiv

  18. [26]

    M obile LLM : Optimizing sub-billion parameter language models for on-device use cases

    Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoorthi, R., Lai, L., and Chandra, V. M obile LLM : Optimizing sub-billion parameter language models for on-device use cases. In Salakhutdinov, R., Kolter, Z., Heller, K., Wel...

    2024

  19. [27]

    D., and Xu, M

    Lu, Z., Li, X., Cai, D., Yi, R., Liu, F., Zhang, X., Lane, N. D., and Xu, M. Small language models: Survey, measurements, and insights. arXiv preprint arXiv:2409.15790, 2024

    2024 arXiv

  20. [28]

    and Sun, W

    Luo, H. and Sun, W. Addition is all you need for energy-efficient language models. arXiv preprint arXiv:2410.00907, 2024

    2024 arXiv

  21. [29]

    L., Daly, R

    Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp.\ 142--150, Portland, Oregon, US...

    2011

  22. [30]

    Ladabert: Lightweight adaptation of bert through hybrid model compression

    Mao, Y., Wang, Y., Wu, C., Zhang, C., Wang, Y., Yang, Y., Zhang, Q., Tong, Y., and Bai, J. Ladabert: Lightweight adaptation of bert through hybrid model compression. arXiv preprint arXiv:2004.04124, 2020

    2004 arXiv

  23. [31]

    H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., et al

    Mehta, S., Sekhavat, M. H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., et al. Openelm: An efficient language model family with open-source training and inference framework. arXiv preprint arXiv:2404.14619, 2024

    2024 arXiv

  24. [32]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Byj72udxe

    2017

  25. [33]

    Oseledets, I. V. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33 0 (5): 0 2295--2317, 2011. doi:10.1137/090752286. URL https://doi.org/10.1137/090752286

    2011 doi

  26. [34]

    Qiu, S., Potapczynski, A., Finzi, M., Goldblum, M., and Wilson, A. G. Compute better spent: Replacing dense layers with structured matrices. arXiv preprint arXiv:2406.06248, 2024

    2024 arXiv

  27. [35]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  28. [36]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  29. [37]

    Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter

    Sanh, V. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019

    1910 arXiv

  30. [38]

    Social iqa: Commonsense reasoning about social interactions

    Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce...

    2019

  31. [39]

    K ronecker BERT : Significant compression of pre-trained language models through kronecker decomposition and knowledge distillation

    Tahaei, M., Charlaix, E., Nia, V., Ghodsi, A., and Rezagholizadeh, M. K ronecker BERT : Significant compression of pre-trained language models through kronecker decomposition and knowledge distillation. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V. (eds.), Proceedin...

    2022

  32. [40]

    Lighttoken: A task and model-agnostic lightweight token embedding framework for pre-trained language models

    Wang, H., Li, R., Jiang, H., Wang, Z., Tang, X., Bi, B., Cheng, M., Yin, B., Wang, Y., Zhao, T., and Gao, J. Lighttoken: A task and model-agnostic lightweight token embedding framework for pre-trained language models. In Proceedings of the 29th ACM SIGKDD Conference on Knowled...

    2023

  33. [41]

    A method to estimate the energy consumption of deep neural networks

    Yang, T.-J., Chen, Y.-H., Emer, J., and Sze, V. A method to estimate the energy consumption of deep neural networks. In 2017 51st Asilomar Conference on Signals, Systems, and Computers, pp.\ 1916--1920, 2017. doi:10.1109/ACSSC.2017.8335698

    2017

  34. [42]

    Asvd: Activation-aware singular value decomposition for compressing large language models

    Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y., and Sun, G. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023

    2023 arXiv

  35. [43]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.