Pith. sign in

REVIEW 5 major objections 5 minor 83 references

Low-rank decomposition and quantization are not orthogonal: combining them compounds error, and the order matters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:22 UTC pith:XBSGG6VE

load-bearing objection The empirical story — SVD before quantization helps and DAM is a useful fix — looks real; the advertised mathematical proofs of non-orthogonality and optimal order are not supported as written. the 5 major comments →

arxiv 2607.20434 v1 pith:XBSGG6VE submitted 2026-05-11 cs.CL cs.AI

Break Through the Compression Bottleneck: From Theory to Practice

Xiusheng Huang , Lu Wang , Yequan Wang , Jun Zhao , Kang Liu This is my paper
classification cs.CL cs.AI
keywords model compressionlow-rank decompositionquantizationnon-orthogonalitycompression orderDiagonal Adhesive Methodlarge language modelsSVD
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to establish that two widely used LLM compression techniques, low-rank decomposition and quantization, are not orthogonal: when applied together they produce combined error larger than the sum of the individual errors. It further claims that the order in which they are applied matters, with low-rank decomposition first followed by quantization being the strictly better order at both the matrix and dot-product level. To recover the accuracy lost to this interaction, the paper introduces a Diagonal Adhesive Method (DAM), a learnable diagonal rescaling of the decomposed weight factors that reduces quantization error from activation outliers. A sympathetic reader would care because model deployers routinely stack these two methods assuming the errors just add, and this paper's proof—if sound—changes that assumption and gives a cheap fix.

Core claim

On its own terms, the paper's central claim is Theorem 3.11: there exist input and weight vectors x, w such that the absolute dot-product error from low-rank-then-quantize is strictly smaller than the error from quantize-then-low-rank, written |E^D_{l,q}(x,w)| < |E^D_{q,l}(x,w)|. At the tensor level, Theorem 3.8 asserts that quantizing before SVD introduces extra error beyond the sum of individual errors, because rounding distorts the singular-value structure that SVD then amplifies. The practical conclusion is that low-rank decomposition should be applied before quantization, and that the remaining outlier-driven loss can be mitigated by a diagonal scaling matrix a chosen to balance quantiz

What carries the argument

The argument is carried by a decomposition of the combined compression error into individual errors plus a cross term: at the tensor level, the squared Frobenius norm has an extra 2·tr(A^T B) term, and the paper's strict inequality for one order hinges on asserting this cross term is 'usually positive'. At the dot-product level, the key quantity is the total energy of quantization error, which the paper argues scales with the number of elements quantized; since low-rank factors have k(n+m) elements versus nm for the full matrix, the low-rank-first order quantizes fewer elements. DAM is a diagonal matrix a inserted between the low-rank factors to equalize per-rank quantization-error variances

Load-bearing premise

The strict inequality for quantization-then-SVD rests on an unproven assertion that the cross term in the error decomposition is 'usually positive'; if that term is negative or negligible for some weight matrices, the theorem's strict inequality collapses.

What would settle it

Compute, for a real LLM weight matrix, the cross term 2tr((SVD_r(Q(W))−Q(W))^T(Q(W)−W)) over a range of ranks r and quantization bit-widths; if a single (layer, r, bit-width) combination yields a negative value, the claimed strict non-orthogonality is false for that case. A secondary check is to compare quantization error energy across matrices of different sizes to test whether it is truly proportional to element count.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any compression pipeline that stacks quantization and low-rank decomposition should apply low-rank first; reversing the order incurs avoidable error that grows with compression ratio.
  • Compression error budgets are not additive: the combined error is a superlinear function of the individual errors, so planners must reserve extra headroom when combining methods.
  • The diagonal rescaling (DAM) offers a retraining-free way to recover a large part of the lost accuracy, at negligible inference cost.
  • The proof framework gives a template for testing non-orthogonality of other compression method pairs (e.g., pruning with quantization), rather than assuming independence.
  • The theoretical ordering result holds at both matrix and dot-product level, so it applies to weight-only and weight-activation quantized inference alike.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension: compute the cross term 2tr((SVD_r(Q(W))−Q(W))^T(Q(W)−W)) on real weight matrices across layers and ranks; if any layer yields a negative cross term, Theorem 3.8's strict inequality reduces to a weaker bound and the 'non-orthogonality' claim would need reframing.
  • The element-count proportionality assumption at the dot-product level is an idealization; real quantization step sizes depend on per-block maxima, so the ordering advantage may vary across layers and bit-widths.
  • The same diagonal-balancing idea could be tested against other factorizations, such as combining pruning with quantization, where a similar cross-term error may appear.
  • The authors' own stated limitation—no evaluation on very large models or varied GPUs—leaves open whether the empirical gains replicate at deployment scale and on different hardware.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies the interaction between low-rank decomposition and quantization for LLM compression. It claims to provide the first mathematical proof that the two operations are non-orthogonal, that applying low-rank decomposition before quantization is strictly better, and that the proposed Diagonal Adhesive Method (DAM) reduces the compound error. The theoretical arguments are developed at the tensor level (Section 3.1) and the dot-product level (Section 3.2), and are supported by large-scale experiments on LLaMA, Qwen, and Mistral models. The paper also derives an optimal diagonal scaling matrix for DAM and validates it with perplexity and zero-shot accuracy comparisons.

Significance. If the central claims were established, the paper would be practically significant: it would settle an ordering question for combining two widely used compression techniques, provide a theoretical basis for their interaction, and offer a cheap fix for outlier-driven degradation. The experimental comparisons are broad and the reported gains of DAM are notable. However, the paper's main contribution is advertised as a mathematical proof, and the proof is the weakest part of the manuscript. Several load-bearing inequalities are asserted without justification, no explicit witness matrices/vectors are constructed, and the DAM optimization derivation is internally inconsistent. The empirical results may still indicate a real effect, but they cannot substitute for the missing proof. Because the theoretical claims are the paper's core contribution and they are not supported, the manuscript does not currently meet the standard for acceptance.

major comments (5)
  1. [Theorem 3.8, Appendix D, Eqs. (56)-(58)] The claimed strict inequality E_{q◦l} > E_{l'} + E_{q'} rests entirely on the cross term 2tr((SVD_r(Q(W))−Q(W))^T(Q(W)−W)) being positive. The proof only states that this term is 'usually positive' and gives no condition on W, no bound, and no explicit construction. If the cross term is non-positive, Eq. (58) fails and Theorem 3.8 does not follow. Equation (52) is also asserted rather than proved. Since this theorem is the tensor-level non-orthogonality result, the central claim is unsupported.
  2. [Theorem 3.11, Appendix E, Eq. (71)] The proof of |E1| < |E2| relies on the lower bound |E2| ≥ |x|(|e_w| − |r'|), which is not a consequence of Cauchy-Schwarz. The reverse triangle inequality gives only |<x,e_w> − <x,r'>| ≥ ||<x,e_w>| − |<x,r'>||, and |<x,e_w>| can be zero when x is orthogonal to e_w. No explicit (x,w) is constructed, and the assertion |e_w| ≫ |r'| is not quantified. Thus the existence claim in Theorem 3.11 is not established.
  3. [Appendix E.3.1, Eqs. (67)-(68)] The proof assumes that total quantization error energy is proportional to the number of quantized elements, so k(n+m) ≪ nm implies |e_A|^2+|e_B|^2 ≪ |e_w|^2. Under the paper's own block-quantization rule (Def. 3.1), per-element error is scaled by max(|block|)/(2^b−1), which differs between A, B, and W. There is no uniform per-element variance, so Eq. (68) does not follow. This assumption is load-bearing for the error comparison.
  4. [Definition 3.9, Appendix E.2] The dot-product error is defined with quantized activations: E^D_{l,q}(x,w) = <x,w> − <q(x), l◦q(w)>. However, the entire proof in Appendix E ignores q(x) and uses x in place of q(x) throughout, e.g., Eqs. (63)-(66). It also drops the e_A e_B term in Eq. (62) without a bound. The proof therefore does not compare the quantities defined in Theorem 3.11.
  5. [Section 4.3, Eqs. (17)-(21)] The DAM optimality derivation is internally inconsistent. After substituting Var(e_{1,i}) = c1 a_i^2 and Var(e_{2,i}) = c2 σ_i^2/a_i^2, the total error in Eq. (18) is c2 σ_i^2 n + c1 σ_i^2 m, which is independent of a_i. Consequently, Eq. (19) cannot be used to 'choose a_i', and Eq. (21), a_i = (c2 σ_i^2 n / (c1 m))^{1/4}, does not follow from minimizing Eq. (14). The proposed optimal diagonal matrix is therefore not derived.
minor comments (5)
  1. [Section 5.2] The text says 'combining quantization and sparsity' but the paper is about low-rank decomposition, not sparsity. Please correct the terminology.
  2. [Definition 3.9 / Appendix E] The dimensions are inconsistent: Definition 3.9 writes x,w ∈ R^n, while Appendix E treats w as an n×m matrix and x as an n-vector. Please clarify the notation and the roles of n and m.
  3. [Section 4.4, Eq. (22)] The reconstruction loss L_recon is introduced after the 'extreme value theorem' statement, but no argument shows that minimizing L_recon is equivalent to minimizing the quantization error in Eq. (14). The connection should be stated explicitly.
  4. [Appendix D, Eq. (52)] The inequality in Eq. (52) is asserted as a consequence of perturbation theory, but no reference or derivation is given. If this inequality is intended as a lemma, it should be proved.
  5. [References] Several references are incomplete: 'Hsu et al.' has no year, and 'Radford et al.' is labeled 'OpenAI'. Please ensure all citations follow the journal style.

Circularity Check

0 steps flagged

No significant circularity; the derivation chain is self-contained, and the identified problems are proof gaps rather than circular reductions.

full rationale

The paper's central derivation chain is self-contained and does not reduce any load-bearing conclusion to its own inputs by construction. Theorem 3.7 is obtained from the triangle inequality on the decomposition of the l-after-q error into quantization and low-rank parts; this is a valid (if weak) inequality, not a hidden assumption of the claimed result. Theorem 3.8 and Theorem 3.11 are attempted existence/inequality proofs using perturbation reasoning and error decomposition; they do not assume the strict inequality they set out to prove. The empirical validation in Section 5.2 uses THo = Po - Loq - Lol, which is simply the additive-loss null hypothesis: if combined loss equals the sum of the individual losses, accuracy should equal THo. This threshold is not a fitted parameter and the combined score is measured independently, so observing a value below THo is a falsifiable result, not a conclusion forced by construction. The DAM optimum is derived from explicit variance-scaling assumptions; while the derivation is internally problematic (the stated assumptions make the total error independent of a_i, so Eq. 20 does not follow), that is a mathematical inconsistency, not a circular definition. There are no load-bearing self-citations: SVD-LLM and ASVD are external prior works, and the self-references in the introduction are not used to justify the technical claims. The Appendix D 'usually positive' cross term and Appendix E element-count/quantization-error comparisons are serious proof gaps and correctness risks, but they are unsupported inferences rather than equivalences between inputs and outputs. Under the review rules, proof invalidity is distinct from circularity, so no circular step is established.

Axiom & Free-Parameter Ledger

2 free parameters · 7 axioms · 0 invented entities

The central theorem is meant to be self-contained error analysis, but it rests on several unproved modeling assumptions about quantization noise and SVD perturbation behavior. DAM adds a per-layer fitted diagonal matrix. No new physical entities are introduced.

free parameters (2)
  • DAM diagonal scaling matrix a (per-layer) = not reported; optimized via Lrecon (Eq. 22) or closed-form Eq. 21
    Central to the proposed DAM method. The closed-form optimum Eq. 21 does not follow from Eq. 18-19 because a_i cancels; in practice a is fitted to a reconstruction loss and acts as a per-layer free parameter.
  • variance constants c1, c2 (or c2/c1) = not estimated; Eq. 19 asserts c2/c1 = m/n
    Introduced in Section 4.3 to model quantization error variance after diagonal scaling. The optimal a_i depends on their ratio, but the ratio is never measured; the balancing condition is chosen rather than derived.
axioms (7)
  • standard math Rounding quantization error for each element is bounded by half the quantization step
    Definition 3.2 / Appendix A. The bound is standard, but the paper's Eq. 1 uses 2^{b-1} while Eq. 2 uses 2^b-1, making the stated step size inconsistent.
  • domain assumption SVD truncation error of the i-th singular value equals σ_i under a whitening transform S=Cholesky(XX^T)
    Appendix B, Eq. 31-34, borrowed from SVD-LLM; depends on input activation distribution and is not a universal property of all weight matrices.
  • domain assumption Quantization error behaves as independent zero-mean additive noise with variance proportional to step size
    Section 4.2, Eq. 15-16. Activation and weight outliers violate this, which is exactly the problem DAM claims to solve.
  • ad hoc to paper After scaling, quantization error variances satisfy Var(e1,i)=c1 a_i^2 and Var(e2,i)=c2 σ_i^2/a_i^2
    Section 4.3, Eq. 17-19. These specific scalings are chosen so that both error terms in Eq. 18 are proportional to σ_i^2; they are not empirically justified, and they cause a_i to cancel from the balance condition.
  • ad hoc to paper The cross term 2tr((SVDr(Q(W))−Q(W))^T(Q(W)−W)) is 'usually positive'
    Appendix D, Eq. 56-58. The strict inequality in Theorem 3.8 requires this to be positive for the existential claim, but no proof or example is given.
  • ad hoc to paper Total quantization error energy is proportional to the number of quantized elements, so k(n+m) << nm implies |eA|^2+|eB|^2 << |ew|^2
    Appendix E.3.1, Eq. 67-68. Ignores dependence on matrix norms and per-element step sizes; used to conclude |E1|<|E2| without constructing x,w.
  • standard math Frobenius norm triangle inequality and trace identity hold
    Appendix C, Eq. 36-43; sound but only proves an upper bound, not non-orthogonality in a strong sense.

pith-pipeline@v1.3.0-alltime-deepseek · 18329 in / 18819 out tokens · 175446 ms · 2026-08-02T14:22:45.444813+00:00 · methodology

0 comments
read the original abstract

As the parameter size of language models continues to grow, effective model compression is required to reduce their computational and memory overhead. Existing compression methods suffer from bottleneck issues: when the compression ratio is increased, performance degrades significantly. Low-rank decomposition and quantization are two prominent compression methods that have been proven to significantly reduce the computational and memory requirements of Large Language Models (LLMs) while maintaining model accuracy. Evidently, combining these two methods will break through the existing compression bottleneck. However, how these two methods interact when combined remains a critical question for developers, as many assume they are orthogonal, meaning their combination would not introduce additional errors beyond those independently introduced by each method. This paper provides the first mathematical proof that low-rank decomposition and quantization are non-orthogonal. We validate these findings through a series of experiments on large language models. Our results demonstrate that these methods are non-orthogonal, and their combination leads to significant performance degradation. Importantly, we propose a novel approach Diagonal Adhesive Method (DAM), which can effectively combine the two methods and mitigate the performance loss. Our research provides deep insights into model compression and lays a solid theoretical and experimental foundation for future related studies.

Figures

Figures reproduced from arXiv: 2607.20434 by Jun Zhao, Kang Liu, Lu Wang, Xiusheng Huang, Yequan Wang.

Figure 1
Figure 1. Figure 1: Visualization of activation values < ΣrV T r >, the left and right images respectively show the 3D visualization of activation values before and after utilizing DAM. The X and Y axes represent the dimensions of the activation matrix, while the Z axis indicates the magnitude of the values. The left figure shows that before applying the DAM method, the activation values are steeply distributed with clear out… view at source ↗
Figure 2
Figure 2. Figure 2: Compression errors (L2) and outliers (Anomaly Values) across different layers. The models utilized include [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prefill and decoding speedup across different [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

83 extracted references · 41 linked inside Pith

  1. [1]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  2. [2]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  3. [3]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  4. [4]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  5. [5]

    Publications Manual , year = "1983", publisher =

    1983

  6. [6]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  7. [7]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  8. [8]

    Dan Gusfield , title =. 1997

    1997

  9. [9]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  10. [10]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  11. [11]

    Linear Algebra and its applications , volume=

    A generalization of the Eckart-Young-Mirsky matrix approximation theorem , author=. Linear Algebra and its applications , volume=. 1987 , publisher=

    1987

  12. [12]

    Language model compression with weighted low-rank factorization , author=

  13. [13]

    arXiv preprint arXiv:2312.05821 , year=

    Asvd: Activation-aware singular value decomposition for compressing large language models , author=. arXiv preprint arXiv:2312.05821 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2403.07378 , year=

    Svd-llm: Truncation-aware singular value decomposition for large language model compression , author=. arXiv preprint arXiv:2403.07378 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2210.17323 , year=

    Gptq: Accurate post-training quantization for generative pre-trained transformers , author=. arXiv preprint arXiv:2210.17323 , year=

    Pith/arXiv arXiv

  16. [16]

    Proceedings of Machine Learning and Systems , volume=

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration , author=. Proceedings of Machine Learning and Systems , volume=

  17. [17]

    arXiv preprint arXiv:2306.02272 , volume=

    Owq: Lessons learned from activation outliers for weight quantization in large language models , author=. arXiv preprint arXiv:2306.02272 , volume=

    Pith/arXiv arXiv

  18. [18]

    Advances in Neural Information Processing Systems , volume=

    Quip: 2-bit quantization of large language models with guarantees , author=. Advances in Neural Information Processing Systems , volume=

  19. [19]

    arXiv preprint arXiv:2402.04396 , year=

    Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks , author=. arXiv preprint arXiv:2402.04396 , year=

    Pith/arXiv arXiv

  20. [20]

    International Conference on Machine Learning , pages=

    Smoothquant: Accurate and efficient post-training quantization for large language models , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  21. [21]

    arXiv preprint arXiv:2308.13137 , year=

    Omniquant: Omnidirectionally calibrated quantization for large language models , author=. arXiv preprint arXiv:2308.13137 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2405.17849 , year=

    I-llm: Efficient integer-only inference for fully-quantized low-bit large language models , author=. arXiv preprint arXiv:2405.17849 , year=

    Pith/arXiv arXiv

  23. [23]

    Advances in Neural Information Processing Systems , volume=

    Quarot: Outlier-free 4-bit inference in rotated llms , author=. Advances in Neural Information Processing Systems , volume=

  24. [24]

    arXiv preprint arXiv:2405.16406 , year=

    Spinquant: Llm quantization with learned rotations , author=. arXiv preprint arXiv:2405.16406 , year=

    Pith/arXiv arXiv

  25. [25]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Editing common sense in transformers , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  26. [26]

    Advances in Neural Information Processing Systems , volume=

    Locating and editing factual associations in GPT , author=. Advances in Neural Information Processing Systems , volume=

  27. [27]

    arXiv preprint arXiv:2210.07229 , year=

    Mass-editing memory in a transformer , author=. arXiv preprint arXiv:2210.07229 , year=

    Pith/arXiv arXiv

  28. [28]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Pmet: Precise model editing in a transformer , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  29. [29]

    On symbolic and neural commonsense knowledge graphs , author=

  30. [30]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Atomic: An atlas of machine commonsense for if-then reasoning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  31. [31]

    arXiv preprint arXiv:2307.12976 , year=

    Evaluating the ripple effects of knowledge editing in language models , author=. arXiv preprint arXiv:2307.12976 , year=

    Pith/arXiv arXiv

  32. [32]

    GPT-J-6B: A 6 billion parameter autoregressive language model , author=

  33. [33]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2012.00363 , year=

    Modifying memories in transformer models , author=. arXiv preprint arXiv:2012.00363 , year=

    Pith/arXiv arXiv 2012

  35. [35]

    arXiv preprint arXiv:2110.11309 , year=

    Fast model editing at scale , author=. arXiv preprint arXiv:2110.11309 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  37. [37]

    arXiv preprint arXiv:2104.10247 , year=

    Modeling event plausibility with consistent conceptual abstraction , author=. arXiv preprint arXiv:2104.10247 , year=

    Pith/arXiv arXiv

  38. [38]

    arXiv preprint arXiv:1706.04115 , year=

    Zero-shot relation extraction via reading comprehension , author=. arXiv preprint arXiv:1706.04115 , year=

    Pith/arXiv arXiv

  39. [39]

    International Conference on Machine Learning , pages=

    Memory-based model editing at scale , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  40. [40]

    arXiv preprint arXiv:2210.03329 , year=

    Calibrating factual knowledge in pretrained language models , author=. arXiv preprint arXiv:2210.03329 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2305.14795 , year=

    Mquake: Assessing knowledge editing in language models via multi-hop questions , author=. arXiv preprint arXiv:2305.14795 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2311.09053 , year=

    Assessing knowledge editing in language models via relation perspective , author=. arXiv preprint arXiv:2311.09053 , year=

    Pith/arXiv arXiv

  43. [43]

    arXiv preprint arXiv:2402.13093 , year=

    Event-level Knowledge Editing , author=. arXiv preprint arXiv:2402.13093 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2402.11324 , year=

    EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundaries , author=. arXiv preprint arXiv:2402.11324 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2104.08696 , year=

    Knowledge neurons in pretrained transformers , author=. arXiv preprint arXiv:2104.08696 , year=

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2201.06009 , year=

    Memory-assisted prompt editing to improve gpt-3 after deployment , author=. arXiv preprint arXiv:2201.06009 , year=

    Pith/arXiv arXiv

  47. [47]

    arXiv preprint arXiv:2305.12740 , year=

    Can We Edit Factual Knowledge by In-Context Learning? , author=. arXiv preprint arXiv:2305.12740 , year=

    Pith/arXiv arXiv

  48. [48]

    arXiv preprint arXiv:2104.08164 , year=

    Editing factual knowledge in language models , author=. arXiv preprint arXiv:2104.08164 , year=

    Pith/arXiv arXiv

  49. [49]

    arXiv preprint arXiv:2311.04661 , year=

    Massive editing for large language models via meta learning , author=. arXiv preprint arXiv:2311.04661 , year=

    Pith/arXiv arXiv

  50. [50]

    Nature Machine Intelligence , volume=

    Parameter-efficient fine-tuning of large-scale pre-trained language models , author=. Nature Machine Intelligence , volume=. 2023 , publisher=

    2023

  51. [51]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2309.16535 , year=

    Klob: a benchmark for assessing knowledge locating methods in language models , author=. arXiv preprint arXiv:2309.16535 , year=

    Pith/arXiv arXiv

  53. [53]

    Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages=

    Named entity recognition via noise aware training mechanism with data filter , author=. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages=

    2021

  54. [54]

    Proceedings of the 29th International Conference on Computational Linguistics , pages=

    Document-level relation extraction via pair-aware and entity-enhanced representation learning , author=. Proceedings of the 29th International Conference on Computational Linguistics , pages=

  55. [55]

    arXiv preprint arXiv:2405.20935 , year=

    Effective interplay between sparsity and quantization: From theory to practice , author=. arXiv preprint arXiv:2405.20935 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint arXiv:1609.07843 , year=

    Pointer sentinel mixture models , author=. arXiv preprint arXiv:1609.07843 , year=

    Pith/arXiv arXiv

  57. [57]

    doi:10.5281/zenodo.12608602 , url =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang...

    doi:10.5281/zenodo.12608602

  58. [58]

    arXiv preprint arXiv:1905.10044 , year=

    Boolq: Exploring the surprising difficulty of natural yes/no questions , author=. arXiv preprint arXiv:1905.10044 , year=

    Pith/arXiv arXiv 1905

  59. [59]

    arXiv preprint arXiv:1905.07830 , year=

    Hellaswag: Can a machine really finish your sentence? , author=. arXiv preprint arXiv:1905.07830 , year=

    Pith/arXiv arXiv 1905

  60. [60]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  61. [61]

    arXiv preprint arXiv:1809.02789 , year=

    Can a suit of armor conduct electricity? a new dataset for open book question answering , author=. arXiv preprint arXiv:1809.02789 , year=

    Pith/arXiv arXiv

  62. [62]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Piqa: Reasoning about physical commonsense in natural language , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  63. [63]

    arXiv preprint arXiv:1904.09728 , year=

    Socialiqa: Commonsense reasoning about social interactions , author=. arXiv preprint arXiv:1904.09728 , year=

    Pith/arXiv arXiv 1904

  64. [64]

    Communications of the ACM , volume=

    Winogrande: An adversarial winograd schema challenge at scale , author=. Communications of the ACM , volume=. 2021 , publisher=

    2021

  65. [65]

    arXiv preprint arXiv:1806.00358 , year=

    A systematic classification of knowledge, reasoning, and context within the ARC dataset , author=. arXiv preprint arXiv:1806.00358 , year=

    Pith/arXiv arXiv

  66. [66]

    arXiv preprint arXiv:1510.00149 , year=

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding , author=. arXiv preprint arXiv:1510.00149 , year=

    Pith/arXiv arXiv

  67. [67]

    Proceedings of the 2016 conference on empirical methods in natural language processing , pages=

    Sequence-level knowledge distillation , author=. Proceedings of the 2016 conference on empirical methods in natural language processing , pages=

    2016

  68. [68]

    IEEE transactions on neural networks and learning systems , volume=

    Dynamical channel pruning by conditional accuracy change for deep neural networks , author=. IEEE transactions on neural networks and learning systems , volume=. 2020 , publisher=

    2020

  69. [69]

    Mechanical Systems and Signal Processing , volume=

    Differentiable neural architecture search augmented with pruning and multi-objective optimization for time-efficient intelligent fault diagnosis of machinery , author=. Mechanical Systems and Signal Processing , volume=. 2021 , publisher=

    2021

  70. [70]

    IEEE transactions on neural networks and learning systems , volume=

    Knowledge distillation using hierarchical self-supervision augmented distribution , author=. IEEE transactions on neural networks and learning systems , volume=. 2022 , publisher=

    2022

  71. [71]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Simdistill: Simulated multi-modal distillation for bev 3d object detection , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  72. [72]

    IEEE Transactions on Parallel and Distributed Systems , year=

    A Generic, High-Performance, Compression-Aware Framework for Data Parallel DNN Training , author=. IEEE Transactions on Parallel and Distributed Systems , year=

  73. [73]

    Advances in neural information processing systems , volume=

    Early-learning regularization prevents memorization of noisy labels , author=. Advances in neural information processing systems , volume=

  74. [74]

    5 technical report , author=

    Qwen2. 5 technical report , author=. arXiv preprint arXiv:2412.15115 , year=

    Pith/arXiv arXiv

  75. [75]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  76. [76]

    Transactions of the Association for Computational Linguistics , volume=

    A survey on model compression for large language models , author=. Transactions of the Association for Computational Linguistics , volume=. 2024 , publisher=

    2024

  77. [77]

    Journal of digital imaging , volume=

    Information entropy measure for evaluation of image quality , author=. Journal of digital imaging , volume=. 2008 , publisher=

    2008

  78. [78]

    2025 , eprint=

    FlatQuant: Flatness Matters for LLM Quantization , author=. 2025 , eprint=

    2025

  79. [79]

    arXiv preprint arXiv:1711.05101 , volume=

    Fixing weight decay regularization in adam , author=. arXiv preprint arXiv:1711.05101 , volume=

    Pith/arXiv arXiv

  80. [80]

    https://zenodo.org/records/12608602 , year=

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang...

    arXiv

Showing first 80 references.