REVIEW 3 major objections 5 minor 26 references
Projected Compression: Trainable Projection for Efficient Transformer Compression
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Frozen weights, small projections beat hard pruning on big models
desk verdict A clean compression trick with a consistent scaling trend, undercut by likely training-loss evaluation that needs to be re-measured on held-out data before the headline claim is credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the projection module $W_C = P_1 W P_2 + W_r$, in which the base weight matrix $W$ is frozen and $P_1 \in \mathbb{R}^{d_{in}^S \times d_{in}}$, $P_2 \in \mathbb{R}^{d_{out} \times d_{out}^S}$ are trainable downsampling projections. The identity $x W_C = ((x P_1) W) P_2$ is what carries the argument: it shows that training the small projection matrices is equivalent to letting each token activate the full frozen parameter space and then projecting back down, so no weight information is discarded a priori. During the backward pass, gradients are accumulated in the small projected weight space, which the paper argues makes the per-step training cost comparable to retraining a hard-pruned model despite the extra projection parameters.
What would settle it
Train the two pipelines identically and evaluate on a held-out C4 validation split or a separate language-modeling benchmark after the same number of tokens; if Projected Compression's lower last-100-step training loss does not translate into lower held-out loss, the central outperformance claim collapses.
Extended reading notes
Core claim
The paper's central claim is that a compressed Transformer weight matrix $W_C = P_1 W P_2 + W_r$ can be trained while the original weight $W$ stays frozen, and this produces a better final model than hard pruning and retraining at the same compute budget. Here $W$ is the full base weight matrix, $P_1$ and $P_2$ are small trainable projections that downsample the input and output dimensions, and $W_r$ is a zero-initialized residual term. Because $x W_C = ((x P_1) W) P_2$, every token still interacts with the full frozen weight matrix during training even though inference uses only the compressed product. The projections are initialized from the same importance scores used by hard pruning, so the most important directions dominate initially while less important directions can re-enter as training proceeds. The experiments compare Projected Compression with hard pruning plus retraining on 300M and 800M GPT-2-style models trained at 20:1 and 80:1 token-to-parameter ratios, at 35%, 50%, and 65% compression, and report lower cross-entropy loss for Projected Compression in the data-rich (80:1) regime, with the gap widening as retraining tokens increase and as the base model grows.
Load-bearing premise
The comparison rests on the assumption that cross-entropy loss averaged over the last 100 training steps measures final model quality; if that figure is training-set loss rather than held-out validation loss, the reported advantage over hard pruning could reflect training dynamics instead of true generalization.
Editorial extensions
If this is right
- The compressed model is a standard Transformer, so it runs in existing inference stacks with no special sparse kernels or extra per-token computation.
- Information from dimensions that hard pruning would delete is not lost: it can re-enter through the learned projections during retraining.
- The advantage over hard pruning strengthens as the base model is trained on more tokens and as the base model size grows, so the method is aimed precisely at the large, well-trained models that are most worth compressing.
- Because the same importance-score initialization is used, Projected Compression can slot into an existing pruning pipeline without changing the pruning criterion.
- The training-time memory cost is higher than hard pruning (projections plus frozen weights), but the paper identifies gradient checkpointing and offloading as mitigations.
Reading between the lines
- The same projection construction could be applied to attention Q/K/V and output projections, not just feedforward width and embedding size, which the paper does not test; if the mechanism transfers, gains could extend to deeper compression of the attention stack.
- Because the method only needs the frozen base weights and two small projections, it could be combined with distillation or quantization of the compressed weights; the paper leaves these combinations untested.
- The scaling trend suggests a testable prediction: on models trained well past the token-rich regime, Projected Compression's margin over hard pruning should keep growing, which could be checked with larger base models than the 800M configuration.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Projected Compression (PC), a method for compressing transformer linear layers by learning projection matrices P1 and P2, plus a residual term Wr, that map a frozen base weight matrix W to a smaller compressed matrix WC = P1 W P2 + Wr. After training, the projections and residual are merged into a standard smaller weight matrix, so inference uses a standard transformer with reduced dimensions. The paper compares PC to hard pruning with retraining (HPR) on GPT-2-style models of 300M and 800M parameters trained with 20:1 and 80:1 token-to-parameter ratios, reporting cross-entropy loss at 35%, 50%, and 65% compression levels. The main empirical claim is that PC outperforms HPR on models trained with more tokens (80:1) and that the margin grows with the number of training tokens.
Significance. If the reported gains hold on held-out data, the paper makes a useful empirical contribution: it demonstrates that a frozen pretrained model can be compressed via learned projections with no additional inference cost, and that the gains over hard-pruning retraining increase with base model quality and training budget. The experimental design has strengths: it matches compute budgets between the two pipelines, uses identical importance scores for both methods, and spans two model sizes, three compression levels, and two token-to-parameter ratios. However, the central claim is not yet established because the evaluation metric is not specified as held-out loss, and the absence of multiple runs or error bars leaves the small reported differences unquantified. The paper does not provide code or release models, which limits independent verification.
major comments (3)
- [Section 4.1, Evaluation] The evaluation metric is described as 'Cross-Entropy loss averaged over the last 100 training steps,' but the text does not state whether this loss is computed on the training split or on a held-out validation/test split. All models are trained on the C4 dataset and no evaluation split is mentioned. If this is training-set loss, the comparisons in Table 4 and Figure 2 largely measure training fit rather than generalization; because PC retains the frozen full base weights and adds learnable residual weights during training, it has a different and potentially larger effective capacity than HPR, so lower training loss would not establish that PC 'outperforms' HPR in deployment. Please specify the evaluation split and report held-out loss, or explicitly state that the reported numbers are held-out if that is the case.
- [Table 4 and Figure 2] No multiple seeds or error bars are reported, and many of the reported differences are small (for example, the 300M 80:1 35% compression row at 10B tokens is 3.0711 for PC versus 3.0775 for HPR, and at 2.5B tokens HPR is 3.1362 versus PC at 3.1361). Without an estimate of run-to-run variance, it is not possible to assess whether the cross-entropy differences are statistically significant or robust. Please report the number of runs and standard deviations (or confidence intervals), or justify why single runs are sufficient for the conclusions drawn.
- [Abstract and Section 3.4] The abstract states that Projected Compression 'matches the base model's per-token computation step in FLOPs,' which is ambiguous: a compressed model with reduced hidden and embedding dimensions should have fewer per-token FLOPs than the base model, and Section 3.4 argues that the training cost per step is equivalent to that of hard pruning with retraining. Please clarify whether the claim concerns training or inference FLOPs, and which baseline (the base model, HPR, or both) it is compared with.
minor comments (5)
- [Section 1, Introduction] The word 'demonstraing' should be 'demonstrating' in the contributions list.
- [Section 4.3] The sentence 'This efficiency trend that compression is most beneficial when applied to large, high-quality base models in shown in Frantar et al. work [26]' is grammatically incorrect ('in shown' should be 'is shown') and the reference to Frantar et al. does not seem to directly support the stated claim; please verify the citation and rephrase.
- [Table 4, Appendix A.2] The entry '3,0217' in the 800M 80:1 50% compression PC row at 1.25B tokens uses a comma instead of a period as the decimal separator; please fix the formatting.
- [Section 3.2] The dimensions of the residual term Wr are given as Rdout_S x din_S, while the compressed matrix WC is defined as Rdin_S x dout_S; please confirm that the orientation is correct and define the dimensions consistently.
- [Section 4.1, Training setting] The experimental setup does not report hyperparameters such as optimizer, learning rate, batch size, sequence length, or the number of training steps; including these details is essential for reproducibility.
Circularity Check
No circularity found: the method and evaluation are empirical comparisons against an independent hard-pruning baseline, with no fitted prediction or self-referential derivation.
full rationale
The paper's central claims are empirical comparisons between Projected Compression (PC) and Hard Pruning with Retraining (HPR) under matched compute and matched importance scores. The construction WC = P1W P2 + Wr defines the compressed weight matrix from frozen base weights and trainable projections; the identity xWC = ((xP1)W)P2 is a direct algebraic restatement of that definition, used only for intuition, not as an input that generates the reported losses. Initializing PC with the same weight-importance scores used to select HPR pruned dimensions is a controlled comparison, not a fitted parameter that predetermines the outcome. The scaling trend described in Section 4.3 is read off Tables 1/4 and Figure 2 rather than fitted from them, so no 'prediction' reduces to its own input. No load-bearing self-citation appears: the cited related work, including the scaling-law reference [26], is external and is not invoked to justify the PC-vs-HPR result. The reader-flagged concern that the evaluation metric may be training loss rather than held-out loss is a validity threat about generalization, not a circularity of the kind defined here. Accordingly, the correct circularity score is 0.
Assumptions & free parameters
assumptions (2)
- domain assumption Cross-entropy loss averaged over the last 100 training steps is a valid measure of final model quality.
- domain assumption For sufficiently large batch sizes, the overhead of recomputing and differentiating through WC = P1 W P2 is negligible, giving PC the same per-step training cost as hard pruning retraining.
Cite this review
Pith. "Pith review of Projected Compression: Trainable Projection for Efficient Transformer Compression." pith.science (2026). https://pith.science/paper/ETCW6N2P
@misc{pith2026250622255,
author = {Pith},
title = {Pith review of: Projected Compression: Trainable Projection for Efficient Transformer Compression},
year = {2026},
howpublished = {\url{https://pith.science/paper/ETCW6N2P}},
note = {Machine review of arXiv:2506.22255}
}
read the original abstract
Large language models have steadily increased in size to achieve improved performance; however, this growth has also led to greater inference time and computational demands. Consequently, there is rising interest in model size reduction methods. To address this issue, we propose Projected Compression, a novel model compression technique, that reduces model weights by utilizing projection modules. Specifically, we first train additional trainable projections weights and preserve access to all the original model parameters. Subsequently, these projections are merged into a lower-dimensional product matrix, resulting in a reduced-size standard Transformer-based model. Unlike alternative approaches that require additional computational overhead, our method matches the base model's per-token computation step in FLOPs. Experimental results show that Projected Compression outperforms the comparable hard pruning and retraining approach on higher quality models. Moreover, the performance margin scales well with the number of tokens.
Figures
Reference graph
Works this paper leans on
-
[1]
Language models are few-shot learners
Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sas- try, A Askell, S Agarwal, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 1:3, 2020
arXiv 2005
-
[2]
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020
2020
-
[3]
On the opportunities and risks of foundation models
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021
arXiv 2021
-
[4]
The rising costs of training frontier ai models
Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, Tamay Besiroglu, and David Owen. The rising costs of training frontier ai models. arXiv preprint arXiv:2405.21015, 2024
arXiv 2024
-
[5]
A survey of sustainability in large language models: Applications, economics, and challenges
Aditi Singh, Nirmal Prakashbhai Patel, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. A survey of sustainability in large language models: Applications, economics, and challenges. In 2025 IEEE 15th Annual Computing and Communication Workshop and Conference (CCWC) , pages 00008–00014. IEEE, 2025
work page 2025
-
[6]
Optimal brain damage
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989
1989
-
[7]
Second order derivatives for network pruning: Optimal brain surgeon
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems , 5, 1992
work page 1992
-
[8]
Learning both weights and connections for efficient neural network
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems , 28, 2015
2015
Show all 26 references
-
[9]
The state of sparsity in deep neural networks
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019
1902 arXiv
-
[10]
Network trimming: A data-driven neuron pruning approach towards efficient deep architectures
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250, 2016
2016 arXiv
-
[11]
To prune, or not to prune: exploring the efficacy of pruning for model compression
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017
2017 arXiv
-
[12]
Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications , 32:2281–2301, 2020
2020
-
[13]
Rigging the lottery: Making all tickets winners
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICML) , 2020
2020
-
[14]
Structured pruning learns compact and accurate models
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00408, 2022
2022 arXiv
-
[15]
Learning sparse neural networks through l_0 regularization
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l_0 regularization. arXiv preprint arXiv:1712.01312, 2017
2017 arXiv
-
[16]
Learning structured sparsity in deep neural networks
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2016
2016
-
[17]
Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen -Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen -Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low -rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 7
2021 arXiv
-
[18]
Prefix-tuning: Optimizing continuous prompts for generation
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021
2021 arXiv
-
[19]
Bitfit: Simple parameter -efficient fine-tuning for transformer-based masked language-models, 2021
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter -efficient fine-tuning for transformer-based masked language-models, 2021
2021
-
[20]
Qlora: Efficient finetuning of quantized llms
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023
2023 arXiv
-
[21]
A token is worth over 1,000 tokens: Efficient knowledge distillation through low-rank clone, 2025
Jitai Hao, Qiang Huang, Hao Liu, Xinyan Xiao, Zhaochun Ren, and Jun Yu. A token is worth over 1,000 tokens: Efficient knowledge distillation through low-rank clone, 2025
2025
-
[22]
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, ...
2019
-
[23]
Automatic differentiation in pytorch
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. preprint, 2017
2017
-
[24]
Training deep nets with sublinear memory cost
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016
2016 arXiv
-
[25]
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. preprint, 2018
2018
-
[26]
Scaling laws for sparsely-connected foundation models, 2023
Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-connected foundation models, 2023. 8 A Experiments A.1 Weights importances in experiments initialization We have tested the influence on the importance of random and magnitude-...
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.