REVIEW 4 major objections 4 minor 14 references
On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that layer pruning a full-size BERT, then fine-tuning it on a Marathi classification task, yields compact models whose accuracy stays close to the original and often beats same-size BERT models trained from scratch.
desk verdict Useful pruning data for Marathi BERT, but the abstract contradicts Table 2 and the central overclaim needs fixing before it can be taken seriously. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is layer pruning applied to a pre-trained Transformer encoder. Whole transformer blocks (6 or 10 of the 12 layers) are removed from the top, middle, or bottom of the stack, the classification head is reattached, and the remaining shallow model is fine-tuned on the target dataset. This contrasts with the baseline path of pre-training an equally small BERT from scratch; the paper's argument is that the surviving layers already contain task-relevant Marathi representations, so fine-tuning them is cheaper and often more accurate than learning representations from scratch. The named strategies—top, middle, and bottom pruning—are the independent variables that carry the comparison.
What would settle it
Train the 2-layer and 6-layer Marathi BERT baselines from scratch with the same fine-tuning budget (steps, batch size, learning rate, and data) used for the pruned models, and test on the same SHC, LPC, and LDC splits; if either baseline reaches or exceeds the best pruned MahaBERT-v2 accuracy on those splits, the claim that pruning beats from-scratch training at equal size is falsified.
Extended reading notes
Core claim
The central discovery the paper argues for is that downstream-task fine-tuning can recover most of a BERT model's accuracy after structured layer removal, so that model depth is substantially redundant for Marathi text classification. More specifically, pruning MahaBERT-v2 or Google-Muril from 12 layers down to 6 or 2 layers—using any of the top/middle/bottom strategies—produces test accuracies that cluster within a few points of the unpruned models, and in several configurations beat the same-size models trained from scratch. The paper also finds that no single pruning position dominates: the best strategy varies by model and dataset, with middle-layer pruning serving as the most consistently competitive option. On the paper's own tables, the strongest pruned-versus-full results occur on SHC and LPC, while on LDC the full 12-layer MahaBERT-v2 (94.78%) remains well ahead of its best pruned variant (90.04%).
Load-bearing premise
The comparison assumes the scratch-trained baselines (the 6-layer and 2-layer Marathi BERT models) were trained well enough to fairly represent what from-scratch training can achieve; the paper reports no compute budget, corpus size, training steps, or hyperparameters for them.
Editorial extensions
If this is right
- A 10-layer-removed MahaBERT-v2 (12 to 2 layers) can stay within roughly 1-3 accuracy points of the full model on SHC and LPC in some configurations, while cutting parameters by about 80%.
- A 6-layer pruned MahaBERT-v2 can beat the scratch-trained 6-layer MahaBERT-Small on SHC (90.47 vs 88.81 for Bottom6) and on LPC (91.05 vs 89.46 for Bottom6).
- Middle-layer pruning is a reasonable default: it is rarely the single best choice everywhere, but it is consistently near the top across model and dataset combinations.
- Because pruning removes the need to pre-train a small model, the same compute spent on one large pre-trained model can produce multiple task-specific small models.
- The optimal pruning position is task- and model-specific, so model selection should include a small grid of pruning strategies rather than a fixed choice.
Reading between the lines
- The paper does not report training compute for the two scratch-trained baselines, so the 'beats from-scratch' conclusion should be treated as conditional; a matched-budget comparison is the natural follow-up.
- If middle-layer pruning generalizes to other languages, it suggests that the informational bottleneck in fine-tuned classifiers is in the lower and upper layers, and that intermediate layers carry the most redundancy—a claim that could be tested with layer-wise probing.
- The monolingual-over-multilingual trend in the pruned models hints that language-specific pretraining leaves more recoverable structure after depth reduction, which would imply that multilingual models need different pruning recipes than monolingual ones.
- Because only Marathi is tested, the no-clear-winner conclusion is a statement about this language and corpus pair; spreading the same pruning grid across other low-resource languages would show whether the pattern is about depth redundancy in general or about Marathi BERT in particular.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates layer pruning as a compression method for two 12-layer BERT models (MahaBERT-v2 and Google-Muril) on three Marathi text-classification datasets (SHC, LPC, LDC). It compares top, middle, and bottom pruning with 6 or 10 layers removed, against the full models and against two smaller scratch-trained baselines (MahaBERT-Small and MahaBERT-Smaller). The authors claim that pruned models perform comparably to full models and consistently outperform similarly sized scratch-trained models, and that middle-layer pruning is generally the most effective strategy.
Significance. If the headline claims were supported, the paper would be a useful practical contribution: layer pruning plus fine-tuning would be a cheaper alternative to training small BERT models from scratch for low-resource languages. The experimental setup is straightforward and the comparison across three datasets and two base models is relevant. However, the central claims are contradicted by the paper's own results in Table 2, and the lack of uncertainty quantification and baseline training details prevents the reader from assessing whether the observed differences are meaningful. The paper therefore does not currently establish its main thesis.
major comments (4)
- [Abstract / Introduction / Table 2] The central claim that pruned models "achieve comparable performance" to full models and "consistently outperform" scratch-trained models of similar size is contradicted by Table 2. On LDC, full MahaBERT-v2 obtains 94.78% while the best pruned MahaBERT-v2 variant (Bottom 6) obtains 90.04%, a 4.74-point gap that is not plausibly "comparable". On LPC, the 2-layer pruned MahaBERT-v2 Middle10 (89.68%) and Bottom10 (89.71%) variants both fall below the 2-layer scratch-trained MahaBERT-Smaller (90.62%), so the 2-layer pruned models do not consistently outperform the same-size scratch baseline. These numbers are load-bearing and should be addressed directly.
- [Table 2 / Section 4] The paper reports no error bars, no multiple seeds, and no statistical significance testing. Several conclusions rest on differences well below 1 percentage point, such as MahaBERT-v2 Bottom6 versus MahaBERT-Smaller on LPC (91.05% versus 90.62%) and MahaBERT-v2 Middle10 versus MahaBERT-Smaller on LPC (89.68% versus 90.62%). Without variance estimates, the claimed comparability or superiority of pruned models is not established, and the reader cannot distinguish genuine effects from run-to-run noise.
- [Section 3.1.1] The scratch-trained baselines MahaBERT-Small and MahaBERT-Smaller are described only by their layer counts. The manuscript gives no pretraining corpus size, training steps, compute budget, or hyperparameters for these models. Since the baselines and the datasets both come from the L3Cube group, the comparison is partly intramural; if the small models were undertrained, the observed advantage of prune-then-finetune over scratch training would be an artifact. The authors should either provide full training details or use independently established baselines with documented training budgets.
- [Contributions / Section 5] The paper's contribution list and conclusion claim that middle-layer pruning is "optimal in most cases" and provides the best balance. This is not reflected in Table 2: for MahaBERT-v2, the best pruning configuration per dataset is Top6 on SHC, Top10 on LPC, and Bottom6 on LDC; for Google-Muril, the best configuration is Bottom6 on LDC, while Middle6 is best only on SHC and LPC. The conclusion should be restated as strongly dataset- and model-dependent, and the manuscript should avoid the general "middle is optimal" claim.
minor comments (4)
- [Section 4] The text says "For detailed validation accuracy, please refer to Tables 2, 3, and 4 in the appendix," but the appendix tables are numbered 3, 4, and 5, and they contain both validation and testing accuracies, not validation alone.
- [Table 1] The first row of Table 1 is labeled "Category" but contains the train/test/validation counts; this appears to be a formatting error, as the row does not correspond to a category distribution.
- [Section 3.1.1 / References] The model is referred to as "Google-Muril" in the text and as "MuRIL" in the cited reference [10]; the naming should be made consistent.
- [Acknowledgements] The acknowledgement that ChatGPT was used to generate the text is transparent, but the authors should verify that this disclosure complies with the target venue's policy on AI-assisted writing.
Circularity Check
No circularity: the study is an empirical comparison, not a derivation, and no claim reduces to its inputs by construction.
full rationale
The paper's central claim is empirical: pruned BERT models are compared with full models and with scratch-trained small models on Marathi classification datasets. There is no fitted parameter that is later relabeled as a prediction, no equation defines the pruned models' accuracy in terms of the scratch baselines, and no uniqueness theorem or ansatz is imported from the authors' prior work to force the conclusion. The baselines MahaBERT-Small and MahaBERT-Smaller are cited from Joshi's prior L3Cube work [9], and the datasets are from the same group, but this is intramural sourcing rather than circularity: the baselines are externally available artifacts with stated architectures, and the comparison is directly falsifiable from the paper's own Table 2. In fact, Table 2 contradicts the abstract's claim that pruned models 'consistently outperform' scratch-trained models of similar size—on LPC, MahaBERT-Smaller scores 90.62% while pruned MahaBERT-v2 Middle10 and Bottom10 score 89.68% and 89.71%—but that contradiction is a correctness or robustness problem, not a circularity problem. The self-citations are not load-bearing in the sense of reducing the argument to an unverified assertion from the same authors; they point to actual models and corpora that the reader could independently run. No step in the paper's reasoning derives its conclusion from the conclusion itself or from a definitional equivalence, so the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- Fine-tuning hyperparameters (learning rate, batch size, epochs, max length) =
not reported
- Number of layers pruned (6 or 10) =
6 or 10
assumptions (3)
- domain assumption Pretrained BERT layers can be removed and the remaining stack fine-tuned to recover task accuracy.
- domain assumption MahaBERT-Small and MahaBERT-Smaller were trained from scratch to a quality representative of what small-model pretraining can achieve for Marathi.
- domain assumption The train/validation/test splits of SHC, LPC, and LDC are correctly constructed and representative.
Cite this review
Pith. "Pith review of On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages." pith.science (2026). https://pith.science/paper/VRVAJU3X
@misc{pith2026250100733,
author = {Pith},
title = {Pith review of: On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages},
year = {2026},
howpublished = {\url{https://pith.science/paper/VRVAJU3X}},
note = {Machine review of arXiv:2501.00733}
}
read the original abstract
This study explores the effectiveness of layer pruning for developing more efficient BERT models tailored to specific downstream tasks in low-resource languages. Our primary objective is to evaluate whether pruned BERT models can maintain high performance while reducing model size and complexity. We experiment with several BERT variants, including MahaBERT-v2 and Google-Muril, applying different pruning strategies and comparing their performance to smaller, scratch-trained models like MahaBERT-Small and MahaBERT-Smaller. We fine-tune these models on Marathi datasets, specifically Short Headlines Classification (SHC), Long Paragraph Classification (LPC) and Long Document Classification (LDC), to assess their classification accuracy. Our findings demonstrate that pruned models, despite having fewer layers, achieve comparable performance to their fully-layered counterparts while consistently outperforming scratch-trained models of similar size. Notably, pruning layers from the middle of the model proves to be the most effective strategy, offering performance competitive with pruning from the top and bottom. However, there is no clear winner, as different pruning strategies perform better in different model and dataset combinations. Additionally, monolingual BERT models outperform multilingual ones in these experiments. This approach, which reduces computational demands, provides a faster and more efficient alternative to training smaller models from scratch, making advanced NLP models more accessible for low-resource languages without compromising classification accuracy.
Figures
Reference graph
Works this paper leans on
-
[1]
Divyanshu Aggarwal, Vivek Gupta, and Anoop Kunchukuttan. 2022. In- dicXNLI: Evaluating Multilingual Inference for Indian Languages. arXiv preprint arXiv:2204.08776 (2022). arXiv:2204.08776 [cs.CL]
arXiv 2022
-
[2]
Mirashi Aishwarya, Sonavane Srushti, Lingayat Purva, Padhiyar Tejas, and Joshi Raviraj. 2023. L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages. InProceedings of the 20th International Conference on Natural Language Processing (ICON) . 442–449
work page 2023
-
[3]
Shi Chen and Qi Zhao. 2019. Shallowing Deep Networks: Layer-Wise Pruning Based on Feature Representations. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 12 (2019), 3048–3056. https://doi.org/10.1109/TPAMI. 2018.2874634
arXiv 2019
-
[4]
Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. 2021. A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommenda- tions. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
work page 2021
-
[5]
Ting-Wu Chin, Cha Zhang, and Diana Marculescu. 2018. Layer-compensated Pruning for Resource-Constrained Convolutional Neural Networks. arXiv preprint arXiv:1810.00518 (2018). arXiv:1810.00518 [cs.CV]
work page Pith review arXiv 2018
-
[6]
Jacob Devlin et al . 2018. BERT: Pre-training of Deep Bidirectional Trans- formers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018). arXiv:1810.04805 [cs.CL]
arXiv 2018
-
[7]
Angela Fan, Edouard Grave, and Armand Joulin. 2019. Reducing Transformer Depth on Demand with Structured Dropout. arXiv preprint arXiv:1909.11556 (2019). arXiv:1909.11556 [cs.LG]
arXiv 2019
-
[8]
Prakhar Ganesh et al. 2021. Compressing Large-Scale Transformer-Based Mod- els: A Case Study on BERT. Transactions of the Association for Computational Linguistics 9 (2021), 1061–1080. https://doi.org/10.1162/tacl_a_00413
Show all 14 references
-
[9]
Raviraj Joshi. 2022. L3Cube-MahaCorpus and MahaBERT: Marathi Monolin- gual Corpus, Marathi BERT Language Models, and Resources. In Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference. 97–101
2022
-
[10]
Simran Khanuja et al . 2021. MuRIL: Multilingual Representations for Indian Languages. arXiv preprint arXiv:2103.10730 (2021). arXiv:2103.10730 [cs.CL]
2021 arXiv
-
[11]
Morteza Mousa Pasandi, Mohsen Hajabdollahi, Nader Karimi, and Shadrokh Samavi. 2018. Modeling of Pruning Techniques for Deep Neural Networks Sim- plification. Technical Report, Department of Electrical and Computer Engineering, Isfahan University of Technology (2018)
2018
-
[12]
Hassan Sajjad et al. 2022. On the Effect of Dropping Layers of Pre-trained Trans- former Models. arXiv preprint arXiv:2004.03844 (2022). arXiv:2004.03844 [cs.CL]
2022 arXiv
-
[13]
Seul-Ki Yeom, Philipp Seeger, Sebastian Lapuschkin, Alexander Binder, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek. 2018. Pruning by Ex- plaining: A Novel Criterion for Deep Neural Network Pruning. arXiv preprint arXiv:1807.10553 (2018). arXiv:1807.10553 [cs.LG]
2018 arXiv
-
[14]
Michael Zhu and Suyog Gupta. 2017. To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression. arXiv preprint arXiv:1710.01878 (2017). arXiv:1710.01878 [stat.ML] FIRE ’24, 12th - 15th December 2024, DA-IICT, Gandhinagar, India Mayur Shirke, Amey Shembad...
2017 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.