MuCRASP prunes VLMs in a CoT-aware manner, outperforming baselines by preserving reasoning quality at 30-50% compression rates on models like Qwen2.5-VL-7B.
To prune, or not to prune: exploring the efficacy of pruning for model compression
13 Pith papers cite this work. Polarity classification is still indexing.
13
Pith papers citing it
abstract
Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports (Han et al., 2015; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size. This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy. We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process. We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint. Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
Experiments across code LLMs show no-review collapses fastest, human-gated filters slow collapse, and AI self-gates lose effect over time, degenerating to ungated self-training under self-confirming acceptance as proven via gated distributional reweighting and spectral analysis.
Systematic experiments reveal that activation steering trades fluency for concept control, is less effective on instruction-tuned models, and that prompting/SFT excel at injection but not removal, with textual metrics correlating to LLM judges.
Putri improves structured LLM pruning by FFN weight updates, sequential FFN pruning, and fine-grained GQA-compatible head removal, enabling usable models at extreme sparsity.
STARFISH recovers accuracy in pruned neural networks by optimizing internal state alignment to the original model with a minimal unlabeled calibration set, outperforming prior recovery methods especially at high pruning ratios.
Orthogonal growth recycles pre-trained MoE checkpoints via layer copying and noisy expert duplication, delivering 10.6% higher accuracy than training from scratch with equivalent extra compute.
A concept-based pruning method for DNNs guided by interpretable concepts and system requirements produces smaller, computationally efficient models that maintain effectiveness on image classification tasks.
SCENIC framework reports up to 99% exact match on structured IoT command generation using sub-0.2B models, with pruned INT8 versions retaining 91% EM@1 after 25% size reduction.
Recursive ViT with dynamic depth/width adjustments reduces parameters by 48.7% and improves reconstruction quality versus baselines in semantic image communication.
DACIS-guided PMP pipeline prunes plant pathology models by 78% while retaining 92.3% accuracy and achieving 7 FPS on Raspberry Pi 4 using few-shot meta-learning.
A unified F-statistic screening and weighted evaluation method prunes both unstructured and structured parameters in FNNs and CNNs, claiming order-of-magnitude size reduction with competitive accuracy on vision datasets.
Proposes a modality relation distillation method that transfers teacher modality relationships via the modality-level Gram Matrix.
The survey introduces personalized federated intelligence (PFI) as a framework integrating federated learning and foundation models to support privacy-aware personalization of AI models.
citing papers explorer
-
MuCRASP: Multimodal Chain-of-thought Reasoning aware Structured Pruning
MuCRASP prunes VLMs in a CoT-aware manner, outperforming baselines by preserving reasoning quality at 30-50% compression rates on models like Qwen2.5-VL-7B.
-
When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMs
Experiments across code LLMs show no-review collapses fastest, human-gated filters slow collapse, and AI self-gates lose effect over time, degenerating to ungated self-training under self-confirming acceptance as proven via gated distributional reweighting and spectral analysis.
-
On The Effectiveness-Fluency Trade-Off In LLM Conditioning: A Systematic Study
Systematic experiments reveal that activation steering trades fluency for concept control, is less effective on instruction-tuned models, and that prompting/SFT excel at injection but not removal, with textual metrics correlating to LLM judges.
-
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
Putri improves structured LLM pruning by FFN weight updates, sequential FFN pruning, and fine-grained GQA-compatible head removal, enabling usable models at extreme sparsity.
-
STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing
STARFISH recovers accuracy in pruned neural networks by optimizing internal state alignment to the original model with a minimal unlabeled calibration set, outperforming prior recovery methods especially at high pruning ratios.
-
Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts
Orthogonal growth recycles pre-trained MoE checkpoints via layer copying and noisy expert duplication, delivering 10.6% higher accuracy than training from scratch with equivalent extra compute.
-
Engineering Resource-constrained Software Systems with DNN Components: a Concept-based Pruning Approach
A concept-based pruning method for DNNs guided by interpretable concepts and system requirements produces smaller, computationally efficient models that maintain effectiveness on image classification tasks.
-
SCENIC: Semantic-Conditioned Edge-Aware Neural Framework for Structured IoT Command Generation
SCENIC framework reports up to 99% exact match on structured IoT command generation using sub-0.2B models, with pruned INT8 versions retaining 91% EM@1 after 25% size reduction.
-
Recursive Vision Transformer with Dynamic Depth and Width Adjustment for Resource-Efficient Image Semantic Communication
Recursive ViT with dynamic depth/width adjustments reduces parameters by 48.7% and improves reconstruction quality versus baselines in semantic image communication.
-
Meta-Learning Guided Pruning for Few-Shot Plant Pathology on Edge Devices
DACIS-guided PMP pipeline prunes plant pathology models by 78% while retaining 92.3% accuracy and achieving 7 FPS on Raspberry Pi 4 using few-shot meta-learning.
-
Exploring Vision Neural Network Pruning via Screening Methodology
A unified F-statistic screening and weighted evaluation method prunes both unstructured and structured parameters in FNNs and CNNs, claiming order-of-magnitude size reduction with competitive accuracy on vision datasets.
-
Multi-Modality Distillation via Learning the teacher's modality-level Gram Matrix
Proposes a modality relation distillation method that transfers teacher modality relationships via the modality-level Gram Matrix.
-
A Survey on Foundation Models for Personalized Federated Intelligence
The survey introduces personalized federated intelligence (PFI) as a framework integrating federated learning and foundation models to support privacy-aware personalization of AI models.