Empirical analysis identifies task-interfering layers in VLMs and proposes TaLo, a test-time method to bypass them for improved performance without training.
The curse of depth in large language models.arXiv preprint arXiv:2502.05795
11 Pith papers cite this work, alongside 1 external citations. Polarity classification is still indexing.
11
Pith papers citing it
1
external citations · external index
citation-role summary
background 1
citation-polarity summary
years
2026 11roles
background 1polarities
background 1representative citing papers
CascadeFormer tapers Transformer width with depth based on gradient fan-in asymmetry to match uniform baselines in perplexity while cutting latency 8.6%.
FPRM is a Transformer-based model using fixed-point convergence for adaptive halting in looped architectures, claimed effective on Sudoku, Maze, state-tracking, and ARC-AGI benchmarks.
AlphaQ performs calibration-free mixed-precision quantization of MoE models by allocating higher bits to experts whose weight spectra exhibit stronger heavy-tailed structure according to HT-SR theory, outperforming calibration-based methods and reaching near full-precision accuracy at 3.5 average bi
Separating non-credit-assigned hierarchical memory from prediction, plus sleep-time accelerated replay, extends effective streaming context beyond short BPTT with linear cost and better retention than recurrent baselines.
Protein language models exhibit consistent depth inefficiency where most task-relevant computation occurs in a subset of layers, mirroring patterns in large language models.
SiameseNorm is a two-stream architecture that reconciles Pre-Norm and Post-Norm in Transformers by coupling streams via shared residual blocks, yielding performance gains with maintained stability on language, vision, and diffusion models.
Pion is an optimizer that preserves the singular values of weight matrices in LLM training by applying orthogonal equivalence transformations.
Pruning pretrained MoE models outperforms training from scratch under fixed budget, different expert compression methods converge after continued training, and progressive pruning plus multi-token KD improves the final 23A2B model.
Position paper claims fixed exponents in scaling laws arise from generic mechanisms while coefficients vary with data and architecture, making the latter the focus for improvements.
citing papers explorer
-
Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models
Empirical analysis identifies task-interfering layers in VLMs and proposes TaLo, a test-time method to bypass them for improved performance without training.
-
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
CascadeFormer tapers Transformer width with depth based on gradient fan-in asymmetry to match uniform baselines in perplexity while cutting latency 8.6%.
-
Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
FPRM is a Transformer-based model using fixed-point convergence for adaptive halting in looped architectures, claimed effective on Sudoku, Maze, state-tracking, and ARC-AGI benchmarks.
-
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
AlphaQ performs calibration-free mixed-precision quantization of MoE models by allocating higher bits to experts whose weight spectra exhibit stronger heavy-tailed structure according to HT-SR theory, outperforming calibration-based methods and reaching near full-precision accuracy at 3.5 average bi
-
SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition
Separating non-credit-assigned hierarchical memory from prediction, plus sleep-time accelerated replay, extends effective streaming context beyond short BPTT with linear cost and better retention than recurrent baselines.
-
From Words to Amino Acids: Does the Curse of Depth Persist?
Protein language models exhibit consistent depth inefficiency where most task-relevant computation occurs in a subset of layers, mirroring patterns in large language models.
-
SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm
SiameseNorm is a two-stream architecture that reconciles Pre-Norm and Post-Norm in Transformers by coupling streams via shared residual blocks, yielding performance gains with maintained stability on language, vision, and diffusion models.
-
Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
Pion is an optimizer that preserves the singular values of weight matrices in LLM training by applying orthogonal equivalence transformations.
-
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
Pruning pretrained MoE models outperforms training from scratch under fixed budget, different expert compression methods converge after continued training, and progressive pruning plus multi-token KD improves the final 23A2B model.
-
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Position paper claims fixed exponents in scaling laws arise from generic mechanisms while coefficients vary with data and architecture, making the latter the focus for improvements.
- On the Limits of Layer Pruning for Generative Reasoning in Large Language Models