Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
read the original abstract
A key factor in the success of deep neural networks is the ability to scale models to improve performance by varying the architecture depth and width. This simple property of neural network design has resulted in highly effective architectures for a variety of tasks. Nevertheless, there is limited understanding of effects of depth and width on the learned representations. In this paper, we study this fundamental question. We begin by investigating how varying depth and width affects model hidden representations, finding a characteristic block structure in the hidden representations of larger capacity (wider or deeper) models. We demonstrate that this block structure arises when model capacity is large relative to the size of the training set, and is indicative of the underlying layers preserving and propagating the dominant principal component of their representations. This discovery has important ramifications for features learned by different models, namely, representations outside the block structure are often similar across architectures with varying widths and depths, but the block structure is unique to each model. We analyze the output predictions of different model architectures, finding that even when the overall accuracy is similar, wide and deep models exhibit distinctive error patterns and variations across classes.
This paper has not been read by Pith yet.
Forward citations
Cited by 17 Pith papers
-
Neural Networks Provably Learn Spectral Representations for Group Composition
Two-layer neural networks provably converge almost surely to irreducible representations of finite groups when trained on the group composition task, with the dynamics governed by Riemannian gradient ascent on a repre...
-
The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior
The grokking delay in encoder-decoder models on one-step Collatz prediction stems from decoder inability to use early-learned encoder representations of parity and residue structure, with numeral base acting as a stro...
-
Resolving superposition in AI for interpretability and cross-modal alignment in patient-neuronal images
Sparse autoencoders resolve superposition in image-based neuron representations, recovering geometric fidelity and enabling scRNA-seq adaptation plus GW-map alignment to reconstruct pathology pathways without spatial ...
-
TopoGeoScore: A Self-Supervised Source-Only Geometric Framework for OOD Checkpoint Selection
TopoGeoScore combines a torsion-inspired Laplacian log-determinant, Ollivier-Ricci curvature, and higher-order topological summaries from source embeddings, with weights learned via self-supervised invariance to geome...
-
Tumor-aware augmentation with task-guided attention analysis improves rectal cancer segmentation from magnetic resonance images
Tumor-aware augmentation and anisotropic cropping improve CT-to-MRI transfer for rectal cancer segmentation in hierarchical transformers by reducing attention dilution from padding and enhancing feature adaptation.
-
Transferable Physics-Informed Representations via Closed-Form Head Adaptation
Pi-PINN learns transferable physics-informed representations and solves known or unseen PDEs via closed-form pseudoinverse head adaptation, achieving 100-1000x faster predictions and 10-100x lower error than standard ...
-
Invariant Features for Global Crop Type Classification
CropNet, a lightweight CNN jointly convolving spectral and temporal dimensions, learns invariant crop signatures from multispectral time series and outperforms larger models under geographic domain shifts on the new C...
-
Primus: Enforcing Attention Usage for 3D Medical Image Segmentation
Primus and PrimusV2 are Transformer-centric models that match or exceed nnU-Net and top CNNs on nine 3D medical segmentation datasets by enforcing attention usage.
-
Resolving superposition in AI for interpretability and cross-modal alignment in patient-neuronal images
Sparse autoencoders resolve superposition in image-based neural representations of neurons, recovering metric geometry and enabling de novo cross-modal alignment to scRNA-seq via Gromov-Wasserstein transport.
-
Attention Alignment Between Humans and Vision-Language Models
LSTM-decoder VLMs reach 80-87% of human noise ceiling in fixation alignment versus 40-59% for Transformer decoders, though LSTM maps are diffuse and less task-specific while CNN-Transformer maps better predict synthet...
-
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
PID applied to MLLMs identifies task-specific modality interaction profiles that generalize across models, extend to tri-modal cases, and yield initial performance gains via reweighting.
-
Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks
SMIT, which combines masked image modeling with self-distillation, delivers the highest segmentation accuracy, fastest convergence, and best few-shot performance across nine CT and MRI tasks compared to contrastive an...
-
Tumor-aware augmentation with task-guided attention analysis improves rectal cancer segmentation from magnetic resonance images
Tumor-aware augmentation and anisotropic cropping mitigate token inefficiency and feature adaptation issues in CT-to-MRI transfer for rectal cancer segmentation, achieving 88.7-91.1% detection rates.
-
Deep Reprogramming Distillation for Medical Foundation Models
DRD introduces a reprogramming module and CKA-based distillation to enable efficient, robust adaptation of medical foundation models to downstream 2D/3D classification and segmentation tasks, outperforming prior PEFT ...
-
TOAST: Transformer Optimization using Adaptive and Simple Transformations
TOAST approximates full transformer blocks in pretrained models via lightweight closed-form mappings to cut parameters and FLOPs without retraining or finetuning.
-
How Data Augmentation Shapes Neural Representations
Data augmentation produces well-behaved trajectories in shape-invariant representation space, with augmentation type steering distinct directions and geometry predicting ensembling gains.
-
TopoGeoScore: A Self-Supervised Source-Only Geometric Framework for OOD Checkpoint Selection
TopoGeoScore learns a non-negative linear combination of geometric and topological features from source embeddings via self-supervised invariance to select robust checkpoints for OOD scenarios.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.