Mesa-optimization arises when learned models act as optimizers with objectives that can differ from their training loss, creating alignment risks in advanced machine learning.
Deep learning generalizes because the parameter-function map is biased towards simple functions
10 Pith papers cite this work, alongside 87 external citations. Polarity classification is still indexing.
10
Pith papers citing it
87
external citations · Pith
abstract
Deep neural networks (DNNs) generalize remarkably well without explicit regularization even in the strongly over-parametrized regime where classical learning theory would instead predict that they would severely overfit. While many proposals for some kind of implicit regularization have been made to rationalise this success, there is no consensus for the fundamental reason why DNNs do not strongly overfit. In this paper, we provide a new explanation. By applying a very general probability-complexity bound recently derived from algorithmic information theory (AIT), we argue that the parameter-function map of many DNNs should be exponentially biased towards simple functions. We then provide clear evidence for this strong simplicity bias in a model DNN for Boolean functions, as well as in much larger fully connected and convolutional networks applied to CIFAR10 and MNIST. As the target functions in many real problems are expected to be highly structured, this intrinsic simplicity bias helps explain why deep networks generalize well on real world problems. This picture also facilitates a novel PAC-Bayes approach where the prior is taken over the DNN input-output function space, rather than the more conventional prior over parameter space. If we assume that the training algorithm samples parameters close to uniformly within the zero-error region then the PAC-Bayes theorem can be used to guarantee good expected generalization for target functions producing high-likelihood training sets. By exploiting recently discovered connections between DNNs and Gaussian processes to estimate the marginal likelihood, we produce relatively tight generalization PAC-Bayes error bounds which correlate well with the true error on realistic datasets such as MNIST and CIFAR10 and for architectures including convolutional and fully connected networks.
citation-role summary
background 3
citation-polarity summary
roles
background 3polarities
background 3representative citing papers
Infinite-width transformers exhibit an inductive bias against high-complexity polynomial-time algorithms, with derived upper bounds on capturable tasks like sorting and string matching.
Shared chat-template tokens piggyback narrow finetuning behaviors onto out-of-domain queries; regularizing their KV states (TReFT) reduces emergent misalignment and other off-topic generalization.
ALU uses public data to suppress unlearning cost quadratically while characterizing distribution mismatch effects, enabling mass unlearning with maintained utility.
Gradient matching empirically recovers implicit regularization effects such as l2 penalties from early stopping and dropout in neural networks.
Feedback alignment in deep networks is limited by low-rank error signals; orthogonal weight updates and activity normalization raise effective rank and boost performance.
A rate-distortion based switching strategy for adaptive state-action abstractions in RL decomposes value error into Bellman residual and bisimulation metric terms to achieve near-optimal performance under lossy compression in tabular settings.
Proposes memorization-guided two-stage scoring to select debiased training subsets, enabling ERM models to achieve better performance than SOTA debiasing techniques using only 10% of data.
Gradient flow in energy-based models for strictly positive binary distributions produces stable data-consistent fixed points and a learning hierarchy that favors lower-order interactions first, mechanistically explaining distributional simplicity bias.
Outcome optimization induces reward-induced manifold collapse in LLMs by favoring low-complexity spurious correlations over high-complexity causal reasoning, with process reward models acting as topological filters to block shortcuts.
citing papers explorer
-
Risks from Learned Optimization in Advanced Machine Learning Systems
Mesa-optimization arises when learned models act as optimizers with objectives that can differ from their training loss, creating alignment risks in advanced machine learning.
-
Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers
Infinite-width transformers exhibit an inductive bias against high-complexity polynomial-time algorithms, with derived upper bounds on capturable tasks like sorting and string matching.
-
The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment
Shared chat-template tokens piggyback narrow finetuning behaviors onto out-of-domain queries; regularizing their KV states (TReFT) reduces emergent misalignment and other off-topic generalization.
-
Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data
ALU uses public data to suppress unlearning cost quadratically while characterizing distribution mismatch effects, enabling mass unlearning with maintained utility.
-
Estimating Implicit Regularization in Deep Learning
Gradient matching empirically recovers implicit regularization effects such as l2 penalties from early stopping and dropout in neural networks.
-
Overcoming Rank Collapse in Feedback Alignment
Feedback alignment in deep networks is limited by low-rank error signals; orthogonal weight updates and activity normalization raise effective rank and boost performance.
-
Adaptive state-action abstractions via rate-distortion
A rate-distortion based switching strategy for adaptive state-action abstractions in RL decomposes value error into Bellman residual and bisimulation metric terms to achieve near-optimal performance under lossy compression in tabular settings.
-
Mitigating Spurious Correlations with Memorization-Guided Dataset De-Biasing
Proposes memorization-guided two-stage scoring to select debiased training subsets, enabling ERM models to achieve better performance than SOTA debiasing techniques using only 10% of data.
-
Distributional simplicity bias and effective convexity in Energy Based Models
Gradient flow in energy-based models for strictly positive binary distributions produces stable data-consistent fixed points and a learning hierarchy that favors lower-order interactions first, mechanistically explaining distributional simplicity bias.
-
The Paradox of Outcome Optimization: A Causal Information-Theoretic Bound on Reasoning Shortcuts in LLMs
Outcome optimization induces reward-induced manifold collapse in LLMs by favoring low-complexity spurious correlations over high-complexity causal reasoning, with process reward models acting as topological filters to block shortcuts.