RICA replaces ICA's global generative model with local Riemannian geometry, introducing a disentanglement tensor based on the Hessian of the log-likelihood and Ricci curvature to measure pointwise disentanglement, which recovers sources across manifolds in controlled tests.
hub
Towards a Definition of Disentangled Representations
21 Pith papers cite this work, alongside 296 external citations. Polarity classification is still indexing.
21
Pith papers citing it
296
external citations · Pith
abstract
How can intelligent agents solve a diverse set of tasks in a data-efficient manner? The disentangled representation learning approach posits that such an agent would benefit from separating out (disentangling) the underlying structure of the world into disjoint parts of its representation. However, there is no generally agreed-upon definition of disentangling, not least because it is unclear how to formalise the notion of world structure beyond toy datasets with a known ground truth generative process. Here we propose that a principled solution to characterising disentangled representations can be found by focusing on the transformation properties of the world. In particular, we suggest that those transformations that change only some properties of the underlying world state, while leaving all other properties invariant, are what gives exploitable structure to any kind of data. Similar ideas have already been successfully applied in physics, where the study of symmetry transformations has revolutionised the understanding of the world structure. By connecting symmetry transformations to vector representations using the formalism of group and representation theory we arrive at the first formal definition of disentangled representations. Our new definition is in agreement with many of the current intuitions about disentangling, while also providing principled resolutions to a number of previous points of contention. While this work focuses on formally defining disentangling - as opposed to solving the learning problem - we believe that the shift in perspective to studying data transformations can stimulate the development of better representation learning algorithms.
hub tools
citation-role summary
background 2
citation-polarity summary
roles
background 2representative citing papers
Linear representations of high-level concepts in LLMs are formalized via counterfactuals in input and output spaces, unified under a causal inner product that enables consistent probing and steering.
WTA bottlenecks enforce highly symbolic representations of categorical latent factors in multi-task deep learning under specific conditions.
KamonBench is a grammar-based dataset of 20,000 synthetic Japanese crests with multi-format annotations that enables direct evaluation of factor recovery beyond caption accuracy in vision-language models.
Mechanistic independence criteria yield identifiability of latent subspaces under nonlinear mixing by focusing on action-based independence rather than latent distributions, with a hierarchy and graph-theoretic view of subspaces.
Proves regular representation must appear in latent space of finite-group equivariant encoders and enforces it via auxiliary loss to match specialized equivariant models without added parameters.
Proposes PrOSe parameterization of latent space as product of orthogonal spheres to improve disentangled representation learning, with closed-form ortho-normality loss under equal block size assumption.
NEO is a probabilistic neural model that induces compositional programs as a learned Language of Thought from non-textual observations and executes them via a shared transition model to enable explanation-driven generalization.
A new framework shows concept subspaces are not unique, estimator choice affects containment and disentanglement, LEACE works well but generalizes poorly, and HuBERT encodes phone info as contained and disentangled from speaker info while speaker info resists compact containment.
Parameter division decomposes group transformations via parameter splitting and homomorphism constraints to enable unsupervised categorization of image transformations like rotation, translation, and scale.
Low-rank graphs induce latents that form causal abstractions, with identifiability results and a practical objective enabling unsupervised learning of high-level SCMs from low-level measurements.
VH-CBM uses a Gaussian process in VLM embedding space to propagate sparse human annotations and improve concept accuracy and calibration over pure VLM-guided concept bottleneck models.
Disentangled representations enable a new auditing procedure to identify proxy features and quantify their influence on model outcomes more effectively than prior methods.
RG-inspired lattice models for piecewise GLMs provide explicit interpretable partitions and a replica-analysis-derived scaling law for regularization that allows increasing complexity without expected rise in generalization loss.
Discrete decentralized learning dynamics on manifolds converge uniformly to an overdamped Langevin SDE whose stationary states produce orthogonally disentangled, linearly separable features.
DEAL disentangles concepts from images and text using causal interventions and enforces equivariance on compositional transformations to boost generalization in VQA, outperforming prior methods on CLEVR-CoGenT and GQA-SGL.
Stimulus symmetries render many neural representations functionally equivalent yet produce qualitatively different RSMs, including drifting ones from SGD or regularization in image-encoding networks.
Empirical tests of VLM-CBMs show VLM supervision differs from expert annotations depending on task and that concept accuracy correlates weakly with quality metrics.
ADIS-GAN disentangles affine transformations in a GAN to achieve over 98% classification accuracy on MNIST within 30 degrees rotation and over 90% under FGSM and PGD attacks while generating rotation and scaling factors.
Authors propose a fibre bundle gauge theory model for disentangled representations and connect it to the relativity twins paradox.
citing papers explorer
-
Disentanglement Beyond Generative Models with Riemannian ICA
RICA replaces ICA's global generative model with local Riemannian geometry, introducing a disentanglement tensor based on the Hessian of the log-likelihood and Ricci curvature to measure pointwise disentanglement, which recovers sources across manifolds in controlled tests.
-
The Linear Representation Hypothesis and the Geometry of Large Language Models
Linear representations of high-level concepts in LLMs are formalized via counterfactuals in input and output spaces, unified under a causal inner product that enables consistent probing and steering.
-
Winner-Take-All bottlenecks enforce disentangled symbolic representations in multi-task learning
WTA bottlenecks enforce highly symbolic representations of categorical latent factors in multi-task deep learning under specific conditions.
-
KamonBench: A Grammar-Based Dataset for Evaluating Compositional Factor Recovery in Vision-Language Models
KamonBench is a grammar-based dataset of 20,000 synthetic Japanese crests with multi-format annotations that enables direct evaluation of factor recovery beyond caption accuracy in vision-language models.
-
Mechanistic Independence: A Principle for Identifiable Disentangled Representations
Mechanistic independence criteria yield identifiability of latent subspaces under nonlinear mixing by focusing on action-based independence rather than latent distributions, with a hierarchy and graph-theoretic view of subspaces.
-
Algebraic Priors for Approximately Equivariant Networks
Proves regular representation must appear in latent space of finite-group equivariant encoders and enforces it via auxiliary loss to match specialized equivariant models without added parameters.
-
Product of Orthogonal Spheres Parameterization for Disentangled Representation Learning
Proposes PrOSe parameterization of latent space as product of orthogonal spheres to improve disentangled representation learning, with closed-form ortho-normality loss under equal block size assumption.
-
Learning to Theorize the World from Observation
NEO is a probabilistic neural model that induces compositional programs as a learned Language of Thought from non-textual observations and executes them via a shared transition model to enable explanation-driven generalization.
-
A framework for analyzing concept representations in neural models
A new framework shows concept subspaces are not unique, estimator choice affects containment and disentanglement, LEACE works well but generalizes poorly, and HuBERT encodes phone info as contained and disentangled from speaker info while speaker info resists compact containment.
-
Transformation Categorization Based on Group Decomposition Theory Using Parameter Division
Parameter division decomposes group transformations via parameter splitting and homomorphism constraints to enable unsupervised categorization of image transformations like rotation, translation, and scale.
-
Unsupervised Causal Abstractions Discovery
Low-rank graphs induce latents that form causal abstractions, with identifiability results and a practical objective enabling unsupervised learning of high-level SCMs from low-level measurements.
-
Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations
VH-CBM uses a Gaussian process in VLM embedding space to propagate sparse human annotations and improve concept accuracy and calibration over pure VLM-guided concept bottleneck models.
-
Disentangling Influence: Using Disentangled Representations to Audit Model Predictions
Disentangled representations enable a new auditing procedure to identify proxy features and quantify their influence on model outcomes more effectively than prior methods.
-
A renormalization-group inspired lattice-based framework for piecewise generalized linear models
RG-inspired lattice models for piecewise GLMs provide explicit interpretable partitions and a replica-analysis-derived scaling law for regularization that allows increasing complexity without expected rise in generalization loss.
-
Continuous Limits of Coupled Flows in Representation Learning
Discrete decentralized learning dynamics on manifolds converge uniformly to an overdamped Langevin SDE whose stationary states produce orthogonally disentangled, linearly separable features.
-
Disentanglement-Based Equivariant Learning for Compositional VQA
DEAL disentangles concepts from images and text using causal interventions and enforces equivariance on compositional transformations to boost generalization in VQA, outperforming prior methods on CLEVR-CoGenT and GQA-SGL.
-
Stimulus symmetries can confound representational similarity analyses
Stimulus symmetries render many neural representations functionally equivalent yet produce qualitatively different RSMs, including drifting ones from SGD or regularization in image-encoding networks.
-
If Concept Bottlenecks are the Question, are Foundation Models the Answer?
Empirical tests of VLM-CBMs show VLM supervision differs from expert annotations depending on task and that concept accuracy correlates weakly with quality metrics.
-
Affine Disentangled GAN for Interpretable and Robust AV Perception
ADIS-GAN disentangles affine transformations in a GAN to achieve over 98% classification accuracy on MNIST within 30 degrees rotation and over 90% under FGSM and PGD attacks while generating rotation and scaling factors.
-
Gauge theory and twins paradox of disentangled representations
Authors propose a fibre bundle gauge theory model for disentangled representations and connect it to the relativity twins paradox.
- Unsupervised Disentanglement Without Compromises : How Functional Orthogonality Enforces Identifiability