REVIEW 6 cited by
On Provable Length and Compositional Generalization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Out-of-distribution generalization capabilities of sequence-to-sequence models can be studied from the lens of two crucial forms of generalization: length generalization -- the ability to generalize to longer sequences than ones seen during training, and compositional generalization: the ability to generalize to token combinations not seen during training. In this work, we provide first provable guarantees on length and compositional generalization for common sequence-to-sequence models -- deep sets, transformers, state space models, and recurrent neural nets -- trained to minimize the prediction error. We show that \emph{limited capacity} versions of these different architectures achieve both length and compositional generalization provided the training distribution is sufficiently diverse. In the first part, we study structured limited capacity variants of different architectures and arrive at the generalization guarantees with limited diversity requirements on the training distribution. In the second part, we study limited capacity variants with less structural assumptions and arrive at generalization guarantees but with more diversity requirements on the training distribution. Further, we also show that chain-of-thought supervision enables length generalization in higher capacity counterparts of the different architectures we study.
Forward citations
Cited by 6 Pith papers
-
Universal priors: solving empirical Bayes via Bayesian inference and pretraining
A simple random prior-on-prior lets pretrained transformers achieve near-optimal empirical Bayes regret uniformly over all test priors, and length generalization matches α-posterior inference.
-
When Does In-Context Search Help? A Sampling-Complexity Theory of Reflection-Driven Reasoning
When reflections localize early errors, in-context search solves exp-small pass-rate problems with poly sequential attempts; otherwise it offers no asymptotic gain over parallel sampling, and the update is learnable a...
-
PatchGen: Learning Soft Intra-Image Predictive Subsets for Visual Generalization
PatchGen learns a sample-dependent soft mask that selects label-predictive image patches, improving visual generalization across domain, category, and combined shifts.
-
From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?
Correlational disentanglement scores do not guarantee that steering a feature changes only its target concept; current featurizers entangle concepts in their causal effects.
-
Extrapolation by Association: Length Generalization Transfer in Transformers
Length generalization on a short-trained main task can be inherited from a longer-trained related auxiliary task trained jointly with it.
-
Saving for the future: Enhancing generalization via partial logic regularization
PL-Reg adds a trainable mask and a defined/undefined classification loss to logic-based regularization, improving unknown-class accuracy across GCD, mDG+GCD, and CIL benchmarks.
Discussion (0). Continue with ORCID to comment.