NEO is a probabilistic neural model that induces compositional programs as a learned Language of Thought from non-textual observations and executes them via a shared transition model to enable explanation-driven generalization.
Najoung Kim and Tal Linzen
10 Pith papers cite this work. Polarity classification is still indexing.
10
Pith papers citing it
abstract
State-of-the-art machine learning methods exhibit limited compositional generalization. At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements. We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks. We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures. We find that they fail to generalize compositionally and that there is a surprisingly strong negative correlation between compound divergence and accuracy. We also demonstrate how our method can be used to create new compositionality benchmarks on top of the existing SCAN dataset, which confirms these findings.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
A transformer trained on random meaningless MicroPy programs generalizes to execute diverse human-written programs, providing empirical evidence it can act as a universal computer.
R-EMID metric with upper bound shows user shifts pose highest risk to role-playing model generalization, with co-evolving RL as most effective mitigation.
SFT supplies entangled compositional traces of atomic skills and routing modules; RL identifies those modules and enables recombination on novel compositions outside the SFT support.
Observational causal-inspired analysis finds prompt optimization failures arise from systematic interactions between edit families and task characteristics rather than random artifacts.
LLMs show strong spatial generalization to unseen maps in shortest-path tasks but fail length scaling due to recursive instability, with data coverage setting hard limits.
LLMs solve compositional factual recall either by computing intermediates or directly, with mechanism choice correlated to translation geometry in embedding spaces.
A 235-item multimodal stress-test shows frontier closed models outpace open-weight peers by ~10% and leaves shared failures on counting, spatial, and character-level tasks.
ClassicLogic is an open-source benchmark using four logic puzzles with a hierarchical strategy knowledge base to evaluate three forms of compositional generalization in AI agents.
DiRL extracts a reasoning-memorization direction from model representations inside GRPO to weight gradients and shape rewards so that exploration favors reasoning trajectories over memorization ones.
citing papers explorer
-
Learning to Theorize the World from Observation
NEO is a probabilistic neural model that induces compositional programs as a learned Language of Thought from non-textual observations and executes them via a shared transition model to enable explanation-driven generalization.
-
Training Transformers as a Universal Computer
A transformer trained on random meaningless MicroPy programs generalizes to execute diverse human-written programs, providing empirical evidence it can act as a universal computer.
-
Understanding Generalization in Role-Playing Models via Information Theory
R-EMID metric with upper bound shows user shifts pose highest risk to role-playing model generalization, with co-evolving RL as most effective mitigation.
-
From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning
SFT supplies entangled compositional traces of atomic skills and routing modules; RL identifies those modules and enables recombination on novel compositions outside the SFT support.
-
Why Prompt Optimization Works, and Why It Sometimes Doesn't: A Causal-Inspired Edit-Level Analysis
Observational causal-inspired analysis finds prompt optimization failures arise from systematic interactions between edit families and task characteristics rather than random artifacts.
-
Generalization in LLM Problem Solving: The Case of the Shortest Path
LLMs show strong spatial generalization to unseen maps in shortest-path tasks but fail length scaling due to recursive instability, with data coverage setting hard limits.
-
How Do Language Models Compose Functions?
LLMs solve compositional factual recall either by computing intermediates or directly, with mechanism choice correlated to translation geometry in embedding spaces.
-
Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models
A 235-item multimodal stress-test shows frontier closed models outpace open-weight peers by ~10% and leaves shared failures on counting, spatial, and character-level tasks.
-
ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization
ClassicLogic is an open-source benchmark using four logic puzzles with a hierarchical strategy knowledge base to evaluate three forms of compositional generalization in AI agents.
-
Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learning
DiRL extracts a reasoning-memorization direction from model representations inside GRPO to weight gradients and shape rewards so that exploration favors reasoning trajectories over memorization ones.