REVIEW 5 cited by
Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large transformer models pretrained on offline reinforcement learning datasets have demonstrated remarkable in-context reinforcement learning (ICRL) capabilities, where they can make good decisions when prompted with interaction trajectories from unseen environments. However, when and how transformers can be trained to perform ICRL have not been theoretically well-understood. In particular, it is unclear which reinforcement-learning algorithms transformers can perform in context, and how distribution mismatch in offline training data affects the learned algorithms. This paper provides a theoretical framework that analyzes supervised pretraining for ICRL. This includes two recently proposed training methods -- algorithm distillation and decision-pretrained transformers. First, assuming model realizability, we prove the supervised-pretrained transformer will imitate the conditional expectation of the expert algorithm given the observed trajectory. The generalization error will scale with model capacity and a distribution divergence factor between the expert and offline algorithms. Second, we show transformers with ReLU attention can efficiently approximate near-optimal online reinforcement learning algorithms like LinUCB and Thompson sampling for stochastic linear bandits, and UCB-VI for tabular Markov decision processes. This provides the first quantitative analysis of the ICRL capabilities of transformers pretrained from offline trajectories.
Forward citations
Cited by 5 Pith papers
-
Learning-To-Measure: In-Context Active Feature Acquisition
One transformer, pretrained with sequence modeling, greedily acquires features to reduce prediction uncertainty across multiple tasks with missing data, matching or beating task-specific AFA baselines without per-task...
-
Sample Complexity and Representation Ability of Test-time Scaling Paradigms
Best-of-n sampling provably needs about 1/Δ samples versus 1/Δ² for self-consistency, and a constructed Transformer can route among experts using verifier feedback to reach near-optimal final responses.
-
Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Models
A hand-constructed O(ln L + T)-layer Transformer is shown to approximate low-rank hidden Markov models in-context, with lower layers extracting local history features and upper layers performing regression-based decoding.
-
Filtering Learning Histories Enhances In-Context Reinforcement Learning
Filtering ICRL pretraining datasets by a simple improvement-and-stability score boosts downstream in-context learning performance across AD, DICP, and DPT baselines.
-
Interaction as Intelligence: Deep Research With Human-AI Partnership
A human-in-the-loop deep research system with transparent, interruptible interaction is claimed to outperform commercial baselines, but the evidence is weakened by small samples and biased instructions.
Discussion (0). Sign in to comment.