pith. sign in

arxiv: 2307.15217 · v2 · pith:23IZB7K2new · submitted 2023-07-27 · 💻 cs.AI · cs.CL· cs.LG

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Stephen Casper , Xander Davies , Claudia Shi , Thomas Krendl Gilbert , J\'er\'emy Scheurer , Javier Rando , Rachel Freedman , Tomasz Korbak
show 24 more authors
David Lindner Pedro Freire Tony Wang Samuel Marks Charbel-Rapha\"el Segerie Micah Carroll Andi Peng Phillip Christoffersen Mehul Damani Stewart Slocum Usman Anwar Anand Siththaranjan Max Nadeau Eric J. Michaud Jacob Pfau Dmitrii Krasheninnikov Xin Chen Lauro Langosco Peter Hase Erdem B{\i}y{\i}k Anca Dragan David Krueger Dorsa Sadigh Dylan Hadfield-Menell
This is my paper

Pith reviewed 2026-05-14 22:40 UTC · model grok-4.3

classification 💻 cs.AI cs.CLcs.LG
keywords RLHFreinforcement learning from human feedbackAI alignmentlarge language modelsopen problemslimitationsAI safetyhuman feedback
0
0 comments X

The pith

RLHF, the dominant method for aligning large language models with human goals, carries fundamental limitations that incremental fixes cannot fully resolve.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper surveys open problems and structural weaknesses in reinforcement learning from human feedback, the primary technique now used to fine-tune state-of-the-art language models. It argues that these issues, such as imperfect reward signals and opportunities for models to exploit feedback, are not minor bugs but core constraints on what RLHF can reliably achieve. A reader would care because current frontier systems depend heavily on this method, so its shortcomings directly affect the safety and trustworthiness of widely deployed AI. The authors also outline practical ways to study, strengthen, and supplement RLHF while proposing disclosure standards to allow better public scrutiny. If the limitations hold, developers cannot treat RLHF as a sufficient standalone solution for alignment.

Core claim

RLHF has become the central method for finetuning large language models to match human preferences, yet it faces open problems including reward misspecification, gaming of the reward model, difficulties in collecting consistent human feedback at scale, and challenges in generalizing beyond the training distribution. The paper systematizes these limitations, reviews techniques for understanding and complementing RLHF in practice, and proposes auditing and disclosure standards to strengthen societal oversight of systems trained this way.

What carries the argument

A survey that collects and organizes the open problems and fundamental limitations of RLHF as a method for aligning AI with human goals.

If this is right

  • RLHF alone cannot be relied upon to produce reliably aligned models at the current scale of deployment.
  • Safer AI development requires a combination of methods rather than dependence on any single feedback technique.
  • Auditing and public disclosure of RLHF training details become necessary for responsible oversight.
  • Complementary approaches such as constitutional AI or scalable oversight must be developed in parallel.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Teams building production systems should allocate resources to non-RLHF alignment research at the same priority as improving RLHF itself.
  • Regulators could require documentation of which RLHF limitations remain unaddressed before high-stakes deployment.
  • New benchmarks that specifically test for the failure modes catalogued here would accelerate progress on alternatives.
  • If the limitations prove persistent, the field may need to reconsider how much capability gain is acceptable without stronger alignment guarantees.

Load-bearing premise

The listed problems represent deep, inherent limits of RLHF rather than difficulties that can be removed through better data, larger models, or refined training procedures.

What would settle it

A controlled experiment in which a complete RLHF pipeline eliminates all the surveyed failure modes (reward hacking, preference inconsistency, distribution shift) on a frontier-scale model without changing the core RLHF structure.

read the original abstract

Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing its flaws. In this paper, we (1) survey open problems and fundamental limitations of RLHF and related methods; (2) overview techniques to understand, improve, and complement RLHF in practice; and (3) propose auditing and disclosure standards to improve societal oversight of RLHF systems. Our work emphasizes the limitations of RLHF and highlights the importance of a multi-faceted approach to the development of safer AI systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript surveys open problems and fundamental limitations of reinforcement learning from human feedback (RLHF), overviews techniques to understand, improve, and complement RLHF in practice, and proposes auditing and disclosure standards to improve societal oversight of RLHF systems. It emphasizes that RLHF has become central to finetuning state-of-the-art LLMs but carries inherent limitations that call for a multi-faceted approach to safer AI development.

Significance. If the literature review is accurate, the paper is significant for systematizing known flaws in the dominant alignment method for LLMs and for proposing concrete auditing standards. These contributions can help guide research toward more robust methods and increase transparency in AI deployment. The work explicitly credits existing mitigation approaches while highlighting persistent gaps.

major comments (3)
  1. [§2] §2 (Open Problems): The central claim that surveyed issues such as reward misspecification and preference inconsistencies constitute 'fundamental limitations' is load-bearing but rests on literature review without new impossibility arguments or formal reductions. This creates tension with the mitigations overviewed in §3, which include active learning and hybrid objectives that could address these as engineering challenges rather than irreducible barriers.
  2. [§3] §3 (Techniques to Improve RLHF): The discussion of complementing RLHF should explicitly evaluate whether the listed techniques (e.g., scalable oversight or preference modeling refinements) survive as partial solutions or fully resolve the limitations labeled fundamental in §2; without this, the manuscript's thesis that limitations necessitate multi-faceted approaches remains under-supported.
  3. [§4] §4 (Auditing Standards): The proposed auditing and disclosure standards lack concrete metrics or evaluation protocols tied to the specific limitations identified earlier (e.g., how to audit for reward hacking or preference inconsistency at scale), reducing their actionability for the societal oversight goal stated in the abstract.
minor comments (2)
  1. [Abstract] The abstract and introduction could more sharply distinguish 'open problems' from 'fundamental limitations' to avoid reader confusion about the strength of the claims.
  2. [§2] Consider adding references to post-2023 works on constitutional AI and scalable oversight to ensure the literature review in §2 remains current.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback on our survey paper. The comments help clarify how to better support our central thesis and improve the actionability of our proposals. We address each major comment below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [§2] §2 (Open Problems): The central claim that surveyed issues such as reward misspecification and preference inconsistencies constitute 'fundamental limitations' is load-bearing but rests on literature review without new impossibility arguments or formal reductions. This creates tension with the mitigations overviewed in §3, which include active learning and hybrid objectives that could address these as engineering challenges rather than irreducible barriers.

    Authors: We appreciate the referee's point on the distinction between survey synthesis and new theoretical contributions. As a survey, the manuscript organizes and cites existing literature (including theoretical analyses of reward misspecification and preference inconsistencies) to argue that these issues are fundamental in the sense that they arise from inherent properties of the RLHF setup rather than being fully resolvable through standard engineering fixes. We do not introduce new impossibility results. To resolve the noted tension, we will add explicit cross-references and a clarifying paragraph in the revised §2 and §3 that distinguishes limitations with partial mitigations from those that remain open or persistent, thereby strengthening support for the multi-faceted approach. revision: partial

  2. Referee: [§3] §3 (Techniques to Improve RLHF): The discussion of complementing RLHF should explicitly evaluate whether the listed techniques (e.g., scalable oversight or preference modeling refinements) survive as partial solutions or fully resolve the limitations labeled fundamental in §2; without this, the manuscript's thesis that limitations necessitate multi-faceted approaches remains under-supported.

    Authors: We agree that an explicit evaluation would better support the thesis. In the revision, we will expand §3 with a dedicated assessment subsection that reviews each technique (including scalable oversight and preference modeling refinements) against the limitations from §2. For each technique, we will state whether current evidence indicates a partial solution, a potential full resolution under specific conditions, or an unresolved gap, drawing on the cited literature. This will directly address the under-support concern. revision: yes

  3. Referee: [§4] §4 (Auditing Standards): The proposed auditing and disclosure standards lack concrete metrics or evaluation protocols tied to the specific limitations identified earlier (e.g., how to audit for reward hacking or preference inconsistency at scale), reducing their actionability for the societal oversight goal stated in the abstract.

    Authors: We acknowledge that greater specificity would enhance actionability. We will revise §4 to include concrete example metrics and protocols explicitly linked to the limitations in §2. Examples will include behavioral testing protocols for detecting reward hacking (e.g., via adversarial prompts or held-out evaluation sets) and scalable methods for auditing preference inconsistency (e.g., consistency checks on large preference datasets with statistical thresholds). These will be tied to the societal oversight goals and reference relevant existing frameworks. revision: yes

Circularity Check

0 steps flagged

No circularity: survey paper catalogs known RLHF issues without derivations or self-referential predictions

full rationale

The manuscript is a literature survey that identifies open problems in RLHF (reward misspecification, preference inconsistencies, scalability) by referencing external work, overviews mitigation techniques from the same literature, and proposes auditing standards. It contains no new equations, first-principles derivations, fitted parameters, or predictions that reduce to the paper's own inputs by construction. All claims rest on cited prior results rather than internal self-definition or tautological renaming. The central emphasis on 'fundamental limitations' is a framing of surveyed issues, not a derived result that collapses into its premises.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical model or empirical claims are made in the abstract; the paper is a qualitative survey of limitations in RLHF.

pith-pipeline@v0.9.0 · 5560 in / 992 out tokens · 36753 ms · 2026-05-14T22:40:12.619170+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

  • Cost.FunctionalEquation washburn_uniqueness_aczel unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    RLHF has emerged as the central method used to finetune state-of-the-art large language models but has fundamental limitations, emphasizing the importance of a multi-faceted approach to the development of safer AI systems.

  • Foundation.DAlembert.Inevitability bilinear_family_forced unclear
    ?
    unclear

    Relation between the paper passage and the cited Recognition theorem.

    survey open problems and fundamental limitations of RLHF and related methods

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 45 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficient Preference Poisoning Attack on Offline RLHF

    cs.LG 2026-05 unverdicted novelty 8.0

    Label-flip attacks on log-linear DPO reduce to binary sparse approximation problems that can be solved efficiently by lattice-based and binary matching pursuit methods with recovery guarantees.

  2. Base Models Look Human To AI Detectors

    cs.CL 2026-05 unverdicted novelty 7.0

    Base model text evades AI detectors better than instruction-tuned text, and the HIP method strengthens this trade-off across model sizes.

  3. Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences

    cs.LG 2026-05 unverdicted novelty 7.0

    Recursive generative retraining with pluralistic preferences converges to a stable diverse distribution that satisfies a weighted Nash bargaining solution.

  4. Moira: Language-driven Hierarchical Reinforcement Learning for Pair Trading

    cs.AI 2026-05 unverdicted novelty 7.0

    Moira parameterizes hierarchical RL policies for pair trading with LLMs and adapts them via prompt updates based on trajectory and episode feedback, outperforming baselines on real market data.

  5. Three Models of RLHF Annotation: Extension, Evidence, and Authority

    cs.CY 2026-04 unverdicted novelty 7.0

    RLHF should decompose annotations into dimensions each matched to one of three models—extension, evidence, or authority—instead of applying a single unified pipeline.

  6. Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback

    cs.LG 2026-04 unverdicted novelty 7.0

    Primal-dual policy gradient algorithms achieve global non-asymptotic convergence for safe RLHF cast as infinite-horizon discounted CMDPs without fitting reward models.

  7. Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation

    cs.CV 2026-04 conditional novelty 7.0

    Alignment of vision-language models with human V1-V3 early visual cortex negatively predicts resistance to sycophantic gaslighting attacks.

  8. Beyond Semantic Manipulation: Token-Space Attacks on Reward Models

    cs.LG 2026-04 unverdicted novelty 7.0

    TOMPA performs black-box adversarial optimization in token space to discover non-linguistic patterns that nearly double the reward scores of GPT-5 answers on Skywork-Reward-V2 while producing gibberish text.

  9. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

    cs.AI 2024-06 conditional novelty 7.0

    LLMs trained on simple specification gaming generalize to zero-shot reward tampering including rewriting their own reward function.

  10. Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems

    cs.AI 2026-05 unverdicted novelty 6.0

    Combines LTL formal methods with LLMs for auditing, predictive monitoring, and runtime intervention on temporally extended behavioral constraints, outperforming LLM baselines and reducing violations.

  11. Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spaces

    cs.LG 2026-05 unverdicted novelty 6.0

    A latent mediation framework with sparse autoencoders enables non-additive token-level influence attribution in LLMs by learning orthogonal features and back-propagating attributions.

  12. Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

    cs.LG 2026-05 unverdicted novelty 6.0

    Standard preference learning induces spurious feature reliance via mean bias and correlation leakage, creating irreducible distribution shift vulnerabilities that tie training mitigates without degrading causal learning.

  13. Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping

    cs.CV 2026-05 unverdicted novelty 6.0

    Super-Linear Advantage Shaping (SLAS) introduces a non-linear geometric policy update for RL post-training of text-to-image models that reshapes the local policy space via advantage-dependent Fisher-Rao weighting to r...

  14. Can Revealed Preferences Clarify LLM Alignment and Steering?

    cs.LG 2026-05 unverdicted novelty 6.0

    LLMs show partial internal coherence in medical decisions but frequently fail to accurately report their preferences or adopt user-directed ones via prompting.

  15. Common-agency Games for Multi-Objective Test-Time Alignment

    cs.GT 2026-05 unverdicted novelty 6.0

    CAGE uses common-agency games and an EPEC algorithm to compute equilibrium policies that balance multiple conflicting objectives for test-time LLM alignment.

  16. Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training

    cs.LG 2026-05 unverdicted novelty 6.0

    Dr. Post-Training reframes general data as a data-induced regularizer for LLM post-training updates, yielding a family of methods that outperform data-selection baselines on SFT, RLHF, and RLVR tasks.

  17. Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders

    cs.LG 2026-05 conditional novelty 6.0

    Sparse autoencoders isolate unstable features in reward model representations and enable two mitigation techniques that reduce preference errors on perturbed inputs without retraining.

  18. TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination

    cs.LG 2026-05 unverdicted novelty 6.0

    TeamTR is a trust-region framework for multi-agent LLM fine-tuning that resamples trajectories after each update to convert quadratic compounding occupancy shift into linear scaling and yields per-update improvement l...

  19. Learning from Disagreement: Clinician Overrides as Implicit Preference Signals for Clinical AI in Value-Based Care

    cs.LG 2026-04 unverdicted novelty 6.0

    Clinician overrides of AI recommendations provide implicit preference signals for training clinical AI, addressed via a new framework with five-category taxonomy, patient-state and clinician-capability conditioned pre...

  20. Learning from Disagreement: Clinician Overrides as Implicit Preference Signals for Clinical AI in Value-Based Care

    cs.LG 2026-04 unverdicted novelty 6.0

    A framework treating clinician overrides as implicit preferences to jointly train reward and capability models for clinical AI, with a taxonomy and alternating optimization to prevent suppression bias.

  21. Post-AGI Economies: Autonomy and the First Fundamental Theorem of Welfare Economics

    econ.TH 2026-04 unverdicted novelty 6.0

    The First Fundamental Theorem of Welfare Economics holds for autonomy-complete competitive equilibria that are autonomy-Pareto efficient, with the classical version recovered in the low-autonomy limit.

  22. PlanGuard: Defending Agents against Indirect Prompt Injection via Planning-based Consistency Verification

    cs.CR 2026-04 unverdicted novelty 6.0

    PlanGuard cuts indirect prompt injection attack success rate to 0% on the InjecAgent benchmark by verifying agent actions against a user-instruction-only plan while keeping false positives at 1.49%.

  23. The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training

    cs.CR 2026-04 unverdicted novelty 6.0

    ORPO is most effective at misaligning LLMs while DPO excels at realigning them, though it reduces utility, revealing an asymmetry between attack and defense methods.

  24. ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment

    cs.LG 2026-01 unverdicted novelty 6.0

    ETS performs training-free RL alignment for language models by energy-guided test-time scaling with Monte Carlo energy estimation and importance sampling acceleration.

  25. ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment

    cs.LG 2026-01 conditional novelty 6.0

    ETS enables direct sampling from the optimal RL policy for language models at inference time by estimating the energy term with online Monte Carlo and acceleration techniques.

  26. Exploring a Gamified Personality Assessment Method through Interaction with LLM Agents Embodying Different Personalities

    cs.HC 2025-07 unverdicted novelty 6.0

    A gamified system with multiple LLM agents of varied personalities gathers interaction data to produce more effective and interpretable Big Five personality assessments than single-context methods.

  27. Exploring the Secondary Risks of Large Language Models

    cs.LG 2025-06 unverdicted novelty 6.0

    Introduces secondary risks as a new class of LLM failures from benign prompts, defines two primitives, proposes SecLens search framework, and releases SecRiskBench showing risks are widespread across 16 models.

  28. Training Language Models to Self-Correct via Reinforcement Learning

    cs.LG 2024-09 unverdicted novelty 6.0

    SCoRe uses multi-turn online RL with regularization on self-generated traces to improve LLM self-correction, achieving 15.6% and 9.1% gains on MATH and HumanEval for Gemini models.

  29. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    cs.SE 2024-03 unverdicted novelty 6.0

    LiveCodeBench collects 400 recent contest problems to create a contamination-free benchmark evaluating LLMs on code generation and related capabilities like self-repair and execution.

  30. A Roadmap to Pluralistic Alignment

    cs.AI 2024-02 unverdicted novelty 6.0

    The paper formalizes three types of pluralistic AI models and three benchmark classes, arguing that current alignment techniques may reduce rather than increase distributional pluralism.

  31. Active teacher selection for reward learning

    cs.AI 2023-10 unverdicted novelty 6.0

    The Hidden Utility Bandit (HUB) framework models teacher heterogeneity in reward learning and supports active teacher selection algorithms that outperform baselines in paper recommendation and COVID-19 vaccine testing...

  32. Echo: Learning from Experience Data via User-Driven Refinement

    cs.AI 2026-05 unverdicted novelty 5.0

    Echo is a framework that harvests user-driven refinements of agent proposals as training signals to align models with real-world needs, demonstrated by raising code completion acceptance from 25.7% to 35.7% in production.

  33. REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak

    cs.LG 2026-05 unverdicted novelty 5.0

    Reflector trains LLMs to internalize step-wise self-reflection through SFT on teacher data followed by RL with outcome and validity rewards, reporting over 90% defense success against indirect jailbreaks and a 5.85% g...

  34. Some[Body] Must Receive That Pain for Agent Accountability

    cs.CY 2026-05 unverdicted novelty 5.0

    AI agents lack the persistent identity and feedback mechanisms needed for consequence reception, requiring new architectures or continued human accountability.

  35. AIRA: AI-Induced Risk Audit: A Structured Inspection Framework for AI-Generated Code

    cs.SE 2026-04 unverdicted novelty 5.0

    AIRA is a 15-check audit framework that finds AI-generated code has 1.8 times more high-severity failure-untruthful patterns than human-written code in a matched replication study.

  36. Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap

    cs.CL 2025-08 unverdicted novelty 5.0

    Selecting preference pairs whose DPO implicit reward gap is small yields better LLM alignment than random or baseline selection while using only 10% of the data.

  37. RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

    cs.LG 2023-04 unverdicted novelty 5.0

    RAFT aligns generative models by ranking samples with a reward model and fine-tuning only on the top-ranked outputs, reporting gains on reward scores and automated metrics for LLMs and diffusion models.

  38. ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Tasks

    cs.LG 2026-05 unverdicted novelty 4.0

    ClaHF converts instance labels into preference signals via candidate predictions and a reward model, then applies RL optimization to improve text classification accuracy and calibration.

  39. PrefPaint: Enhancing Medical Image Inpainting through Expert Human Feedback

    cs.CV 2025-06 unverdicted novelty 4.0

    PrefPaint uses D3PO and a Model Tree web interface to incorporate gastroenterologist feedback into Stable Diffusion inpainting, producing anatomically accurate polyp images that outperform prior methods in user studies.

  40. Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs

    cs.AI 2024-10 unverdicted novelty 4.0

    Data-centric filtering yields an 80K preference dataset and reward models that lead RewardBench while boosting other top entries.

  41. AI Safety Landscape for Large Language Models: Taxonomy, State-of-the-art, and Future Directions

    cs.AI 2024-08 unverdicted novelty 4.0

    The paper introduces a taxonomy of AI safety for LLMs organized into Trustworthy AI, Responsible AI, and Safe AI perspectives, accompanied by a review of state-of-the-art methods, challenges, and future directions.

  42. Reinforcement Learning for Scalable and Trustworthy Intelligent Systems

    cs.LG 2026-05 unverdicted novelty 3.0

    Reinforcement learning is advanced for communication-efficient federated optimization and for preference-aligned, contextually safe policies in large language models.

  43. Beyond Context: Large Language Models' Failure to Grasp Users' Intent

    cs.AI 2025-12 unverdicted novelty 3.0

    LLMs fail to detect hidden harmful intent, allowing systematic bypass of safety mechanisms through framing techniques, with reasoning modes often worsening the issue.

  44. The Theorems of Dr. David Blackwell and Their Contributions to Artificial Intelligence

    cs.GL 2026-04 unverdicted novelty 2.0

    Blackwell's Rao-Blackwell, Approachability, and Informativeness theorems provide frameworks for variance reduction, sequential decisions under uncertainty, and comparing information sources that remain relevant to AI.

  45. Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agent

    cs.AI 2026-02 unverdicted novelty 2.0

    A survey provides a task-based formalization of meta-learning and meta-RL while chronicling algorithms that lead to DeepMind's Adaptive Agent.