Pith. sign in

REVIEW 9 cited by

Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.23150 v1 pith:TGBTB2W2 submitted 2025-05-29 cs.LG

Michal Nauman , Marek Cygan , Carmelo Sferrazza , Aviral Kumar , Pieter Abbeel This is my paper
classification cs.LG
keywords multi-taskmodelstrainingapproachhigh-capacityonlinepoliciessingle-task
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advances in language modeling and vision stem from training large models on diverse, multi-task data. This paradigm has had limited impact in value-based reinforcement learning (RL), where improvements are often driven by small models trained in a single-task context. This is because in multi-task RL sparse rewards and gradient conflicts make optimization of temporal difference brittle. Practical workflows for generalist policies therefore avoid online training, instead cloning expert trajectories or distilling collections of single-task policies into one agent. In this work, we show that the use of high-capacity value models trained via cross-entropy and conditioned on learnable task embeddings addresses the problem of task interference in online RL, allowing for robust and scalable multi-task training. We test our approach on 7 multi-task benchmarks with over 280 unique tasks, spanning high degree-of-freedom humanoid control and discrete vision-based RL. We find that, despite its simplicity, the proposed approach leads to state-of-the-art single and multi-task performance, as well as sample-efficient transfer to new tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Debiased Model-based Representations for Sample-efficient Continuous Control

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    DR.Q debiases model-based representations for Q-learning by maximizing mutual information between state-action and next-state representations and applying faded prioritized experience replay, achieving competitive or ...

  2. When Does Non-Uniform Replay Matter in Reinforcement Learning?

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Non-uniform replay helps off-policy RL mainly at low replay volumes, high-entropy sampling matters even at similar recency, and Truncated Geometric replay offers a low-overhead practical solution.

  3. FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    FlashSAC improves training speed and final performance of off-policy RL on high-dimensional robot tasks by reducing update frequency, increasing model scale, and bounding norms to limit critic error accumulation.

  4. FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    FlashSAC scales up Soft Actor-Critic with fewer updates, larger models, higher data throughput, and norm bounds to deliver faster, more stable training than PPO on high-dimensional robot control tasks across dozens of...

  5. What Does Flow Matching Bring To TD Learning?

    cs.LG 2026-03 conditional novelty 6.0 of 10

    Flow matching critics outperform monolithic ones in RL by 2x performance and 5x sample efficiency via test-time error recovery through integration and multi-point velocity supervision that preserves feature plasticity.

  6. When Does Non-Uniform Replay Matter in Reinforcement Learning?

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Non-uniform replay improves RL sample efficiency mainly in low replay-volume regimes, with high-entropy sampling being key even at comparable recency.

  7. When Does Non-Uniform Replay Matter in Reinforcement Learning?

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Non-uniform replay helps most when replay volume is low; high-entropy sampling remains important, and a truncated geometric distribution delivers better sample efficiency with negligible overhead.

  8. RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning

    cs.LG 2026-01 conditional novelty 5.0 of 10

    A discretized categorical actor paired with a pre-LayerNorm residual MLP (RN-D) improves PPO's final performance and sample efficiency on continuous-control benchmarks.

  9. Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A survey that categorizes deep reinforcement learning scaling strategies into data, network, and training budget dimensions and outlines challenges for scaling DRL systems.

Pith tools