Pith. sign in

REVIEW 5 cited by

Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.05038 v3 pith:TRLBGPV3 submitted 2021-10-11 cs.LG cs.AIcs.RO

Tianwei Ni , Benjamin Eysenbach , Ruslan Salakhutdinov This is my paper
classification cs.LGcs.AIcs.RO
keywords model-freepomdpsrecurrentimplementationmethodsbaselineenvironmentsfind
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Many problems in RL, such as meta-RL, robust RL, generalization in RL, and temporal credit assignment, can be cast as POMDPs. In theory, simply augmenting model-free RL with memory-based architectures, such as recurrent neural networks, provides a general approach to solving all types of POMDPs. However, prior work has found that such recurrent model-free RL methods tend to perform worse than more specialized algorithms that are designed for specific types of POMDPs. This paper revisits this claim. We find that careful architecture and hyperparameter decisions can often yield a recurrent model-free implementation that performs on par with (and occasionally substantially better than) more sophisticated recent techniques. We compare to 21 environments from 6 prior specialized methods and find that our implementation achieves greater sample efficiency and asymptotic performance than these methods on 18/21 environments. We also release a simple and efficient implementation of recurrent model-free RL for future work to use as a baseline for POMDPs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Does Reward Teach State? A Hidden-Automaton Instrument and the Group-Language Boundary

    cs.LG 2026-07 conditional novelty 7.0 of 10

    High reward in sparse RL does not imply latent-state recovery; a hidden-DFA instrument separates perception from planning gaps and flags group-language structure as a pre-training warning.

  2. Recurrent Structural Policy Gradient for Partially Observable Mean Field Games

    cs.AI 2026-02 conditional novelty 6.0 of 10

    RSPG extends structural policy gradients to shared-observation partially observable mean-field games, learning recurrent policies with order-of-magnitude faster convergence than model-free RL.

  3. Active Electrosensing and Communication in MARL-trained Weakly Electric Fish Collectives

    cs.NE 2025-11 conditional novelty 6.0 of 10

    Multi-agent reinforcement learning agents with bio-inspired electrosensing reproduce heavy-tailed electric-organ-discharge statistics and emergent social behaviors such as freeloading and dominance-dependent foraging.

  4. GBC: Generalized Behavior-Cloning Framework for Whole-Body Humanoid Imitation

    cs.RO 2025-08 conditional novelty 5.0 of 10

    GBC unifies MoCap retargeting and imitation learning into one framework that trains whole-body humanoid policies across multiple robot morphologies in simulation.

  5. Decoupled Hierarchical Reinforcement Learning with State Abstraction for Discrete Grids

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A decoupled hierarchical RL framework with a rule-based low-level policy and DeepMDP state abstraction outperforms PPO on two custom discrete grid environments, but with a single baseline and sparse experimental detail.

Pith tools