Pith. sign in

REVIEW 4 cited by

Reinformer: Max-Return Sequence Modeling for Offline RL

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.08740 v3 pith:OG723QMI submitted 2024-05-14 cs.LG

Zifeng Zhuang , Dengyun Peng , Jinxin Liu , Ziqi Zhang , Donglin Wang This is my paper · ORCID
classification cs.LG
keywords sequencereinformermodelmodelingobjectivereturnreturnstrajectory
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

As a data-driven paradigm, offline reinforcement learning (RL) has been formulated as sequence modeling that conditions on the hindsight information including returns, goal or future trajectory. Although promising, this supervised paradigm overlooks the core objective of RL that maximizes the return. This overlook directly leads to the lack of trajectory stitching capability that affects the sequence model learning from sub-optimal data. In this work, we introduce the concept of max-return sequence modeling which integrates the goal of maximizing returns into existing sequence models. We propose Reinforced Transformer (Reinformer), indicating the sequence model is reinforced by the RL objective. Reinformer additionally incorporates the objective of maximizing returns in the training phase, aiming to predict the maximum future return within the distribution. During inference, this in-distribution maximum return will guide the selection of optimal actions. Empirically, Reinformer is competitive with classical RL methods on the D4RL benchmark and outperforms state-of-the-art sequence model particularly in trajectory stitching ability. Code is public at https://github.com/Dragon-Zhuang/Reinformer.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only

    cs.LG 2025-05 conditional novelty 6.0 of 10

    PORL fine-tunes a pre-trained policy online by first collecting epsilon-greedy interaction data and learning a fresh Q-function from scratch, removing the need for pre-trained critics or offline data.

  2. ReBRAC-v2: The Return of the King

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A fixed-recipe offline RL method combining normalizing-flow actors, categorical critics, staged training, and test-time refinement beats recent flow-based baselines by 22.5 points averaged over ten OGBench categories.

  3. Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models

    cs.RO 2025-09 conditional novelty 5.0 of 10

    ARFM adaptively adjusts a scaling factor in the flow-matching loss so that offline RL advantage signals are preserved while gradient variance is controlled, improving VLA robot policy fine-tuning.

  4. Closing the Gap between TD Learning and Supervised Learning with $Q$-Conditioned Maximization

    cs.LG 2025-06 conditional novelty 5.0 of 10

    GCReinSL adds Q-conditioned maximization to supervised offline RL, using normalizing flows to estimate goal-reaching probabilities and expectile regression to condition actions on the best in-distribution value, impro...

Pith tools