Pith. sign in

REVIEW 3 major objections 6 minor 96 references

ReBRAC-v2: The Return of the King

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a modernized behavior-regularized actor-critic, with an exact-likelihood normalizing-flow actor and a frozen shared recipe, achieves the strongest aggregate offline-RL results among the compared methods on OGBench and

desk verdict A strong empirical recipe with a genuinely large OGBench win, but the AntMaze 'strongest' claim rests on a reproduction gap the paper cannot explain, and the abstract overstates the result. read the letter →

arxiv 2608.01205 v1 pith:IJPJNYZ7 submitted 2026-08-02 cs.LG cs.AI

Denis Tarasov , Robert K. Katzschmann This is my paper
classification cs.LGcs.AI
keywords offlinereinforcementlearningbehavior-regularizedactor-criticnormalizing-flowpolicycategoricalvalueestimationtest-timeactionrefinementstagedtrainingOGBenchD4RL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ReBRAC-v2 keeps the ordinary offline-RL actor-critic loop—Bellman value updates, policy improvement, behavior cloning—and modernizes each component: an exact-likelihood conditional normalizing-flow actor trained by critic gradients and mixed likelihood/MSE/MAE cloning, four categorical residual critics, staged training, and multi-sample test-time action refinement. The paper's central claim is that this integration, frozen into one shared configuration and adapted only through two behavior-cloning coefficients, yields the strongest aggregate scores in its comparisons: 74.8 average over ten state-based OGBench categories versus 52.3 for the next best, and strongest suite averages on D4RL AntMaze (90.2) and Adroit (33.6). If true, this matters because it would show that expressive generative policies and specialized value-guidance machinery are not required for competitive offline RL; disciplined, transferable engineering of a minimalist actor-critic can match or beat the more elaborate methods.

What carries the argument

The carrying object is the exact-likelihood conditional normalizing-flow actor: an invertible state-conditioned transformation of a base variable that yields both reparameterized one-pass action samples (used for critic gradients and evaluation) and exact log-likelihoods (used for behavior cloning). Around it, the paper assembles the supporting machinery: four categorical residual critics with HL-Gauss 201-bin value classification, a mean Bellman target, and minimum-over-ensemble selection; three-stage training (BC warm-up, critic warm-up, joint delayed-actor updates); the Kron optimizer with selected regularizers; and evaluation-time sampling of 32 candidate actions refined by two normalize

What would settle it

Apply each baseline's own published evaluation protocol (e.g., GFP/DriftQL's 800K–1M late-checkpoint averaging; FAC's native rule, if recoverable) to the released implementations with the same seeds. If any compared baseline then produces an average at or above ReBRAC-v2's reported suite averages (74.8 OGBench, 90.2 AntMaze, 33.6 Adroit), the aggregate-superiority claim is falsified.

Watch

Extended reading notes

Core claim

The paper's discovery, on its own terms, is that a conventional behavior-regularized actor-critic remains highly competitive when its design choices are systematically modernized. ReBRAC-v2 trains a single exact-likelihood conditional normalizing flow as the RL actor; the same parameters receive critic gradients for policy improvement and a mixed behavior-cloning objective (flow likelihood plus MSE and MAE attraction to dataset actions). Four HL-Gauss categorical residual critics (201 bins, mean Bellman target, minimum over the ensemble for the actor gradient and for evaluation) provide value learning, and three training stages—100,000 behavior-cloning updates, 200,000 critic-only updates wi

Load-bearing premise

The load-bearing premise is that the reruns of FAC, GFP, and DriftQL under the paper's final-checkpoint protocol fairly represent those methods; if those reruns understate the baselines' true capabilities, the reported aggregate margins shrink or disappear.

Editorial extensions

If this is right

  • If the aggregate results hold, a single invertible flow actor can replace the two-policy structures used by recent offline RL methods (a generative behavior model plus a distilled one-step actor), removing iterative generation at deployment without sacrificing score.
  • The frozen-recipe interface means practitioners could apply ReBRAC-v2 to a new environment by adjusting only two coefficients over 16 configurations, rather than re-searching architecture and optimization choices.
  • The ablation evidence singles out the mixed cloning objective (likelihood plus MSE plus MAE), staged training, flow depth (at least eight coupling layers), and multi-sample inference as the components that drive most of the gain; the paper keeps these as the core of the recipe.
  • Under late-checkpoint averaging (800K to 1M updates), ReBRAC-v2's OGBench average is 73.1 rather than 75.0, and the gap to the compared baselines' published averages persists, indicating the outcome does not hinge on evaluating the exact final checkpoint.
  • The Cube Double failure (8.3 versus GFP's 40.1) is a documented counter-example within the paper; the authors could not conclusively explain it and flag it as a direction for critic-design investigation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: because the paper's reproduction changes FAC's OGBench average from 60.3 to 50.2 and DriftQL's AntMaze average from 84.2 to 77.3, part of the headline margin may be an artifact of the final-checkpoint protocol rather than a pure algorithmic difference; a neutral audit using each baseline's native checkpoint rule would separate the two effects.
  • My inference: the two-coefficient selection is still guided by environment returns, so the published numbers are not a fully offline selection benchmark; a dataset-only validation criterion (for example, held-out behavior-cloning fit or in-sample Q) would be a stronger test of the claimed transfer.
  • My inference: Cube Double was one of the six development tasks yet is the method's worst category, suggesting the Bayesian search can overfit its selection objective on hard compositional tasks; targeted perturbation of critic regularization and residual connections on Cube Double could test whether the recipe needs a critic fix rather than a readjustment of cloning coefficients.
  • My inference: the multi-sample inference lever is cheap and complementary to training; one could investigate whether selecting candidates by critic disagreement or variance (instead of minimum Q) further improves executed actions, although the paper's own grid suggests diminishing and task-dependent returns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces ReBRAC-v2, an offline RL actor-critic that replaces the Gaussian policy in ReBRAC with an exact-likelihood conditional normalizing-flow actor, trains it with a mixed likelihood+MSE+MAE cloning objective, uses a four-critic categorical residual critic with auxiliary next-state prediction, applies staged warm-up training, and performs multi-sample test-time Q-guided action refinement. A single structural recipe is selected by roughly 576 Bayesian proposals on six OGBench tasks and then frozen; downstream adaptation changes only two behavior-regularization coefficients on a 16-point grid. The authors report an average of 74.8 on ten state-based OGBench categories (next best 52.3), first place in eight categories, and suite averages of 90.2 on D4RL AntMaze and 33.6 on D4RL Adroit. The paper also provides fixed-recipe ablations, baseline reruns under a final-checkpoint protocol, a late-checkpoint robustness analysis for ReBRAC-v2, and an explicit limitations section.

Significance. If the empirical claims hold, the contribution is substantial: it demonstrates that a behavior-regularized actor-critic without a generative proxy or distilled actor can match or exceed specialized flow-based offline RL methods on these benchmarks while exposing a much simpler two-coefficient adaptation interface. The manuscript is unusually candid and reproducible: it publishes pinned commits for all baseline reruns, complete task-level tables, fixed-recipe ablations that explicitly refuse to retune alternatives, and a sensitivity analysis of the checkpoint-selection protocol for OGBench. The OGBench and Adroit aggregate advantages are large and robust to the main identified threats. However, the headline AntMaze claim is not robust to the published FAC result, and the D4RL numbers are obtained with per-dataset coefficient selection on datasets that also appear in the main table. With appropriate qualification and sensitivity analyses, the core OGBench and Adroit conclusions are likely to stand.

major comments (3)
  1. [Abstract; Source Protocols and Reproduction Differences (Tables 11–13)] The claim that ReBRAC-v2 'obtains the strongest averages in our comparisons on D4RL AntMaze (90.2)' is not robust to the published FAC value. FAC's source-paper AntMaze average is 90.5 (Table 12), which is above ReBRAC-v2's 90.2; the authors' rerun of FAC is 85.3, and the manuscript states that the 5.2-point gap cannot be attributed to any specific cause. Under the natural 'use the published baseline value' reading, the AntMaze ordering flips. Because this is part of the abstract's central empirical claim, the paper should condition the statement explicitly on the final-checkpoint rerun protocol and report a sensitivity analysis that substitutes the published baseline averages for the three D4RL suites. The OGBench and Adroit comparisons are much less sensitive to this substitution and would likely survive it.
  2. [Experimental Setup: Shared Development and Adaptation; Table 7; Limitations and Future Work] The D4RL 'transfer' narrative is weakened by the fact that the two behavior-regularization coefficients are selected per D4RL dataset using simulator returns on the same datasets that appear in Table 1. Similarly, the six OGBench development tasks are themselves included in the headline 50-task average. The paper discloses this ('environment-assisted model selection') and discusses post-selection optimism, but the conclusion's phrase 'the same recipe, without structural changes, obtains the strongest averages' overstates what is frozen: two coefficients are still tuned per downstream dataset with access to the reported benchmark's returns. I recommend that the main table distinguish tasks used in development/coefficient selection from fully held-out tasks/datasets, and that the D4RL claim be stated as 'with per-dataset coefficient selection' rather than as a pure transfer result.
  3. [Baseline Provenance; Why Final-Checkpoint Evaluation?; Tables 11–13] The matched-protocol reproduction changes some baseline results by large amounts (FAC OGBench 60.3→50.2, FAC AntMaze 90.5→85.3, DriftQL AntMaze 84.2→77.3, DriftQL Adroit 14.5→12.5), while GFP changes little. The final-checkpoint rule is defended and the OGBench robustness to late-checkpoint averaging is shown only for ReBRAC-v2, not for the baselines. Since the main-table comparisons are built entirely on the reruns, the paper should include a side-by-side summary that shows the effect of using published baseline values instead of the reruns, and either a late-checkpoint analysis of the D4RL baselines analogous to Table 10 or an explicit statement of how FAC's unspecified checkpoint-selection rule could affect the comparison. This would make the protocol-dependence of the headline aggregates transparent.
minor comments (6)
  1. [Equation (10)] GSP is defined with sinc(x); please state the convention for sinc (normalized or unnormalized) to avoid ambiguity.
  2. [Table 6] The selected value 'Actor BC noise 1.153e-03' is not defined in the main text or the appendix. Please specify what operation this noise applies to (e.g., additive Gaussian noise on dataset actions in the cloning loss).
  3. [Method and Algorithm Details] The statement that the flow base is a 'uniform base' is potentially misleading: the implementation samples u ~ U([-1,1]) and sets z = atanh(u), so the latent z is not uniform. Please clarify that the base distribution is the transformed uniform density, not a uniform distribution on z.
  4. [Explored Variants That Did Not Improve Results] The abbreviation 'NS' in 'the strong NS baseline studied in Decoupled Q-Chunking' is not defined at first use. Please expand it or give a one-line description.
  5. [Figure 2 caption; RLiable Construction] The RLiable intervals are omitted for ReBRAC and FQL because only task-level means are available. This should be stated in the caption of Figure 2, not only in the appendix, so that readers do not interpret the shaded regions as comparable across all curves.
  6. [Algorithm 1] The line 'Polyak-update target critics on schedule' is vague. Specify the update rule and the exact schedule (every step, every d steps, or per critic step) so the pseudocode is self-contained.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the method is benchmark-fitted, but the headline empirical claims rest on direct, disclosed runs against external baselines; self-citations are motivational, not load-bearing.

full rationale

The paper contains no formal derivation whose conclusion is equivalent to an assumption. The central claim is empirical: ReBRAC-v2's aggregate OGBench, AntMaze, and Adroit scores. The selection procedure is disclosed: roughly 600 Bayesian proposals on six OGBench tasks, followed by freezing all structural choices and tuning only (alpha_NF, alpha_aux) over a 16-point grid, with OGBench pairs transferred from each category's default task and per-dataset D4RL pairs selected. The manuscript explicitly labels this 'environment-assisted model selection' and warns that 'one-seed development scores are noisy and post-selection' and that transfer 'does not eliminate post-selection optimism.' This is honest model selection, not a hidden fit masquerading as prediction. The evaluation also contains genuine held-out structure: four OGBench categories absent from search, four held-out tasks within each developed category, and all D4RL datasets. The headline comparisons are therefore not forced by construction. Self-citations (ReBRAC, CORL, actor regularization, classification-critic adaptation, SERNF) motivate individual design choices, but the recipe's contribution is evaluated by direct reruns of external baselines (FAC, GFP, DriftQL) under a stated protocol, and the flow actor is credited to an external work (NF-RLBC). No uniqueness theorem or prior result is invoked to forbid alternatives. The FAC checkpoint-reproduction gap can flip the AntMaze ordering and is a legitimate robustness/correctness concern, but it is not a circularity: the paper reports the discrepancy without assigning a cause. Consistent with the rubrics, this is a 0-2 case: self-citations are present but not load-bearing, and the empirical claims have independent content.

Assumptions & free parameters 4 free parameters · 8 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The load-bearing commitments are the benchmark comparability assumption, the transferability of the frozen recipe, and the empirical choice of the mixed cloning objective and inference settings. The two behavior-regularization coefficients and the shared structural hyperparameters are fitted values in the sense that they were selected using environment returns from the evaluated suites.

free parameters (4)
  • α_aux (auxiliary cloning coefficient) = 0.03 to 1.0 on OGBench; 0.005 to 0.1 on D4RL (Table 7)
    Selected per category or dataset over a 4x4 grid using environment returns on the benchmark suites being reported.
  • α_NF (flow likelihood coefficient) = 1e-4 to 1e-2 on OGBench; 1e-3 to 3e-2 on D4RL (Table 7)
    Same grid selection; these two coefficients are the only adaptation interface after the shared recipe is frozen.
  • Shared structural hyperparameters (14 flow layers, 4 critics, 201 value bins, 5-layer residual critic, GSP activation, K = Listed in Table 6
    Chosen by 576 Bayesian proposals on six OGBench development tasks, maximizing mean normalized score; frozen for all reported results.
  • Inference settings (K samples, J refinement steps, η step size) = K=32, J=2, η=0.01
    Fixed in preliminary experiments before Bayesian development; the retrospective grid in Figure 3 shows (32,1) achieves 64.4 vs 63.9 for the selected setting, so this choice is not uniquely optimal.
assumptions (8)
  • standard math Change-of-variables formula for invertible conditional normalizing flows yields exact action likelihoods (Eq. 3).
    Used to define the flow actor's likelihood for behavior regularization; standard result from Rezende and Mohamed, Dinh et al.
  • domain assumption Q-min policy gradient with per-sample normalization λ improves the policy in expectation (Eq. 5).
    Standard actor-critic assumption inherited from TD3+BC and ReBRAC; no proof is given that this specific normalized objective is an unbiased policy improvement step.
  • domain assumption Categorical HL-Gauss value prediction with 201 bins produces usable Q-values for policy improvement (Eq. 4, Farebrother et al. 2024).
    Adopted from prior work and treated as given; no analysis of bias introduced by binning for these benchmarks is provided.
  • domain assumption Benchmark normalized scores (OGBench, D4RL) are comparable across methods and are the right measure of aggregate performance.
    The central claim is an aggregate score comparison; comparability rests on the benchmarks' normalization and the reproduction protocol.
  • ad hoc to paper The frozen structural recipe selected on six development tasks transfers to other OGBench categories and to D4RL.
    This is the paper's core empirical premise; it is tested on held-out categories but not guaranteed. The paper acknowledges one-seed selection noise.
  • ad hoc to paper The mixed likelihood plus MSE plus MAE cloning objective is a reasonable behavior regularizer (Eq. 6).
    The paper states the combination is empirical rather than derived as an optimal objective.
  • domain assumption Test-time Q-refinement with frozen critic gradients improves executed actions (Eqs. 8 and 9).
    Assumes the critic's Q-min gradient points toward higher-return actions; the ablation shows task-dependent, sometimes negative, effects.
  • domain assumption Final-checkpoint evaluation is the correct estimand for comparing methods.
    The authors explicitly justify this against late-checkpoint averaging; if the field-standard protocol differs, baseline comparisons could change.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReBRAC-v2: The Return of the King." pith.science (2026). https://pith.science/paper/IJPJNYZ7

@misc{pith2026260801205,
  author       = {Pith},
  title        = {Pith review of: ReBRAC-v2: The Return of the King},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJPJNYZ7}},
  note         = {Machine review of arXiv:2608.01205}
}
read the original abstract

Recent offline reinforcement learning methods increasingly rely on expressive generative policies and specialized value-guidance mechanisms. We ask whether comparable progress can instead come from systematically modernizing a conventional behavior-regularized actor-critic while preserving its algorithmic simplicity. We introduce ReBRAC-v2, which directly trains an exact-likelihood normalizing flow as the RL actor, combines likelihood, MSE, and MAE behavior regularization, and integrates a classification-based residual critic, staged optimization, and multi-sample test-time action selection. Rather than tuning this recipe separately for every task, we develop a single shared configuration via roughly 600 Bayesian proposals on six challenging OGBench tasks, freeze all structural and optimization choices, and adapt only two behavior-regularization coefficients over a 16-point grid. Across ten common state-based OGBench categories, ReBRAC-v2 averages 74.8 compared to 52.3 for the next-best aggregate result and ranks first in eight categories. The same recipe, without structural changes, obtains the strongest averages in our comparisons on D4RL AntMaze (90.2) and Adroit (33.6). Fixed-recipe ablations show the largest sensitivity to the selected mixed cloning objective, staged training, sufficient flow capacity, and multi-sample inference, while showing that several smaller choices depend on the values of other hyperparameters. These results show that disciplined, transferable engineering can achieve state-of-the-art aggregate performance without abandoning a minimalist offline RL foundation.

Figures

Figures reproduced from arXiv: 2608.01205 by the authors.

Figure 1
Figure 1. ReBRAC-v2 pipeline. Mixed behavior cloning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Aggregate OGBench and D4RL performance. ReBRAC-v2 has the strongest performance profile, median, interquartile [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Inference-time compute and flow-capacity ablations. (a) Across the six OGBench development tasks, sampling provides [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Task-level sensitivity to inference-time compute. Each heatmap reports normalized score as a function of the number of [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

96 extracted references · 17 canonical work pages

  1. [1]

    Scaling Learning Algorithms Towards

    Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards

  2. [2]

    and Osindero, Simon and Teh, Yee Whye , journal =

    Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume =

  3. [3]

    2016 , publisher=

    Deep learning , author=. 2016 , publisher=

    2016

  4. [4]

    arXiv preprint arXiv:2210.14215 , year=

    In-context reinforcement learning with algorithm distillation , author=. arXiv preprint arXiv:2210.14215 , year=

    arXiv

  5. [5]

    arXiv preprint arXiv:1312.5602 , year=

    Playing atari with deep reinforcement learning , author=. arXiv preprint arXiv:1312.5602 , year=

    arXiv

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    Conservative q-learning for offline reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  7. [7]

    arXiv preprint arXiv:2110.06169 , year=

    Offline reinforcement learning with implicit q-learning , author=. arXiv preprint arXiv:2110.06169 , year=

    arXiv

  8. [8]

    Advances in Neural Information Processing Systems , year=

    Attention is all you need , author=. Advances in Neural Information Processing Systems , year=

Show all 96 references
  1. [9]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  2. [10]

    Advances in Neural Information Processing Systems , volume=

    Supervised pretraining can learn in-context reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  3. [11]

    arXiv preprint arXiv:2411.01958 , year=

    N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs , author=. arXiv preprint arXiv:2411.01958 , year=

    arXiv

  4. [12]

    arXiv preprint arXiv:2312.13327 , year=

    In-context reinforcement learning for variable action spaces , author=. arXiv preprint arXiv:2312.13327 , year=

    arXiv

  5. [13]

    arXiv preprint arXiv:2410.07071 , year=

    Retrieval-augmented decision transformer: External memory for in-context rl , author=. arXiv preprint arXiv:2410.07071 , year=

    arXiv

  6. [14]

    arXiv preprint arXiv:2403.06826 , year=

    In-context Exploration-Exploitation for Reinforcement Learning , author=. arXiv preprint arXiv:2403.06826 , year=

    arXiv

  7. [15]

    DISTILLING REINFORCEMENT LEARNING ALGO-RITHMS FOR IN-CONTEXT MODEL-BASED PLANNING , author=

  8. [16]

    arXiv preprint arXiv:2310.09971 , year=

    Amago: Scalable in-context reinforcement learning for adaptive agents , author=. arXiv preprint arXiv:2310.09971 , year=

    arXiv

  9. [17]

    arXiv preprint arXiv:2410.02751 , year=

    ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI , author=. arXiv preprint arXiv:2410.02751 , year=

    arXiv

  10. [18]

    arXiv preprint arXiv:2005.01643 , year=

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems , author=. arXiv preprint arXiv:2005.01643 , year=

    2005 arXiv

  11. [19]

    International Conference on Machine Learning , pages=

    Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  12. [20]

    arXiv preprint arXiv:2405.17098 , year=

    Q-value regularized transformer for offline reinforcement learning , author=. arXiv preprint arXiv:2405.17098 , year=

    arXiv

  13. [21]

    arXiv preprint arXiv:2405.08740 , year=

    Reinformer: Max-return sequence modeling for offline rl , author=. arXiv preprint arXiv:2405.08740 , year=

    arXiv

  14. [22]

    arXiv preprint arXiv:2405.20692 , year=

    In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought , author=. arXiv preprint arXiv:2405.20692 , year=

    arXiv

  15. [23]

    arXiv preprint arXiv:2301.08028 , year=

    A survey of meta-reinforcement learning , author=. arXiv preprint arXiv:2301.08028 , year=

    arXiv

  16. [24]

    International conference on machine learning , pages=

    Off-policy deep reinforcement learning without exploration , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  17. [25]

    Advances in neural information processing systems , volume=

    A minimalist approach to offline reinforcement learning , author=. Advances in neural information processing systems , volume=

  18. [26]

    Advances in neural information processing systems , volume=

    Uncertainty-based offline reinforcement learning with diversified q-ensemble , author=. Advances in neural information processing systems , volume=

  19. [27]

    Advances in neural information processing systems , volume=

    Decision transformer: Reinforcement learning via sequence modeling , author=. Advances in neural information processing systems , volume=

  20. [28]

    Advances in Neural Information Processing Systems , volume=

    Revisiting the minimalist approach to offline reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  21. [29]

    Advances in Neural Information Processing Systems , year=

    Deep Reinforcement Learning at the Edge of the Statistical Precipice , author=. Advances in Neural Information Processing Systems , year=

  22. [30]

    arXiv preprint arXiv:2206.11871 , year=

    Offline rl for natural language generation with implicit language q learning , author=. arXiv preprint arXiv:2206.11871 , year=

    arXiv

  23. [31]

    arXiv preprint arXiv:2111.04714 , year=

    A Dataset Perspective on Offline Reinforcement Learning , author=. arXiv preprint arXiv:2111.04714 , year=

    arXiv

  24. [32]

    Advances in Neural Information Processing Systems , volume=

    CORL: Research-oriented deep offline reinforcement learning library , author=. Advances in Neural Information Processing Systems , volume=

  25. [33]

    Machine learning , volume=

    Q-learning , author=. Machine learning , volume=. 1992 , publisher=

    1992

  26. [34]

    arXiv preprint arXiv:1412.6980 , year=

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    arXiv

  27. [35]

    Advances in Neural Information Processing Systems , volume=

    Katakomba: tools and benchmarks for data-driven NetHack , author=. Advances in Neural Information Processing Systems , volume=

  28. [36]

    Advances in Neural Information Processing Systems , volume=

    The nethack learning environment , author=. Advances in Neural Information Processing Systems , volume=

  29. [37]

    arXiv preprint arXiv:2312.12044 , year=

    XLand-minigrid: Scalable meta-reinforcement learning environments in JAX , author=. arXiv preprint arXiv:2312.12044 , year=

    arXiv

  30. [38]

    arXiv preprint arXiv:2406.08973 , year=

    XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning , author=. arXiv preprint arXiv:2406.08973 , year=

    arXiv

  31. [39]

    Conference on robot learning , pages=

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning , author=. Conference on robot learning , pages=. 2020 , organization=

    2020

  32. [40]

    arXiv preprint arXiv:2312.03801 , year=

    Generalization to new sequential decision making tasks with in-context learning , author=. arXiv preprint arXiv:2312.03801 , year=

    arXiv

  33. [41]

    2025 , eprint=

    Vintix: Action Model via In-Context Reinforcement Learning , author=. 2025 , eprint=

    2025

  34. [42]

    arXiv preprint arXiv:2312.12275 , year=

    Emergence of In-Context Reinforcement Learning from Noise Distillation , author=. arXiv preprint arXiv:2312.12275 , year=

    arXiv

  35. [43]

    Conference on Robot Learning , pages=

    Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble , author=. Conference on Robot Learning , pages=. 2022 , organization=

    2022

  36. [44]

    arXiv preprint arXiv:2006.09359 , year=

    Awac: Accelerating online reinforcement learning with offline datasets , author=. arXiv preprint arXiv:2006.09359 , year=

    2006 arXiv

  37. [45]

    arXiv preprint arXiv:2204.05618 , year=

    When should we prefer offline reinforcement learning over behavioral cloning? , author=. arXiv preprint arXiv:2204.05618 , year=

    arXiv

  38. [46]

    arXiv preprint arXiv:2502.07978 , year=

    A Survey of In-Context Reinforcement Learning , author=. arXiv preprint arXiv:2502.07978 , year=

    arXiv

  39. [47]

    2012 IEEE/RSJ international conference on intelligent robots and systems , pages=

    Mujoco: A physics engine for model-based control , author=. 2012 IEEE/RSJ international conference on intelligent robots and systems , pages=. 2012 , organization=

    2012

  40. [48]

    International conference on machine learning , pages=

    Addressing function approximation error in actor-critic methods , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  41. [49]

    International conference on machine learning , pages=

    Efficient off-policy meta-reinforcement learning via probabilistic context variables , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  42. [50]

    International conference on machine learning , pages=

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  43. [51]

    Araújo , title =

    Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo , title =. Journal of Machine Learning Research , year =

  44. [52]

    arXiv preprint arXiv:1607.06450 , year=

    Layer normalization , author=. arXiv preprint arXiv:1607.06450 , year=

    arXiv

  45. [53]

    arXiv preprint arXiv:2401.12973 , year=

    In-context language learning: Architectures and algorithms , author=. arXiv preprint arXiv:2401.12973 , year=

    arXiv

  46. [54]

    arXiv preprint arXiv:2402.08609 , year=

    Mixtures of experts unlock parameter scaling for deep rl , author=. arXiv preprint arXiv:2402.08609 , year=

    arXiv

  47. [55]

    arXiv preprint arXiv:1701.06538 , year=

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer , author=. arXiv preprint arXiv:1701.06538 , year=

    arXiv

  48. [56]

    arXiv preprint arXiv:2403.03950 , year=

    Stop regressing: Training value functions via classification for scalable deep rl , author=. arXiv preprint arXiv:2403.03950 , year=

    arXiv

  49. [57]

    arXiv preprint arXiv:2406.06309 , year=

    Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning? , author=. arXiv preprint arXiv:2406.06309 , year=

    arXiv

  50. [58]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    2001 arXiv

  51. [59]

    arXiv preprint arXiv:2508.14881 , year=

    Compute-optimal scaling for value-based deep rl , author=. arXiv preprint arXiv:2508.14881 , year=

    arXiv

  52. [60]

    arXiv preprint arXiv:1911.11361 , year=

    Behavior regularized offline reinforcement learning , author=. arXiv preprint arXiv:1911.11361 , year=

    1911 arXiv

  53. [61]

    arXiv preprint arXiv:2409.07606 , year=

    The role of deep learning regularizations on actors in offline rl , author=. arXiv preprint arXiv:2409.07606 , year=

    arXiv

  54. [62]

    arXiv preprint arXiv:2602.05688 , year=

    Mining Generalizable Activation Functions , author=. arXiv preprint arXiv:2602.05688 , year=

  55. [63]

    Advances in Neural Information Processing Systems , volume=

    Normalizing Flows are Capable Models for Continuous Control , author=. Advances in Neural Information Processing Systems , volume=

  56. [64]

    training vla models with normalizing flows , author=

    Nina: Normalizing flows in action. training vla models with normalizing flows , author=. arXiv preprint arXiv:2508.16845 , year=

  57. [65]

    Advances in Neural Information Processing Systems , volume=

    Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning , author=. Advances in Neural Information Processing Systems , volume=

  58. [66]

    arXiv preprint arXiv:2602.09580 , year=

    SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows , author=. arXiv preprint arXiv:2602.09580 , year=

    arXiv

  59. [67]

    Advances in Neural Information Processing Systems , volume=

    Is value learning really the main bottleneck in offline RL? , author=. Advances in Neural Information Processing Systems , volume=

  60. [68]

    International conference on machine learning , pages=

    Variational inference with normalizing flows , author=. International conference on machine learning , pages=. 2015 , organization=

    2015

  61. [69]

    arXiv preprint arXiv:1605.08803 , year=

    Density estimation using real nvp , author=. arXiv preprint arXiv:1605.08803 , year=

    arXiv

  62. [70]

    arXiv preprint arXiv:2211.11096 , year=

    Let offline rl flow: Training conservative agents in the latent space of normalizing flows , author=. arXiv preprint arXiv:2211.11096 , year=

    arXiv

  63. [71]

    International conference on machine learning , pages=

    A distributional perspective on reinforcement learning , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  64. [72]

    arXiv preprint arXiv:2509.21073 , year=

    Normalizing flows are capable visuomotor policy learning models , author=. arXiv preprint arXiv:2509.21073 , year=

  65. [73]

    arXiv preprint arXiv:2110.02034 , year=

    Dropout q-functions for doubly efficient reinforcement learning , author=. arXiv preprint arXiv:2110.02034 , year=

    arXiv

  66. [74]

    International Conference on Machine Learning , pages=

    Efficient online reinforcement learning with offline data , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  67. [75]

    Advances in Neural Information Processing Systems , volume=

    Efficient diffusion policies for offline reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  68. [76]

    arXiv preprint arXiv:2208.06193 , year=

    Diffusion policies as an expressive policy class for offline reinforcement learning , author=. arXiv preprint arXiv:2208.06193 , year=

    arXiv

  69. [77]

    Advances in Neural Information Processing Systems , volume=

    Diffusion policies creating a trust region for offline reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  70. [78]

    arXiv preprint arXiv:2502.04778 , year=

    Behavior-regularized diffusion policy optimization for offline reinforcement learning , author=. arXiv preprint arXiv:2502.04778 , year=

    arXiv

  71. [79]

    arXiv preprint arXiv:2505.14139 , year=

    FlowQ: Energy-guided flow policies for offline reinforcement learning , author=. arXiv preprint arXiv:2505.14139 , year=

    arXiv

  72. [80]

    International Conference on Machine Learning (ICML) , year=

    Flow Q-Learning , author=. International Conference on Machine Learning (ICML) , year=

  73. [81]

    International conference on machine learning , pages=

    Improving regression performance with distributional losses , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  74. [82]

    arXiv preprint arXiv:2506.15544 , year=

    Stable gradients for stable learning at scale in deep reinforcement learning , author=. arXiv preprint arXiv:2506.15544 , year=

  75. [83]

    arXiv preprint arXiv:2211.15144 , year=

    Offline q-learning on diverse multi-task data both scales and generalizes , author=. arXiv preprint arXiv:2211.15144 , year=

    arXiv

  76. [84]

    arXiv preprint arXiv:2302.01470 , year=

    Learning to optimize for reinforcement learning , author=. arXiv preprint arXiv:2302.01470 , year=

    arXiv

  77. [85]

    International Conference on Learning Representations , volume=

    Butterfly effects of sgd noise: Error amplification in behavior cloning and autoregression , author=. International Conference on Learning Representations , volume=

  78. [86]

    arXiv preprint arXiv:2602.18015 , year=

    Flow actor-critic for offline reinforcement learning , author=. arXiv preprint arXiv:2602.18015 , year=

  79. [87]

    arXiv preprint arXiv:2512.03973 , year=

    Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning , author=. arXiv preprint arXiv:2512.03973 , year=

  80. [88]

    arXiv preprint arXiv:2606.00350 , year=

    Drift Q-Learning , author=. arXiv preprint arXiv:2606.00350 , year=

    arXiv

  81. [89]

    Advances in Neural Information Processing Systems , volume=

    Reinforcement learning with action chunking , author=. Advances in Neural Information Processing Systems , volume=

  82. [90]

    arXiv preprint arXiv:2512.10926 , year=

    Decoupled Q-Chunking , author=. arXiv preprint arXiv:2512.10926 , year=

  83. [91]

    Advances in neural information processing systems , volume=

    Neural spline flows , author=. Advances in neural information processing systems , volume=

  84. [92]

    International Conference on Learning Representations , volume=

    Ogbench: Benchmarking offline goal-conditioned rl , author=. International Conference on Learning Representations , volume=

  85. [93]

    arXiv preprint arXiv:2004.07219 , year=

    D4rl: Datasets for deep data-driven reinforcement learning , author=. arXiv preprint arXiv:2004.07219 , year=

    2004 arXiv

  86. [94]

    Advances in neural information processing systems , volume=

    Glow: Generative flow with invertible 1x1 convolutions , author=. Advances in neural information processing systems , volume=

  87. [95]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    arXiv

  88. [96]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

    Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=. 2024 , publisher=

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.