Pith. sign in

REVIEW 3 major objections 5 minor 51 references

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Auto-regressive soft advantages let value-based RL learn better from suboptimal demonstrations.

desk verdict Clean theory and a plausible algorithm, but the paper's own appendix muddies the headline 1.62x claim by switching BC losses on six datasets without saying whether the baseline was switched too. read the letter →

arxiv 2502.00288 v2 pith:LVVGDS22 submitted 2025-02-01 cs.LG cs.RO

Jijia Liu , Feng Gao , Qingmin Liao , Chao Yu , Yu Wang This is my paper · ORCID
classification cs.LGcs.RO
keywords auto-regressivesoftQ-learningcontinuouscontrolvalue-basedreinforcementlearningsuboptimaldemonstrationsdimensionaladvantagecoarse-to-finediscretizationD4RLRLBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that value-based reinforcement learning for continuous control fails on suboptimal data because it scores each action dimension independently, ignoring correlations, so that frequent suboptimal behaviors dominate the Q estimate. It proposes Auto-Regressive Soft Q-learning (ARSQ), which discretizes the action space coarse-to-fine and predicts a normalized dimensional soft advantage for each action dimension conditioned on the previously chosen ones. The key identity is that when each dimensional advantage is normalized to a probability distribution, the ordinary soft advantage equals the sum of dimensional advantages, so the Q-function remains a valid soft Q while the policy is generated one dimension at a time. On D4RL, ARSQ reports an average 1.62x performance improvement over the state-of-the-art value-based baseline; on RLBench it outperforms actor-critic and imitation baselines. If right, value-based RL can extract correct behavior from datasets where optimal and suboptimal demonstrations are mixed.

What carries the argument

The central object is the dimensional soft advantage $A_d(s,a_{-d},a_d)$: the per-dimension advantage of choosing bin $a_d$ after earlier dimensions are fixed, together with the normalization constraint that turns $\exp(A_d/\alpha)$ into a conditional distribution. The log-sum-exp subtraction in Eq. (16) enforces that constraint in the network, so the sum of dimensional advantages equals the full soft advantage (Theorem 4.3). Around this core, ARSQ wraps a coarse-to-fine discretization of the continuous action space and a margin-based behavior-cloning loss (Eq. 15), with a log-softmax variant (Eq. 22) used on some datasets, to keep expert actions preferred during online fine-tuning.

What would settle it

Run the D4RL medium and medium-expert comparisons with CQN given the same behavior-cloning loss variant (Eq. 22) that ARSQ uses; if CQN then closes the gap, the reported gain is not caused by the auto-regressive advantage.

Watch

Extended reading notes

Core claim

The central claim is that the soft Q-function of a multi-dimensional continuous action can be written auto-regressively: with dimensional soft advantages $A_d(s,a_{-d},a_d)$ satisfying $\sum_{a_d} \exp(A_d/\alpha)=1$, the identity $\sum_d A_d(s,a_{-d},a_d)=A(s,a)$ holds, so the sum of the dimensional advantages is the full soft advantage and the product of the conditional policies equals the global soft policy. ARSQ realizes this with a shared network with separate heads, hard-normalizing each head by log-sum-exp subtraction, and by conditioning both on previously sampled action dimensions and on coarser discretization levels. The authors argue that this removes the bias toward frequent suboptimal modes that independent per-dimension Q estimation inherits, and support it with a one-step toy example, a fully offline evaluation, and online-with-demonstration benchmarks.

Load-bearing premise

The central comparison assumes ARSQ and the baseline faced the identical behavior-cloning loss; the appendix reveals a different loss variant for some datasets, so part of the gain could reflect that change rather than the auto-regressive advantage.

Editorial extensions

If this is right

  • If the identity holds, any soft Q-function for continuous control can be represented by per-dimension conditional advantages, so value-based methods can handle correlations between action dimensions without enumerating all joint action bins.
  • Learning from datasets with many suboptimal demonstrations should improve automatically, because the policy samples from normalized conditional distributions instead of a blurred per-dimension Q average.
  • The coarse-to-fine hierarchy keeps the number of output heads small, so fine control precision does not require an exponential growth in discretized actions.
  • The method also works fully offline, matching or exceeding dedicated offline RL and offline imitation baselines on the D4RL aggregate score.
  • When demonstration quality degrades, the paper reports the advantage over the value-based baseline grows, reaching about 2x on the bottom 30% of trajectories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the paper leaves open is that the normalized-conditional-advantage construction is a recipe, not just an algorithm: any value network whose heads are normalized this way yields a factorized stochastic policy, so the identity could be grafted onto other offline RL methods.
  • An untested prediction of the paper's logic is that the gain over per-dimension baselines grows with the strength of cross-action correlation; on near-independent action dimensions the two approaches should converge.
  • Grouping action dimensions into blocks before the auto-regressive pass is a natural latency fix; the identity holds at group level (normalize over each block's bins) and would directly address the higher inference time the authors report.
  • The paper's error analysis suggests a cheap deployment diagnostic: compare the auto-regressive Q value with a plain value head on the same state-action pairs; large deviations would indicate the factorization is not faithfully representing the soft Q-function.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Auto-Regressive Soft Q-Network (ARSQ), a value-based RL method for continuous control that discretizes actions in a coarse-to-fine hierarchy and predicts normalized 'dimensional soft advantages' auto-regressively over action dimensions. A theoretical result (Theorem 4.3) shows that, under a per-dimension normalization condition, the soft advantage A(s,a) equals the sum of dimensional advantages. The method combines this representation with a behavior-cloning objective and is evaluated on D4RL (with suboptimal offline data) and RLBench (with expert demonstrations plus online data), reporting an average 1.62x improvement over the CQN baseline on D4RL.

Significance. If the empirical claims are substantiated, ARSQ offers a useful value-based alternative for continuous control with suboptimal data, and the factored advantage representation is a valid soft-Q parameterization. The theoretical derivation is correct but is essentially a mathematical identity that follows from the normalization the authors enforce in Eq. (16); the paper's contribution therefore rests primarily on the experimental comparison. The paper is clearly written, provides code-level implementation details, and includes ablations that isolate the coarse-to-fine and auto-regressive components. The main risk is that the headline comparison with CQN currently does not cleanly isolate the auto-regressive mechanism because of an unstated change in the behavior-cloning loss for part of the D4RL evaluation.

major comments (3)
  1. [Sec. 5.1 and Appendix B.2] Section 5.1 states that 'both ARSQ and CQN incorporate the BC objective (Eq. (15)) towards offline dataset.' Appendix B.2 introduces a different objective, Eq. (22), and says 'we adopt this variant objective when working with such datasets,' referring to the medium and medium-expert series of D4RL datasets. These are six of the nine datasets used in the main D4RL comparison. The paper never states whether the CQN baseline was also switched to Eq. (22) on those datasets. Because the BC loss directly controls how strongly the learned policy is pulled toward dataset actions, and because the appendix reports that Eq. (22) 'achieves better performance' in these regimes, the reported average 1.62x improvement over CQN may be partly attributable to the BC-loss change rather than to the auto-regressive advantage decomposition. Please clarify, for each D4RL dataset, which objective each method used, and if CQN was not updated, rerun the baseline with the same variant and re-report the results.
  2. [Abstract and Fig. 4; Appendix D] The abstract's headline claim of an 'average 1.62x performance improvement' over the SOTA value-based baseline is not backed by a per-dataset numeric table. Figure 4 shows learning curves without numerical converged values, and the additional curves in Appendix D (Fig. 15) also lack a table. Please provide a table reporting the mean and standard deviation over seeds for each of the nine D4RL datasets for ARSQ and CQN, together with the per-dataset ratio, so the 1.62x figure can be verified and the contribution of each dataset assessed.
  3. [Sec. 5.1 and Figs. 4-8] The main empirical claims rest on comparisons that are reported only as learning curves averaged over three random seeds, with no error bars or shaded regions in Figs. 4-8. Given that the central message is a quantitative advantage over CQN, the absence of variance information makes it impossible to assess whether the reported gaps are statistically meaningful. Please add error bars or confidence bands, and state the number of evaluation episodes used for each point.
minor comments (5)
  1. [Appendix A, Eq. (19)] The proof of Theorem 4.3 uses both Z(s,a_-d) and Z_d(s,a_-d) with slightly inconsistent notation; please unify the notation to avoid confusion.
  2. [Algorithm 2] Algorithm 2 takes the minimum over two advantage networks, A_d(a_d) = min_i A^{θ_i}_d, and then renormalizes. Because each individual network satisfies the normalization in Eq. (16), the min of two normalized exponentials does not in general satisfy it, and the renormalization is a heuristic not covered by Theorem 4.3. Please add a sentence clarifying that the double-Q min is an algorithmic choice outside the theoretical identity.
  3. [Appendix B.2] There is a grammatical error in the first sentence: 'we incorporate an behavior cloning objective' should be 'we incorporate a behavior cloning objective.'
  4. [Sec. 5.4 and Fig. 7] The 'Plain' ablation removes both the coarse-to-fine structure and dimensional conditioning, but it is not clear whether this variant corresponds exactly to an independent per-dimension Q-network with a single discretization level; please clarify the architecture of this baseline.
  5. [Sec. 5.3, Table 1] Table 1 states that most baseline numbers are sourced from their respective papers while only DWBC is re-evaluated (marked with '*'); please specify the evaluation protocol (number of trajectories, number of seeds) used for the cited baselines so that the comparison with ARSQ, evaluated with 10 trajectories over 3 seeds, is meaningful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the factorization identity is a definitional mathematical identity and the main empirical gains are evaluated against external baselines.

full rationale

I find no circularity that reduces the paper's claims to its inputs. The theoretical centerpiece, Theorem 4.3, is a parameterization identity: Definition 4.2 sets pi(a_d|s,a_-d) proportional to exp(A_d/alpha), Eq. (16) enforces the normalization sum exp(A_d/alpha)=1, and the proof in Appendix A then uses the chain rule of conditional probability together with the soft-policy form Eq. (10) to obtain sum A_d = A. This is a mathematical consequence of the authors' representation, not an empirical prediction fitted from data, and it is not used to justify the algorithm's empirical success. The 1.62x D4RL claim is an external benchmark comparison against CQN and other baselines, and the ablations in Sec. 5.4 test the auto-regressive mechanism against variants, so the main empirical claims do not reduce to a fit or to self-citation. References to the authors' own prior work (e.g., Yu et al. 2022) are not load-bearing. I do note an internal-validity concern that is not circularity: Sec. 5.1 says both ARSQ and CQN use BC loss Eq. (15), but Appendix B.2 states that on medium and medium-expert datasets ARSQ adopts variant Eq. (22) because it 'achieves better performance'; the paper never states whether the CQN baseline was updated to Eq. (22), so part of the reported gain may be confounded. This affects interpretation of the experiments but is not a circular derivation.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The factored advantage identity (Theorem 4.3) is derived from the chain rule of probability with an enforced normalization; it is not an independent law. The learning algorithm further depends on standard soft Q-learning, on the unproven convergence of the factored TD update, and on chosen discretization resolutions. Hyperparameters alpha, C_m, L, B, and a per-dataset BC-loss variant are tuned and materially affect the results.

free parameters (4)
  • Temperature coefficient alpha = 0.01 (D4RL), 0.001 (RLBench)
    Balances entropy and reward in soft Q-learning; Fig. 14 shows strong sensitivity, so the chosen value is tuned per domain.
  • BC margin C_m = -1 (D4RL), -0.01 (RLBench)
    Controls how strongly expert actions are preferred in the behavior cloning loss; affects the learning signal from demonstrations.
  • Coarse-to-fine levels L and bins B = L=2, B=7 (D4RL); L=3, B=5 (RLBench)
    Defines action discretization granularity. If too coarse, fine-grained control accuracy is capped; the main experiments do not ablate this resolution.
  • BC loss variant selection = Eq. (22) for medium/medium-expert D4RL datasets, Eq. (15) otherwise
    The paper adopts a different BC objective after observing better performance on certain datasets; this per-dataset tuning affects the comparison with CQN.
assumptions (3)
  • standard math The soft Bellman equation and the soft policy form (Eqs. 4-5) hold for the maximum-entropy RL objective (Eq. 1).
    The paper adopts results from Ziebart et al. (2008) and Haarnoja et al. (2017) without re-proving them; all downstream updates rely on this.
  • domain assumption The factored TD objective (Eq. 18), optimized over V and the normalized advantage heads, has a fixed point equal to the soft Q-function.
    Section 4.2 states 'Q-iteration follows the same update rule as soft Q-learning', but the residual loss with a sum-of-advantages parameterization is not proven to converge to that fixed point under function approximation.
  • domain assumption The coarse-to-fine grid (L=2, B=7 on D4RL; L=3, B=5 on RLBench) is fine enough to express near-optimal continuous actions.
    Discretization resolution is chosen per benchmark and is not swept in the main experiments; Appendix G analyzes discretization error only in a toy 2D environment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network." pith.science (2026). https://pith.science/paper/LVVGDS22

@misc{pith2026250200288,
  author       = {Pith},
  title        = {Pith review of: Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LVVGDS22}},
  note         = {Machine review of arXiv:2502.00288}
}
abstract

Reinforcement learning (RL) for continuous control often requires large amounts of online interaction data. Value-based RL methods can mitigate this burden by offering relatively high sample efficiency. Some studies further enhance sample efficiency by incorporating offline demonstration data to "kick-start" training, achieving promising results in continuous control. However, they typically compute the Q-function independently for each action dimension, neglecting interdependencies and making it harder to identify optimal actions when learning from suboptimal data, such as non-expert demonstration and online-collected data during the training process. To address these issues, we propose Auto-Regressive Soft Q-learning (ARSQ), a value-based RL algorithm that models Q-values in a coarse-to-fine, auto-regressive manner. First, ARSQ decomposes the continuous action space into discrete spaces in a coarse-to-fine hierarchy, enhancing sample efficiency for fine-grained continuous control tasks. Next, it auto-regressively predicts dimensional action advantages within each decision step, enabling more effective decision-making in continuous control tasks. We evaluate ARSQ on two continuous control benchmarks, RLBench and D4RL, integrating demonstration data into online training. On D4RL, which includes non-expert demonstrations, ARSQ achieves an average $1.62\times$ performance improvement over SOTA value-based baseline. On RLBench, which incorporates expert demonstrations, ARSQ surpasses various baselines, demonstrating its effectiveness in learning from suboptimal online-collected data. Project page is at https://sites.google.com/view/ar-soft-q

Figures

Figures reproduced from arXiv: 2502.00288 by the authors.

Figure 1
Figure 1. A motivating example of how Q decomposition influences policy training, as detailed in Appendix C.1. dimensional actions (a1, a2) ∈ A = [−1, 1]2 ⊂ R 2 , shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The ARSQ algorithm. The action space is discretized using a coarse-to-fine approach. By predicting dimensional soft advantages, ARSQ generates actions in an auto-regressive manner within a single decision-making step. 4.1. Coarse-to-fine Action Discretization To apply Q-learning (Mnih et al., 2015) in a continuous domain, a straightforward approach is to discretize the ac￾tion space (Tang & Agrawal, 2020; Seo et al.… view at source ↗
Figure 3
Figure 3. Network architecture of ARSQ. The soft value Vsoft and the dimensional soft advantage A d are predicted by two separate networks. The advantage network utilizes a shared backbone, and advantage constraints are applied to its output. puts. A d (st, a −d , ad ) = u d (st, a −d , ad ) − αlogX ad′ exp  1 α u d (st, a −d , ad ′ )  (16) where u d is the output of the d-th output head. Furthermore, to stabilize training … view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: D4RL main results. mr, m, and me represent medium-replay, medium, and medium-expert, respectively. feature ACT (Zhao et al., 2023) and a CQN-style behavior cloning (BC) policy among our baselines. Details about the baselines can be found in Appendix C.3. 5.1. Performan…
Figure 5
Figure 5. Figure 5: D4RL results on different demonstration quality aver￾aged over 3 tasks, with each task containing 3 datasets respectively. We report the normalized return provided by D4RL. Analysis on Demonstration Quality. To better investi￾gate the influence of dataset quality, we r…
Figure 6
Figure 6. Figure 6: RLBench results on different tasks. Each experiment begins with 100 expert demonstrations, and all RL methods include a behavior cloning objective. 5.2. Performance on RLBench To further evaluate ARSQ’s performance, we focus on six tasks from RLBench (James et al., 202…
Figure 8
Figure 8. Figure 8: Ablation on shared backbone in D4RL (left) and RL￾Bench (right). 6. Conclusion In this paper, we introduced Auto-Regressive Soft Q￾learning (ARSQ), a novel value-based RL approach tailored for continuous control tasks with suboptimal data. ARSQ addresses the limitation…
Figure 9
Figure 9. Figure 9: D4RL Gym tasks used in experiment. D4RL Dataset. In D4RL, we use the medium-replay, medium, and medium-expert datasets for tasks involving half-cheetah, hopper, and walker2d. In Section 5.1, to examine the impact of dataset quality, we rank trajectories based on episod…
Figure 10
Figure 10. Figure 10: Histogram of reward in D4RL datasets. In RLBench, we adopt DrQ-v2+, an optimized variant of DrQ-v2 proposed by (Seo et al., 2024), as our baseline. DrQ-v2+ incorporates several optimization strategies introduced in the CQN algorithm. Specifically, compared to DrQ-v2, …
Figure 11
Figure 11. Figure 11: Example of RLBench tasks used in experiment. pos: {0, 1} vel: {0.5, 0.2} acc: {0.01, 2.0} feedback: … observa�on Infer 𝑽𝒔𝒐𝒇𝒕 𝒔𝒕 𝑨𝟏(𝒔𝒕, ∅,⋅) 𝒂𝒕 𝟏 Head 1 Infer ∅ 𝒖𝟐(𝒔𝒕, 𝒂−𝟐 ,⋅) Infer Head 2 𝒖𝟏(𝒔𝒕, ∅,⋅) 𝒂𝒕 𝟐 Head 3 Infer Head 4 𝒂𝒕 𝟑 … … … Infer 𝑨𝟐(𝒔𝒕, 𝒂−𝟐 ,⋅) 𝑨𝟑(𝒔𝒕, 𝒂−𝟑 …
Figure 12
Figure 12. Figure 12: Network architecture of Separate backbone baseline in ablation study. D4RL Results per Task for Different Demonstration Quality. In Sec. 5.1, we present the D4RL results, averaged over all 9 datasets, based on varying demonstration quality. The results for each task a…
Figure 13
Figure 13. Figure 13: Network architecture of Level Shared backbone baseline in ablation study. = 1.0 = 0.1 = 0.01 = 1e-3 = 1e-4 0 50 100 Normalized Return 76.2 95.7 97.5 83.4 92.2 0 10k 20k 30k Environment Steps 0 25 50 75 100 Success Rate (%) = 0.1 = 0.01 = 0.001 [PITH_FULL_IMAGE:figure…
Figure 14
Figure 14. Figure 14: Sensitivity of temperature coefficient α, evaluated on hopper-medium from D4RL and Open Oven from RLBench over three random seeds. using three random seeds. We also illustrate the performance of both vanilla and offline ARSQ using the one of the hopper dataset, specif…
Figure 15
Figure 15. Figure 15: Training curves of D4RL main results, evaluated over three random seeds. decomposition results in a significant error increase compared to ARSQ and continuous Q learning. Additionally, ARSQ without coarse-to-fine action discretization also results in higher Q error, f…
Figure 16
Figure 16. Figure 16: D4RL results per task on different demonstration quality, evaluated over three random seeds [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: RLBench results in all 20 tasks. 0 100k 200k 300k 400k Env. Steps 0 50 100 Normalized Return ARSQ ARSQ (online) ARSQ (offline) CQN (online) PPO [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: Performance under fully online settings, on hopper task or hopper-medium-replay dataset. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Visualization of Q prediction with different action discretization strategy. (a) Independent action decom￾position. (b) ARSQ w/o coarse-to-fine discretization. (c) ARSQ [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Q prediction errors with different action discretization strategy. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 30 canonical work pages

  1. [1]

    Ba, J. L. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  2. [2]

    J., Smith, L., Kostrikov, I., and Levine, S

    Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. Efficient online reinforcement learning with offline data. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.\ 1577--1594. PMLR, 23-...

    work page 2023

  3. [3]

    Dota 2 with large scale deep reinforcement learning

    Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019

    arXiv 1912

  4. [4]

    Offline rl without off-policy evaluation

    Brandfonbrener, D., Whitney, W., Ranganath, R., and Bruna, J. Offline rl without off-policy evaluation. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 4933--4946. Curran Associates, Inc., 2021

    work page 2021

  5. [5]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.,...

    1901

  6. [6]

    A., Salazar, G., Tran, H

    Chebotar, Y., Vuong, Q., Hausman, K., Xia, F., Lu, Y., Irpan, A., Kumar, A., Yu, T., Herzog, A., Pertsch, K., Gopalakrishnan, K., Ibarz, J., Nachum, O., Sontakke, S. A., Salazar, G., Tran, H. T., Peralta, J., Tan, C., Manjunath, D., Singh, J., Zitkovich, B., Jackson, T., Rao, K., Finn, C., and Levine, S. Q-transformer: Scalable offline reinforcement learn...

    work page 2023

  7. [7]

    Decision transformer: Reinforcement learning via sequence modeling

    Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 15084--15097. Curran Associa...

    work page 2021

  8. [8]

    Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022

    Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=S874XAIpkR-

    work page 2022

Show all 51 references
  1. [9]

    Counterfactual multi-agent policy gradients

    Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, 32 0 (1), Apr. 2018. doi:10.1609/aaai.v32i1.11794

    2018 doi

  2. [10]

    D4rl: Datasets for deep data-driven reinforcement learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  3. [11]

    and Gu, S

    Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 20132--20145. Curran Associates, Inc., 2021

    2021

  4. [12]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 1587--1...

    2018

  5. [13]

    Reinforcement learning with deep energy-based policies

    Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based policies. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 135...

    2017

  6. [14]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings...

    2018

  7. [15]

    Modem: Accelerating visual model-based reinforcement learning with demonstrations

    Hansen, N., Lin, Y., Su, H., Wang, X., Kumar, V., and Rajeswaran, A. Modem: Accelerating visual model-based reinforcement learning with demonstrations. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=JdTnc9gjVfJ

    2023

  8. [16]

    Neural networks: a comprehensive foundation

    Haykin, S. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1998

    1998

  9. [17]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  10. [18]

    Deep q-learning from demonstrations

    Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., Dulac-Arnold, G., Agapiou, J., Leibo, J., and Gruslys, A. Deep q-learning from demonstrations. Proceedings of the AAAI Conference on Artificial Intellige...

    2018 doi

  11. [19]

    B ayesian design principles for offline-to-online reinforcement learning

    Hu, H., Yang, Y., Ye, J., Wu, C., Mai, Z., Hu, Y., Lv, T., Fan, C., Zhao, Q., and Zhang, C. B ayesian design principles for offline-to-online reinforcement learning. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), ...

    2024

  12. [20]

    R., and Davison, A

    James, S., Ma, Z., Arrojo, D. R., and Davison, A. J. Rlbench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters, 5 0 (2): 0 3019--3026, 2020. doi:10.1109/LRA.2020.2974707

    2020

  13. [21]

    Offline reinforcement learning with implicit q-learning

    Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=68n2s9ZJWF8

    2022

  14. [22]

    Conservative q-learning for offline reinforcement learning

    Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 1179--1191. Curran Associ...

    2020

  15. [23]

    Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble

    Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Faust, A., Hsu, D., and Neumann, G. (eds.), Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine...

    2022

  16. [24]

    Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization

    LEI, K., He, Z., Lu, C., Hu, K., Gao, Y., and Xu, H. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=tbFBh3LMKi

    2024

  17. [25]

    A survey of convolutional neural networks: Analysis, applications, and prospects

    Li, Z., Liu, F., Yang, W., Peng, S., and Zhou, J. A survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Transactions on Neural Networks and Learning Systems, 33 0 (12): 0 6999--7019, 2022. doi:10.1109/TNNLS.2021.3084827

    2022

  18. [26]

    Continuous control with deep reinforcement learning

    Lillicrap, T. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  19. [27]

    Discrete sequential prediction of continuous actions for deep rl

    Metz, L., Ibarz, J., Jaitly, N., and Davidson, J. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv:1705.05035, 2017

    2017 arXiv

  20. [28]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  21. [29]

    Overcoming exploration in reinforcement learning with demonstrations

    Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 6292--6299, 2018. doi:10.1109/ICRA.2018.8463162

    2018

  22. [30]

    Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning

    Nakamoto, M., Zhai, S., Singh, A., Sobol Mark, M., Ma, Y., Finn, C., Kumar, A., and Levine, S. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural I...

    2023

  23. [31]

    Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018

    Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018

    2018

  24. [32]

    Rudner, T. G. J., Lu, C., Osborne, M. A., Gal, Y., and Teh, Y. On pathologies in kl-regularized reinforcement learning from expert demonstrations. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Syst...

    2021

  25. [33]

    Mastering atari, go, chess and shogi by planning with a learned model

    Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588 0 (7839): 0 604--609, 2020

    2020

  26. [34]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  27. [35]

    and Abbeel, P

    Seo, Y. and Abbeel, P. Reinforcement learning with action sequence for data-efficient robot learning. arXiv preprint arXiv:2411.12155, 2024

    2024

  28. [36]

    Continuous control with coarse-to-fine reinforcement learning

    Seo, Y., Uru c , J., and James, S. Continuous control with coarse-to-fine reinforcement learning. In 8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id=WjDR48cL3O

    2024

  29. [37]

    Solving continuous control via q-learning

    Seyde, T., Werner, P., Schwarting, W., Gilitschenski, I., Riedmiller, M., Rus, D., and Wulfmeier, M. Solving continuous control via q-learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=U5XOGxAgccS

    2023

  30. [38]

    Growing Q -networks: S olving continuous control tasks with adaptive control resolution

    Seyde, T., Werner, P., Schwarting, W., Wulfmeier, M., and Rus, D. Growing Q -networks: S olving continuous control tasks with adaptive control resolution. In Abate, A., Cannon, M., Margellos, K., and Papachristodoulou, A. (eds.), Proceedings of the 6th Annual Learning for Dyna...

    2024

  31. [39]

    Mastering the game of go without human knowledge

    Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550 0 (7676): 0 354--359, 2017

    2017

  32. [40]

    and Agrawal, S

    Tang, Y. and Agrawal, S. Discretizing continuous action space for on-policy optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 34 0 (04): 0 5981--5988, Apr. 2020. doi:10.1609/aaai.v34i04.6059

    2020 doi

  33. [41]

    Action branching architectures for deep reinforcement learning

    Tavakoli, A., Pardo, F., and Kormushev, P. Action branching architectures for deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 32 0 (1), Apr. 2018. doi:10.1609/aaai.v32i1.11798

    2018 doi

  34. [42]

    Learning to represent action values as a hypergraph on the action vertices

    Tavakoli, A., Fatemi, M., and Kormushev, P. Learning to represent action values as a hypergraph on the action vertices. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=Xv_s64FiXTv

    2021

  35. [43]

    Deep reinforcement learning with double q-learning

    van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30 0 (1), Mar. 2016. doi:10.1609/aaai.v30i1.10295

    2016 doi

  36. [44]

    Discriminator-weighted offline imitation learning from suboptimal demonstrations

    Xu, H., Zhan, X., Yin, H., and Qin, H. Discriminator-weighted offline imitation learning from suboptimal demonstrations. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learnin...

    2022

  37. [45]

    Hd-cnn: Hierarchical deep convolutional neural networks for large scale visual recognition

    Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., and Yu, Y. Hd-cnn: Hierarchical deep convolutional neural networks for large scale visual recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015

    2015

  38. [46]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=_SJ-_yyes8

    2022

  39. [47]

    The surprising effectiveness of ppo in cooperative multi-agent games

    Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and WU, Y. The surprising effectiveness of ppo in cooperative multi-agent games. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, ...

    2022

  40. [48]

    Policy expansion for bridging offline-to-online reinforcement learning

    Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-online reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=-Y34L45JR6z

    2023

  41. [49]

    Z., Kumar, V., Levine, S., and Finn, C

    Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learning fine-grained bimanual manipulation with low-cost hardware, 2023

    2023

  42. [50]

    D., Maas, A., Bagnell, J

    Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3, AAAI'08, pp.\ 1433–1438. AAAI Press, 2008. ISBN 9781577353683

    2008

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.