Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning

Abdelghani Ghanem; Mounir Ghogho

REVIEW 3 major objections 3 minor 68 references

Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that an upper-expectile loss on n-step action-value residuals yields a gamma^n-contraction with a unique fixed point, and shows that a fixed tau=0.8, n=4 setting is competitive with the long-horizon Q-learning baseline…

desk verdict Solid theory, honest paper, but the headline empirical comparison is confounded because the baselines were imported with different hyperparameters; worth refereeing for the operator analysis alone. read the letter →

arxiv 2608.02034 v1 pith:FS33GFSG submitted 2026-08-03 cs.LG

Abdelghani Ghanem , Mounir Ghogho This is my paper · ORCID

classification cs.LG

keywords expectileQ-learningmulti-stepreturnsoff-policyreinforcementlearningn-stepTDlong-horizonvaluecontractionoperatoroffline-to-onlineRLpessimismbias

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-step returns spread reward information quickly in off-policy reinforcement learning, but they drag along the logged behavior policy's suboptimal choices, creating a pessimistic bias that grows with horizon. The paper introduces Expectile n-step Q-learning (ENQ), which replaces the symmetric squared n-step TD loss with an asymmetric loss that up-weights positive action-value residuals through an expectile level tau and down-weights negative ones. It proves that the resulting n-step Bellman operator is a gamma^n-contraction for every tau in (0,1] and horizon n, so it always has a unique fixed point; under deterministic dynamics with an optimal-coverage condition, the tau=1 endpoint removes the pessimistic bias exactly at Q* on supported pairs, and the fixed point satisfies the lower-bound optimality inequality used by long-horizon Q-learning at separations n, 2n, .... With a single setting (tau=0.8, n=4) held fixed across 27 manipulation and navigation task instances, ENQ is competitive with the long-horizon baseline and shows higher training throughput in a matched profile. If these guarantees carry over to practical neural critics, ENQ would be a drop-in, hyperparameter-light replacement for n-step TD that needs no importance ratios, auxiliary value networks, or trajectory-level constraints.

What carries the argument

The load-bearing mechanism is the asymmetric squared expectile loss on the scalar n-step TD residual. With expectile level $\tau > 1/2$, positive residuals (target above current estimate) receive weight $\tau$ and negative residuals weight $1-\tau$, so the critic is pulled toward an upper expectile of the distribution of n-step targets induced by logged actions and transition noise; this is what down-weights suboptimal continuations without any likelihood ratio. The contraction proof couples two value functions on the same trajectory, showing $|Z_n^{Q_1} - Z_n^{Q_2}| \le \gamma^n \|Q_1 - Q_2\|_\infty$ almost surely, then uses the 1-Lipschitz property of expectiles (Lemma 1(v)) to transfer the bound; the Banach fixed-point theorem yields the unique fixed point. The bias analysis rests on the telescoping identity $W_n = \sum_{k=0}^{n-1} \gamma^k \epsilon_k - \sum_{k=1}^{n-1} \gamma^k \Delta^*(s_k,a_k)$, which separates the mean-zero Bellman noise $\epsilon_k$ from the nonnegative suboptimality gaps $\Delta^*$ of the logged actions; the monotonicity of expectiles in $\tau$ converts this decomposition into a one-sided bias that can be tuned by $\tau$.

What would settle it

Take a deterministic MDP where the behavior policy, at some supported (s,a), always follows a strictly suboptimal action within the next n-1 steps, so the cumulative discounted gap has a positive lower bound on every logged continuation. Then, by the definition of the operator, $T_n^{1}$ Q*(s,a) will be strictly below Q*(s,a) on that pair, contradicting Theorem 2(c); observing exactly that inequality in a finite experiment would also flag missing near-optimal coverage in the dataset.

Watch

Extended reading notes

Core claim

The paper's central claim is that placing an upper-expectile loss directly on the n-step action-value residual corrects the off-policy pessimism of multi-step returns while keeping their fast reward propagation. Formally, define $Z_n^Q(s,a)$ as the n-step target whose intermediate actions are drawn from the behavior policy and whose bootstrap is greedy; the expectile n-step Bellman operator is $(T_n^{\tau} Q)(s,a) = e_{\tau}[Z_n^Q(s,a)]$. Theorem 1 states that for every $\tau \in (0,1]$ and $n \ge 1$, with arbitrary stochastic dynamics and behavior policies, $\|T_n^{\tau} Q_1 - T_n^{\tau} Q_2\|_\infty \le \gamma^n \|Q_1 - Q_2\|_\infty$, so $T_n^{\tau}$ is a $\gamma^n$-contraction with a unique fixed point $Q_n^{\tau}$. Under deterministic dynamics, the pathwise gap to $Q^*$ decomposes as $W_n = \sum_{k=0}^{n-1} \gamma^k \epsilon_k - \sum_{k=1}^{n-1} \gamma^k \Delta^*(s_k,a_k)$, a mean-zero noise term minus the discounted behavioral drift; therefore the $\tau=1/2$ operator is exactly as pessimistic as uncorrected n-step TD, the fixed point $Q_n^{\tau}$ is nondecreasing in $\tau$ and bounded above by $Q^*$, and under optimal coverage the $\tau=1$ operator fixes $Q^*$ on covered in-support pairs, its fixed point obeying the separation-n lower-bound inequality of LQL. Under stochastic dynamics, the one-application bias satisfies $-\mu_n(s,a) \le B_{\tau}(s,a) \le -\mu_n(s,a) + \kappa_{\tau} \sigma_n(s,a)$ with $\kappa_{\tau} = \frac{2\tau-1}{2(1-\tau)}$ and horizon-independent variance constants, yielding a fixed-point error bound with the same constants as the LQL stochasticity analysis. Empirically, with $\tau=0.8$ and $n=4$ fixed, ENQ attains the higher aggregate mean in four of five comparisons against the LQL baseline across 27 task instances, with a 1.27-1.77x training-throughput advantage in a matched profiling study.

Load-bearing premise

Assumption 2 requires that for every supported state-action pair, the logged data place positive probability on n-step continuations whose total discounted suboptimality gap is arbitrarily close to zero; if such near-optimal continuations are absent, Theorem 2(c)'s tau=1 endpoint correction no longer holds and the fixed point can fall strictly below Q*.

Editorial extensions

If this is right

ENQ is a drop-in replacement for n-step TD: it adds only the expectile level tau, and needs no action likelihoods, importance ratios, auxiliary value networks, or trajectory-level pairwise penalties.
The gamma^n-contraction property gives a well-posed learning target for every tau and horizon: a unique fixed point exists and iteration converges geometrically with modulus gamma^n.
At the tau=1 endpoint, the fixed point satisfies the separation-jn lower-bound inequality of LQL for j=1,2,..., so the trajectory-level optimality constraint is met by construction rather than by hinge penalties.
The fixed tau=0.8, n=4 configuration reaches the higher aggregate mean in four of five comparisons against LQL across 27 manipulation and navigation task instances, and achieves 1.27-1.77x higher measured training-step throughput.
The stochastic-dynamics bound produces a safe-expectile threshold tau_safe(s,a) = (sigma_n + 2 mu_n)/(2 sigma_n + 2 mu_n): larger behavioral drift permits larger tau, while with noisy upper tails a conservative ensemble penalty is needed to counter over-optimism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The theorems analyze an exact population operator with greedy bootstrap, so they do not quantify function-approximation or finite-sample error; the natural next step, which the paper leaves open, is a finite-sample bound for expectile n-step regression under approximate operators.
The monotone dependence of the fixed point on tau suggests tau can serve as a practical bias dial; the ablation's finding that the preferred tau grows with n on one domain but not another implies tau and n should be selected jointly, an implication the paper states but does not resolve.
ENQ's throughput advantage widens with ensemble size, so as a backbone ENQ may scale better with critic-ensemble compute than trajectory-constraint methods; this is an extrapolation beyond the paper's data rather than a claim it makes.
The direct placement of the expectile on the action-value residual outperformed an n-step IQL-style baseline that routes the same asymmetry through a state-value function; a testable extension would vary the baseline's value-network capacity to see whether the placement effect persists.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, the core idea is genuinely clean: apply an upper-expectile loss directly to the n-step TD residual on the action-value error, with tau the only added hyperparameter. That placement is new relative to IQL (state-value expectile), XQL, and LEQ (lower expectile on model rollouts). Second, the theory is the strong part and it holds up: Theorem 1's gamma^n contraction is a direct expectile-Lipschitz argument, Theorem 2's bias decomposition under deterministic dynamics is correct and the tau=1 endpoint connection to LQL's lower-bound inequality is a nice observation. The appendix proofs check out, and the paper is honest about its idealizations (exact operator, greedy bootstrap) in Appendix F.

The soft spot is exactly where the stress-test note lands. The main empirical tables compare ENQ against TD, TD-n, and LQL numbers copied from Abraham et al. (2026), and those cells carry different action-chunk sizes (h=5 vs h=4), different gamma on humanoidmaze-medium (0.995 vs 0.99), different alpha on antmaze-giant (3 vs 5), and different ensemble aggregation on RoboMimic. So the aggregate margins (e.g., OGBench FQL total 69.0 vs 59.0) cannot be attributed to the ENQ loss. The paper does disclose this in Appendix G, which is good, but disclosure doesn't remove the confound. To establish the empirical claim they need to rerun the baselines in their own codebase with matched hyperparameters. The ablations are more convincing because they are self-contained, but they don't fix the headline comparison.

Is the central argument sound? The theoretical mechanism is plausible and rigorously derived; the empirical mechanism is plausible but underdetermined. Assumption 2 (optimal coverage) is strong but explicitly stated, and the paper doesn't hide that the tau=1 endpoint needs it. The practical tau=0.8 choice is motivated by ablations, but those ablations were run on the benchmark itself, so there is some selection risk on the fixed hyperparameters.

Who is this for? Researchers working on off-policy value learning with expressive policies, multi-step returns, and offline-to-online RL. The operator theory and the expectile-on-action-value idea will be useful even if the empirical claims need another pass. I'd cite the theoretical part. The paper deserves a serious referee: the math is solid, the idea is simple and worth testing, and the empirical issues are fixable by rerunning baselines under matched settings. I'd send it to review with a clear request for that rerun, not desk-reject it.

Referee Report

3 major / 3 minor

Summary. The paper proposes Expectile n-step Q-learning (ENQ), a modification of off-policy n-step TD in which the squared loss on the action-value error is replaced by an asymmetric expectile loss. The authors prove that the resulting population operator is a gamma^n-contraction for every tau in (0,1] and any dynamics and behavior policy (Theorem 1), decompose the pessimistic bias of n-step TD under deterministic dynamics (Theorem 2), show that at tau=1 the operator is unbiased at Q* under an optimal-coverage assumption, and derive two-sided bias bounds plus a fixed-point error bound under stochastic dynamics (Theorem 3). Corollary 1 connects the tau=1 fixed point to LQL's lower-bound inequality at separations n, 2n, and so on. Empirically, using tau=0.8 and n=4 across 27 OGBench and RoboMimic instances, ENQ is reported to attain a higher aggregate mean than LQL in four of five comparisons, with additional ablations on expectile level, horizon, ensemble size, expectile placement, and stochastic transitions.

Significance. The theoretical core is a genuine contribution: the contraction theorem is clean, the bias decomposition in Equation (8) is illuminating, and the connection to LQL's lower-bound inequality is novel. The appendix proofs are detailed and internally consistent, and the experimental protocol is careful in several respects, notably matching the number of sampled transitions per update across ENQ and LQL and reporting bootstrap confidence intervals for all newly run ENQ results. However, the headline empirical claim—that ENQ is competitive with or better than LQL on aggregate—is currently underdetermined because the baseline cells in the main comparison tables were copied from a prior paper with different hyperparameters rather than rerun under matched settings. The theoretical results stand on their own, but the paper's central empirical contribution needs either matched baseline runs or a substantially narrowed claim.

major comments (3)

[Appendix G, 'Policy extraction and baseline provenance'; Tables 1–3] The main comparative tables compare newly run ENQ results against TD, TD-n, LQL, ReBRAC, and IQL cells that are 'taken directly from the results reported by Abraham, Shi, and Finn (2026)' and were not rerun. The ENQ runs in the same tables use different action-chunk sizes (h=4 versus h=5), different discount factors on humanoidmaze-medium (gamma=0.995 versus 0.99), different policy coefficients on antmaze-giant (alpha=3 versus 5), and different target-ensemble aggregation on RoboMimic (ensemble-mean versus ensemble-minimum). These settings can materially affect success rates, so aggregate margins such as the OGBench FQL total (69.0 versus 59.0 in Table 1) and the humanoidmaze-giant total (80.9 versus 75.7 in Table 2) cannot be attributed to the expectile loss. This undermines the central empirical claim in 'Comparative Evaluation' that ENQ attains the higher aggregate mean in four of five comparisons.
[Tables 1–3] Baseline cells have no confidence intervals or seed-level data; only ENQ cells carry 95% bootstrap intervals. Consequently, even the direction of the aggregate comparisons is not statistically assessable: a margin of, for example, 10 points on the OGBench FQL total could be within seed-level noise, and the absence of baseline intervals makes the reported ranking unverifiable. The authors should either rerun the baselines with the same seeds and report their own intervals, or clearly restrict the comparative claims to settings where all entries were produced by the same pipeline.
[Theorem 2(c) and Assumption 2; Appendix F] The theoretical mechanism that upper expectiles correct n-step pessimism is established only under Assumption 2 (optimal coverage), which requires that for every in-support (s,a) the behavior policy places positive probability on continuations whose cumulative suboptimality gap is arbitrarily small. This is a strong condition that may fail in real offline datasets. The paper is transparent about this, but the consequence is that the theoretical support for the method's headline configuration (tau=0.8, n=4) is indirect: Theorem 2(c) concerns the tau=1 endpoint, and the stochastic-dynamics bounds in Theorem 3 are stated in terms of unobservable drift and variance quantities. The authors should state more explicitly that the empirical gains at tau=0.8 are not directly predicted by the endpoint result, and ideally provide a diagnostic of approximate optimal coverage on the benchmark datasets.

minor comments (3)

[Table 5 and Table 2] There is an inconsistency about the TD-n baseline horizon: Table 5 lists the published TD-n horizon as 'n=8 (default)', while Table 2 labels the baseline 'TD-4' and the experimental section states that for humanoidmaze-giant the published TD-n baseline uses n=4. Please reconcile these statements.
[Abstract and Section 'Expectile n-step Q-learning'] The abstract and introduction say tau is 'the only method-specific hyperparameter added beyond n-step TD', but the ENQ results also depend on the conservative coefficient rho and on the action-chunk size h, which vary across comparisons. Since rho and h are inherited from the ensemble aggregation and policy-extraction recipes, the phrasing should clarify that tau is the only new hyperparameter of the loss itself, not of the full method.
[Figure 3, left panel] The throughput profiling uses a single seed (seed 10001) for each ensemble size. This is acceptable for a profiling study, but the 1.27x to 1.77x speedup ratios in Table 7 should be described as single-seed measurements without implying tight error bars.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ENQ's contraction and bias results are derived from the expectile operator's definition and explicit coverage assumptions; the copied baseline cells create a comparison confound, not a circular dependence.

full rationale

The theoretical derivation chain is self-contained. The ENQ operator T_n^τ is defined as the τ-expectile of the n-step target (Eq. 7), and Theorem 1's contraction proof couples two targets on the same trajectory, applies the elementary Lipschitz property of expectiles (Lemma 1(v), proved in Appendix A), and invokes the Banach fixed-point theorem; nothing that ENQ is supposed to predict is used as an input. Theorem 2's pessimism decomposition follows from the telescoping identity (8) derived in Appendix B, and the endpoint correction at τ=1 is a direct consequence of Assumption 2 plus the definition of the essential supremum; the assumption explicitly states the coverage condition and is not entailed by the conclusion. Corollary 1 is transparently labeled 'by construction': the τ=1 fixed-point equation Q(s,a)=ess sup[G+γ^n max Q] implies the LQL lower-bound inequality, so the paper is reporting a mathematical implication rather than claiming an independent empirical discovery. The empirical claim that ENQ beats LQL on aggregate is weakened by Appendix G's admission that TD, TD-n, and LQL cells were taken from Abraham, Shi, and Finn (2026) without rerunning, with different hyperparameters; this is a validity/confounding concern, not a circularity, because the baseline numbers are not fitted inputs and no reported prediction reduces to its own construction. The fixed τ=0.8 and n=4 across all tasks also rules out the fitted-input-called-prediction pattern. There are no load-bearing self-citations by the present authors. I therefore find no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The theory is self-contained in the sense that the operator results follow from the definitions, but it rests on two substantive domain assumptions (deterministic dynamics and optimal coverage) that are not verified in the empirical setups, and on the standard idealization that the population operator describes the neural algorithm. The practical results additionally depend on hyperparameters (tau, n, rho, h) set by the authors or inherited from prior work.

free parameters (4)

expectile level tau = 0.8
Method-specific hyperparameter; fixed at 0.8 after ablation A1 on cube-triple-task2 and antmaze-giant-task1, so not derived from first principles and selected partly on the evaluation benchmark.
backup horizon n = 4
Bootstrap horizon; fixed at n=4 after ablation A1; intermediate n=8 was often best in the ablation, but 4 was chosen for the main runs.
conservative coefficient rho = 0 (humanoid maze domains), 0.5 (elsewhere)
Task-specific coefficient inherited from QAM/LQL recipes, not a unified constant; interacts with the expectile level per Theorem 3.
action-chunk size h = 4 for ENQ (5 for published LQL baselines)
Action chunk size differs between ENQ and the published LQL baseline, confounding the algorithm comparison.

assumptions (5)

domain assumption Assumption 1: Deterministic dynamics, s_{k+1}=f(s_k,a_k)
Used in Theorem 2 and Corollary 1 to make the bias pathwise and the tau=1 endpoint correction exact; real environments in the experiments are stochastic, so the main theoretical justification of the bias correction only holds approximately.
domain assumption Assumption 2: Optimal coverage, ess inf over P_{s,a} of sum_{k=1}^{n-1} gamma^k Delta*(s_k,a_k) = 0 for every in-support (s,a)
Needed for Theorem 2(c) that the tau=1 operator bias vanishes at Q*. This is a segment-level optimal-coverage condition on the behavior policy; it is plausible for mixed datasets but not verified for the benchmark datasets.
standard math Bounded rewards and discounting: |R| <= R_max, gamma in [0,1)
Standard RL assumptions used for the contraction argument and the norm bounds.
standard math Expectile well-defined for square-integrable targets
Newey and Powell (1987) definition; used throughout the analysis.
domain assumption Idealized operator: population expectile, exact greedy bootstrap, no function approximation
Appendix F states the theorems concern the exact operator, not the practical neural algorithm with target network and actor proxy; the transfer to practice is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning." pith.science (2026). https://pith.science/paper/FS33GFSG

@misc{pith2026260802034,
  author       = {Pith},
  title        = {Pith review of: Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FS33GFSG}},
  note         = {Machine review of arXiv:2608.02034}
}

abstract

Multi-step returns accelerate reward propagation in off-policy reinforcement learning, but couple the evaluation of each decision to the suboptimal logged actions that follow it, inducing a pessimistic bias that grows with the horizon. We propose Expectile $n$-step Q-learning (ENQ), which replaces the symmetric $n$-step temporal-difference (TD) loss with an asymmetric expectile loss on the action-value error, with expectile level $\tau$ as the only method-specific hyperparameter added beyond $n$-step TD. We prove that the ENQ operator is a $\gamma^{n}$-contraction. Under deterministic dynamics, at $\tau=1$, its bias vanishes at the optimal action-value function $Q^*$ on covered in-support pairs, and the corresponding fixed point satisfies the separation-$n$ instance and its multiples of the lower-bound inequality used by Long-Horizon Q-learning (LQL). Under stochastic dynamics, the operator bias admits two-sided bounds with horizon-independent noise constants. Using a single expectile level $\tau=0.8$ and a fixed backup horizon across 27 manipulation and navigation task instances, ENQ is competitive with LQL on aggregate, achieves higher measured training-step throughput in our profiling study, and benefits more from a ten-critic ensemble in a controlled scaling experiment.

Figures

Figures reproduced from arXiv: 2608.02034 by the authors.

**Figure 1.** Bootstrap error and behavioral bias in longhorizon value learning. (a) One-step TD compounds bootstrap errors across successive backups. (b) LQL limits this propagation using trajectory-level constraints. (c) n-step TD removes intermediate bootstrapping but inherits pessimistic bias from suboptimal logged actions. (d) ENQ fits an upper expectile of logged n-step targets, reducing the influence of poor continuation… view at source ↗

**Figure 4.** Offline-to-online success for ENQ and the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

**Figure 2.** Final success across two tasks and two policy [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Figures from the paper (2 more)

**Figure 3.** Figure 3: Left: Training throughput on antmaze-giant across critic-ensemble sizes, with ENQ and LQL using identical batch size B = 128 and horizon length n = L = 8. Right: Final success on the five cube-triple tasks with four seeds per task. Shading shows 95% bootstrap confiden…

**Figure 6.** Figure 6: Evaluated OGBench and RoboMimic domains. Each OGBench group contains five single-task variants; can and square use the RoboMimic multi-human datasets. The antmaze-giant and humanoidmaze-giant environments share the giant-maze layout but use different agents [PITH_FULL…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 44 canonical work pages

[1]

Machine learning , volume=

Learning to predict by the methods of temporal differences , author=. Machine learning , volume=. 1988 , publisher=

1988
[2]

Machine learning , volume=

Q-learning , author=. Machine learning , volume=. 1992 , publisher=

1992
[3]

1998 , publisher=

Reinforcement learning: An introduction , author=. 1998 , publisher=

1998
[4]

Proceedings of the AAAI conference on artificial intelligence , volume=

Rainbow: Combining improvements in deep reinforcement learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=
[5]

Proceedings of the AAAI conference on artificial intelligence , volume=

Multi-step reinforcement learning: A unifying algorithm , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

work page
[6]

and Singh, Satinder P

Precup, Doina and Sutton, Richard S. and Singh, Satinder P. , title =. Proceedings of the Seventeenth International Conference on Machine Learning , pages =. 2000 , isbn =

work page 2000
[7]

Advances in neural information processing systems , volume=

Safe and efficient off-policy reinforcement learning , author=. Advances in neural information processing systems , volume=
[8]

International conference on machine learning , pages=

Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures , author=. International conference on machine learning , pages=. 2018 , organization=

2018

Show all 68 references

[9]

arXiv preprint arXiv:1611.01606 , year=

Learning to play in a day: Faster deep reinforcement learning by optimality tightening , author=. arXiv preprint arXiv:1611.01606 , year=

arXiv
[10]

arXiv preprint arXiv:2605.05812 , year=

Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities , author=. arXiv preprint arXiv:2605.05812 , year=

arXiv
[11]

arXiv preprint arXiv:2110.06169 , year=

Offline reinforcement learning with implicit q-learning , author=. arXiv preprint arXiv:2110.06169 , year=

arXiv
[12]

arXiv preprint arXiv:2301.02328 , year=

Extreme q-learning: Maxent rl without entropy , author=. arXiv preprint arXiv:2301.02328 , year=

arXiv
[13]

Econometrica: Journal of the Econometric Society , pages=

Asymmetric least squares estimation and testing , author=. Econometrica: Journal of the Econometric Society , pages=. 1987 , publisher=

1987
[14]

Advances in neural information processing systems , volume=

Uncertainty-based offline reinforcement learning with diversified q-ensemble , author=. Advances in neural information processing systems , volume=
[15]

Forty-second International Conference on Machine Learning , year=

Flow Q-Learning , author=. Forty-second International Conference on Machine Learning , year=
[16]

Advances in Neural Information Processing Systems , volume=

Reinforcement learning with action chunking , author=. Advances in Neural Information Processing Systems , volume=
[17]

International Conference on Learning Representations , volume=

Ogbench: Benchmarking offline goal-conditioned rl , author=. International Conference on Learning Representations , volume=
[18]

arXiv preprint arXiv:2108.03298 , year=

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation , author=. arXiv preprint arXiv:2108.03298 , year=

arXiv
[19]

Revisiting the Minimalist Approach to Offline Reinforcement Learning , url =

Tarasov, Denis and Kurenkov, Vladislav and Nikulin, Alexander and Kolesnikov, Sergey , booktitle =. Revisiting the Minimalist Approach to Offline Reinforcement Learning , url =
[20]

arXiv preprint arXiv:2210.02747 , year=

Flow matching for generative modeling , author=. arXiv preprint arXiv:2210.02747 , year=

arXiv
[21]

The International Journal of Robotics Research , volume=

Diffusion policy: Visuomotor policy learning via action diffusion , author=. The International Journal of Robotics Research , volume=. 2025 , publisher=

2025
[22]

The Thirteenth International Conference on Learning Representations , year=

Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning , author=. The Thirteenth International Conference on Learning Representations , year=
[23]

2026 , eprint=

Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns , author=. 2026 , eprint=

2026
[24]

Value-Based Deep

Oleh Rybkin and Michal Nauman and Preston Fu and Charlie Victor Snell and Pieter Abbeel and Sergey Levine and Aviral Kumar , booktitle=. Value-Based Deep. 2025 , url=

2025
[25]

The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous control , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=
[26]

2025 , url=

Tianyuan Chen and Ronglong Cai and Faguo Wu and Xiao Zhang , booktitle=. 2025 , url=

2025
[27]

Zhang, Yinmin and Liu, Jie and Li, Chuming and Niu, Yazhe and Yang, Yaodong and Liu, Yu and Ouyang, Wanli , title =. Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence an...

2024 doi
[28]

, title =

Daley, Brett and White, Martha and Machado, Marlos C. , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

2024
[29]

2024 , eprint=

Stop Regressing: Training Value Functions via Classification for Scalable Deep RL , author=. 2024 , eprint=

2024
[30]

2026 , eprint=

Q-learning with Adjoint Matching , author=. 2026 , eprint=

2026
[31]

arXiv preprint arXiv:1812.02648 , year=

Deep reinforcement learning and the deadly triad , author=. arXiv preprint arXiv:1812.02648 , year=

arXiv
[32]

Proceedings of the 1993 connectionist models summer school , pages=

Issues in using function approximation for reinforcement learning , author=. Proceedings of the 1993 connectionist models summer school , pages=. 2014 , organization=

1993
[33]

Advances in neural information processing systems , volume=

Double Q-learning , author=. Advances in neural information processing systems , volume=
[34]

Sur les op

Banach, Stefan , journal=. Sur les op. 1922 , publisher=

1922
[35]

2014 , publisher=

Markov decision processes: discrete stochastic dynamic programming , author=. 2014 , publisher=

2014
[36]

Generalized quantiles as risk measures , journal =

Fabio Bellini and Bernhard Klar and Alfred Müller and Emanuela. Generalized quantiles as risk measures , journal =. 2014 , issn =. doi:https://doi.org/10.1016/j.insmatheco.2013.10.015 , url =

2014 doi
[37]

2013 , month =

Boucheron, Stéphane and Lugosi, Gábor and Massart, Pascal , title =. 2013 , month =. doi:10.1093/acprof:oso/9780199535255.001.0001 , url =

2013
[38]

arXiv preprint arXiv:2005.01643 , year =

Levine, Sergey and Kumar, Aviral and Tucker, George and Fu, Justin , title =. arXiv preprint arXiv:2005.01643 , year =

2005 arXiv
[39]

Proceedings of the 36th International Conference on Machine Learning (ICML) , year =

Fujimoto, Scott and Meger, David and Precup, Doina , title =. Proceedings of the 36th International Conference on Machine Learning (ICML) , year =
[40]

Advances in Neural Information Processing Systems 32 (NeurIPS) , year =

Kumar, Aviral and Fu, Justin and Soh, Matthew and Tucker, George and Levine, Sergey , title =. Advances in Neural Information Processing Systems 32 (NeurIPS) , year =
[41]

Advances in Neural Information Processing Systems 33 (NeurIPS) , year =

Kumar, Aviral and Zhou, Aurick and Tucker, George and Levine, Sergey , title =. Advances in Neural Information Processing Systems 33 (NeurIPS) , year =
[42]

Advances in Neural Information Processing Systems 34 (NeurIPS) , year =

Fujimoto, Scott and Gu, Shixiang Shane , title =. Advances in Neural Information Processing Systems 34 (NeurIPS) , year =
[43]

International Conference on Learning Representations (ICLR) , year =

Chen, Xinyue and Wang, Che and Zhou, Zijian and Ross, Keith , title =. International Conference on Learning Representations (ICLR) , year =
[44]

International Conference on Learning Representations (ICLR) , year =

Bai, Chenjia and Wang, Lingxiao and Yang, Zhuoran and Deng, Zhihong and Garg, Animesh and Liu, Peng and Wang, Zhaoran , title =. International Conference on Learning Representations (ICLR) , year =
[45]

Advances in Neural Information Processing Systems 35 (NeurIPS) , year =

Ghasemipour, Seyed Kamyar Seyed and Gu, Shixiang Shane and Nachum, Ofir , title =. Advances in Neural Information Processing Systems 35 (NeurIPS) , year =
[46]

Offline Reinforcement Learning as Anti-Exploration , booktitle =

Rezaeifar, Shideh and Dadashi, Robert and Vieillard, Nino and Hussenot, L. Offline Reinforcement Learning as Anti-Exploration , booktitle =
[47]

Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

Nikulin, Alexander and Kurenkov, Vladislav and Tarasov, Denis and Kolesnikov, Sergey , title =. Proceedings of the 40th International Conference on Machine Learning (ICML) , year =
[48]

Proceedings of the 38th International Conference on Machine Learning (ICML) , year =

Jin, Ying and Yang, Zhuoran and Wang, Zhaoran , title =. Proceedings of the 38th International Conference on Machine Learning (ICML) , year =
[49]

Advances in Neural Information Processing Systems 34 (NeurIPS) , year =

Rashidinejad, Paria and Zhu, Banghua and Ma, Cong and Jiao, Jiantao and Russell, Stuart , title =. Advances in Neural Information Processing Systems 34 (NeurIPS) , year =
[50]

Proceedings of the 35th International Conference on Machine Learning (ICML) , year =

Oh, Junhyuk and Guo, Yijie and Singh, Satinder and Lee, Honglak , title =. Proceedings of the 35th International Conference on Machine Learning (ICML) , year =
[51]

Proceedings of the 37th International Conference on Machine Learning (ICML) , year =

Fedus, William and Ramachandran, Prajit and Agarwal, Rishabh and Bengio, Yoshua and Larochelle, Hugo and Rowland, Mark and Dabney, Will , title =. Proceedings of the 37th International Conference on Machine Learning (ICML) , year =
[52]

Revisiting

Kozuno, Tadashi and Tang, Yunhao and Rowland, Mark and Munos, R. Revisiting. Proceedings of the 38th International Conference on Machine Learning (ICML) , pages =
[53]

Advances in Neural Information Processing Systems 38 (NeurIPS) , year =

Park, Seohong and Frans, Kevin and Mann, Deepinder and Eysenbach, Benjamin and Kumar, Aviral and Levine, Sergey , title =. Advances in Neural Information Processing Systems 38 (NeurIPS) , year =
[54]

and Munos, R

Dabney, Will and Rowland, Mark and Bellemare, Marc G. and Munos, R. Distributional Reinforcement Learning with Quantile Regression , booktitle =
[55]

Statistics and Samples in Distributional Reinforcement Learning , booktitle =

Rowland, Mark and Dadashi, Robert and Kumar, Saurabh and Munos, R. Statistics and Samples in Distributional Reinforcement Learning , booktitle =
[56]

International Conference on Learning Representations (ICLR) , year =

Xu, Haoran and Jiang, Li and Li, Jianxiong and Yang, Zhuoran and Wang, Zhaoran and Chan, Victor Wai Kin and Zhan, Xianyuan , title =. International Conference on Learning Representations (ICLR) , year =
[57]

International Conference on Learning Representations (ICLR) , year =

Xiao, Chenjun and Wang, Han and Pan, Yangchen and White, Adam and White, Martha , title =. International Conference on Learning Representations (ICLR) , year =
[58]

and Zhou, Mingyuan , title =

Wang, Zhendong and Hunt, Jonathan J. and Zhou, Mingyuan , title =. International Conference on Learning Representations (ICLR) , year =
[59]

and Kumar, Vikash and Levine, Sergey and Finn, Chelsea , title =

Zhao, Tony Z. and Kumar, Vikash and Levine, Sergey and Finn, Chelsea , title =. Robotics: Science and Systems (RSS) , year =
[60]

arXiv preprint arXiv:2410.24164 , year =

Black, Kevin and Brown, Noah and Driess, Danny and Esmail, Adnan and Equi, Michael and Finn, Chelsea and others , title =. arXiv preprint arXiv:2410.24164 , year =

arXiv
[61]

Advances in Neural Information Processing Systems 36 (NeurIPS) , year =

Nakamoto, Mitsuhiko and Zhai, Yuexiang and Singh, Anikait and Mark, Max Sobol and Ma, Yi and Finn, Chelsea and Kumar, Aviral and Levine, Sergey , title =. Advances in Neural Information Processing Systems 36 (NeurIPS) , year =
[62]

and Smith, Laura and Kostrikov, Ilya and Levine, Sergey , title =

Ball, Philip J. and Smith, Laura and Kostrikov, Ilya and Levine, Sergey , title =. Proceedings of the 40th International Conference on Machine Learning (ICML) , year =
[63]

and Bellemare, Marc G

Agarwal, Rishabh and Schwarzer, Max and Castro, Pablo Samuel and Courville, Aaron C. and Bellemare, Marc G. , title =. Advances in Neural Information Processing Systems 34 (NeurIPS) , year =
[64]

Proceedings of the AAAI Conference on Artificial Intelligence , volume =

Henderson, Peter and Islam, Riashat and Bachman, Philip and Pineau, Joelle and Precup, Doina and Meger, David , title =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =
[65]

Proceedings of the AAAI Conference on Artificial Intelligence , volume =

van Hasselt, Hado and Guez, Arthur and Silver, David , title =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =
[66]

International Conference on Learning Representations (ICLR) , year =

Ghugare, Raj and Geist, Matthieu and Berseth, Glen and Eysenbach, Benjamin , title =. International Conference on Learning Representations (ICLR) , year =
[67]

Advances in Neural Information Processing Systems 37 (NeurIPS) , year =

Park, Seohong and Frans, Kevin and Levine, Sergey and Kumar, Aviral , title =. Advances in Neural Information Processing Systems 37 (NeurIPS) , year =
[68]

Atti del Congresso Internazionale dei Matematici, Bologna , volume =

Cantelli, Francesco Paolo , title =. Atti del Congresso Internazionale dei Matematici, Bologna , volume =

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Machine learning , volume=

Learning to predict by the methods of temporal differences , author=. Machine learning , volume=. 1988 , publisher=

1988

[2] [2]

Machine learning , volume=

Q-learning , author=. Machine learning , volume=. 1992 , publisher=

1992

[3] [3]

1998 , publisher=

Reinforcement learning: An introduction , author=. 1998 , publisher=

1998

[4] [4]

Proceedings of the AAAI conference on artificial intelligence , volume=

Rainbow: Combining improvements in deep reinforcement learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

[5] [5]

Proceedings of the AAAI conference on artificial intelligence , volume=

Multi-step reinforcement learning: A unifying algorithm , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

work page

[6] [6]

and Singh, Satinder P

Precup, Doina and Sutton, Richard S. and Singh, Satinder P. , title =. Proceedings of the Seventeenth International Conference on Machine Learning , pages =. 2000 , isbn =

work page 2000

[7] [7]

Advances in neural information processing systems , volume=

Safe and efficient off-policy reinforcement learning , author=. Advances in neural information processing systems , volume=

[8] [8]

International conference on machine learning , pages=

Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures , author=. International conference on machine learning , pages=. 2018 , organization=

2018