Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes a hybrid offline/online PPO policy with TWTL-based reward shaping and proves a monotonic improvement bound and optimal-policy preservation for delayed-reward tasks.

desk verdict The hybrid mixing and TWTL shaping idea is plausible, but the paper's two load-bearing theoretical guarantees—optimality preservation and the improvement bound—are wrong as written, and the experiments are too thin to rescue it. read the letter →

arxiv 2411.17861 v3 pith:MFAZRZIH submitted 2024-11-26 cs.LG cs.AI

Ahmad Ahmad , Mehdi Kermanshah , Kevin Leahy , Zachary Serlin , Ho Chit Siu , Makai Mann , Cristian-Ioan Vasile , Roberto Tron
show 1 more author
Calin Belta
This is my paper · ORCID
classification cs.LGcs.AI
keywords delayedrewardsproximalpolicyoptimizationofflinemixingTWTLrewardshapingtemporallogictaskpredictionimprovementbound
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Delayed rewards weaken the credit-assignment signal in PPO, so this paper tries to make PPO learn faster and better by adding two mechanisms: a policy that mixes a fixed offline policy (trained on demonstrations) with the online PPO policy, and a reward shaper built from Time Window Temporal Logic (TWTL). The central theoretical claim is that the mixed policy improves monotonically over both the offline policy and earlier iterations, with a worst-case gap of $(2\varsigma\gamma\alpha^2)/(1-\gamma)^2$, where $\alpha$ is the mixing weight and $\gamma$ the discount factor. The second claim is that the TWTL-based reward shaper preserves the optimal policy of the original problem, so the dense temporal-logic feedback does not distort the task. If both claims hold, practitioners can safely combine any offline or expert policy with PPO and add formal temporal-logic hints without losing the guarantee of converging to the task's true optimum. The experiments on inverted pendulum and lunar lander are offered as evidence that these mechanisms accelerate learning and improve final performance.

What carries the argument

Three objects carry the argument. First, the hybrid policy $\pi_\theta=(1-\alpha)\pi_\rho+\alpha\pi_\beta$ with learnable mixing weight $\alpha$ makes the offline policy a permanent prior and turns the choice of how much to trust it into a parameter of the network's final layer. Second, the $\alpha$-coupling and trust-region bound taken from TRPO bounds how far the mixed policy's return can deviate from the offline policy's return in terms of the expected advantage $\varsigma$ and the discount factor $\gamma$, producing the monotonic improvement guarantee. Third, the TWTL robustness degree $\varrho$ and the shaping function $F$ convert a temporal-logic task specification into dense per-step feedback via a predictor that completes the partial trajectory; the paper intends this shaping to be potential-based, so it cancels out of policy comparisons and preserves the original optimum.

What would settle it

Solve for a function $\Phi$ in the equation $\kappa\varrho(\mathrm{Pred}(x_t),\phi)-\varrho(\mathrm{Pred}(x_{t+1}),\phi)=\gamma\Phi(x_{t+1})-\Phi(x_t)$; coefficient matching forces $\gamma\kappa=1$, which is impossible for the paper's $0<\kappa<1$, $0<\gamma<1$. Running policy iteration on a small tabular MDP with this shaping and $\kappa\ne 1/\gamma$ would then show whether the optimal policy actually changes.

Watch

Extended reading notes

Core claim

The paper claims two theoretical results. First, for the policy $\pi_\theta(u|x)=(1-\alpha)\pi_\rho(u|x)+\alpha\pi_\beta(u|x)$ that mixes a frozen offline policy $\pi_\rho$ with a trainable online PPO policy $\pi_\beta$, the total return satisfies $\eta(\pi_\theta)\ge L_{\pi_\rho}(\pi_\theta)-2\varsigma\gamma\alpha^2/(1-\gamma)^2$, which implies the mixed policy never falls more than the stated gap below the offline policy and, for successive iterations, improves with each trust-region update. Second, the reward-shaping term $F(x_t,u_t,x_{t+1},\phi)=\kappa\,\varrho(\mathrm{Pred}(x_t),\phi)-\varrho(\mathrm{Pred}(x_{t+1}),\phi)$, where $\varrho$ is the TWTL robustness degree and $\mathrm{Pred}$ is a learned LSTM state predictor, is claimed to be a potential-based shaping function, so by the classical potential-based reward-shaping theorem the optimal policy of the shaped MDP equals that of the original MDP. Empirically, the paper reports that the combined method learns faster and reaches better final returns than vanilla PPO and offline-only baselines in LunarLander-v2 and Inverted Pendulum, even when the offline policy is intentionally degraded.

Load-bearing premise

The guarantee that the temporal-logic reward shaper does not change the task's optimal policy rests on the shaper being a difference of a single state-dependent score, the special form that makes it cancel out over full trajectories; the paper asserts this without showing the algebra, and the displayed formula does not have that form as written.

Editorial extensions

If this is right

  • A frozen offline policy can remain active throughout training, and the worst-case return loss relative to it is bounded by $2\varsigma\gamma\alpha^2/(1-\gamma)^2$, so the method will not silently collapse below the offline baseline.
  • Each trust-region update of the mixed policy is guaranteed to improve on the previous iteration, preserving PPO's monotonic-improvement spirit under the hybrid architecture.
  • TWTL reward shaping gives immediate feedback about temporally extended objectives, so the agent does not have to wait until the end of an episode to learn whether its behavior was good.
  • Because the shaping is claimed not to change the optimal policy, task knowledge encoded in TWTL can be added freely as a learning aid rather than as a separate, possibly conflicting objective.
  • The method tolerates poor offline policies: even a deliberately degraded expert policy accelerates online learning in the reported environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's optimality-preservation claim would hold if the shaping were literally $\gamma\Phi(x_{t+1})-\Phi(x_t)$; setting $\kappa=1/\gamma$ or redefining the potential is a direct fix, and a small tabular MDP test would confirm whether the optimal policy is preserved.
  • Because the mixing weight $\alpha$ is learned rather than fixed, the $\alpha$-coupling argument is applied at a snapshot; an extension would anneal $\alpha$ based on the relative performance of the offline and online policies and re-derive a time-varying bound.
  • The same task-prediction-plus-robustness construction should transfer to other quantitative temporal logics, such as signal temporal logic; the dense-feedback benefit would remain, but the optimality argument would need the potential-form condition verified case by case.
  • The empirical support comes only from two gymnasium benchmarks; a longer-horizon sparse-reward domain where the LSTM predictor's errors compound would test how much of the early-learning gain depends on prediction accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes two enhancements to PPO for environments with delayed rewards: a hybrid policy architecture that mixes a fixed offline policy πρ with an online PPO policy πβ through πθ = (1−α)πρ + απβ, and a reward-shaping mechanism based on Time Window Temporal Logic (TWTL) robustness using a learned state predictor. The main theoretical claims are that the hybrid policy improves over the offline policy and over previous iterations with a bounded performance gap of (2ςγα²)/(1−γ)², and that the TWTL-based reward shaping preserves the optimal policy of the original MDP. The paper also reports experiments on LunarLander and Pendulum environments.

Significance. If the theoretical guarantees were valid, the paper would offer a useful practical recipe for combining offline data, temporal-logic specifications, and policy-gradient training. The TWTL formulation is concrete, and the architectural idea of maintaining an offline policy as an active mixing component is reasonable. However, the central guarantees are not established: the optimality-preservation lemma is mathematically false as stated, and the improvement-bound proof contains algebraic and conceptual errors. The experimental section is also too thin to compensate, with two environments and no reported statistical variability. The paper does not ship code or machine-checked proofs.

major comments (5)
  1. [§4.2, Eq. (11), Lemma 4] Lemma 4 is unsupported, and its assertion is false for the stated parameter range. Ng et al.'s potential-based shaping theorem requires F(s,a,s′) = γΦ(s′)−Φ(s) in a discounted MDP. The paper defines F(x_t,u_t,x_{t+1},φ) = κ·ϱ(Pred(x_t),φ) − ϱ(Pred(x_{t+1}),φ) with 0<κ<1 and 0<γ<1. Coefficient matching forces κγ=1, which is impossible in the stated range. The proof merely labels F a potential function without verifying the identity, so the appeal to Theorem 1 of Ng et al. (1999) does not go through. In addition, the return in Eq. (1) is finite-horizon and undiscounted, whereas the Ng et al. theorem is for discounted MDPs; no extension is provided.
  2. [§4.3, Eqs. (19)–(24), Proposition 7] The proof of Proposition 7 is internally inconsistent. Eq. (19) defines Lπρ(πθ) as η(πθ)+E_{u∼πθ}[Σ_i ¯A_i(x)], but Eq. (14) defines Lπρ(π) with η(πρ), not η(πθ). The subtraction leading to Eq. (22) is therefore algebraically invalid. The proof then replaces the undiscounted sum in Eq. (23) with the discounted sum Σ_i γ^i ·4α(1−(1−α)^i)·ς without any justification, despite the fact that the paper's value functions are undiscounted finite-horizon sums. Finally, the constants do not match: the theorem statement gives 2ςγα²/(1−γ)², while the derivation in Eq. (24) obtains 4εα/(1−γ)² with ε never defined. The claimed bound is not proven.
  3. [§4.3, Eq. (21)] The inequality chain in Eq. (21) is not a valid derivation from Lemma 6. It writes E_{u∼πθ}[¯A_i(x_i)] − E_{u∼πρ}[¯A_i(x_i)] ≤ 2α ¯A_i(x_i) ≤ 4α(1−(1−α)^i)·max_{x∈X} A^{πβ,k}_i(x_i,u_i), but ¯A_i(x_i) is itself an expectation, and the inequality is applied as if it were a pointwise random quantity. The outer bound in terms of max_x A^{πβ,k}_i also omits absolute values, which are needed for the subsequent bound on |η(πθ)−Lπρ(πθ)|. No rigorous coupling argument connecting Eq. (12) to the α-coupled condition (20) is supplied beyond the informal 'probability of event' statement.
  4. [§4.3, Theorem 8] Theorem 8, which is the paper's iterative-improvement guarantee, is not proven. Its proof states that it follows 'the same logical structure as Theorem 6', but no Theorem 6 appears in the paper (there is Lemma 6 and Proposition 7). More substantively, the proof asserts that the α-coupling argument can be replaced by the total-variation quantity TV_{θ_i}^{θ_{i+1}} without deriving the analogue of Lemma 6 for this quantity. Since the iterative improvement over previous iterations is a central claimed contribution, this omission is load-bearing.
  5. [§4.2, Lemma 4 and §4.4, Algorithm 1] Lemma 4 concerns the shaped reward r'_{φ,t} = r_{φ,t} + F, but F is evaluated on Pred(x_t), the output of a learned LSTM predictor, rather than on the true future trajectory. Even if the ideal shaping term had the potential-based form, replacing the true trajectory with a learned prediction changes the reward actually used for policy optimization and can alter the optimal policy. No assumption or bound on the predictor error appears in the lemma or its proof, so the optimality-preservation claim is not valid for the implemented algorithm.
minor comments (6)
  1. [Section 1, Paper structure] The paper-structure paragraph says 'Sec. 4' twice and omits the actual Sec. 3 from the outline; the section numbers should be corrected.
  2. [Eq. (15)] The definition of ¯A_i(x) is confusing: the left-hand side has index i, while the right-hand side writes E_{u_i∼πθ}[Σ_{i=1}^N A^{πρ}_i], reusing i as both a free index and a summation index.
  3. [Eq. (16)] Lemma 6 bounds the advantage of πθ relative to πρ, but Eq. (16) states the bound in terms of max |A^{πβ}_0|; the base policy in the advantage should be πρ for consistency with the subsequent definition of ς.
  4. [Algorithm 1] Algorithm 1 is underspecified: the order of action collection, advantage computation, and policy optimization inside an episode is unclear, and the loop count N is reused for both the horizon and the number of iterations.
  5. [Figure 2 and Section 5] The experimental comparison reports no seeds, confidence intervals, hyperparameters, or ablations; single learning curves are not sufficient to support the claimed 'faster learning and better asymptotic performance'.
  6. [Definition 3] Definition 3 contains a corrupted subscript in 'u_{i−1,i+||φ||−1} := u_{i−1}u_t...u_{N−i+||φ||−1}'; the action sequence should be written with consistent indices.

Circularity Check

1 steps flagged · score 6.0 of 10

The reward-shaping optimality guarantee is obtained by labeling Eq. (11) a potential function and importing Ng et al.'s conclusion, without verifying the potential condition; the hybrid-policy improvement bound is an external TRPO-style derivation and is not circular.

  1. ansatz smuggled in via citation [Lemma 4 and Eq. (11), Section 4.2]
    "F (xt, ut, xt+1, ϕ) := κ · ϱ(Pred(xt), ϕ) − ϱ(Pred(xt+1), ϕ) (11) ... By Theorem 1 in Ng et al. (1999), F being a potential function, as defined in (11), guarantees optimal policy consistency."

    Ng et al.'s theorem applies only when F(s,a,s') = γΦ(s') − Φ(s). For (11), setting Φ(x) = −κϱ(Pred(x), ϕ) + C gives γΦ(x_{t+1}) − Φ(x_t) = κϱ(Pred(x_t), ϕ) − γκϱ(Pred(x_{t+1}), ϕ) + (γ−1)C, which matches (11) only if γκ = 1. Since 0<κ<1 and γ∈(0,1), this is impossible. The lemma therefore does not prove that (11) is a potential function; it simply names it one and reads off optimality preservation from Ng et al. The central reward-shaping guarantee is assumed by labeling rather than derived from the stated F. Ng et al.'s theorem is also for discounted MDPs, while the paper's value functions (1) are undiscounted.

full rationale

The only genuinely circular step is Lemma 4: the proof imports the optimality-preservation conclusion by calling Eq. (11) a potential function, and coefficient matching shows the necessary potential identity cannot hold for the stated 0<κ<1 and γ∈(0,1). This makes one of the paper's two headline guarantees an assertion rather than a derivation. The hybrid-policy improvement bound, by contrast, is a direct adaptation of external TRPO/Kakade-Langford results with the mixing coefficient α playing the role of the coupling parameter; whatever algebraic slips it contains, it does not presuppose its own conclusion. The TWTL formalism and robustness definitions are stated in the paper, and the self-citations to Vasile et al. and Ahmad et al. provide definitions rather than load-bearing uniqueness claims. So the circularity is partial but central: the reward-shaping guarantee is effectively assumed by construction, while the policy-mixing guarantee has independent external content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the potential-function assertion for the shaping term, on the predictor's accuracy, and on the standard TRPO machinery. The learnable mixing parameter α and shaping weight κ are hand-chosen or learned without error analysis. No new physical entities are introduced.

free parameters (2)
  • κ = unspecified (declared 0<κ<1)
    Reward shaping weight in Eq. (11); chosen by hand, no value or sensitivity analysis reported.
  • Offline policy degradation noise = unspecified
    Experiments create offline policies by adding 'controlled noise' to pre-trained PPO parameters; the noise level is not reported, so the experimental condition is not reproducible.
assumptions (4)
  • ad hoc to paper The reward shaping function (11) is a potential-based shaping function in the sense of Ng et al. (1999).
    Lemma 4 asserts this without verifying that Eq. (11) can be written as γΦ(s') - Φ(s); the actual expression has the opposite placement of γ and κ.
  • domain assumption The observation predictor Pred produces sequences such that ϱ(Pred(x_t),φ) defines a valid state-dependent potential.
    The shaping uses predicted future observations; no error bound or consistency condition on the predictor is given, so the optimality theorem does not extend to imperfect prediction.
  • standard math The Kakade-Langford/TRPO policy improvement bound applies unchanged to the mixed policy.
    Lemma 5 relies on Lemma 1 of Schulman et al. 2015a; standard result, though the subsequent algebra in Proposition 7 is not a faithful application.
  • standard math The total variation coupling inequality between πθ and πρ holds with the stated α.
    Used in Proposition 7 and Theorem 8; the coupling probability calculation is sketched but the inequality chain is incomplete.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards." pith.science (2026). https://pith.science/paper/MFAZRZIH

@misc{pith2026241117861,
  author       = {Pith},
  title        = {Pith review of: Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MFAZRZIH}},
  note         = {Machine review of arXiv:2411.17861}
}
abstract

In this paper, we tackle the challenging problem of delayed rewards in reinforcement learning (RL). While Proximal Policy Optimization (PPO) has emerged as a leading Policy Gradient method, its performance can degrade under delayed rewards. We introduce two key enhancements to PPO: a hybrid policy architecture that combines an offline policy (trained on expert demonstrations) with an online PPO policy, and a reward shaping mechanism using Time Window Temporal Logic (TWTL). The hybrid architecture leverages offline data throughout training while maintaining PPO's theoretical guarantees. Building on the monotonic improvement framework of Trust Region Policy Optimization (TRPO), we prove that our approach ensures improvement over both the offline policy and previous iterations, with a bounded performance gap of $(2\varsigma\gamma\alpha^2)/(1-\gamma)^2$, where $\alpha$ is the mixing parameter, $\gamma$ is the discount factor, and $\varsigma$ bounds the expected advantage. Additionally, we prove that our TWTL-based reward shaping preserves the optimal policy of the original problem. TWTL enables formal translation of temporal objectives into immediate feedback signals that guide learning. We demonstrate the effectiveness of our approach through extensive experiments on an inverted pendulum and a lunar lander environments, showing improvements in both learning speed and final performance compared to standard PPO and offline-only approaches.

Figures

Figures reproduced from arXiv: 2411.17861 by the authors.

Figure 1
Figure 1. The Actor-Critic RL framework. The Actor’s architecture, policy πθ, consists of an offline pol￾icy, πρ, and an adaptive policy, πβ, where the two policies are mixed using the parameters of the FCL, α. The critic consists of an MLP that approximates the value function. The task pre￾dictor LSTM network and the reward shaping are depicted in cyan. 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Training Steps 1e6 200 100 … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ANSR-DT: A Neuro-Symbolic Framework for Adaptive and Explainable Digital Twins

    cs.AI 2025-01 reject novelty 5.0 of 10

    ANSR-DT combines neural detection, symbolic rules, and reinforcement learning into one digital twin framework, reporting high accuracy on synthetic data and claiming transfer to the SKAB benchmark.

Reference graph

Works this paper leans on

31 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Robustness measures and monitors for time window temporal logic

    Ahmad Ahmad, Cristian-Ioan Vasile, Roberto Tron, and Calin Belta. Robustness measures and monitors for time window temporal logic. In 2023 62nd IEEE Conference on Decision and Control (CDC), pages 6841--6846. IEEE, 2023

    work page 2023

  2. [2]

    Q-learning for robust satisfaction of signal temporal logic specifications

    Derya Aksaray, Austin Jones, Zhaodan Kong, Mac Schwager, and Calin Belta. Q-learning for robust satisfaction of signal temporal logic specifications. In 2016 IEEE 55th Conference on Decision and Control (CDC), pages 6565--6570. IEEE, 2016

    work page 2016

  3. [3]

    u diger Ehlers, Bettina K \

    Mohammed Alshiekh, Roderick Bloem, R \"u diger Ehlers, Bettina K \"o nighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    work page 2018

  4. [4]

    Temporal-logic-constrained hybrid reinforcement learning to perform optimal aerial monitoring with delivery drones

    Ahmet Semi Asarkaya, Derya Aksaray, and Yasin Yaz c o g lu. Temporal-logic-constrained hybrid reinforcement learning to perform optimal aerial monitoring with delivery drones. In 2021 International Conference on Unmanned Aircraft Systems (ICUAS), pages 285--294. IEEE, 2021

    work page 2021

  5. [5]

    Principles of model checking

    Christel Baier and Joost-Pieter Katoen. Principles of model checking. MIT press, 2008

    work page 2008

  6. [6]

    Structured reward shaping using signal temporal logic specifications

    Anand Balakrishnan and Jyotirmoy V Deshmukh. Structured reward shaping using signal temporal logic specifications. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3481--3486. IEEE, 2019

    work page 2019

  7. [7]

    Efficient online reinforcement learning with offline data

    Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577--1594. PMLR, 2023

    2023

  8. [8]

    Overcoming exploration: Deep reinforcement learning for continuous control in cluttered environments from temporal logic specifications

    Mingyu Cai, Erfan Aasi, Calin Belta, and Cristian-Ioan Vasile. Overcoming exploration: Deep reinforcement learning for continuous control in cluttered environments from temporal logic specifications. IEEE Robotics and Automation Letters, 8 0 (4): 0 2158--2165, 2023

    work page 2023

Show all 31 references
  1. [9]

    Provably efficient exploration in policy optimization

    Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, page...

    2020

  2. [10]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 0 15084--15097, 2021

    2021

  3. [11]

    Mirror learning: A unifying framework of policy optimisation

    Jakub Grudzien, Christian A Schroeder De Witt, and Jakob Foerster. Mirror learning: A unifying framework of policy optimisation. In International Conference on Machine Learning, pages 7825--7844. PMLR, 2022

    2022

  4. [12]

    Imitation bootstrapped reinforcement learning

    Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation bootstrapped reinforcement learning. arXiv preprint arXiv:2311.02198, 2023

    2023 arXiv

  5. [13]

    A tutorial on mm algorithms

    David R Hunter and Kenneth Lange. A tutorial on mm algorithms. The American Statistician, 58 0 (1): 0 30--37, 2004

    2004

  6. [14]

    Reward machines: Exploiting reward function structure in reinforcement learning

    Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A McIlraith. Reward machines: Exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research, 73: 0 173--208, 2022

    2022

  7. [15]

    Approximately optimal approximate reinforcement learning

    Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267--274, 2002

    2002

  8. [16]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 1179--1191, 2020

    2020

  9. [17]

    Reinforcement learning with temporal logic rewards

    Xiao Li, Cristian-Ioan Vasile, and Calin Belta. Reinforcement learning with temporal logic rewards. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3834--3839. IEEE, 2017

    2017

  10. [18]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  11. [19]

    Advice-guided reinforcement learning in a non-markovian environment

    Daniel Neider, Jean-Raphael Gaglione, Ivan Gavran, Ufuk Topcu, Bo Wu, and Zhe Xu. Advice-guided reinforcement learning in a non-markovian environment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9073--9080, 2021

    2021

  12. [20]

    Policy invariance under reward transformations: Theory and application to reward shaping

    Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278--287, 1999

    1999

  13. [21]

    Implicit human perception learning in complex and unknown environments

    Amirhossein Ravari, Seyede Fatemeh Ghoreishi, and Mahdi Imani. Implicit human perception learning in complex and unknown environments. In American Control Conference (ACC), 2024

    2024

  14. [22]

    Agnostic system identification for model-based reinforcement learning

    St \'e phane Ross and J Andrew Bagnell. Agnostic system identification for model-based reinforcement learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 1905--1912, 2012

    1905

  15. [23]

    Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data

    Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XVIII 16, pages...

    2020

  16. [24]

    Kickstarting deep reinforcement learning

    Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835, 2018

    2018 arXiv

  17. [25]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889--1897. PMLR, 2015 a

    2015

  18. [26]

    High-dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015 b

    2015 arXiv

  19. [27]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  20. [28]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  21. [29]

    Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, Andrew Copplestone-Bruce, Gianluca Lutati, Rousslan Fernand Julien Dossa, Antonin Famularo, Filip Pytlak, Marc Ballester, Hao Guo, Jiayi Hu, and Mikayel Awad. Gymnasium. https://github.com/Farama-Foundation/Gymnasium, 2023. URL ...

    2023

  22. [30]

    Time window temporal logic

    Cristian-Ioan Vasile, Derya Aksaray, and Calin Belta. Time window temporal logic. Theoretical Computer Science, 691: 0 27--54, 2017

    2017

  23. [31]

    Joint inference of reward machines and policies for reinforcement learning

    Zhe Xu, Ivan Gavran, Yousef Ahmad, Rupak Majumdar, Daniel Neider, Ufuk Topcu, and Bo Wu. Joint inference of reward machines and policies for reinforcement learning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 590--598, 2020

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.