Pith. sign in

REVIEW 3 major objections 5 minor 90 references

Recursive Reward Aggregation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that the standard discounted-sum Bellman equations generalize to any recursively computed reward aggregation—max, min, mean, variance, and Sharpe ratio—so RL agents can optimize these objectives directly without touching…

desk verdict Elegant fold-based framework for policy evaluation, but the Bellman optimality theorem fails for non-monotone aggregations like variance; the value-based optimization claims are overbroad. read the letter →

arxiv 2507.08537 v2 pith:2LZ4GKND submitted 2025-07-11 cs.LG math.CT

Yuting Tang , Yivan Zhang , Johannes Ackermann , Yu-Jie Zhang , Soichiro Nishimori , Masashi Sugiyama This is my paper
classification cs.LGmath.CT MSC 90C40
keywords recursiverewardaggregationBellmanequationalgebrafusiondynamicprogrammingSharperatiodistributionalreinforcementlearningMarkovdecisionprocess
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the Bellman equations of reinforcement learning are not special to the discounted sum of rewards: they hold for any objective that can be computed by folding a reward sequence with a fixed initial value and update rule. The authors introduce "recursive statistic aggregation," in which a statistic (a single number or a small vector) is updated one reward at a time, and show that composing this fold with the recursive generation of rewards yields a generalized Bellman equation. That means value-based and actor-critic algorithms can directly optimize objectives such as discounted max, discounted min, mean, range, variance, and the Sharpe ratio, without changing the reward function or enlarging the state space. If correct, this shifts goal specification in RL from "what to reward" to "how to evaluate reward sequences," a cheaper and more flexible design axis.

What carries the argument

The central object is the recursive statistic aggregation $\mathrm{agg}_{\mathrm{init},\triangleright}$: a fold over reward lists defined by an initial statistic $\mathrm{init} \in T$, an update function $\triangleright : R \times T \to T$, and a post-processing function $\mathrm{post} : T \to R$ that extracts a scalar value (Definition 3.1). The machinery that carries the argument is algebra fusion: composing the coalgebra that generates rewards one step at a time (the step function $\mathrm{step}_\pi$) with this fold as a catamorphism collapses the two recursions into one, yielding the Bellman equation $\tau_\pi(s) = r_\pi(s) \triangleright \tau_\pi(p_\pi(s))$ for the state statistic function. This single recursive identity lets a Q-learning, PPO, or TD3 update target compute any listed aggregation by replacing the scalar "discounted addition" with the appropriate update rule.

What would settle it

In a two-step MDP where the objective is variance (or a variance-penalized sum), compute the optimal trajectory by exhaustive enumeration and compare it with the policy produced by the Bellman optimality update $\tau_*(s) = \sup_a (r(s,a) \triangleright \tau_*(p(s,a)))$ using the paper's variance update; different outcomes would show Theorem 3.8 does not hold for that aggregation.

Watch

Extended reading notes

Core claim

The central claim is that the Bellman equation is a fusion of two recursive structures: an MDP generates a reward list recursively through a step function, and a recursive statistic aggregation consumes that list through an update function $\triangleright$ and an initial statistic. For any such aggregation (Definition 3.1), the composed state statistic function $\tau_\pi$ satisfies $\tau_\pi(s) = \mathrm{init}$ at terminal states and $\tau_\pi(s) = r_\pi(s) \triangleright \tau_\pi(p_\pi(s))$ elsewhere (Theorem 3.2), and the analogous optimality equation holds with a supremum over actions (Theorem 3.8). This reproduces the standard discounted sum while also covering discounted max, discounted min, log-sum-exp, range, mean, variance, and top-k, each with an explicit update rule in Table 1; the Sharpe ratio follows because it is post-processed from recursively maintained mean and variance statistics. The paper further shows the generalized Bellman operator has unique fixed points when the update function is contractive (Theorem 3.6), and it extends the recursion to stochastic transitions as a distributional Bellman equation.

Load-bearing premise

Picking the best-looking next step at every state is assumed to add up to the best whole trajectory, but that holds only when the update rule and the way statistics are compared are aligned (order-preserving), which is not guaranteed for objectives like variance.

Editorial extensions

If this is right

  • Value-based algorithms (Q-learning, DQN) and actor-critic algorithms (PPO, TD3) can optimize any objective expressible as a recursive aggregation by swapping the Bellman update, with no change to state space or reward function.
  • Objectives that previously required state augmentation or approximate reward shaping—discounted max, worst-case min, mean, range, and variance-regularized sums—become direct training objectives with the same per-step computational cost.
  • The Sharpe ratio can be optimized exactly, rather than through an approximate differential surrogate, by maintaining mean and variance statistics recursively; the portfolio experiment suggests this improves test Sharpe ratio.
  • In stochastic settings, the recursion yields a distributional Bellman equation for any recursive aggregation, so distributional RL methods extend beyond discounted sums.
  • The contractive-update condition (Theorem 3.6) guarantees uniqueness of the fixed point of the generalized Bellman operator, supporting iterative policy evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of Theorem 3.8's scope would compare the policy from the Bellman optimality update against an exhaustive trajectory search in a small MDP for a non-monotone objective like variance; a mismatch would show the pointwise-maximization step requires an order-preservation assumption the paper does not state.
  • The framework opens a design axis where the aggregation function itself could be learned from data (e.g., a parameterized $\triangleright$), rather than hand-picked, which the paper lists only as future work.
  • Non-recursive objectives like the median and CVaR fall outside the exact framework (the paper says so); sketching approximations might still combine with the same Bellman structure, but the per-step approximation error would need to be controlled.
  • The same fold-and-fuse lens applies beyond rewards to any per-step quantity—costs, constraint violations, or information signals—potentially unifying risk measures and constrained control within one Bellman-style recursion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an algebraic framework for reinforcement learning in which the standard discounted-sum return is replaced by a recursive reward aggregation defined by an initial statistic init, an update function ▷, and a post-processing function post. The central claim is that composing a recursively generated reward sequence with such an aggregation yields a generalized Bellman equation for the state statistic function (Theorem 3.2), that fixed points of the associated Bellman operator are unique under a contractiveness condition (Theorem 3.6), and that an analogous Bellman optimality equation holds for optimal policies (Theorem 3.8). The authors extend the framework to state-action statistics, stochastic transitions, and actor-critic and value-based algorithms, and they report experiments in grid-world, continuous control, and portfolio optimization, including optimization of discounted max, min, mean, variance-regularized sum, and Sharpe ratio objectives.

Significance. The policy-evaluation side of the paper is elegant and potentially useful: the fusion perspective gives a clean derivation of a generalized Bellman equation for a large class of recursive aggregations, and the fixed-point uniqueness result is a natural generalization of the classical contraction argument. The paper also makes a welcome effort to provide explicit recursive formulations, a table of aggregations, and empirical validation, all of which contribute to reproducibility. However, the policy-optimization claim is not supported as stated: Theorem 3.8's Bellman optimality equation is false for non-monotone aggregations such as variance, mean, range, and Sharpe ratio, which are precisely among the paper's advertised examples. This is a load-bearing gap because value-based algorithms (Q-learning, DQN) and the grid-world experiments rely on that theorem. The stochastic/portfolio claims are also weaker than stated because the PPO implementation optimizes aggregated expected rewards rather than expected aggregated rewards.

major comments (3)
  1. [Section 3.3, Theorem 3.8 and Eq. (12)] The step labelled 'pointwise maximization' in the proof (the line marked as (99) in Appendix F) is valid only if the update function satisfies a monotonicity condition with respect to the preorder ≤T: for every reward r, τ1 ≤T τ2 must imply r ▷ τ1 ≤T r ▷ τ2. This assumption is not stated in Theorem 3.8, and it fails for several aggregations in Table 1, including variance, mean, range, and Sharpe ratio. Concretely, consider the variance aggregation with lower variance preferred, so that sup in the preorder means minimum variance. Let states s, u, v1, v2 with transitions s→u (reward 0), u→v1 (reward 100) then v1→T (reward −100), and u→v2 (reward 1000) then v2 terminal. From u, the suffixes [100,−100] and [1000] have variances 10,000 and 0, respectively, so [1000] is better at u; from s, the full trajectories [0,100,−100] and [0,1000] have variances 6,666.7 and 250,000, respectively, so [100,−100] is better at s. Hence no stationary policy is optimal at both states, τ* is not well-defined, and Eq. (12) fails because the right-hand side using the optimal statistic at u gives variance 250,000 rather than the true optimal value 6,666.7. The theorem can be repaired by adding the missing monotonicity assumption and restricting the value-based claims to aggregations that satisfy it, but as stated the theorem is false.
  2. [Section 4, Appendix H.2, and Section 5.4] The stochastic and portfolio claims are weaker than the text suggests. Equation (13) is a distributional recursion for the distribution of the aggregated statistic, but the PPO implementation in Algorithm 2 uses a point-valued critic and sampled transitions, which computes the statistic of expected rewards (aggregated expected rewards) rather than the expected aggregated statistic. The paper itself acknowledges in Appendix H.2 that this differs from the expected aggregated rewards. Therefore the statement in Section 5.4 that the method estimates and optimizes the exact Sharpe ratio over the entire trajectory is not justified for the stochastic PPO setting: expectation does not commute with the Sharpe ratio, so the optimized objective is not the expected Sharpe ratio. The deterministic policy-evaluation results are unaffected, but the stochastic and portfolio claims need to be reworded and the experiments re-described accordingly.
  3. [Appendix G.1, Algorithm 1] The Q-learning update τ(s,a) ← τ(s,a) + α(max_{a′} r ▷ τ(s′,a′) − τ(s,a)) is not well-defined for most of the statistics in Table 1. For the mean aggregation, T = N × R, so τ(s,a) is a pair (length, sum) and the expression τ(s,a) + α(·) is undefined; similar problems arise for variance, range, and top-k buffers. The paper does not specify the vector-space structure on T or an alternative update for these cases. As written, Algorithm 1 supports only aggregations whose statistic space is a vector space, such as the discounted sum. The grid-world experiments in Section 5.1 therefore need either a concrete definition of the statistic space and the addition operation used in the implementation, or a different update rule.
minor comments (5)
  1. [Appendix A.3, Theorem A.1] The theorem statement says the state-action statistic function has type τ S×Aπ : S → T, but the equation below it and the surrounding text use S × A → T; the domain should be S × A.
  2. [Definition 3.1] There is a typo in the definition: 'a recursive statistic aggregation function agginit,▷ : [R] → T of is defined' should read 'is defined'.
  3. [Appendix F, Lemma D.7] The proof of Lemma D.7 uses arg sup_{a∈A} f(a), which need not be unique; the argument can be repaired by choosing witnesses or by proving the two inequalities directly without naming an argmax.
  4. [Section 4, Eq. (13)] In Eq. (13), the notation r ▷ τ′ is used with r and τ′ as random variables, but the joint distribution and measurability conditions on the stochastic update are not specified; please define the pushforward of the product measure explicitly.
  5. [Table 1 and Section 5.4] The Sharpe ratio is advertised as a key example, but Table 1 does not contain an explicit Sharpe-ratio row; the paper only says that mean and variance are recursive. Adding the explicit statistic, update function, and post-processing for the Sharpe ratio would make the example precise.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the Bellman recursion is a direct consequence of the recursive aggregation definition, and the main theoretical gap is a missing monotonicity assumption, not a circular fit.

full rationale

The paper's central derivation is not circular in the sense that matters. Theorem 3.2 (Eq. 10) is obtained by composing the recursive reward generation function (Eq. 4) with the recursive aggregation function (Definition 3.1, Eq. 8); the proof in Appendix F explicitly unfolds gen and agg, so the Bellman equation for the state statistic function is a definition-tracking fold/fusion result rather than a fitted parameter renamed as a prediction. No term in the Bellman equation is estimated from data, and the portfolio experiment evaluates the Sharpe ratio on a separate test period against external baselines (DiffSharpe, NCMDP), so the empirical claims are not self-confirming. The self-citations present (e.g., Morimura et al. 2010a,b and Ackermann et al. 2024) are contextual and are not load-bearing for the main claim. The one serious issue is Theorem 3.8: the proof's 'pointwise maximization' step (Eq. 99) requires that r(s,a) ▷ tau*(p(s,a)) be monotone with respect to the preorder on statistics, an assumption that is not stated and that fails for variance/mean-type aggregations; however, a missing assumption or false theorem is a correctness concern, not circularity. Accordingly, the circularity pass finds no reduction of a claimed prediction to its own inputs, and the score is low, reflecting only minor non-load-bearing self-citations.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The framework itself has no fitted free parameters; experimental weights such as the 0.2 variance penalty in wind-world are task-specific hyperparameters. The central derivation relies on the fusion law (standard mathematics), the contractiveness and monotonicity assumptions on the update function (the latter unstated), and the domain assumption that objectives are computable as single-pass recursive folds. No new entities (particles, forces, dimensions) are introduced.

assumptions (4)
  • standard math Fusion law for recursive coalgebras/algebras (Hinze et al. 2010, Section 4)
    Used to derive the Bellman equation in Theorem 3.2 from the definitions of recursive generation and aggregation.
  • domain assumption Contractiveness of the update function ▷ with respect to a premetric dT (Definition 3.5)
    Needed for uniqueness of fixed points (Theorem 3.6); not all aggregations in Table 1 are shown to satisfy it under a single premetric.
  • ad hoc to paper Monotonicity (order-preservation) of ▷ with respect to the preorder <=T for the Bellman optimality equation
    Needed for the 'pointwise maximization' step in the proof of Theorem 3.8 (Appendix F); not stated in the theorem but required for the greedy sup to yield the optimal policy.
  • domain assumption Objectives expressible as recursive aggregations with bounded-size accumulators
    Stated in limitations: median and semivariance are excluded because they cannot be computed with a bounded-size accumulator in a single pass.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Recursive Reward Aggregation." pith.science (2026). https://pith.science/paper/2LZ4GKND

@misc{pith2026250708537,
  author       = {Pith},
  title        = {Pith review of: Recursive Reward Aggregation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2LZ4GKND}},
  note         = {Machine review of arXiv:2507.08537}
}
read the original abstract

In reinforcement learning (RL), aligning agent behavior with specific objectives typically requires careful design of the reward function, which can be challenging when the desired objectives are complex. In this work, we propose an alternative approach for flexible behavior alignment that eliminates the need to modify the reward function by selecting appropriate reward aggregation functions. By introducing an algebraic perspective on Markov decision processes (MDPs), we show that the Bellman equations naturally emerge from the recursive generation and aggregation of rewards, allowing for the generalization of the standard discounted sum to other recursive aggregations, such as discounted max and Sharpe ratio. Our approach applies to both deterministic and stochastic settings and integrates seamlessly with value-based and actor-critic algorithms. Experimental results demonstrate that our approach effectively optimizes diverse objectives, highlighting its versatility and potential for real-world applications.

Figures

Figures reproduced from arXiv: 2507.08537 by the authors.

Figure 1
Figure 1. Illustration of three deterministic policies in a simple environment, shown as colored paths [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. By combining the recursive generation and aggregation of rewards, we can express the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The recursive structures of (left) the expected discounted sum of rewards E[r +γ s] and (right) the discounted sum of expected rewards E[r] +γ E[s]. Stochastic aggregation functions Note that this framework also accommodates the traditional expected discounted sum of rewards E hPΩ t=1 γ t−1 rt i learning objective, by selecting δ0 as init, the (pushforward through) discounted addition function +γ : R × R → R as the … view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Grid-world: Fig. 4a shows the discrete environment and the reward function r(s, a), where the agent starts from the top-left corner • and needs to reach the goal at the bottom-right corner ■. Figs. 4b to 4e show the optimal state-action value functions q∗ (s, a) under …
Figure 5
Figure 5. Figure 5: Wind-world: Fig. 5a shows the continuous environment, where the agent encounters wind disturbances (visualized with streamlines) and receives higher rewards near the center (depicted with colored contours). Fig. 5b illustrates the trajectories of agents trained using d…
Figure 6
Figure 6. Figure 6: Lunar Lander Continuous: Comparison of five reward aggregation methods. (Left) Radar plot showing performance across seven evaluation metrics, averaged over four random seeds. (Right) Sample trajectories illustrating the qualitative behaviors induced by each aggregatio…
Figure 7
Figure 7. Figure 7: State statistic bidirectional process τ S π : S → T S T R S T . . . . . . {∗} T π p πθ r ▷ init st st+1 τt τt+1 rt+1 [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: State statistic bidirectional process (with different behavior and target policies) [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]
Figure 9
Figure 9. Figure 9: State statistic bidirectional process (with state as the residual) [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]
Figure 10
Figure 10. Figure 10: State-action statistic bidirectional process [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: State-action statistic bidirectional process (with different behavior and target policies) [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: State-action statistic bidirectional process (with state-action as the residual) [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: τ S π = τ S×A π ◦ ⟨idS, π⟩ and τ S×A π = r ▷ (τ S π ◦ p) Theorem A.2 (Relationship between state and state-action statistic functions). Given a recursive reward generation function genπ,p,r,ω (Definition 2.1) and a recursive statistic aggregation function agginit,▷ (D…
Figure 14
Figure 14. Figure 14: max − λ range = λ min + (1 − λ) max H Experiments In this section, we provide detailed descriptions of the environments used in our experiments and the specific configurations and hyperparameters employed for each task. We also present additional results for the grid-…
Figure 15
Figure 15. Figure 15: Hopper: Comparison of five reward aggregation methods. (Left) Radar plot showing performance across six evaluation metrics, averaged over four random seeds. (Right) Sample trajectories illustrating the qualitative behaviors induced by each aggregation method. max!.## …
Figure 16
Figure 16. Figure 16: Ant: Comparison of five reward aggregation methods. (Left) Radar plot showing performance across six evaluation metrics, averaged over four random seeds. (Right) Sample trajectories illustrating the qualitative behaviors induced by each aggregation method [PITH_FULL_…
Figure 17
Figure 17. Figure 17: A year-by-year comparison of Sharpe ratios obtained by different methods during the test [PITH_FULL_IMAGE:figures/full_fig_p052_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

90 extracted references · 36 canonical work pages

  1. [1]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, 2017. URL https://proceedings.mlr.press/v70/achiam17a.html

    2017

  2. [2]

    Offline reinforcement learning from datasets with structured non-stationarity

    Johannes Ackermann, Takayuki Osa, and Masashi Sugiyama. Offline reinforcement learning from datasets with structured non-stationarity. In Reinforcement Learning Conference, 2024. URL https://openreview.net/forum?id=qowNlhKcPw

    2024

  3. [3]

    Concrete problems in AI safety

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man \'e . Concrete problems in AI safety. arXiv preprint, 2016. URL https://arxiv.org/abs/1606.06565

    arXiv 2016

  4. [4]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Neural Information Processing Systems, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/453fadbd8a1a3af50a9df4df899537b5-Abstract.html

    2017

  5. [5]

    A survey of inverse reinforcement learning: Challenges, methods and progress

    Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297: 0 103500, 2021. URL https://doi.org/10.1016/j.artint.2021.103500

    arXiv 2021

  6. [6]

    The probability of backtest overfitting

    David H Bailey, Jonathan Borwein, Marcos Lopez de Prado, and Qiji Jim Zhu. The probability of backtest overfitting. Journal of Computational Finance (Risk Journals), 2015. URL https://dx.doi.org/10.2139/ssrn.2326253

    doi:10.2139/ssrn.2326253 2015

  7. [7]

    Leemon C. Baird. Reinforcement learning in continuous time: Advantage updating. In IEEE International Conference on Neural Networks, volume 4, pp.\ 2448--2453. IEEE, 1994. URL https://doi.org/10.1109/ICNN.1994.374604

    work page arXiv 1994

  8. [8]

    Barto, Richard S

    Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13 0 (5): 0 834--846, 1983. URL https://doi.org/10.1109/TSMC.1983.6313077

    arXiv 1983

Show all 90 references
  1. [9]

    A distributional perspective on reinforcement learning

    Marc G Bellemare, Will Dabney, and R \'e mi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, 2017. URL https://proceedings.mlr.press/v70/bellemare17a.html

    2017

  2. [10]

    Distributional Reinforcement Learning

    Marc G Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. MIT Press, 2023. URL https://doi.org/10.7551/mitpress/14207.001.0001

    2023 doi

  3. [11]

    Dynamic programming

    Richard Bellman. Dynamic programming. Science, 153 0 (3731): 0 34--37, 1966. URL https://doi.org/10.1126/science.153.3731.34

    1966 doi

  4. [12]

    Abstract Dynamic Programming

    Dimitri Bertsekas. Abstract Dynamic Programming. Athena Scientific, 2022

    2022

  5. [13]

    Algebra of Programming

    Richard Bird and Oege de Moor. Algebra of Programming. Prentice Hall, 1997

    1997

  6. [14]

    Investments

    Zvi Bodie, Alex Kane, and Alan J Marcus. Investments. McGraw-hill, 2011

    2011

  7. [15]

    Settling the reward hypothesis

    Michael Bowling, John D Martin, David Abel, and Will Dabney. Settling the reward hypothesis. In International Conference on Machine Learning, 2023. URL https://proceedings.mlr.press/v202/bowling23a.html

    2023

  8. [16]

    JAX : composable transformations of Python + NumPy programs, 2018

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake Vander P las, Skye Wanderman- M ilne, and Qiao Zhang. JAX : composable transformations of Python + NumPy programs, 2018. URL http://github.com/jax-ml/jax

    2018

  9. [17]

    OpenAI Gym

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym . arXiv preprint, 2016. URL https://arxiv.org/abs/1606.01540

    2016 arXiv

  10. [18]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4...

    2017

  11. [19]

    Categorical foundations of gradient-based learning

    Geoffrey SH Cruttwell, Bruno Gavranovi \'c , Neil Ghani, Paul Wilson, and Fabio Zanasi. Categorical foundations of gradient-based learning. In European Symposium on Programming, pp.\ 1--28, 2022. URL https://doi.org/10.1007/978-3-030-99336-8_1

    2022 doi

  12. [20]

    Reinforcement learning with non-cumulative objective

    Wei Cui and Wei Yu. Reinforcement learning with non-cumulative objective. IEEE Transactions on Machine Learning in Communications and Networking, 1: 0 124--137, 2023. URL https://doi.org/10.1109/TMLCN.2023.3285543

    2023

  13. [21]

    Categories, relations and dynamic programming

    Oege De Moor. Categories, relations and dynamic programming. Mathematical Structures in Computer Science, 4 0 (1): 0 33--69, 1994. URL https://doi.org/10.1017/S0960129500000360

    1994 doi

  14. [22]

    Goal misgeneralization in deep reinforcement learning

    Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning, 2022. URL https://proceedings.mlr.press/v162/langosco22a.html

    2022

  15. [23]

    Offline meta reinforcement learning -- identifiability challenges and effective data collection strategies

    Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement learning -- identifiability challenges and effective data collection strategies. In Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=IBdEfhLveS

    2021

  16. [24]

    Infinite-horizon model predictive control for periodic tasks with contacts

    Tom Erez, Yuval Tassa, and Emanuel Todorov. Infinite-horizon model predictive control for periodic tasks with contacts. In Robotics: Science and Systems VII. The MIT Press, 2012. URL https://doi.org/10.7551/mitpress/9481.003.0015

    2012 doi

  17. [25]

    Lenses and learners

    Brendan Fong and Michael Johnson. Lenses and learners. In International Workshop on Bidirectional Transformations, 2019. URL https://arxiv.org/abs/1903.03671

    2019 arXiv

  18. [26]

    A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statistics

    Tobias Fritz. A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statistics. Advances in Mathematics, 370: 0 107239, 2020. URL https://doi.org/10.1016/j.aim.2020.107239. https://arxiv.org/abs/1908.07021

    2020

  19. [27]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018. URL https://proceedings.mlr.press/v80/fujimoto18a.html

    2018

  20. [28]

    Reinforcement learning for constrained Markov decision processes

    Ather Gattami, Qinbo Bai, and Vaneet Aggarwal. Reinforcement learning for constrained Markov decision processes. In International Conference on Artificial Intelligence and Statistics, 2021. URL https://proceedings.mlr.press/v130/gattami21a.html

    2021

  21. [29]

    Space-time tradeoffs of lenses and optics via higher category theory

    Bruno Gavranovi \'c . Space-time tradeoffs of lenses and optics via higher category theory. arXiv preprint, 2022. URL https://arxiv.org/abs/2209.09351

    2022 arXiv

  22. [30]

    A categorical approach to probability theory

    Mich \`e le Giry. A categorical approach to probability theory. Categorical Aspects of Topology and Analysis, pp.\ 68--85, 1982. URL https://doi.org/10.1007/BFb0092872

    1982 doi

  23. [31]

    What every computer scientist should know about floating-point arithmetic

    David Goldberg. What every computer scientist should know about floating-point arithmetic. ACM computing surveys (CSUR), 23 0 (1): 0 5--48, 1991. URL https://doi.org/10.1145/103162.103163

    1991

  24. [32]

    Maximum reward formulation in reinforcement learning

    Sai Krishna Gottipati, Yashaswi Pathak, Rohan Nuttall, Raviteja Chunduru, Ahmed Touati, Sriram Ganapathi Subramanian, Matthew E Taylor, and Sarath Chandar. Maximum reward formulation in reinforcement learning. arXiv preprint, 2020. URL https://arxiv.org/abs/2010.03744

    2020 arXiv

  25. [33]

    Space-efficient online computation of quantile summaries

    Michael Greenwald and Sanjeev Khanna. Space-efficient online computation of quantile summaries. ACM SIGMOD Record, 30 0 (2): 0 58--66, 2001. URL https://doi.org/10.1145/376284.375670

    2001

  26. [34]

    Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov

    William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. MineRL : A large-scale dataset of Minecraft demonstrations. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2...

    2019 doi

  27. [35]

    Soft Actor-Critic : Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic : Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018. URL https://proceedings.mlr.press/v80/haarnoja18b.html

    2018

  28. [36]

    The off-switch game

    Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game. In Workshop on AI, Ethics, and Society at the Thirty-First AAAI Conference on Artificial Intelligence, 2017. URL https://arxiv.org/abs/1611.08219

    2017 arXiv

  29. [37]

    Harris, K

    Charles R. Harris, K. Jarrod Millman, St \' e fan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jai...

    2020

  30. [38]

    Value iteration is optic composition

    Jules Hedges and Riu Rodr \' guez Sakamoto. Value iteration is optic composition. In International Conference on Applied Category Theory, 2022. URL https://arxiv.org/abs/2206.04547

    2022 arXiv

  31. [39]

    Ralf Hinze, Thomas Harper, and Daniel W. H. James. Theory and practice of fusion. In Symposium on Implementation and Application of Functional Languages, pp.\ 19--37, 2010. URL https://doi.org/10.1007/978-3-642-24276-2_2

    2010 doi

  32. [40]

    Imitation learning: A survey of learning methods

    Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50 0 (2): 0 1--35, 2017. URL https://doi.org/10.1145/3054912

    2017 doi

  33. [41]

    A tutorial on the universality and expressiveness of fold

    Graham Hutton. A tutorial on the universality and expressiveness of fold. Journal of Functional Programming, 9 0 (4): 0 355--372, 1999. URL https://doi.org/10.1017/S0956796899003500

    1999 doi

  34. [42]

    AI alignment: A comprehensive survey

    Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O'Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, S...

    2023 arXiv

  35. [43]

    Deep reinforcement learning approach for trading automation in the stock market

    Taylan Kabbani and Ekrem Duman. Deep reinforcement learning approach for trading automation in the stock market. IEEE Access, 10: 0 93564--93574, 2022. URL https://doi.org/10.1109/ACCESS.2022.3203697

    2022

  36. [44]

    Deep reinforcement learning for autonomous driving: A survey

    B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick P \'e rez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23 0 (6): 0 4909--4926, 2021. URL ht...

    2021

  37. [45]

    Reinforcement learning in robotics: A survey

    Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32 0 (11): 0 1238--1274, 2013. URL https://doi.org/10.1177/0278364913495721

    2013 doi

  38. [46]

    gymnax : A JAX -based reinforcement learning environment library, 2022

    Robert Tjarko Lange. gymnax : A JAX -based reinforcement learning environment library, 2022. URL http://github.com/RobertTLange/gymnax

    2022

  39. [47]

    AI safety gridworlds

    Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. AI safety gridworlds. arXiv preprint, 2017. URL https://arxiv.org/abs/1711.09883

    2017 arXiv

  40. [48]

    Dynamic datasets and market environments for financial reinforcement learning

    Xiao-Yang Liu, Ziyi Xia, Hongyang Yang, Jiechao Gao, Daochen Zha, Ming Zhu, Christina Dan Wang, Zhaoran Wang, and Jian Guo. Dynamic datasets and market environments for financial reinforcement learning. Machine Learning, 113 0 (5): 0 2795--2839, 2024. URL https://doi.org/10.10...

    2024 arXiv

  41. [49]

    Mean-variance optimization in Markov decision processes

    Shie Mannor and John Tsitsiklis. Mean-variance optimization in Markov decision processes. In International Conference on Machine Learning, 2011. URL https://dl.acm.org/doi/abs/10.5555/3104482.3104505. https://icml.cc/2011/papers/156_icmlpaper.pdf

    2011

  42. [50]

    Functional programming with bananas, lenses, envelopes and barbed wire

    Erik Meijer, Maarten Fokkinga, and Ross Paterson. Functional programming with bananas, lenses, envelopes and barbed wire. In Conference on Functional Programming Languages and Computer Architecture, pp.\ 124--144, 1991. URL https://doi.org/10.1007/3540543961_7

    1991 doi

  43. [51]

    Playing Atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint, 2013. URL https://arxiv.org/abs/1312.5602

    2013 arXiv

  44. [52]

    Rusu, Joel Veness, Marc G

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, ...

    2015 doi

  45. [53]

    Asynchronous methods for deep reinforcement learning

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, 2016. URL https://proceedings.mlr.pres...

    2016

  46. [54]

    Learning to trade via direct reinforcement

    John Moody and Matthew Saffell. Learning to trade via direct reinforcement. IEEE transactions on neural Networks, 12 0 (4): 0 875--889, 2001. URL https://doi.org/10.1109/72.935097

    2001 doi

  47. [55]

    Performance functions and reinforcement learning for trading systems and portfolios

    John Moody, Lizhong Wu, Yuansong Liao, and Matthew Saffell. Performance functions and reinforcement learning for trading systems and portfolios. Journal of forecasting, 17 0 (5-6): 0 441--470, 1998. URL https://doi.org/10.1002/(SICI)1099-131X(1998090)17:5/6

    1998 doi

  48. [56]

    Nonparametric return distribution approximation for reinforcement learning

    Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. In International Conference on Machine Learning, 2010 a . URL https://dblp.org/rec/conf/icml/MorimuraSKHT10....

    2010

  49. [57]

    Parametric return density estimation for reinforcement learning

    Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. In Conference on Uncertainty in Artificial Intelligence, 2010 b . URL https://dblp.org/rec/conf/uai/MorimuraSKHT10.html...

    2010

  50. [58]

    Handbook of floating-point arithmetic, volume 1

    Jean-Michel Muller, Nicolas Brunie, Florent De Dinechin, Claude-Pierre Jeannerod, Mioara Joldes, Vincent Lef \`e vre, Guillaume Melquiond, Nathalie Revol, and Serge Torres. Handbook of floating-point arithmetic, volume 1. Springer, 2018. URL https://doi.org/10.1007/978-3-319-76526-6

    2018 doi

  51. [59]

    Reinforcement learning: An overview

    Kevin Murphy. Reinforcement learning: An overview. arXiv preprint, 2024. URL https://arxiv.org/abs/2412.05265

    2024

  52. [60]

    a gele, Jan Olle, Thomas F \

    Maximilian N \"a gele, Jan Olle, Thomas F \"o sel, Remmy Zen, and Florian Marquardt. Tackling decision processes with non-cumulative objectives using reinforcement learning. arXiv preprint, 2024. URL https://arxiv.org/abs/2405.13609

    2024 arXiv

  53. [61]

    Ng, Daishi Harada, and Stuart J

    Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, 1999. URL https://dl.acm.org/doi/10.5555/645528.657613

    1999

  54. [62]

    Learnable commutative monoids for graph neural networks

    Euan Ong and Petar Veli c kovi \'c . Learnable commutative monoids for graph neural networks. In Learning on Graphs Conference, 2022. URL https://proceedings.mlr.press/v198/ong22a.html

    2022

  55. [63]

    Compositional computational systems

    Aleksandar Petrov. Compositional computational systems. Master's thesis, ETH Zurich, 2020. URL https://doi.org/10.3929/ethz-b-000463467

    2020 doi

  56. [64]

    Consistent aggregation of objectives with diverse time preferences requires non-Markovian rewards

    Silviu Pitis. Consistent aggregation of objectives with diverse time preferences requires non-Markovian rewards. Advances in Neural Information Processing Systems, 2023. URL https://proceedings.neurips.cc/paper/2023/hash/08342dc6ab69f23167b4123086ad4d38-Abstract.html

    2023

  57. [65]

    Puterman

    Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994. URL https://doi.org/10.1002/9780470316887

    1994 doi

  58. [66]

    Maximum reward reinforcement learning: A non-cumulative reward criterion

    Kian Hong Quah and Chai Quek. Maximum reward reinforcement learning: A non-cumulative reward criterion. Expert Systems with Applications, 31 0 (2): 0 351--359, 2006. URL https://doi.org/10.1016/j.eswa.2005.09.054

    2006 doi

  59. [67]

    Stable-Baselines3 : Reliable reinforcement learning implementations

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-Baselines3 : Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 0 (268): 0 1--8, 2021. URL http://jmlr.org/papers/v22/20-1364.html. ...

    2021

  60. [68]

    Categories of optics

    Mitchell Riley. Categories of optics. arXiv preprint, 2018. URL https://arxiv.org/abs/1809.00738

    2018 arXiv

  61. [69]

    Jordan, and Pieter Abbeel

    John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016. URL https://arxiv.org/abs/1506.02438

    2016 arXiv

  62. [70]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  63. [71]

    William F. Sharpe. Mutual fund performance. The Journal of Business, 39 0 (1): 0 119--138, 1966. URL http://www.jstor.org/stable/2351741

    1966

  64. [72]

    A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, ...

    2018 doi

  65. [73]

    Clere Smithe

    Toby St. Clere Smithe. Bayesian updates compose optically. arXiv preprint, 2020. URL https://arxiv.org/abs/2006.01631

    2020 arXiv

  66. [74]

    Matthew J. Sobel. The variance of discounted Markov decision processes. Journal of Applied Probability, 19 0 (4): 0 794--802, 1982. URL https://doi.org/10.2307/3213832

    1982 doi

  67. [75]

    Deep reinforcement learning for optimal portfolio allocation: A comparative study with mean-variance optimization

    Srijan Sood, Kassiani Papasotiriou, Marius Vaiciulis, and Tucker Balch. Deep reinforcement learning for optimal portfolio allocation: A comparative study with mean-variance optimization. FinPlan, pp.\ 21, 2023. URL https://icaps23.icaps-conference.org/papers/finplan/FinPlan23_...

    2023

  68. [76]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 1998. URL http://incompleteideas.net/book/the-book.html

    1998

  69. [77]

    Policy gradients with variance related risk criteria

    Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In International Conference on Machine Learning, 2012. URL https://dl.acm.org/doi/10.5555/3042573.3042784. https://icml.cc/2012/papers/489.pdf

    2012

  70. [78]

    MuJoCo : A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo : A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012. URL https://doi.org/10.1109/IROS.2012.6386109

    2012

  71. [79]

    To the max: Reinventing reward in reinforcement learning

    Grigorii Veviurko, Wendelin B \"o hmer, and Mathijs de Weerdt. To the max: Reinventing reward in reinforcement learning. In International Conference on Machine Learning, 2024. URL https://proceedings.mlr.press/v235/veviurko24a.html

    2024

  72. [80]

    A survey of constraint formulations in safe reinforcement learning

    Akifumi Wachi, Xun Shen, and Yanan Sui. A survey of constraint formulations in safe reinforcement learning. In International Joint Conference on Artificial Intelligence, 2024. URL https://doi.org/10.24963/ijcai.2024/913

    2024 doi

  73. [81]

    Planning with general objective functions: Going beyond total rewards

    Ruosong Wang, Peilin Zhong, Simon S Du, Russ R Salakhutdinov, and Lin Yang. Planning with general objective functions: Going beyond total rewards. In Neural Information Processing Systems, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/a6a767bbb2e3513233f942e0ff24272...

    2020

  74. [82]

    Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8: 0 279--292, 1992. URL https://doi.org/10.1007/BF00992698

    1992 doi

  75. [83]

    Learning from delayed rewards

    Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, King's College, Cambridge United Kingdom, 1989. URL http://www.cs.rhul.ac.uk/ chrisw/new_thesis.pdf

    1989

  76. [84]

    Note on a method for calculating corrected sums of squares and products

    Barry Payne Welford. Note on a method for calculating corrected sums of squares and products. Technometrics, 4 0 (3): 0 419--420, 1962. URL https://doi.org/10.1080/00401706.1962.10490022

    1962

  77. [85]

    Foundations of multivariate distributional reinforcement learning

    Harley Wiltzer, Jesse Farebrother, Arthur Gretton, and Mark Rowland. Foundations of multivariate distributional reinforcement learning. In Neural Information Processing Systems, 2024. URL https://proceedings.neurips.cc/paper/2024/hash/b76bec34ef5e0c0ceedff6edfbefc9f5-Abstract.html

    2024

  78. [86]

    Adaptive stock trading strategies with deep reinforcement learning methods

    Xing Wu, Haolei Chen, Jianjia Wang, Luigi Troiano, Vincenzo Loia, and Hamido Fujita. Adaptive stock trading strategies with deep reinforcement learning methods. Information Sciences, 538: 0 142--158, 2020. URL https://doi.org/10.1016/j.ins.2020.05.066

    2020 doi

  79. [87]

    Fantastic morphisms and where to find them: A guide to recursion schemes

    Zhixuan Yang and Nicolas Wu. Fantastic morphisms and where to find them: A guide to recursion schemes. In International Conference on Mathematics of Program Construction, pp.\ 222--267, 2022. URL https://doi.org/10.1007/978-3-031-16912-0_9

    2022 doi

  80. [88]

    Deep sets

    Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Neural Information Processing Systems, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/f22e4747da1aa27e363d86d40ff442fe-Abstract.html

    2017

  81. [89]

    Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving

    Meixin Zhu, Yinhai Wang, Ziyuan Pu, Jingyun Hu, Xuesong Wang, and Ruimin Ke. Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving. Transportation Research Part C: Emerging Technologies, 117: 0 102662, 2020. URL https://doi.or...

    2020

  82. [90]

    Ziebart, Andrew Maas, J

    Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In National Conference on Artificial Intelligence, 2008. URL https://dl.acm.org/doi/abs/10.5555/1620270.1620297

    2008

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.