Pith. sign in

REVIEW 4 major objections 5 minor 50 references

Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Low-rank tensor policies can match neural-network returns in reinforcement learning while using fewer parameters and converging faster.

desk verdict Solid Gaussian-policy tensor low-rank extension, but the softmax policy score drops the temperature β; a fixable bug that undermines the discrete-action results as written. read the letter →

arxiv 2501.04879 v1 pith:RQICY4NV submitted 2025-01-08 cs.LG

Sergio Rozada , Hoi-To Wai , Antonio G. Marques This is my paper · ORCID
classification cs.LG
keywords reinforcementlearningpolicygradientmethodstensorlow-rankapproximationPARAFACdecompositionactor-critictrust-regionoptimizationproximalsamplecomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Policy-based reinforcement learning typically maps states to action-distribution parameters with a neural network. This paper claims that for many problems those parameter maps have hidden low-dimensional structure: if the parameters are arranged in a tensor over discretized states, a PARAFAC low-rank model with a small rank captures them nearly as well as a neural network. The paper derives the policy-score formulas for tensor low-rank Gaussian and softmax policies, plugs them into policy-gradient, actor-critic, TRPO, and PPO updates, and proves that a projected version of the tensor low-rank policy-gradient method converges to a stationary point at rate $O(1/\sqrt{H})$. The payoff, if the claim is right, is that RL agents can be trained with far fewer parameters and fewer samples while achieving comparable returns.

What carries the argument

The central object is the PARAFAC decomposition of a policy-parameter tensor: a tensor $\Theta\in\mathbb{R}^{N_1\times\cdots\times N_D}$ of rank $K$ is written as $\sum_{k=1}^K \Theta_1(:,k)\circ\cdots\circ \Theta_D(:,k)$, with factors $\Theta_d\in\mathbb{R}^{N_d\times K}$. This replaces $\prod_d N_d$ parameters by $K\sum_d N_d$ parameters. Its role in the argument is to make the policy score computable and sparse: the key identity shows that $\partial \log\pi_\Theta(a|s)/\partial[\Theta_d]_{i,k}$ is nonzero only when $i$ matches the $d$-th coordinate of state $s$, and equals the standard log-density derivative times $\prod_{j\ne d}[\Theta_j]_{i_j,k}$. For Gaussian policies this gives the score formula for the mean tensor, for softmax policies the score formula for the logit tensor, and the same scores feed the critic gradient, the TRPO Fisher matrix, and the PPO clipping objective.

What would settle it

Train a neural-network policy on a control task whose optimal policy is intrinsically high rank, for example a state space where nearby states require unrelated actions, then discretize the states and PARAFAC-decompose the resulting parameter tensor at small rank. If the normalized Frobenius error stays large even for rank comparable to the tensor's maximum rank, or if the rank needed for small error grows with the number of discretization bins, the low-rank premise fails and the tensor policy cannot match the neural network.

Watch

Extended reading notes

Core claim

The central claim is that the state-dependent parameters of a policy—the means of a Gaussian policy or the logits of a softmax policy—can be collected into a tensor and constrained to have low PARAFAC rank without sacrificing performance. Under this parametrization the policy score $\partial \log \pi_\Theta(a|s)/\partial [\Theta_d]_{i,k}$ factorizes into the scalar derivative of the log-density with respect to the tensor entry times the product of the other factors, so every policy-gradient-family algorithm only needs these sparse entrywise scores. The paper's Theorem 1 states that the projected tensor low-rank PG update reaches an $O(1/\sqrt{H})$-stationary point of the expected-return objective under bounded rewards and Lipschitz and bounded-score assumptions. Empirically, PARAFAC decompositions of neural-network-trained policies in Pendulum show negligible normalized Frobenius error at rank 5 for Gaussian means and rank 20 for softmax logits, and the tensor low-rank variants of actor-critic, TRPO, and PPO match or exceed neural-network returns in MountainCar, Pendulum, and a wireless scheduling problem while using fewer parameters and converging faster.

Load-bearing premise

The argument rests on the premise that, after discretizing the state space, the tensor of policy parameters for a good policy is well approximated by a PARAFAC model with small rank; if a problem lacks that structure, the tensor policy cannot match neural-network capacity.

Editorial extensions

If this is right

  • Tensor low-rank variants of REINFORCE, actor-critic, TRPO, and PPO exist and require only the policy scores, so any policy-gradient algorithm can adopt the model by substituting these scores.
  • Parameter count drops from exponential in the state dimension to linear in the factor matrices, which is what makes training faster and less sample-hungry if the low-rank premise holds.
  • The projected tensor low-rank PG algorithm converges to an $O(1/\sqrt{H})$-stationary point, giving a convergence guarantee for low-rank tensor policies.
  • In the tested benchmarks, median returns are similar to neural-network policies, with most tensor variants converging faster and with lower variance, so practitioners can expect cheaper training with comparable final performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the empirical low-rankness observed in Pendulum holds broadly, then low-rank tensor policies could serve as a compression target for trained neural-network policies: discretize the learned parameter map and PARAFAC-decompose it to obtain a smaller deployable policy.
  • The practical gains should grow with state-space dimension, but only if a sufficiently fine discretization of the state space is affordable; for high-dimensional continuous states the discretization step itself may dominate the cost, and adaptive or hierarchical grids would be a natural extension the paper does not develop.
  • The same factorization could be applied to model-based settings: if transition and reward tensors are also low rank, a fully low-rank pipeline of dynamics, value, and policy could reduce sample complexity further than actor-only low rank.
  • A testable prediction is that the rank needed for near-optimal returns is controlled by the intrinsic dimensionality of the MDP's latent state, not by the nominal state dimension; environments with high nominal dimension but low latent dimension should show the largest gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes tensor low-rank (PARAFAC) parameterizations of Gaussian and softmax policies for policy-based reinforcement learning. The parameters are arranged into a tensor indexed by the (discretized) state and, for softmax policies, the action, and low rank is enforced via the PARAFAC decomposition. The authors derive policy-score formulas for these models and integrate them into policy gradient, actor-critic, TRPO, and PPO algorithms. They provide a convergence theorem for a projected tensor low-rank policy-gradient method and report experiments on MountainCar, Pendulum, and a wireless communications setup, claiming faster convergence and fewer parameters than neural-network baselines.

Significance. If the technical derivations are correct, the idea is valuable: it offers a systematic way to impose low-rank structure on policy parameters, with closed-form scores that are easy to plug into several policy-based algorithms. The modularity of the approach, the availability of the code, and the explicit convergence analysis for a projected variant are strengths. However, the softmax policy-score formula omits the temperature factor, which is a load-bearing error for all discrete-action algorithms. The theoretical claims are also narrower than the abstract suggests: only a projected TLRPG variant is analyzed, and no sample-complexity comparison is provided. With those issues fixed or clearly qualified, the paper could make a solid contribution to the literature on structured policy parameterizations.

major comments (4)
  1. [Section III-A, Eq. (14)] The softmax policy score omits the temperature factor β. Given the definition Z(z,β)_i = e^{β z_i} / Σ_j e^{β z_j}, the log-policy is log π_Θ(a|s) = β z_{s,a} − log Σ_b e^{β z_{s,b}}, whose derivative with respect to z_{s,b} is β(1_{a=b} − Z_b). Equation (14) instead uses (1_{a=b} − Z_b) without the factor β. Because β is exposed in Algorithms 2, 5, and 6 and is described as a tunable temperature, the discrete-action TLRPG, TLRAC, TRTLRPO, and PTLRPO updates are not gradients of the stated objective unless β=1, which is not assumed. The omission propagates into the TRPO gradient and Fisher matrix in Eqs. (22)–(23), the PPO gradient in Eqs. (25)–(27), and the softmax verification in Appendix A. Please insert the missing β throughout, or explicitly restrict to β=1 and justify that restriction.
  2. [Section III-D, Theorem 1 and Eq. (28)] The convergence guarantee is proved for the projected update in Eq. (28), but Algorithm 3 as stated has no projection onto the convex set O. The text calls this a slight modification, yet the analyzed algorithm is not the implemented one unless O is the entire parameter space, in which case the boundedness used to verify Assumptions 2–4 must be justified separately. The theorem also covers only TLRPG, while contribution C3 and parts of the abstract suggest guarantees for the proposed methods more generally; no convergence analysis is given for TLRAC, TRTLRPO, or PTLRPO. Please clarify the relationship between the projected update and Algorithm 3, state whether O is active in the experiments, and temper the convergence claims accordingly.
  3. [Abstract; Section I] The abstract claims that the proposed methods "reduce computational and sample complexities" relative to NN models, but no sample-complexity analysis is provided. Theorem 1 yields an O(1/√H) stationarity rate for the projected TLRPG variant, which is a standard rate for stochastic nonconvex optimization and does not quantify sample complexity in comparison with NN policies. The experimental evidence supports claims about parameter efficiency and faster empirical convergence in terms of return per episode, but not a formal reduction in sample complexity. Please revise the claims in the abstract and introduction to match what is actually shown.
  4. [Section IV-A, Fig. 1] The low-rank premise is directly validated only for the Pendulum environment. For MountainCar and the wireless communications setup, no analogous low-rankness diagnostics are provided, so the assertion that "many MDPs admit lower-dimensional representations" is not established for the other benchmarks. Since the expressiveness of the proposed policy is bounded by the chosen PARAFAC rank, the paper should either provide similar low-rankness evidence for the remaining environments or explicitly state the scope of the empirical claim.
minor comments (5)
  1. [Section III-D, proof of Theorem 1] In the displayed bound of the proof, the first term is missing the gradient operator on the second log-policy term; it should read ∇_{Θ_i} log p_g(Θ')(τ_T) rather than log p_g(Θ')(τ_T).
  2. [Appendix A, softmax case] The statement that the softmax function is 1-Lipschitz should account for the temperature parameter β; with the definition in Section III-A, the Lipschitz constant is proportional to β, which changes the constants in Assumptions 2–4 but not their validity for bounded β.
  3. [Section II-B] The sentence claiming that the only hyper-parameter needing tuning in the PARAFAC model is the rank K overlooks the state-space discretization grid and the distributional parameters σ and β, which are also tuned in the experiments; please qualify the statement.
  4. [Section III-A, Eq. (12)] The notation I_{\hat{s}=i_s} is confusing because \hat{s} is a state index while i_s is a tensor index; consider defining the delta explicitly to avoid ambiguity.
  5. [Section IV-A] The statement that NN methods learn low-rank representations of policies is inferred from a single environment; consider softening the wording to reflect the limited empirical basis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the tensor low-rank policy derivation is self-contained and empirically benchmarked against external baselines.

full rationale

The paper's central derivation is self-contained rather than circular. The PARAFAC low-rank policy model is introduced explicitly as a modeling assumption in Section III-A ('we propose using multi-linear policy models that leverage the fact that many MDPs admit lower-dimensional representations'), and the policy scores in Eqs. (13) and (14) are obtained by direct chain-rule differentiation of the Gaussian and softmax log-policies under that parametrization. No quantity is defined in terms of the result it is later claimed to predict, and no fitted parameter is renamed as a prediction. The convergence guarantee in Theorem 1 follows from a standard weakly convex stochastic optimization result (Davis and Drusvyatskiy, [47]) under Assumptions 1-4, whose verification in Appendix A is algebraic and independent of the empirical outcomes. The numerical experiments compare tensor low-rank algorithms against linear RBF baselines and neural-network baselines on external Gym and wireless benchmarks, so the reported performance is not forced by construction. The paper does cite prior work by the same authors (e.g., [18], [19], [34], [45]), but those citations are used for motivation and background on low-rank value functions and matrix low-rank policy methods, not as load-bearing support for the new PARAFAC policy-score derivations or the convergence proof. The empirical low-rankness check in Fig. 1 is an independent verification step, not an input to the algorithm. Even the possible omission of the temperature factor in Eq. (14), if substantiated, would be a correctness issue in the gradient formula, not a circularity, because it does not make an output quantity identical to an input by construction. Therefore no circular step is present.

Assumptions & free parameters 3 free parameters · 7 assumptions · 0 invented entities

The main contribution is a new policy parameterization, not a new physical entity. The load-bearing premises are the low-rankness of policy parameter tensors and the technical smoothness/boundedness conditions used for the convergence theorem.

free parameters (3)
  • PARAFAC rank K = 5 for Gaussian means and 20 for softmax logits in Pendulum (Fig.
    Rank controls model capacity. The paper provides no a priori rule for setting K, and the empirical advantages depend on a small K being sufficient.
  • State-space discretization grid = 100x100x100 for Pendulum in Fig. 1; other grids not fully specified
    Continuous state spaces are discretized to build the policy parameter tensor. Grid resolution is a user-chosen modeling choice that affects approximation quality.
  • Gaussian standard deviation sigma and softmax temperature beta = Set to the same values across compared methods; numerical values are not stated in the paper
    These exploration hyperparameters are fixed for fairness rather than fitted to produce the result, but they are user-chosen and influence gradients.
assumptions (7)
  • domain assumption The reward function r(s,a,s') is uniformly bounded (Assumption 1).
    Used in Section III-D and Appendix A to bound returns G_T and to verify the Lipschitz properties needed for convergence.
  • domain assumption Policy score gradients are Lipschitz and have bounded second moment (Assumptions 2-4).
    These are stated as assumptions and verified in Appendix A only when factor matrices are bounded and actions are bounded. They underpin Theorem 1.
  • domain assumption Factor tensors remain in a compact convex set O with entries bounded by B.
    Required for Appendix A verifications. Algorithm 3 as written has no projection onto such a set, creating a gap between theory and practice.
  • domain assumption Actions are bounded for Gaussian policies, |a| <= A.
    Used in Appendix A Eq. (33) to bound the Gaussian policy score and verify Assumption 4.
  • domain assumption The parameter tensors of near-optimal policies are approximately low rank under PARAFAC with small K.
    Core modeling premise stated in Section III-A. Empirical support is limited to Fig. 1 on Pendulum.
  • domain assumption The state space is discrete or discretized into a Cartesian product grid S1 x ... x SD.
    The tensor policy model requires indexing states by a D-dimensional grid, so continuous states must be discretized.
  • standard math The softmax function is 1-Lipschitz.
    Used in Appendix A to verify Assumption 3 for softmax policies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learning." pith.science (2026). https://pith.science/paper/RQICY4NV

@misc{pith2026250104879,
  author       = {Pith},
  title        = {Pith review of: Multilinear Tensor Low-Rank Approximation for Policy-Gradient Methods in Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RQICY4NV}},
  note         = {Machine review of arXiv:2501.04879}
}
read the original abstract

Reinforcement learning (RL) aims to estimate the action to take given a (time-varying) state, with the goal of maximizing a cumulative reward function. Predominantly, there are two families of algorithms to solve RL problems: value-based and policy-based methods, with the latter designed to learn a probabilistic parametric policy from states to actions. Most contemporary approaches implement this policy using a neural network (NN). However, NNs usually face issues related to convergence, architectural suitability, hyper-parameter selection, and underutilization of the redundancies of the state-action representations (e.g. locally similar states). This paper postulates multi-linear mappings to efficiently estimate the parameters of the RL policy. More precisely, we leverage the PARAFAC decomposition to design tensor low-rank policies. The key idea involves collecting the policy parameters into a tensor and leveraging tensor-completion techniques to enforce low rank. We establish theoretical guarantees of the proposed methods for various policy classes and validate their efficacy through numerical experiments. Specifically, we demonstrate that tensor low-rank policy models reduce computational and sample complexities in comparison to NN models while achieving similar rewards.

Figures

Figures reproduced from arXiv: 2501.04879 by the authors.

Figure 1
Figure 1. NFE between the tensor of Gaussian and softmax parameters obtained via NN-based PPO and their low-rank PARAFAC decomposition in the continuous and discrete variants of the Pendulum environment. The error decreases as the rank of the approximation increases. the NNs via exhaustive search to identify the most compact models, in terms of parameters, capable of yielding high returns. For fairness in comparison, we have … view at source ↗
Figure 2
Figure 2. Median return per episode of tensor low-rank policies (TLR) against NN-based policies (NN) across [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Median return per episode of tensor low-rank policies (TLR) against NN-based policies (NN) across [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Median return per episode of tensor low-rank policies (TLR) against [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 44 canonical work pages

  1. [1]

    R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. The MIT Press, 2018

    work page 2018

  2. [2]

    D. P. Bertsekas, Reinforcement learning and optimal control . Athena Scientific, 2019

    work page 2019

  3. [3]

    Mastering the game of Go with deep neural networks and tree search,

    D. Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature Publishing Group, vol. 529, no. 7587, pp. 484– 489, 2016

    work page 2016

  4. [4]

    Mastering the game of Go without human knowledge,

    ——, “Mastering the game of Go without human knowledge,” Nature Publishing Group, vol. 550, no. 7676, pp. 354–359, 2017

    work page 2017

  5. [5]

    Language models are few-shot learners,

    T. Brown et al. , “Language models are few-shot learners,” in Conf. Neural Information Processing Syst. (NeurIPS), vol. 33, 2020, pp. 1877– 1901

    work page 2020

  6. [6]

    D. P. Bertsekas, Dynamic programming and optimal control: Vol. 1 . Athena Scientific, 2000

    work page 2000

  7. [7]

    D. P. Bertsekas and J. N. Tsitsiklis, Neuro-dynamic programming . Athena Scientific, 1996

    work page 1996

  8. [8]

    Policy gradient methods for reinforcement learning with function approximation,

    R. S. Sutton, D. McAllester, S. Singh, and Y . Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Conf. Neural Information Processing Syst. (NeurIPS) , vol. 12, 1999

    work page 1999

Show all 50 references
  1. [9]

    Stochastic policy gradient ascent in reproducing kernel Hilbert spaces,

    S. Paternain, J. A. Bazerque, A. Small, and A. Ribeiro, “Stochastic policy gradient ascent in reproducing kernel Hilbert spaces,”IEEE Trans. Auto. Control, vol. 66, no. 8, pp. 3429–3444, 2020

    2020

  2. [10]

    Communication- efficient policy gradient methods for distributed reinforcement learning,

    T. Chen, K. Zhang, G. B. Giannakis, and T. Bas ¸ar, “Communication- efficient policy gradient methods for distributed reinforcement learning,” IEEE Trans. Control of Netw. Syst. , vol. 9, no. 2, pp. 917–929, 2021. 13

    2021

  3. [11]

    Learning online alignments with continuous rewards policy gradient,

    Y . Luo, C.-C. Chiu, N. Jaitly, and I. Sutskever, “Learning online alignments with continuous rewards policy gradient,” in IEEE Intl. Conf. Acoust., Speech and Signal Process. (ICASSP) . IEEE, 2017, pp. 2801– 2805

    2017

  4. [12]

    Deep reinforcement learning: A brief survey,

    K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” IEEE Signal Process. Mag., vol. 34, no. 6, pp. 26–38, 2017

    2017

  5. [13]

    Compressed conditional mean embeddings for model-based reinforcement learning,

    G. Lever, J. Shawe-Taylor, R. Stafford, and C. Szepesv ´ari, “Compressed conditional mean embeddings for model-based reinforcement learning,” in AAAI Conf. Artificial Intelligence , vol. 30, no. 1, 2016

    2016

  6. [14]

    Nonparametric stochastic compositional gradient descent for Q-learning in continuous markov decision problems,

    E. Tolstaya, A. Koppel, E. Stump, and A. Ribeiro, “Nonparametric stochastic compositional gradient descent for Q-learning in continuous markov decision problems,” in Ann. American Control Conf. (ACC) . IEEE, 2018, pp. 6608–6615

    2018

  7. [15]

    DDPG-driven deep-unfolding with adaptive depth for channel estimation with sparse Bayesian learning,

    Q. Hu, S. Shi, Y . Cai, and G. Yu, “DDPG-driven deep-unfolding with adaptive depth for channel estimation with sparse Bayesian learning,” IEEE Trans. Signal Process. , vol. 70, pp. 4665–4680, 2022

    2022

  8. [16]

    Harnessing structures for value-based planning and reinforcement learning,

    Y . Yang, G. Zhang, Z. Xu, and D. Katabi, “Harnessing structures for value-based planning and reinforcement learning,” in Intl. Conf. Learning Representations (ICLR) , 2019

    2019

  9. [17]

    Tensor-based reinforcement learning for network routing,

    K.-C. Tsai, Z. Zhuang, R. Lent, J. Wang, Q. Qi, L.-C. Wang, and Z. Han, “Tensor-based reinforcement learning for network routing,” IEEE J. Sel. Topics Signal Process., vol. 15, no. 3, pp. 617–629, 2021

    2021

  10. [18]

    Tensor and matrix low- rank value-function approximation in reinforcement learning,

    S. Rozada, S. Paternain, and A. G. Marques, “Tensor and matrix low- rank value-function approximation in reinforcement learning,” IEEE Transactions on Signal Processing , 2024

    2024

  11. [19]

    Matrix low-rank approximation for policy gradient methods,

    S. Rozada and A. G. Marques, “Matrix low-rank approximation for policy gradient methods,” in IEEE Intl. Conf. Acoust., Speech and Signal Process. (ICASSP). IEEE, 2023, pp. 1–5

    2023

  12. [20]

    The approximation of one matrix by another of lower rank,

    C. Eckart and G. Young, “The approximation of one matrix by another of lower rank,” Psychometrika, vol. 1, no. 3, pp. 211–218, 1936

    1936

  13. [21]

    Markovsky, Low rank approximation

    I. Markovsky, Low rank approximation . Springer, 2012

    2012

  14. [22]

    Generalized low rank models,

    M. Udell, C. Horn, R. Zadeh, S. Boyd et al. , “Generalized low rank models,” Foundations and Trends® in Machine Learning , vol. 9, no. 1, pp. 1–118, 2016

    2016

  15. [23]

    Tensor decompositions and applications,

    T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” SIAM Review, vol. 51, no. 3, pp. 455–500, 2009

    2009

  16. [24]

    Tensor decomposition for signal processing and machine learning,

    N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang, E. E. Papalex- akis, and C. Faloutsos, “Tensor decomposition for signal processing and machine learning,” IEEE Trans. Signal Process. , vol. 65, no. 13, pp. 3551–3582, 2017

    2017

  17. [25]

    Flambe: Structural complexity and representation learning of low rank MDPs,

    A. Agarwal, S. Kakade, A. Krishnamurthy, and W. Sun, “Flambe: Structural complexity and representation learning of low rank MDPs,” in Conf. Neural Information Processing Syst. (NeurIPS), 2020, pp. 20 095– 20 107

    2020

  18. [26]

    Representation learning for online and offline RL in low-rank MDPs,

    M. Uehara, X. Zhang, and W. Sun, “Representation learning for online and offline RL in low-rank MDPs,” in Intl. Conf. Learning Representa- tions (ICLR), 2021

    2021

  19. [27]

    Incremental stochastic factorization for online reinforcement learning,

    A. M. Barreto, R. L. Beirigo, J. Pineau, and D. Precup, “Incremental stochastic factorization for online reinforcement learning,” inAAAI Conf. Artificial Intelligence, vol. 30, no. 1, 2016

    2016

  20. [28]

    Contextual decision processes with low Bellman rank are pac-learnable,

    N. Jiang, A. Krishnamurthy, A. Agarwal, J. Langford, and R. E. Schapire, “Contextual decision processes with low Bellman rank are pac-learnable,” in Intl. Conf. Machine Learning (ICML), vol. 70. JMLR. org, 2017, pp. 1704–1713

    2017

  21. [29]

    Reinforcement learning of POMDPs using spectral methods,

    K. Azizzadenesheli, A. Lazaric, and A. Anandkumar, “Reinforcement learning of POMDPs using spectral methods,” in Conf. Learning Theory. PMLR, 2016, pp. 193–256

    2016

  22. [30]

    Tesseract: Tensorised actors for multi-agent reinforcement learning,

    A. Mahajan, M. Samvelyan, L. Mao, V . Makoviychuk, A. Garg, J. Kos- saifi, S. Whiteson, Y . Zhu, and A. Anandkumar, “Tesseract: Tensorised actors for multi-agent reinforcement learning,” in Intl. Conf. Machine Learning (ICML). PMLR, 2021, pp. 7301–7312

    2021

  23. [31]

    Overcoming the long horizon barrier for sample-efficient reinforcement learning with latent low-rank structure,

    T. Sam, Y . Chen, and C. L. Yu, “Overcoming the long horizon barrier for sample-efficient reinforcement learning with latent low-rank structure,” ACM Measurement and Anal. Comput. Syst. , vol. 7, no. 2, pp. 1–60, 2023

    2023

  24. [32]

    Sample efficient reinforcement learning via low-rank matrix estimation,

    D. Shah, D. Song, Z. Xu, and Y . Yang, “Sample efficient reinforcement learning via low-rank matrix estimation,” in Conf. Neural Information Processing Syst. (NeurIPS) , vol. 33, 2020, pp. 12 092–12 103

    2020

  25. [33]

    Low-rank state-action value-function approximation,

    S. Rozada, V . Tenorio, and A. G. Marques, “Low-rank state-action value-function approximation,” in European Signal Process. Conf. (EU- SIPCO). IEEE, 2021, pp. 1471–1475

    2021

  26. [34]

    Matrix low-rank trust region policy optimization,

    S. Rozada and A. G. Marques, “Matrix low-rank trust region policy optimization,” in IEEE Intl. Wrksp. Computat. Advances Multi-Sensor Adaptive Process. (CAMSAP) . IEEE, 2023, pp. 1–5

    2023

  27. [35]

    Optimization for reinforcement learning: From a single agent to cooperative agents,

    D. Lee, N. He, P. Kamalaruban, and V . Cevher, “Optimization for reinforcement learning: From a single agent to cooperative agents,”IEEE Signal Process. Mag. , vol. 37, no. 3, pp. 123–135, 2020

    2020

  28. [36]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning,

    R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine Learning, vol. 8, no. 3, pp. 229–256, 1992

    1992

  29. [37]

    High- dimensional continuous control using generalized advantage estimation,

    J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High- dimensional continuous control using generalized advantage estimation,” arXiv preprint arXiv:1506.02438 , 2015

    2015 arXiv

  30. [38]

    Variance reduction tech- niques for gradient estimates in reinforcement learning

    E. Greensmith, P. L. Bartlett, and J. Baxter, “Variance reduction tech- niques for gradient estimates in reinforcement learning.” J. Machine Learning Research, vol. 5, no. 9, 2004

    2004

  31. [39]

    Actor-critic algorithms,

    V . Konda and J. Tsitsiklis, “Actor-critic algorithms,” in Conf. Neural Information Processing Syst. (NeurIPS) , vol. 12, 1999, pp. 1008–1014

    1999

  32. [40]

    A natural policy gradient,

    S. M. Kakade, “A natural policy gradient,” Conf. Neural Information Processing Syst. (NeurIPS) , vol. 14, 2001

    2001

  33. [41]

    Approximately optimal approximate rein- forcement learning,

    S. Kakade and J. Langford, “Approximately optimal approximate rein- forcement learning,” in Intl. Conf. Machine Learning (ICML) , 2002, pp. 267–274

    2002

  34. [42]

    Trust region policy optimization,

    J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in Intl. Conf. Machine Learning (ICML) . PMLR, 2015, pp. 1889–1897

    2015

  35. [43]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  36. [44]

    PARAFAC. tutorial and applications,

    R. Bro, “PARAFAC. tutorial and applications,” Chemometrics and intelligent laboratory systems , vol. 38, no. 2, pp. 149–171, 1997

    1997

  37. [45]

    Tensor low-rank approximation of finite- horizon value functions,

    S. Rozada and A. G. Marques, “Tensor low-rank approximation of finite- horizon value functions,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 5975–5979

    2024

  38. [46]

    Amari, Differential-geometrical methods in statistics

    S. Amari, Differential-geometrical methods in statistics . Springer Science & Business Media, 2012, vol. 28

    2012

  39. [47]

    Stochastic model-based minimization of weakly convex functions,

    D. Davis and D. Drusvyatskiy, “Stochastic model-based minimization of weakly convex functions,” SIAM Journal on Optimization , vol. 29, no. 1, pp. 207–239, 2019

    2019

  40. [48]

    Online code repository: Tensor low-rank approximation for policy-gradient methods in reinforcement learning,

    S. Rozada, “Online code repository: Tensor low-rank approximation for policy-gradient methods in reinforcement learning,” https://github.com/ sergiorozada12/tensor-low-rank-pg, 2024

    2024

  41. [49]

    OpenAI Gym,

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schul- man, J. Tang, and W. Zaremba, “OpenAI Gym,” arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  42. [50]

    Radial basis functions,

    M. D. Buhmann, “Radial basis functions,” Acta numerica , vol. 9, pp. 1–38, 2000

    2000

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.