{"id":"e3d3ba0e-7c1f-4f10-90f8-3e01f0a45961","arxiv_id":"2501.04879","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Tensor low-rank (PARAFAC) policies for Gaussian and softmax policy-gradient methods match neural-network returns on several RL benchmarks while using fewer parameters and converging faster.","lead":"The paper introduces policy models for reinforcement learning that store policy parameters in a tensor and force it to be low rank using the PARAFAC decomposition, greatly reducing the number of parameters to learn. The methods plug into standard policy-gradient algorithms and, on control benchmarks, converge faster than neural-network policies while reaching similar rewards.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Softmax policy score in Eq. (14) omits the temperature factor β, so the discrete-action TLR algorithms use a biased policy gradient unless β=1; this affects all discrete-action experiments and the related convergence claims.","rationale":"The paper's central claim is that tensor low-rank policies match NN returns with fewer parameters and faster convergence for both Gaussian and softmax policy classes. The Gaussian score derivation is coherent, but the softmax score in Eq. (14) is missing the β factor that appears in the softmax definition. This is the most load-bearing concern because Eq. (14) is not an optional modeling choice: it is the policy score used by every softmax variant of TLRPG, TLRAC, TRTLRPO, and PTLRPO, and it also enters the TRPO Fisher matrix. The omission is internally verifiable and directly affects the discrete-action experiments, whereas the low-rank premise is a common inductive-bias assumption that the paper partially supports through Fig. 1 and the wireless experiment. The projected-vs-unprojected convergence gap noted by the reader is real but secondary: the paper explicitly analyzes a projected variant, and the empirical algorithms are not claimed to satisfy that theorem. The β omission is more serious because it makes the stated algorithms inconsistent with the stated objective unless β is fixed to 1. If β is constant, a missing positive factor could in principle be absorbed into the learning rate for PG/AC, but for TRPO/PPO and for any annealing schedule the effect is not merely a global rescaling, and the paper does not report the β values used. The reader already issued a conditional verdict and listed the dropped β as one concern; my stress test agrees with that concern but not with the reader's choice of the weakest assumption, hence 'partial'. The proposed test settles the issue: if the corrected and published scores produce the same returns after learning-rate tuning, the concern is minor; if not, the discrete-action claims need to be revised.","tokens_in":20500,"tokens_out":7604,"duration_ms":82341,"concrete_test":"Re-derive Eq. (14) with the β factor, and compare it against automatic differentiation of the softmax log-policy for a random PARAFAC tensor with β=2 and β=0.5; the ratio should be β. Then rerun the PendulumDisc PTLRPO experiment using the corrected score and the published score with β≠1; if the return curves move outside the reported interquartile ranges, the omission is material to the empirical claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section III-A, the softmax policy is defined as Z(z, β)_i = e^{β z_i} / Σ_j e^{β z_j}. Differentiating log π_Θ(a|s) = β z_{s,a} − log Σ_b e^{β z_{s,b}} with respect to z_{s,b} yields β(I_{a=b} − Z_b), yet Eq. (14) has no β factor. The factor β is exposed in Algorithms 2, 5, and 6, and the experiments state that β is set as a hyperparameter. Because Eq. (14) is used as the policy score, the omission propagates into the TRPO gradient and Fisher matrix in Eqs. (22)–(23), and into the PPO gradient in Eqs. (25)–(27). If β=1, the formulas are correct, but the paper never makes that restriction and introduces β as a temperature to be tuned. If β is different from 1 or varies over training, the implemented updates are not gradients of the stated objective, and the convergence guarantee for softmax policies does not apply to the algorithm as implemented. This is an internal correctness issue, not merely an external validity concern about low-rank structure, and it can be checked immediately.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes tensor low-rank (PARAFAC) parameterizations of Gaussian and softmax policies for policy-based reinforcement learning. The parameters are arranged into a tensor indexed by the (discretized) state and, for softmax policies, the action, and low rank is enforced via the PARAFAC decomposition. The authors derive policy-score formulas for these models and integrate them into policy gradient, actor-critic, TRPO, and PPO algorithms. They provide a convergence theorem for a projected tensor low-rank policy-gradient method and report experiments on MountainCar, Pendulum, and a wireless communications setup, claiming faster convergence and fewer parameters than neural-network baselines.","tokens_in":20772,"tokens_out":11837,"duration_ms":115603,"significance":"If the technical derivations are correct, the idea is valuable: it offers a systematic way to impose low-rank structure on policy parameters, with closed-form scores that are easy to plug into several policy-based algorithms. The modularity of the approach, the availability of the code, and the explicit convergence analysis for a projected variant are strengths. However, the softmax policy-score formula omits the temperature factor, which is a load-bearing error for all discrete-action algorithms. The theoretical claims are also narrower than the abstract suggests: only a projected TLRPG variant is analyzed, and no sample-complexity comparison is provided. With those issues fixed or clearly qualified, the paper could make a solid contribution to the literature on structured policy parameterizations.","major_comments":[{"comment":"The softmax policy score omits the temperature factor β. Given the definition Z(z,β)_i = e^{β z_i} / Σ_j e^{β z_j}, the log-policy is log π_Θ(a|s) = β z_{s,a} − log Σ_b e^{β z_{s,b}}, whose derivative with respect to z_{s,b} is β(1_{a=b} − Z_b). Equation (14) instead uses (1_{a=b} − Z_b) without the factor β. Because β is exposed in Algorithms 2, 5, and 6 and is described as a tunable temperature, the discrete-action TLRPG, TLRAC, TRTLRPO, and PTLRPO updates are not gradients of the stated objective unless β=1, which is not assumed. The omission propagates into the TRPO gradient and Fisher matrix in Eqs. (22)–(23), the PPO gradient in Eqs. (25)–(27), and the softmax verification in Appendix A. Please insert the missing β throughout, or explicitly restrict to β=1 and justify that restriction.","section":"Section III-A, Eq. (14)"},{"comment":"The convergence guarantee is proved for the projected update in Eq. (28), but Algorithm 3 as stated has no projection onto the convex set O. The text calls this a slight modification, yet the analyzed algorithm is not the implemented one unless O is the entire parameter space, in which case the boundedness used to verify Assumptions 2–4 must be justified separately. The theorem also covers only TLRPG, while contribution C3 and parts of the abstract suggest guarantees for the proposed methods more generally; no convergence analysis is given for TLRAC, TRTLRPO, or PTLRPO. Please clarify the relationship between the projected update and Algorithm 3, state whether O is active in the experiments, and temper the convergence claims accordingly.","section":"Section III-D, Theorem 1 and Eq. (28)"},{"comment":"The abstract claims that the proposed methods \"reduce computational and sample complexities\" relative to NN models, but no sample-complexity analysis is provided. Theorem 1 yields an O(1/√H) stationarity rate for the projected TLRPG variant, which is a standard rate for stochastic nonconvex optimization and does not quantify sample complexity in comparison with NN policies. The experimental evidence supports claims about parameter efficiency and faster empirical convergence in terms of return per episode, but not a formal reduction in sample complexity. Please revise the claims in the abstract and introduction to match what is actually shown.","section":"Abstract; Section I"},{"comment":"The low-rank premise is directly validated only for the Pendulum environment. For MountainCar and the wireless communications setup, no analogous low-rankness diagnostics are provided, so the assertion that \"many MDPs admit lower-dimensional representations\" is not established for the other benchmarks. Since the expressiveness of the proposed policy is bounded by the chosen PARAFAC rank, the paper should either provide similar low-rankness evidence for the remaining environments or explicitly state the scope of the empirical claim.","section":"Section IV-A, Fig. 1"}],"minor_comments":[{"comment":"In the displayed bound of the proof, the first term is missing the gradient operator on the second log-policy term; it should read ∇_{Θ_i} log p_g(Θ')(τ_T) rather than log p_g(Θ')(τ_T).","section":"Section III-D, proof of Theorem 1"},{"comment":"The statement that the softmax function is 1-Lipschitz should account for the temperature parameter β; with the definition in Section III-A, the Lipschitz constant is proportional to β, which changes the constants in Assumptions 2–4 but not their validity for bounded β.","section":"Appendix A, softmax case"},{"comment":"The sentence claiming that the only hyper-parameter needing tuning in the PARAFAC model is the rank K overlooks the state-space discretization grid and the distributional parameters σ and β, which are also tuned in the experiments; please qualify the statement.","section":"Section II-B"},{"comment":"The notation I_{\\hat{s}=i_s} is confusing because \\hat{s} is a state index while i_s is a tensor index; consider defining the delta explicitly to avoid ambiguity.","section":"Section III-A, Eq. (12)"},{"comment":"The statement that NN methods learn low-rank representations of policies is inferred from a single environment; consider softening the wording to reflect the limited empirical basis.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The missing β factor in Eq. (14) is the main technical problem and is clearly fixable in the equations. I recommend that the editor ask the authors to state explicitly whether the released code already includes the β factor; if it does, the paper should be corrected to match the implementation. I would also ask that the authors clarify whether the convergence theorem applies to the implemented unprojected updates, because the current text analyzes a projected variant that is not presented as the practical algorithm. The novelty over the authors' prior matrix low-rank policy work is incremental, but the tensor extension and the breadth of algorithms make it suitable for a methods-oriented journal if the technical issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper is a clean, useful extension of the same group's matrix low-rank policy work to PARAFAC tensor policies, and the Gaussian half is in good shape. The softmax half has a genuine bug: Eq. (14) drops the temperature factor β when differentiating the softmax log-policy. Since β is a tunable parameter throughout, the discrete-action algorithms (TLRAC, TRTLRPO, PTLRPO) are not optimizing the stated objective unless β=1. That needs fixing before the discrete results can be trusted.\n\nThe novel content is real: PARAFAC structure on Gaussian means and softmax logits, multi-dimensional actions, four algorithms, and a convergence theorem for the projected PG variant. The Gaussian policy scores are correct, the algorithms are clearly specified, the experiments are honest (100 runs, interquartile ranges, code available), and the wireless example is a good non-toy data point. The claim that tensor low-rank policies match NN returns with fewer parameters is supported in most tested settings.\n\nSoft spots, in order. First, the β omission. Differentiating β z_a − log Σ e^{β z_b} gives β(I_{a=b} − softmax_b), yet Eq. (14) lacks β. This propagates into the TRPO/PPO gradients and Fisher matrix, and the convergence theorem's assumptions are verified with the same incorrect expression. The paper never restricts β=1. So the discrete-action experiments and softmax convergence guarantees are invalid as written. It is fixable (include the factor or absorb β into the logits) but it must be fixed.\n\nSecond, Theorem 1 covers a projected version of TLRPG, while Algorithm 3 has no projection. The paper flags the modification honestly, but the conclusion still claims the algorithms are 'convergent under mild conditions' without the caveat. Overstatement, though minor.\n\nThird, the low-rank premise is empirically checked only on Pendulum (Fig. 1). The other experiments imply it holds for MountainCar and the wireless problem, but direct evidence is thin. Not fatal, but worth knowing.\n\nWho this is for: researchers working on sample-efficient policy-based RL with structured state spaces, or tensor methods in RL. The Gaussian parts are a solid contribution; the softmax parts are promising but currently unreliable.\n\nYes, send it to peer review. The β issue is exactly what referees are for, and the positive core deserves evaluation. A revised version that fixes Eq. (14), re-runs the discrete experiments, and qualifies the theorem would be worth publishing.","headline":"Solid Gaussian-policy tensor low-rank extension, but the softmax policy score drops the temperature β; a fixable bug that undermines the discrete-action results as written.","tokens_in":21265,"tokens_out":3840,"would_cite":false,"duration_ms":35593,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low-rank tensor policies can match neural-network returns in reinforcement learning while using fewer parameters and converging faster.","keywords":["reinforcement learning","policy gradient methods","tensor low-rank approximation","PARAFAC decomposition","actor-critic","trust-region policy optimization","proximal policy optimization","sample complexity"],"falsifier":"Train a neural-network policy on a control task whose optimal policy is intrinsically high rank, for example a state space where nearby states require unrelated actions, then discretize the states and PARAFAC-decompose the resulting parameter tensor at small rank. If the normalized Frobenius error stays large even for rank comparable to the tensor's maximum rank, or if the rank needed for small error grows with the number of discretization bins, the low-rank premise fails and the tensor policy cannot match the neural network.","tokens_in":20322,"feed_emoji":"🤖","tokens_out":6848,"duration_ms":65032,"temperature":0.7,"pith_summary":"Policy-based reinforcement learning typically maps states to action-distribution parameters with a neural network. This paper claims that for many problems those parameter maps have hidden low-dimensional structure: if the parameters are arranged in a tensor over discretized states, a PARAFAC low-rank model with a small rank captures them nearly as well as a neural network. The paper derives the policy-score formulas for tensor low-rank Gaussian and softmax policies, plugs them into policy-gradient, actor-critic, TRPO, and PPO updates, and proves that a projected version of the tensor low-rank policy-gradient method converges to a stationary point at rate $O(1/\\sqrt{H})$. The payoff, if the claim is right, is that RL agents can be trained with far fewer parameters and fewer samples while achieving comparable returns.","feed_headline":"Tensor low-rank policies match neural networks with fewer parameters","feed_subtitle":"PARAFAC low-rank policies converge faster and use fewer parameters while matching returns in tested RL benchmarks.","key_machinery":"The central object is the PARAFAC decomposition of a policy-parameter tensor: a tensor $\\Theta\\in\\mathbb{R}^{N_1\\times\\cdots\\times N_D}$ of rank $K$ is written as $\\sum_{k=1}^K \\Theta_1(:,k)\\circ\\cdots\\circ \\Theta_D(:,k)$, with factors $\\Theta_d\\in\\mathbb{R}^{N_d\\times K}$. This replaces $\\prod_d N_d$ parameters by $K\\sum_d N_d$ parameters. Its role in the argument is to make the policy score computable and sparse: the key identity shows that $\\partial \\log\\pi_\\Theta(a|s)/\\partial[\\Theta_d]_{i,k}$ is nonzero only when $i$ matches the $d$-th coordinate of state $s$, and equals the standard log-density derivative times $\\prod_{j\\ne d}[\\Theta_j]_{i_j,k}$. For Gaussian policies this gives the score formula for the mean tensor, for softmax policies the score formula for the logit tensor, and the same scores feed the critic gradient, the TRPO Fisher matrix, and the PPO clipping objective.","core_discovery":"The central claim is that the state-dependent parameters of a policy—the means of a Gaussian policy or the logits of a softmax policy—can be collected into a tensor and constrained to have low PARAFAC rank without sacrificing performance. Under this parametrization the policy score $\\partial \\log \\pi_\\Theta(a|s)/\\partial [\\Theta_d]_{i,k}$ factorizes into the scalar derivative of the log-density with respect to the tensor entry times the product of the other factors, so every policy-gradient-family algorithm only needs these sparse entrywise scores. The paper's Theorem 1 states that the projected tensor low-rank PG update reaches an $O(1/\\sqrt{H})$-stationary point of the expected-return objective under bounded rewards and Lipschitz and bounded-score assumptions. Empirically, PARAFAC decompositions of neural-network-trained policies in Pendulum show negligible normalized Frobenius error at rank 5 for Gaussian means and rank 20 for softmax logits, and the tensor low-rank variants of actor-critic, TRPO, and PPO match or exceed neural-network returns in MountainCar, Pendulum, and a wireless scheduling problem while using fewer parameters and converging faster.","pith_inferences":["If the empirical low-rankness observed in Pendulum holds broadly, then low-rank tensor policies could serve as a compression target for trained neural-network policies: discretize the learned parameter map and PARAFAC-decompose it to obtain a smaller deployable policy.","The practical gains should grow with state-space dimension, but only if a sufficiently fine discretization of the state space is affordable; for high-dimensional continuous states the discretization step itself may dominate the cost, and adaptive or hierarchical grids would be a natural extension the paper does not develop.","The same factorization could be applied to model-based settings: if transition and reward tensors are also low rank, a fully low-rank pipeline of dynamics, value, and policy could reduce sample complexity further than actor-only low rank.","A testable prediction is that the rank needed for near-optimal returns is controlled by the intrinsic dimensionality of the MDP's latent state, not by the nominal state dimension; environments with high nominal dimension but low latent dimension should show the largest gains."],"forward_implications":["Tensor low-rank variants of REINFORCE, actor-critic, TRPO, and PPO exist and require only the policy scores, so any policy-gradient algorithm can adopt the model by substituting these scores.","Parameter count drops from exponential in the state dimension to linear in the factor matrices, which is what makes training faster and less sample-hungry if the low-rank premise holds.","The projected tensor low-rank PG algorithm converges to an $O(1/\\sqrt{H})$-stationary point, giving a convergence guarantee for low-rank tensor policies.","In the tested benchmarks, median returns are similar to neural-network policies, with most tensor variants converging faster and with lower variance, so practitioners can expect cheaper training with comparable final performance."],"supporting_citations":[{"why":"Supplies evidence that value functions of classical control problems are empirically low rank when arranged as tensors, the observation this paper extends from value parameters to policy parameters.","marker":"[18]"},{"why":"Introduces matrix low-rank Gaussian policy models for actor-critic and TRPO that this paper generalizes to tensors and softmax policies.","marker":"[19]"},{"why":"Supplies the premise that many MDPs have low-rank or low-dimensional structure, the structural reason to expect low-rank policies.","marker":"[25]"},{"why":"Defines TRPO, whose actor update is reduced to policy scores and is therefore directly reusable with tensor low-rank policies.","marker":"[42]"},{"why":"Defines PPO, whose clipping objective and gradient in terms of policy scores is used by the proximal tensor low-rank algorithm.","marker":"[43]"},{"why":"Provides the weakly convex optimization framework and Moreau envelope used in the proof of Theorem 1.","marker":"[47]"},{"why":"Defines the PARAFAC decomposition that is the paper's central parametrization of policy parameters.","marker":"[44]"},{"why":"Establishes the policy gradient theorem, the basic identity that all score-based policy updates rely on.","marker":"[8]"}],"fun_headline_variants":["PARAFAC low-rank policies cut RL sample and compute costs","Low-rank tensor policies match neural nets with fewer parameters","Tensor low-rank RL: faster convergence, leaner than NNs","RL policy gradients via low-rank tensors: less is more","Multilinear low-rank policies rival neural networks in RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the premise that, after discretizing the state space, the tensor of policy parameters for a good policy is well approximated by a PARAFAC model with small rank; if a problem lacks that structure, the tensor policy cannot match neural-network capacity.","fun_headline_variants_meta":{"raw":{"variants":["PARAFAC low-rank policies cut RL sample and compute costs","Low-rank tensor policies match neural nets with fewer parameters","Tensor low-rank RL: faster convergence, leaner than NNs","RL policy gradients via low-rank tensors: less is more","Multilinear low-rank policies rival neural networks in RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000451,"raw_usage":{"total_tokens":2289,"prompt_tokens":980,"completion_tokens":1309,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":1226}},"tokens_in":596,"tokens_out":1309,"duration_ms":10079,"temperature":1.0,"reasoning_tokens":1226,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:23:02.204967+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a neural-network policy on a control task whose optimal policy is intrinsically high rank, for example a state space where nearby states require unrelated actions, then discretize the states and PARAFAC-decompose the resulting parameter tensor at small rank. If the normalized Frobenius error stays large even for rank comparable to the tensor's maximum rank, or if the rank needed for small error grows with the number of discretization bins, the low-rank premise fails and the tensor policy cannot match the neural network.","supporting_citations":[{"cited_title":"Tensor and matrix low- rank value-function approximation in reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies evidence that value functions of classical control problems are empirically low rank when arranged as tensors, the observation this paper extends from value parameters to policy parameters."},{"cited_title":"Matrix low-rank approximation for policy gradient methods,","cited_arxiv_id":null,"evidence_quote":"Introduces matrix low-rank Gaussian policy models for actor-critic and TRPO that this paper generalizes to tensors and softmax policies."},{"cited_title":"Flambe: Structural complexity and representation learning of low rank MDPs,","cited_arxiv_id":null,"evidence_quote":"Supplies the premise that many MDPs have low-rank or low-dimensional structure, the structural reason to expect low-rank policies."},{"cited_title":"Trust region policy optimization,","cited_arxiv_id":null,"evidence_quote":"Defines TRPO, whose actor update is reduced to policy scores and is therefore directly reusable with tensor low-rank policies."},{"cited_title":"Stochastic model-based minimization of weakly convex functions,","cited_arxiv_id":null,"evidence_quote":"Provides the weakly convex optimization framework and Moreau envelope used in the proof of Theorem 1."},{"cited_title":"PARAFAC. tutorial and applications,","cited_arxiv_id":null,"evidence_quote":"Defines the PARAFAC decomposition that is the paper's central parametrization of policy parameters."},{"cited_title":"Policy gradient methods for reinforcement learning with function approximation,","cited_arxiv_id":null,"evidence_quote":"Establishes the policy gradient theorem, the basic identity that all score-based policy updates rely on."}],"review_version":1}