{"id":"dd4bc590-f328-48e4-a955-dc0051ecf176","arxiv_id":"2505.04193","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"TERL minimizes an estimated upper bound on action-trajectory entropy, producing more periodic and more robust locomotion policies in simulation.","lead":"TERL adds a reward bonus for actions that are easy to predict from recent states and the previous action, nudging reinforcement learning policies toward simple, repetitive behaviors. The authors report better scores and smaller performance drops under noise and mass changes than several baselines on simulated locomotion tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The trajectory-entropy objective optimizes an unvalidated variational upper bound; the paper never measures the gap between H and H_u, so the claimed entropy-minimization mechanism is not established.","rationale":"The paper's empirical package is credible: 20 seeds per task, comparisons against LZ-SAC, RPC, and SAC, and independent compression/periodicity evidence that TERL produces more compressible action sequences. The central theoretical scaffolding, however, has a real soft spot: Eq. (6) maximizes a variational lower bound of the stated objective, and the tightness of the bound is assumed rather than verified. Because q_ψ is trained on the agent's own replay buffer and is also the source of the reward bonus, the policy and the density estimator co-adapt; without a gap measurement, the method may be reducing a learned proxy rather than the true trajectory entropy. This concern is load-bearing because the paper's causal claim is that minimizing trajectory entropy is what confers predictability and robustness. The reader's weakest_assumption correctly identified the q_ψ estimator and the missing gap measurement, but its factorization sub-claim is wrong: conditioning on future states does reduce conditional action entropy even for a Markov policy, since future states depend on past actions. Correcting that error does not rescue the paper; it sharpens the issue to the variational gap. The concrete test of measuring H_u - Ĥ on held-out trajectories would settle whether the bound is tight enough to support the claim. Pending that check, the conditional verdict remains appropriate: the empirical results are promising and plausible, but the mechanism is not yet established. No change to the reader's verdict is needed.","tokens_in":14800,"tokens_out":8673,"duration_ms":88219,"concrete_test":"For each of the six tasks, after training the final TERL policy, collect a held-out set of 1000 trajectories (z_{1:T}, a_{1:T-1}) from the environment. Train a high-capacity conditional density estimator (e.g., a normalizing flow or a large autoregressive transformer) on a separate training split to approximate p(a_{1:T-1}|z_{1:T}), and evaluate its average negative log-likelihood on the held-out split as an estimate Ĥ of the true trajectory entropy. Also evaluate the learned q_ψ average negative log-likelihood H_u on the same held-out split. If H_u - Ĥ is large (e.g., more than 10% of H_u), the upper bound is loose and the method is not demonstrably minimizing trajectory entropy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that minimizing trajectory entropy yields predictable, high-scoring, robust policies. However, the method never minimizes H(a_{1:T-1}|z_{1:T}) directly. In Section IV-A, Eq. (5) defines an upper bound H_u = -E[Σ log q_ψ(a_t|z_t,z_{t+1},a_{t-1})], and Eq. (6) maximizes E[Σ(r + α log q_ψ)], i.e., maximizes a lower bound of the stated objective in Eq. (1) because H ≤ H_u implies E[Σr] - αH_u ≤ E[Σr] - αH. The gap is the expected KL divergence D_KL(p(a_{1:T-1}|z_{1:T}) || q(a_{1:T-1}|z_{1:T})). This gap is never measured; the paper reports only bzip2 compression as a proxy. Since q_ψ is trained on the agent's own replay buffer and simultaneously used as the reward model, the policy and the estimator co-adapt. A policy can reduce H_u by making actions predictable to q_ψ without necessarily reducing the true conditional entropy H if q_ψ is overconfident or poorly calibrated. If the bound is loose, the reported robustness gains (Tables IV-VI) could stem from reduced action variance rather than from trajectory-level compressibility, weakening the causal story. Note also that the reader's factorization objection is technically incorrect: with a Markov policy, future states z_{t+1:T} depend on past actions, so H(A_t|Z_t,Z_{t+1:T}) ≤ H(A_t|Z_t); the trajectory-level entropy is genuinely smaller than the per-step sum. The load-bearing concern is therefore the unverified tightness of the variational upper bound, not the factorization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Trajectory Entropy Reinforcement Learning (TERL), a regularizer for SAC that penalizes an estimate of the conditional entropy of action trajectories given a learned state-representation trajectory. The claimed motivation is that low trajectory entropy induces simple, predictable, and robust policies. The authors derive a variational upper bound on this entropy using an autoregressive action-prediction model q_psi, add alpha times log q_psi to the reward, and jointly train the policy, encoder, and prediction model. Experiments on six continuous-control locomotion tasks compare TERL with SAC, LZ-SAC, RPC, and the original RPC implementation, reporting higher average scores on most tasks, smaller performance drops under mass/gravity changes, action noise, and observation noise, and higher bzip2 compressibility of action trajectories. The derivation in Eq. (3) is correct as stated, and the empirical study is substantial (20 seeds per task, 10^6 training steps).","tokens_in":15202,"tokens_out":8192,"duration_ms":90801,"significance":"If the mechanism were established, the paper would offer a simple, task-independent inductive bias for robustness in high-dimensional control, with potential value for legged locomotion and human-interactive systems. The strengths are the correct variational inequality in Eq. (3), the use of a public SAC codebase, the breadth of tasks, and the falsifiable compression proxy. However, the central causal claim is currently supported only indirectly: the paper never measures the gap between the true trajectory entropy and the variational upper bound, and the reward model is trained on the agent's own replay buffer while being used to shape the policy. These issues do not invalidate the empirical results, but they leave the interpretation of the gains as 'trajectory entropy minimization' unverified. The paper's own limitation paragraph in Section VI acknowledges that the simplicity bias can hurt performance on tasks requiring complex behavior and that alpha is chosen by grid search, but it does not acknowledge the bound-mismatch or co-adaptation concerns.","major_comments":[{"comment":"The implemented objective is not the stated objective, and the direction of the approximation is not flagged. Eq. (3) gives H <= H_u, so the objective in Eq. (1), E[sum r] - alpha H, satisfies E[sum r] - alpha H >= E[sum r] - alpha H_u. Eq. (6) maximizes E[sum(r + alpha log q_psi)] = E[sum r] - alpha H_u, which is a lower bound of Eq. (1), not an equivalent form. The slack is the expected KL divergence D_KL(p(a_{1:T-1}|z_{1:T}) || q_psi(a_{1:T-1}|z_{1:T})), and this gap is never measured. Because q_psi is trained on the agent's own replay buffer and simultaneously used as the reward model, the policy and the estimator can co-adapt, so reducing H_u need not reduce the true conditional entropy H. Please report an empirical estimate of the gap (for example, held-out log-likelihood of recorded trajectories, or a comparison between H_u and a direct Monte Carlo entropy estimate), and revise the wording so the paper does not claim to minimize H itself. I note that the trajectory-level entropy is not identical to the sum of per-step entropies, because future states z_{t+1:T} can carry information about a_t under the Markov policy; the load-bearing issue is the unmeasured bound tightness, not the factorization.","section":"Section IV-A/B, Eqs. 3-6"},{"comment":"The encoder e_phi is optimized jointly with the policy and the prediction model. Since z_{t+1} is a function of s_{t+1}, and s_{t+1} depends on a_t, the encoder can reduce the upper bound by encoding information about the current action into the next representation, rather than by making the policy's action distribution more predictable. In the extreme, if z_{t+1} encodes a_t, then q_psi can predict a_t almost perfectly regardless of how erratic the policy is. This means the reported low conditional entropy may reflect representation leakage rather than simplicity of the policy. The paper should add a control experiment with a fixed or information-bottlenecked encoder, or separately report the marginal entropy of action sequences and the conditional entropy given z, to show that the regularizer is not being gamed by the learned representation.","section":"Section IV-C, Eq. 11, and Figure 3"},{"comment":"The robustness comparison is presented primarily as percentage performance drops relative to each method's unperturbed score. Because TERL's unperturbed scores in Table III are much higher than those of the baselines, the percentage metric can look favorable even when absolute losses are comparable or larger. The aggregated normalized curves in Figure 5 are more informative, but the paper reports no confidence intervals or significance tests for the drop percentages. Please report raw perturbed rewards with intervals (the 20 seeds would support paired bootstrap or Wilcoxon tests), and state whether the differences in Tables IV-VI are statistically significant.","section":"Section V-C, Tables IV-VI, and Figure 5"}],"minor_comments":[{"comment":"The text after Eq. (2) says the expectation is over the joint distribution of action sequences and 'representation sequences a_{1:T-1}'; the second occurrence should be z_{1:T}.","section":"Section III, Eq. (2)"},{"comment":"In Algorithm 1, the line 'Compute upper bound: u <- E[log q_psi]' is inconsistent with Eq. (7), where the reward bonus is the per-transition log q_psi(a_t|z_t,z_{t+1},a_{t-1}), not an expectation over a batch. Please clarify whether u is a per-sample value or a batch mean.","section":"Algorithm 1, Section IV-B"},{"comment":"The text says the robustness test 'scales the gravity of the robot body,' while the tables and captions describe 'mass changes'; these are different perturbations and the paper should use consistent terminology.","section":"Section V-C, first paragraph"},{"comment":"The hyperparameter search is described as a grid over alpha in [10^-5, 10^-4], but Table VII sweeps alpha = 0.001 and 0.01. Please state whether the ablation extends outside the tuned range and whether those larger values were considered for the other tasks.","section":"Section V-A and Table VII"},{"comment":"The caption says the joint optimization uses the 'lower bound of the trajectory entropy,' while Eq. (5) derives an upper bound; the figure caption should match the equation.","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper has promising empirical results and a correct variational starting point, but the central mechanism is not yet established because the implemented objective is a lower bound of the stated one and the bound gap is never measured. The encoder-leakage concern is particularly important and needs an experimental control. I recommend major revision rather than rejection; the deficiencies are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [colleague],\n\nQuick take on arXiv:2505.04193 (TERL). The paper proposes adding a learned action-prediction bonus to SAC, where the predictor conditions on the previous action and two consecutive state representations, and frames this as minimizing trajectory entropy. The specific method is new relative to LZ-SAC (which predicts from actions only) and RPC, and the experiments are solid: 20 seeds, six locomotion tasks, with consistent improvements over SAC, LZ-SAC, and RPC, plus robustness checks under mass changes and noise. The bzip2 compression results give an external sanity check that the learned policies really do produce more compressible action trajectories.\n\nWhere the paper gets shaky is the information-theoretic framing. The derivation in Eq. 3 is correct as an upper bound on trajectory entropy, but the paper never measures how tight that bound is, and the objective actually maximized is a lower bound of the stated reward-entropy trade-off. Since the predictor q_psi is trained on the same replay buffer that the policy is shaping, the policy can decrease the surrogate by making actions predictable to the current q_psi without necessarily reducing the true conditional entropy. The paper's own compression measure (bzip2) suggests the effect is real, but it doesn't close the gap; we never learn whether the upper bound is anywhere near tight. A revision should either estimate the KL gap or at least soften the entropy-minimization claim.\n\nOne point where I part ways with the reader's initial assessment: the claim that the trajectory entropy factorizes into per-step entropies under a Markov policy is wrong. Future states carry information about past actions, so the trajectory-level conditional entropy is genuinely lower than the sum of per-step entropies. The trajectory framing is not vacuous, even if the practical estimator only conditions on one-step-ahead states.\n\nOther soft spots: no code release, and the alpha grid search covers only two values. Neither is disqualifying.\n\nBottom line: this is a plausible regularizer with good empirical support, but the causal story is overclaimed. I'd send it to review with a request for major revision tightening the theoretical claims and either measuring the bound gap or adding an ablation that isolates the effect of the predictor's self-reference. Worth engaging with; I'd bring it to a reading group.","headline":"A useful incremental regularizer with strong locomotion results, but the trajectory-entropy mechanism is asserted more strongly than the evidence supports.","tokens_in":15676,"tokens_out":4025,"would_cite":true,"duration_ms":40358,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Minimizing trajectory entropy makes reinforcement-learned policies more predictable, higher-scoring, and more robust to perturbation.","keywords":["trajectory entropy","simplicity inductive bias","reinforcement learning","robust control","action prediction","variational inference","locomotion","predictable policies"],"falsifier":"Estimate the true trajectory entropy of a trained TERL policy by Monte Carlo sampling with a high-capacity conditional density estimator and compare it to the variational upper bound from Eq. (5); if the bound is loose or the true entropy does not decrease during training, the method is not actually optimizing the stated objective. Alternatively, replace $q_\\psi$ with a fixed or random predictor during training: if the reported performance and robustness gains persist, the entropy-minimization objective is not the cause.","tokens_in":14615,"feed_emoji":"🤖","tokens_out":4069,"duration_ms":40467,"temperature":0.7,"pith_summary":"The paper claims that adding a trajectory-entropy regularizer to a deep reinforcement learning objective biases policies toward simple, repetitive action sequences, and that this improves both reward and robustness on high-dimensional locomotion tasks. It estimates the trajectory entropy $H(a_{1:T-1} | z_{1:T})$ with a learned variational action prediction model $q_\\psi(a_t | z_t, z_{t+1}, a_{t-1})$, then rewards actions that this model can anticipate. The resulting method, Trajectory Entropy Reinforcement Learning (TERL), is built as an extension of soft actor-critic in which the environment reward is augmented with an information-based bonus. On six simulated locomotion tasks, the learned policies produce more cyclical and consistent gaits, achieve higher average scores than the baselines on most tasks, and show smaller performance drops under mass changes, action noise, and observation noise.","feed_headline":"Trajectory entropy regularizer makes RL policies robust","feed_subtitle":"A predictor rewards predictable actions, giving cyclical gaits and smaller drops under noise and mass changes.","key_machinery":"The central object is the trajectory entropy $H(a_{1:T-1} | z_{1:T})$, the number of bits needed to describe an action sequence after seeing the corresponding state-representation sequence. Since this quantity is intractable, the paper introduces an autoregressive variational distribution $q_\\psi(a_t | z_t, z_{t+1}, a_{t-1})$ that upper-bounds it, and turns the bound into an information-regularized reward $r^*(s_t, a_t) = r(s_t, a_t) + \\alpha \\log q_\\psi(a_t | z_t, z_{t+1}, a_{t-1})$. This reward bonus pushes the policy toward actions that are easy to predict from the previous action and two consecutive state representations, which in practice produces periodic, compressible behaviors.","core_discovery":"The paper's central claim is that minimizing the conditional entropy of entire action trajectories given state-representation trajectories is a tractable and effective simplicity inductive bias for reinforcement learning. Because the true conditional distribution of actions is unknown, the entropy is replaced by a variational upper bound built from an autoregressive action prediction model, and this upper bound is folded into the reward. The discovery, as the authors state it, is that policies optimized for this information-regularized reward are more predictable, achieve higher scores on the majority of the evaluated locomotion tasks, and degrade more gracefully under environmental perturbations than policies trained by prior baseline methods.","pith_inferences":["Editorial inference: because the policy is Markov and the variational model factorizes autoregressively, the trajectory-level entropy reduces to a sum of per-step conditional entropies, so the trajectory formulation may not be essential; a per-step action-prediction bonus could produce equivalent behavior.","Editorial inference: if the benefit comes from smoothness and compressibility, simpler regularizers such as action-rate penalties or Lipschitz constraints on the policy might match TERL's results, and a direct comparison would clarify whether the information-theoretic form adds anything beyond these.","Editorial inference: the robustness gains are reported zero-shot in simulation, which suggests a concrete testable extension of transferring TERL-trained policies to physical legged robots with added mass and actuator noise.","Editorial inference: the paper relies on grid search to set $\\alpha$; a testable improvement would be to tie $\\alpha$ to task properties such as action dimension or state dimension, which the authors list as future work."],"forward_implications":["Policies trained with TERL produce action trajectories that are more cyclical and consistent, and that compress to smaller sizes under a lossless compression algorithm such as bzip2.","TERL achieves higher average scores than LZ-SAC, RPC, and SAC on the majority of the six locomotion tasks, with the largest gains on the high-dimensional H1 Walk and Humanoid Walk tasks.","TERL shows smaller performance drops than the baselines under mass scaling, action noise, and observation noise, especially at small noise strengths.","Because SAC is a special case of TERL with $\\alpha = 0$, the reported improvements are attributed specifically to the trajectory entropy regularizer rather than to the underlying actor-critic machinery.","The simplicity bias can harm performance on tasks that inherently demand complex and unpredictable behavior, a limitation the paper itself acknowledges.","The hyperparameter $\\alpha$ trades off reward against trajectory entropy, and increasing it monotonically increases the compressibility of learned behavior on the Walker Walk task."],"supporting_citations":[{"why":"Supplies the LZ-SAC baseline that learns simple policies from sequence-compression priors, and motivates the trajectory-compression evaluation using bzip2.","marker":"[12]"},{"why":"Provides the RPC baseline that induces simple policies by minimizing mutual information between state sequences and representation sequences.","marker":"[11]"},{"why":"Gives the soft actor-critic algorithm that TERL extends and the policy-entropy baseline SAC, which is the $\\alpha = 0$ special case.","marker":"[19]"},{"why":"Supplies five of the six evaluation tasks from the DeepMind Control Suite.","marker":"[24]"},{"why":"Supplies the H1 Walk task from HumanoidBench, the highest-dimensional evaluation environment.","marker":"[23]"},{"why":"Provides the open-sourced PyTorch SAC implementation that both TERL and the LZ-SAC baseline are built on for a fair comparison.","marker":"[48]"}],"fun_headline_variants":["Trajectory entropy makes RL agents robust","Action path entropy for predictable, robust RL","Reduce trajectory entropy, gain control robustness","Entropy-regulated actions yield robust policies","Simplicity via trajectory entropy for robust RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The learned prediction model $q_\\psi$ is assumed to estimate the true conditional distribution of actions accurately enough that $\\alpha \\log q_\\psi$ is a faithful surrogate for reducing true trajectory entropy; if this assumption fails, the reward bonus rewards actions that are predictable to the learned model rather than actions that actually lower trajectory entropy.","fun_headline_variants_meta":{"raw":{"variants":["Trajectory entropy makes RL agents robust","Action path entropy for predictable, robust RL","Reduce trajectory entropy, gain control robustness","Entropy-regulated actions yield robust policies","Simplicity via trajectory entropy for robust RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3169,"prompt_tokens":863,"completion_tokens":2306,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":2242}},"tokens_in":479,"tokens_out":2306,"duration_ms":16977,"temperature":1.0,"reasoning_tokens":2242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:35:15.608273+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Estimate the true trajectory entropy of a trained TERL policy by Monte Carlo sampling with a high-capacity conditional density estimator and compare it to the variational upper bound from Eq. (5); if the bound is loose or the true entropy does not decrease during training, the method is not actually optimizing the stated objective. Alternatively, replace $q_\\psi$ with a fixed or random predictor during training: if the reported performance and robustness gains persist, the entropy-minimization objective is not the cause.","supporting_citations":[{"cited_title":"Reinforcement learning with simple sequence priors,","cited_arxiv_id":null,"evidence_quote":"Supplies the LZ-SAC baseline that learns simple policies from sequence-compression priors, and motivates the trajectory-compression evaluation using bzip2."},{"cited_title":"Robust predictable control,","cited_arxiv_id":null,"evidence_quote":"Provides the RPC baseline that induces simple policies by minimizing mutual information between state sequences and representation sequences."},{"cited_title":"Improving sample efficiency in model-free reinforcement learning from images,","cited_arxiv_id":null,"evidence_quote":"Provides the open-sourced PyTorch SAC implementation that both TERL and the LZ-SAC baseline are built on for a fair comparison."}],"review_version":1}