{"id":"e33841cf-7f6a-456c-b433-c196298151c3","arxiv_id":"2505.16734","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A reinforcement learning regularizer that maximizes trajectory-level total correlation produces simpler, more compressible policies that are more robust to perturbations.","lead":"This paper adds a new objective to reinforcement learning: make the whole trajectory of states and actions as predictable as possible, not just the next step. The resulting policies are more periodic and compressible, and they hold up better under noise and changes to the robot's dynamics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The optimized surrogate is a state-conditional KL regularizer, not a total-correlation objective; the central claim needs evidence that optimizing it actually increases true total correlation.","rationale":"The reader identified the vacuous lower bound as the weakest assumption. My stress-test goes one step further: the bound is not merely a poor estimator; its functional form shows it minimizes KL(f||q_history) and KL(π||q_history), i.e. it makes current-state information redundant given history. This is a conditional-independence objective, and it can be maximized at zero total correlation. Therefore the connection between the optimized objective and the claimed total-correlation mechanism is not just numerically loose but directionally suspect. The empirical results are substantial: 20 seeds on DMC, ablations, code release, and honest disclosure of the bound in Section 6. Those results support a weaker claim, namely that a history-based consistency regularizer improves robustness and compressibility. The concern is about attribution and framing, not about the existence of the empirical effect. The proposed test—estimating true total correlation and checking monotonicity in Ip—would settle whether the title's mechanism is actually at work. Since this is a major revision of the paper's interpretation rather than a demonstration that the method fails, the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed.","tokens_in":23191,"tokens_out":12370,"duration_ms":118462,"concrete_test":"Estimate the true total correlation of trajectories produced by MTC and baselines, and check whether it tracks the surrogate. On Cheetah Run and Walker Stand, train MTC with Ip in {-0.5, -3, -7} and SAC (same seeds). For each final stochastic policy/encoder, sample 10^4 length-100 trajectories from the true initial-state distribution and policy/encoder noise, recording (z_t, a_t). Estimate C_true = Σ_t I(z_t; z_{<t}, a_{<t}) + Σ_t I(a_t; z_{<t}, a_{<t}) with a consistent kNN/Kraskov estimator. Then compare (i) the ranking of estimated C_true across MTC-Ip values and SAC, and (ii) the ranking of the optimized surrogate eC from Eq. 2. If MTC does not produce higher estimated C_true than SAC, or if increasing Ip does not increase C_true, the central total-correlation claim is unsupported and the paper should be reframed as a conditional-independence/temporal-consistency regularizer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that adding a total-correlation objective to RL induces periodic, compressible, and robust policies. The algorithm optimizes Eq. 2, which Section 6 admits is always negative and not useful for estimating total correlation. The problem is not only vacuousness: Eq. 2 is a sum of negated KL divergences, i.e. eC = -Σ_t E[KL(fθ(z_{t+1}|s_{t+1}) || qη(z_{t+1}|history)) + KL(πφ(a_t|s_t) || qχ(a_t|history))]. Maximizing eC therefore minimizes the conditional dependence of current latent states and actions on the current state given the history. It rewards making the current state redundant, not making the trajectory statistically dependent. A policy that ignores the state and outputs iid actions, with q matching the marginals, maximizes eC (value 0) while having zero total correlation. Conversely, a deterministic periodic trajectory can have large true total correlation across episodes (due to shared phase randomness), yet the bound is loose and can be 0 because both f and q predict equally sharply. Thus the bound's gradients point toward conditional independence, not toward the total-correlation quantity in the title. The robustness and compressibility gains may still be real, but they are currently attributed to a mechanism the optimized objective does not implement. This mismatch is load-bearing because the novelty and interpretation of every empirical result rest on the total-correlation framing, and it is flagged by the authors' own Section 6 disclosure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Maximum Total Correlation Reinforcement Learning (MTC-RL), a modification of the RL objective that adds a total-correlation term over the induced trajectory of latent states and actions, justified as an inductive bias toward simple, compressible, and robust behavior. The authors derive a variational lower bound (Eq. 2) using history-based latent dynamics and action prediction models, integrate it into SAC-style off-policy optimization with an adaptive Lagrangian coefficient, and evaluate the method on DMC, Metaworld, and image-based DMC benchmarks. They report improved asymptotic performance, enhanced zero-shot robustness to observation/action noise and dynamics changes, higher trajectory compressibility, and stronger action predictability compared to SAC, RPC, LZ-SAC, and SPAC. The paper also includes a discussion section acknowledging that the lower bound is always negative and hence not useful for estimating the true total correlation.","tokens_in":23453,"tokens_out":4716,"duration_ms":39988,"significance":"If the central claim were established, MTC-RL would provide a principled, task-independent simplicity bias for RL with broad empirical appeal. The experimental evaluation is substantial and careful: 20 seeds on DMC, multiple benchmarks, ablations, and public code. The variational derivation in Appendix A.1 is mathematically correct as a lower bound, and the robustness and compressibility improvements are reproducible in principle. However, the theoretical interpretation is not supported: the optimized surrogate (Eq. 2) is a sum of negated KL divergences and is always non-positive, so maximizing it is not equivalent to maximizing total correlation. The paper explicitly concedes this in Section 6. The empirical gains may be real, but they are currently attributed to a mechanism that the optimized objective does not implement. This mismatch is load-bearing for the title, abstract, and interpretation of every empirical result. The paper therefore requires major revision, either by providing evidence that optimizing the surrogate increases true total correlation or by reframing the contribution as conditional-KL regularization.","major_comments":[{"comment":"The variational lower bound eC in Eq. (2) is always non-positive because it is a sum of negated KL divergences. The authors acknowledge this in Section 6, but the implication for the paper's central claim is stronger than \"not useful for estimation\": maximizing eC does not correspond to maximizing total correlation. Specifically, eC = -Σ_t E[KL(fθ(z_{t+1}|s_{t+1}) || qη(z_{t+1}|history)) + KL(πφ(a_t|s_t) || qχ(a_t|history))]. A policy that ignores the state and outputs independent actions drawn from the marginal, together with q matching the encoder and policy marginals, attains eC = 0, the theoretical maximum of the bound, while the true total correlation of the trajectory is near zero. Conversely, a deterministic periodic trajectory can have large true total correlation while eC is near zero because both fθ and qη are sharply peaked. Thus the bound's gradients push toward conditional independence of each variable from the current state given the history, not toward the multi-variable dependence measured by total correlation. The title and abstract claim that the method \"maximizes the total correlation within the induced trajectories\"; this claim is not established. To support it, the paper should measure the true total correlation on a tractable task (e.g., a low-dimensional MDP) and show that it increases when optimizing Eq. (2), or it should explicitly reframe the method as state-conditional KL regularization.","section":"§4.2, Eq. (2); §6"},{"comment":"The trajectory-compression result in Section 5.3 and Fig. 3 is presented as evidence that the method maximizes total correlation. However, compressibility is a direct consequence of the optimized objective: Eq. (2) explicitly minimizes the KL divergence between the encoder/policy and history-based predictive models, which by construction makes trajectories predictable and hence compressible. This is not an independent confirmation of the total-correlation mechanism; the same compression improvement would be expected from any regularizer that penalizes KL to a history-based predictor, including a simple action-smoothing baseline. To make the empirical claim non-circular, the paper should compare against a baseline that optimizes the same KL regularizer without the total-correlation interpretation, and should additionally report an independent estimate of total correlation (e.g., a non-parametric plug-in estimate) on the same trajectories.","section":"§5.3, Fig. 3"},{"comment":"The claim in §4.2 that the lower bound \"captures important aspects of the total correlation\" is substantiated only by a qualitative trajectory plot (Fig. 1), which does not establish a quantitative relationship between eC and the true total correlation. The gap between the true C and eC is D_KL(p(z_{1:T},a_{1:T-1}) || q(z_{1:T},a_{1:T-1})) plus additional non-negative terms from the marginal-substitution step in Eq. (11). This gap can be large and input-dependent, so the optimization of eC need not track the optimization of C. The paper should provide at least one empirical or analytic example where eC and C move in opposite directions, or a small-scale measurement showing that increasing eC indeed increases C. Without this, the central mechanism remains unsupported.","section":"§4.2, Eq. (2); Appendix A.1"}],"minor_comments":[{"comment":"The text in §5.4 states that MTC \"achieves higher average rewards than baselines on all tasks\" for Metaworld, while the caption of Fig. 4 says \"MTC is competitive to baselines\"; these statements should be reconciled.","section":"§5.4 vs. Fig. 4 caption"},{"comment":"The entry \"Replay buffer capacity 1 00 000\" contains a typo; it should read \"100 000\".","section":"Table 3, Appendix B.9"},{"comment":"Equation (3) writes r(s_T, a_T) separately in front of the sum over t=1 to T-1, which is redundant with the t=T term inside the sum and may confuse the indexing; please clean up the notation.","section":"Eq. (3)"},{"comment":"The statement that a vacuous lower bound can be useful for optimization \"as in the case of subtracting a constant offset\" is imprecise: a lower bound with an input-dependent gap is not the same as a constant offset, and maximizing a lower bound is only justified if the gap is controlled or if the bound is tight at the optimum. This point should be clarified or removed.","section":"§6"},{"comment":"Table 1 reports \"means over 20 seeds with 90% confidence interval\" but the values are shown as ± intervals; please clarify whether these are standard errors, standard deviations, or quantile-based intervals.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The empirical study is broad and the code is publicly available, but the theoretical framing in the title and abstract overstates what is optimized. The stress-test concern lands: the always-negative lower bound in Eq. (2) is a sum of negated KL divergences, and maximizing it is not equivalent to maximizing total correlation. The paper can likely be made acceptable if the authors either (a) measure true total correlation on a small problem and show the link, or (b) explicitly reposition the contribution as conditional-KL regularization and adjust the title and claims accordingly. Given the ICML format, the requested evidence is feasible within a revision cycle."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. The empirical recipe is real and the evaluation is serious, but the paper's central theoretical claim doesn't survive contact with its own objective. The authors propose to maximize total correlation of trajectories to induce simple, robust policies. The variational lower bound in Eq. 2 is derived correctly, but as Section 6 admits, it is always negative—a sum of negated KL divergences—so it is a vacuous estimator of total correlation. The stress-test note is right that this is worse than vacuousness: maximizing that bound pushes the policy toward conditional independence of the current state given history, not toward statistical dependence across the trajectory. A state-ignoring iid policy with matching marginals attains the bound's maximum (zero) while having literally zero total correlation. So the title's mechanism isn't what the algorithm implements.\n\nCredit where due. The empirical work is broad: 20 seeds on DMC, Metaworld, image-based DMC, robustness to observation noise, action noise, and mass changes, plus a compressibility metric. Code is public. The finding that the regularizer yields more compressible and more robust policies is credible and reproducible. The Appendix proof that RPC can be derived from this formulation is a genuinely nice conceptual link.\n\nSoft spots, in order. First, the total-correlation framing is an overclaim. The surrogate is a conditional-predictability regularizer; the authors need to either rename the contribution or show that maximizing the surrogate actually increases true total correlation. Second, Section 5.4 states MTC achieves higher average rewards than baselines on all Metaworld tasks, but their own Table 8 contradicts that: MTC is worse than SAC on Button-press-wall under action noise and worse than SAC on Plate-slide-back under observation noise. That overclaim should be corrected. Third, the hyperparameter 'coefficient m' is 1e-6 for DMC and 1e-4 for image-based, yet the action bound is weighted by m. With such tiny values, the action-consistency term is effectively ignored, while the ablation in Section 5.5 shows removing it matters. That inconsistency needs explanation; right now it undercuts the action-consistency story.\n\nVerdict: send it to serious peer review, but expect heavy revision. The empirical contribution is worth refereeing; the framing and internal consistency need real work. I'd bring it to reading group to discuss the surrogate gap.","headline":"Solid empirical regularizer, but the title's total-correlation mechanism is not what the optimized objective does; needs reframing and a few consistency fixes.","tokens_in":23988,"tokens_out":6482,"would_cite":true,"duration_ms":52968,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Total-correlation maximization makes RL trajectories simpler and more resilient to perturbations.","keywords":["total correlation","reinforcement learning","trajectory compressibility","robust control","variational lower bound","soft actor-critic","temporal consistency","simplicity bias"],"falsifier":"Train the same agent with the total-correlation reward replaced by a constant negative offset of matched magnitude: if compressibility and perturbation robustness are unchanged, the bound's internal predictive structure is not what matters. Alternatively, estimate the true total correlation of rollout trajectories from joint and marginal entropies before and after MTC training; if true total correlation decreases while the lower bound increases, the paper's proposed mechanism is refuted.","tokens_in":22956,"feed_emoji":"🤖","tokens_out":8380,"duration_ms":63998,"temperature":0.7,"pith_summary":"This paper tries to establish that adding an auxiliary objective to reinforcement learning (maximize the total correlation among the states and actions inside a trajectory) is a task-independent way to induce simpler behavior and, with it, better generalization. Total correlation measures how much information is saved by encoding an entire trajectory jointly instead of compressing each time step separately, so maximizing it pushes policies toward periodic, predictable, open-loop-like behavior. The authors derive a variational lower bound on this trajectory-level objective, fold it into soft actor-critic as a regularized reward, and report that the resulting agent (MTC) out-scores SAC, RPC, LZ-SAC, and SPAC on most DeepMind Control tasks, matches or exceeds them on Metaworld and image-based control, and degrades less under observation noise, action noise, and body-mass changes. If the claims hold, a single extra information-theoretic term could serve as a general robustness regularizer for continuous control.","feed_headline":"Total-correlation training yields simpler, sturdier RL policies","feed_subtitle":"Adding a trajectory-compressibility term to soft actor-critic improves periodic behavior and zero-shot noise resistance.","key_machinery":"The central object is the total correlation of a trajectory, $\\mathcal{C}(z_1; a_1; \\ldots; a_{T-1}; z_T) = \\mathbb{E}[\\log p(z_1, a_1, \\ldots, a_{T-1}, z_T) / \\prod_t p(z_t) \\prod_t p(a_t)]$, which quantifies how many nats are saved by encoding the whole trajectory rather than each state and action separately. The argument is carried by the variational lower bound in Eq. (2), which replaces the unknown marginals with the learned encoder $f_\\theta(z_{t+1}|s_{t+1})$ and policy $\\pi_\\phi(a_t|s_t)$ and adds a history-based dynamics model $q_\\eta(z_{t+1}|z_{1:t}, a_{1:t})$ and action prediction model $q_\\chi(a_t|z_{1:t}, a_{1:t-1})$. This bound turns total-correlation maximization into a stepwise, differentiable reward bonus that favors states and actions whose future is predictable from the past, and it is what allows the objective to be optimized with a standard actor-critic loop.","core_discovery":"On its own terms, the paper's central claim is that trajectories induced by a policy can be made simpler and more robust by maximizing their total correlation, defined as the KL divergence between the joint distribution of the sequence of latent states and actions and the product of their per-time-step marginals. Because the exact quantity is intractable, the paper optimizes a variational lower bound built from a history-based latent dynamics model and a history-based action prediction model; the bound rewards situations in which the next latent state and the next action are predictable from the past, relative to the irreducible randomness of the encoder and policy. The resulting algorithm, MTC, is soft actor-critic with an extra reward term plus a dual constraint that automatically tunes the strength of the total-correlation term. Empirically, the paper reports that MTC's trajectories are the most compressible in bzip2 file-size comparisons, that its actions are the most predictable to a t-step-ahead predictor, and that it maintains higher returns than baselines under Gaussian observation noise, action noise, mass-scaling perturbations, and added irrelevant state dimensions.","pith_inferences":["Editorial inference: since the optimized lower bound is always negative, the practical objective is a weighted sum of KL divergences between predictive models and policy/encoder distributions; if that is the true driver, similar gains might come from any information-bottleneck-style regularizer, not specifically from total correlation.","Editorial inference: the robustness gains are reported at fixed noise and mass scales; a sharper test would be to check whether MTC's advantage grows with trajectory length and with the amount of redundant periodic structure in the task, which would confirm the compressibility mechanism.","Editorial inference: combining MTC with domain randomization could reduce the amount of simulator variation needed for sim-to-real transfer, since the consistency bias already removes some dependence on state-level details; the paper does not test this combination."],"forward_implications":["If the central claim is right, a single trajectory-level information-theoretic term can improve both task performance and zero-shot robustness on locomotion, manipulation, and image-based control benchmarks.","MTC-learned policies should produce measurably more compressible trajectories and more predictable future actions; the paper reports both (bzip2 sizes and t-step-ahead prediction errors).","The regularizer should generalize across tasks without per-task reward engineering, because it acts only on the induced trajectory distribution, not on task-specific structure.","The approach can be layered onto any off-policy actor-critic method that can accept a modified reward; the paper demonstrates this on soft actor-critic and reports similar gains on DMC, Metaworld, and image-based DMC tasks.","The action-prediction component specifically carries part of the robustness benefit; the paper's ablation MTC-NoA indicates that removing it reduces robustness under several perturbations."],"supporting_citations":[{"why":"Defines total correlation, the information-theoretic quantity central to the paper's objective.","marker":"Watanabe, 1960"},{"why":"Provides the multi-information formulation of total correlation used in the problem definition.","marker":"Studený & Vejnarová, 1998"},{"why":"RPC is the sequence-regularization baseline that MTC extends and compares against, and is derived from the same framework in Appendix A.2.","marker":"Eysenbach et al., 2021"},{"why":"LZ-SAC and SPAC are the action-consistency baselines that motivate the simple-sequence-prior comparison.","marker":"Saanum et al., 2023"},{"why":"Soft actor-critic is the base algorithm onto which the total-correlation regularizer and dual constraint are added.","marker":"Haarnoja et al., 2018"},{"why":"Supplies the PyTorch SAC implementation and SAC-AE encoder used as the codebase and image-based baseline.","marker":"Yarats et al., 2021"},{"why":"DeepMind Control Suite provides the eight continuous-control tasks used for the main performance and robustness experiments.","marker":"Tassa et al., 2018"},{"why":"Metaworld provides the non-periodic manipulation tasks used to test the regularizer beyond locomotion.","marker":"Yu et al., 2020"},{"why":"PlaNet benchmark provides the six image-based DMC tasks used to evaluate MTC with convolutional encoders.","marker":"Hafner et al., 2019"}],"fun_headline_variants":["Maximizing trajectory predictability makes RL policies robust to noise","Total-correlation reward simplifies RL trajectories and boosts robustness","Predictable RL trajectories: simpler policies, sturdier performance","RL with total-correlation objective: simpler behavior, better resilience","Correlating RL trajectory states yields simpler, noiseproof policies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that maximizing a bound that is guaranteed to be negative, and therefore cannot estimate total correlation at all, still shifts policies toward genuinely higher total correlation and produces the reported robustness gains.","fun_headline_variants_meta":{"raw":{"variants":["Maximizing trajectory predictability makes RL policies robust to noise","Total-correlation reward simplifies RL trajectories and boosts robustness","Predictable RL trajectories: simpler policies, sturdier performance","RL with total-correlation objective: simpler behavior, better resilience","Correlating RL trajectory states yields simpler, noiseproof policies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000446,"raw_usage":{"total_tokens":2222,"prompt_tokens":883,"completion_tokens":1339,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":1253}},"tokens_in":499,"tokens_out":1339,"duration_ms":9755,"temperature":1.0,"reasoning_tokens":1253,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:56:25.232708+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same agent with the total-correlation reward replaced by a constant negative offset of matched magnitude: if compressibility and perturbation robustness are unchanged, the bound's internal predictive structure is not what matters. Alternatively, estimate the true total correlation of rollout trajectories from joint and marginal entropies before and after MTC training; if true total correlation decreases while the lower bound increases, the paper's proposed mechanism is refuted.","supporting_citations":[{"cited_title":"log p(z1:T |s1:T ) p(z1:T ) # − E","cited_arxiv_id":null,"evidence_quote":"RPC is the sequence-regularization baseline that MTC extends and compares against, and is derived from the same framework in Appendix A.2."}],"review_version":1}