REVIEW 3 major objections 5 minor 13 references
Maximum Total Correlation Reinforcement Learning
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Total-correlation maximization makes RL trajectories simpler and more resilient to perturbations.
desk verdict Solid empirical regularizer, but the title's total-correlation mechanism is not what the optimized objective does; needs reframing and a few consistency fixes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the total correlation of a trajectory, $\mathcal{C}(z_1; a_1; \ldots; a_{T-1}; z_T) = \mathbb{E}[\log p(z_1, a_1, \ldots, a_{T-1}, z_T) / \prod_t p(z_t) \prod_t p(a_t)]$, which quantifies how many nats are saved by encoding the whole trajectory rather than each state and action separately. The argument is carried by the variational lower bound in Eq. (2), which replaces the unknown marginals with the learned encoder $f_\theta(z_{t+1}|s_{t+1})$ and policy $\pi_\phi(a_t|s_t)$ and adds a history-based dynamics model $q_\eta(z_{t+1}|z_{1:t}, a_{1:t})$ and action prediction model $q_\chi(a_t|z_{1:t}, a_{1:t-1})$. This bound turns total-correlation maximization into a stepwise, differentiable reward bonus that favors states and actions whose future is predictable from the past, and it is what allows the objective to be optimized with a standard actor-critic loop.
What would settle it
Train the same agent with the total-correlation reward replaced by a constant negative offset of matched magnitude: if compressibility and perturbation robustness are unchanged, the bound's internal predictive structure is not what matters. Alternatively, estimate the true total correlation of rollout trajectories from joint and marginal entropies before and after MTC training; if true total correlation decreases while the lower bound increases, the paper's proposed mechanism is refuted.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that trajectories induced by a policy can be made simpler and more robust by maximizing their total correlation, defined as the KL divergence between the joint distribution of the sequence of latent states and actions and the product of their per-time-step marginals. Because the exact quantity is intractable, the paper optimizes a variational lower bound built from a history-based latent dynamics model and a history-based action prediction model; the bound rewards situations in which the next latent state and the next action are predictable from the past, relative to the irreducible randomness of the encoder and policy. The resulting algorithm, MTC, is soft actor-critic with an extra reward term plus a dual constraint that automatically tunes the strength of the total-correlation term. Empirically, the paper reports that MTC's trajectories are the most compressible in bzip2 file-size comparisons, that its actions are the most predictable to a t-step-ahead predictor, and that it maintains higher returns than baselines under Gaussian observation noise, action noise, mass-scaling perturbations, and added irrelevant state dimensions.
Load-bearing premise
The load-bearing premise is that maximizing a bound that is guaranteed to be negative, and therefore cannot estimate total correlation at all, still shifts policies toward genuinely higher total correlation and produces the reported robustness gains.
Editorial extensions
If this is right
- If the central claim is right, a single trajectory-level information-theoretic term can improve both task performance and zero-shot robustness on locomotion, manipulation, and image-based control benchmarks.
- MTC-learned policies should produce measurably more compressible trajectories and more predictable future actions; the paper reports both (bzip2 sizes and t-step-ahead prediction errors).
- The regularizer should generalize across tasks without per-task reward engineering, because it acts only on the induced trajectory distribution, not on task-specific structure.
- The approach can be layered onto any off-policy actor-critic method that can accept a modified reward; the paper demonstrates this on soft actor-critic and reports similar gains on DMC, Metaworld, and image-based DMC tasks.
- The action-prediction component specifically carries part of the robustness benefit; the paper's ablation MTC-NoA indicates that removing it reduces robustness under several perturbations.
Reading between the lines
- Editorial inference: since the optimized lower bound is always negative, the practical objective is a weighted sum of KL divergences between predictive models and policy/encoder distributions; if that is the true driver, similar gains might come from any information-bottleneck-style regularizer, not specifically from total correlation.
- Editorial inference: the robustness gains are reported at fixed noise and mass scales; a sharper test would be to check whether MTC's advantage grows with trajectory length and with the amount of redundant periodic structure in the task, which would confirm the compressibility mechanism.
- Editorial inference: combining MTC with domain randomization could reduce the amount of simulator variation needed for sim-to-real transfer, since the consistency bias already removes some dependence on state-level details; the paper does not test this combination.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Maximum Total Correlation Reinforcement Learning (MTC-RL), a modification of the RL objective that adds a total-correlation term over the induced trajectory of latent states and actions, justified as an inductive bias toward simple, compressible, and robust behavior. The authors derive a variational lower bound (Eq. 2) using history-based latent dynamics and action prediction models, integrate it into SAC-style off-policy optimization with an adaptive Lagrangian coefficient, and evaluate the method on DMC, Metaworld, and image-based DMC benchmarks. They report improved asymptotic performance, enhanced zero-shot robustness to observation/action noise and dynamics changes, higher trajectory compressibility, and stronger action predictability compared to SAC, RPC, LZ-SAC, and SPAC. The paper also includes a discussion section acknowledging that the lower bound is always negative and hence not useful for estimating the true total correlation.
Significance. If the central claim were established, MTC-RL would provide a principled, task-independent simplicity bias for RL with broad empirical appeal. The experimental evaluation is substantial and careful: 20 seeds on DMC, multiple benchmarks, ablations, and public code. The variational derivation in Appendix A.1 is mathematically correct as a lower bound, and the robustness and compressibility improvements are reproducible in principle. However, the theoretical interpretation is not supported: the optimized surrogate (Eq. 2) is a sum of negated KL divergences and is always non-positive, so maximizing it is not equivalent to maximizing total correlation. The paper explicitly concedes this in Section 6. The empirical gains may be real, but they are currently attributed to a mechanism that the optimized objective does not implement. This mismatch is load-bearing for the title, abstract, and interpretation of every empirical result. The paper therefore requires major revision, either by providing evidence that optimizing the surrogate increases true total correlation or by reframing the contribution as conditional-KL regularization.
major comments (3)
- [§4.2, Eq. (2); §6] The variational lower bound eC in Eq. (2) is always non-positive because it is a sum of negated KL divergences. The authors acknowledge this in Section 6, but the implication for the paper's central claim is stronger than "not useful for estimation": maximizing eC does not correspond to maximizing total correlation. Specifically, eC = -Σ_t E[KL(fθ(z_{t+1}|s_{t+1}) || qη(z_{t+1}|history)) + KL(πφ(a_t|s_t) || qχ(a_t|history))]. A policy that ignores the state and outputs independent actions drawn from the marginal, together with q matching the encoder and policy marginals, attains eC = 0, the theoretical maximum of the bound, while the true total correlation of the trajectory is near zero. Conversely, a deterministic periodic trajectory can have large true total correlation while eC is near zero because both fθ and qη are sharply peaked. Thus the bound's gradients push toward conditional independence of each variable from the current state given the history, not toward the multi-variable dependence measured by total correlation. The title and abstract claim that the method "maximizes the total correlation within the induced trajectories"; this claim is not established. To support it, the paper should measure the true total correlation on a tractable task (e.g., a low-dimensional MDP) and show that it increases when optimizing Eq. (2), or it should explicitly reframe the method as state-conditional KL regularization.
- [§5.3, Fig. 3] The trajectory-compression result in Section 5.3 and Fig. 3 is presented as evidence that the method maximizes total correlation. However, compressibility is a direct consequence of the optimized objective: Eq. (2) explicitly minimizes the KL divergence between the encoder/policy and history-based predictive models, which by construction makes trajectories predictable and hence compressible. This is not an independent confirmation of the total-correlation mechanism; the same compression improvement would be expected from any regularizer that penalizes KL to a history-based predictor, including a simple action-smoothing baseline. To make the empirical claim non-circular, the paper should compare against a baseline that optimizes the same KL regularizer without the total-correlation interpretation, and should additionally report an independent estimate of total correlation (e.g., a non-parametric plug-in estimate) on the same trajectories.
- [§4.2, Eq. (2); Appendix A.1] The claim in §4.2 that the lower bound "captures important aspects of the total correlation" is substantiated only by a qualitative trajectory plot (Fig. 1), which does not establish a quantitative relationship between eC and the true total correlation. The gap between the true C and eC is D_KL(p(z_{1:T},a_{1:T-1}) || q(z_{1:T},a_{1:T-1})) plus additional non-negative terms from the marginal-substitution step in Eq. (11). This gap can be large and input-dependent, so the optimization of eC need not track the optimization of C. The paper should provide at least one empirical or analytic example where eC and C move in opposite directions, or a small-scale measurement showing that increasing eC indeed increases C. Without this, the central mechanism remains unsupported.
minor comments (5)
- [§5.4 vs. Fig. 4 caption] The text in §5.4 states that MTC "achieves higher average rewards than baselines on all tasks" for Metaworld, while the caption of Fig. 4 says "MTC is competitive to baselines"; these statements should be reconciled.
- [Table 3, Appendix B.9] The entry "Replay buffer capacity 1 00 000" contains a typo; it should read "100 000".
- [Eq. (3)] Equation (3) writes r(s_T, a_T) separately in front of the sum over t=1 to T-1, which is redundant with the t=T term inside the sum and may confuse the indexing; please clean up the notation.
- [§6] The statement that a vacuous lower bound can be useful for optimization "as in the case of subtracting a constant offset" is imprecise: a lower bound with an input-dependent gap is not the same as a constant offset, and maximizing a lower bound is only justified if the gap is controlled or if the bound is tight at the optimum. This point should be clarified or removed.
- [Table 1] Table 1 reports "means over 20 seeds with 90% confidence interval" but the values are shown as ± intervals; please clarify whether these are standard errors, standard deviations, or quantile-based intervals.
Circularity Check
No circularity: the lower-bound derivation is genuine (though vacuous), and the admitted always-negative surrogate is a soundness gap, not a self-referential input-output identity.
full rationale
The paper's derivation chain is self-contained: Eq. 1 defines the total-correlation-augmented objective, Eq. 2 is a variational lower bound on that total correlation (Appendix A.1), and Eq. 3 optimizes the bound with SAC. The bound is mathematically valid, so the algorithm is not defined in terms of the conclusion. Section 6 explicitly discloses the serious limitation: 'our lower bound of the total correlation corresponds to a sum of negated KL divergences, and is therefore always negative. Hence, it is not useful for estimating the actual total correlation.' Because eC <= 0 and total correlation C >= 0, the inequality C >= eC is trivially true, so the derivation does not establish that maximizing eC maximizes C. This is a correctness/soundness gap in the total-correlation interpretation, not a circular reduction: the paper does not silently rename the surrogate as the target, and the robustness/compressibility results are zero-shot empirical evaluations against external baselines rather than fitted quantities renamed as predictions. No load-bearing self-citation appears: co-author citations (Peters et al. 2010; Memmel et al. 2022; Reddi et al. 2023) are contextual related work, and the RPC/LZ-SAC discussion relies on external prior work. The finding is therefore a non-finding for circularity, with the vacuous-bound issue recorded here as the main correctness risk.
Assumptions & free parameters
free parameters (3)
- Information constraint Ip =
-0.5, -7.0, -10.0, -0.1 per task
- History length =
8 for state-based tasks, 3 or 5 for image-based tasks
- Bound combination coefficient m =
1e-6 for state-based, 1e-4 for image-based
assumptions (5)
- standard math The variational lower bound in Eq. 2 is a valid lower bound on total correlation
- standard math Substituting the encoder conditional fθ(zt+1|st+1) and policy conditional πϕ(at|st) for the unknown marginals p(zt+1) and p(at) preserves a lower bound
- ad hoc to paper Maximizing the always-negative lower bound is a useful proxy for maximizing total correlation
- domain assumption Optimizing the prediction models on the replay buffer (off-policy) instead of on-policy samples does not invalidate the bound
- domain assumption A history-free policy using only the current state is sufficient despite the history-dependent reward
Cite this review
Pith. "Pith review of Maximum Total Correlation Reinforcement Learning." pith.science (2026). https://pith.science/paper/P3HND7KT
@misc{pith2026250516734,
author = {Pith},
title = {Pith review of: Maximum Total Correlation Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/P3HND7KT}},
note = {Machine review of arXiv:2505.16734}
}
read the original abstract
Simplicity is a powerful inductive bias. In reinforcement learning, regularization is used for simpler policies, data augmentation for simpler representations, and sparse reward functions for simpler objectives, all that, with the underlying motivation to increase generalizability and robustness by focusing on the essentials. Supplementary to these techniques, we investigate how to promote simple behavior throughout the episode. To that end, we introduce a modification of the reinforcement learning problem that additionally maximizes the total correlation within the induced trajectories. We propose a practical algorithm that optimizes all models, including policy and state representation, based on a lower-bound approximation. In simulated robot environments, our method naturally generates policies that induce periodic and compressible trajectories, and that exhibit superior robustness to noise and changes in dynamics compared to baseline methods, while also improving performance in the original tasks.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[2]
For other details, please refer to the provided code. B.4. Extended Description of Baseline Implementations SAC. We obtain the results for SAC by running the PyTorch implementations of SAC (Yarats et al., 2021). We use the same hyperparameters for SAC as our algorithm to ensure a fair comparison. We found that our obtained results for SAC are stronger tha...
work page 2023
-
[3]
Hyperparameters used in Image-based tasks. Parameter Value Ip for Finger and Ball-In-Cup -10.0 Ip for other tasks -0.1 history length for Finger and Ball-In-Cup 3 history length for other tasks 5 Replay buffer capacity 1 00 000 Optimizer Adam Critic learning rate 10−4 Critic Q-function soft-update rate 0.01 Critic target update frequency 2 Actor learning ...
-
[7]
Tassa, Y ., Doron, Y ., Muldal, A., Erez, T., Li, Y ., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690,
-
[12]
log p(z1:T |s1:T ) p(z1:T ) # − E
does not always hold as it was derived by replacing p(z1:T |s1:T ) by QT −1 t=1 p(zt+1|st+1) (Eysenbach et al., 2021, Appendix C1). These distributions are in general not the same because information about future state observations can decrease uncertainty about the current latent state, and therefore p(zt+1|z1:t, s1:T ) ̸= p(zt+1|st+1). We will now show ...
work page 2021
-
[13]
Hyperparameters used in MTC. Parameter Value Ip for Cheetah Run, Hopper and Walker Stand -0.5 Ip for other tasks -7.0 history length 8 Replay buffer capacity 1 000 000 Optimizer Adam Critic learning rate 10−4 Critic Q-function soft-update rate 0.01 Critic target update frequency 2 Actor learning rate 10−4 Actor update frequency 1 Actor log stddev bounds [...
work page 2021
-
[14]
State sequences of our method show more repeating and periodic patterns
Visualizations of state sequences generated by our method and baselines on the Finger Spin task. State sequences of our method show more repeating and periodic patterns. 22 Maximum Total Correlation Reinforcement Learning 0 25 50 75 100 125 150 175 200 1 0 1 Action Actuator 0 0 25 50 75 100 125 150 175 200 1 0 1 Actuator 1 0 25 50 75 100 125 150 175 200 1...
work page 2001
-
[1934]
URL http://www.jstor.org/ stable/jj.8441753.32
ISBN 9780520364257. URL http://www.jstor.org/ stable/jj.8441753.32. Oord, A. v. d., Li, Y ., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv:1807.03748,
-
[1995]
STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learning
Chakraborty, S., Bedi, A. S., Koppel, A., Wang, M., Huang, F., and Manocha, D. Steering: Stein information di- rected exploration for model-based reinforcement learn- ing. arXiv preprint arXiv:2301.12038,
Show all 13 references
-
[2010]
and Zaslavsky, N
Tishby, N. and Zaslavsky, N. Deep learning and the infor- mation bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1–5. IEEE,
2015
-
[2018]
for more descriptions of tasks. B.2. Implementation Details SAC codebase. We implement our algorithm on top of the common PyTorch implementation of the SAC algorithm (Yarats et al., 2021). We used the default hyperparameters from that implementation unless specified otherwise....
2021
-
[2021]
Dynamics generalization via information bottleneck in deep rein- forcement learning
Lu, X., Lee, K., Abbeel, P., and Tiomkin, S. Dynamics generalization via information bottleneck in deep rein- forcement learning. arXiv preprint arXiv:2008.00614,
2008 arXiv
-
[2022]
Soft actor-critic algorithms and applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V ., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905,
-
[2024]
doi: 10.1126/scirobotics. adi9579. URL https://www.science.org/doi/ abs/10.1126/scirobotics.adi9579. Rakelly, K., Gupta, A., Florensa, C., and Levine, S. Which mutual-information representation learning objectives are sufficient for control? Advances in Neural Information Proc...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.