Pith. sign in

REVIEW 3 major objections 5 minor 32 references

Replacing the fixed latent distance in bisimulation-based visual RL with a pair-conditioned positive-definite quadratic form—PAMD—improves final returns and keeps the encoder from being under-trained.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 16:21 UTC pith:AZX7WW75

load-bearing objection PAMD is a genuinely useful plug-in distance for bisimulation RL, with a sharp diagnostic, but the DBC-family headline gains rest on a modified baseline whose comparability to original DBC is shaky. the 3 major comments →

arxiv 2607.18004 v1 pith:AZX7WW75 submitted 2026-07-20 cs.AI cs.LG

PAMD: Structured Adaptive Distances for Bisimulation Representations in Visual Reinforcement Learning

Daegyeong Roh , Juho Bae , Han-Lim Choi This is my paper
classification cs.AI cs.LG
keywords bisimulation metricsrepresentation learningvisual reinforcement learningmetric learningMahalanobis distancestate similaritydeep reinforcement learningpositive-definite quadratic form
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the latent-space distance used to embed behavioral state similarity is a first-order design choice in bisimulation-based visual reinforcement learning, not an incidental detail. Fixed global norms such as ℓ1 are too rigid, while an unconstrained pairwise network can drive the metric loss to zero on its own, leaving the encoder weakly shaped. The paper introduces PAMD, a pairwise-conditioned Mahalanobis distance built as the square root of a positive-definite quadratic form on the latent displacement, and shows it occupies the useful middle: expressive per pair, yet structured so the fixed-point objective can only be satisfied by actually changing the representation. Plugging PAMD into existing DBC and MICo/SimSR pipelines consistently improves final returns on pixel-based DMControl tasks, and a frozen-encoder diagnostic attributes this to preserved learning pressure on the encoder. A careful reader would care because it pinpoints the latent geometry—not just the behavioral target—as a controllable lever in representation learning.

Core claim

PAMD parameterizes the latent dissimilarity as d_θ(z,z′) = √(Δᵀ G̃_θ(z,z′) Δ + ε) with Δ = z − z′. The matrix G̃ is built by an MLP that maps the pair to the lower-triangular entries of a Cholesky factor, forming a positive-definite matrix, symmetrizing it over the pair order, adding a ridge, and trace-normalizing per pair to fix scale. The claim is that this displacement-tied, positive-definite structure prevents the distance head from absorbing the bisimulation supervision by itself: with the encoder frozen, the residual of the one-step fixed-point equation plateaus above zero, whereas an unconstrained MLP distance of matched size drives it near zero; with a trainable encoder, PAMD's resid

What carries the argument

The central object is the pairwise-conditioned positive-definite quadratic form d_θ(z,z′) = √(Δᵀ G̃_θ(z,z′) Δ + ε). G̃_θ(z,z′) is constructed by an MLP that takes [z;z′], outputs the lower-triangular entries of a Cholesky factor L, forms G₀ = LLᵀ, symmetrizes as G = G₀(z,z′) + G₀(z′,z), adds a ridge λI, and applies per-pair trace normalization. This single construction guarantees symmetry, nonnegativity, and self-zero by design, and it ties the distance to the relative displacement rather than to the absolute positions of the pair. The work it does is to keep the distance expressive—pair-dependent anisotropy, including off-diagonal cross-coordinate terms—while making the metric loss solvable

Load-bearing premise

The controlled DBC comparison assumes that DBC-Det, the deterministic-transition variant that can accept PAMD, is an adequate stand-in for the original probabilistic DBC—because PAMD cannot be inserted into the Gaussian-Wasserstein term without changing the transition-discrepancy estimator—so the reported DBC-family gains rest on the two baselines being broadly comparable.

What would settle it

Train a parameter-matched unconstrained pairwise MLP that is forced to depend only on the displacement Δ = z − z′ (e.g., by symmetrizing its inputs and subtracting its value on identical pairs), and run the same frozen-encoder residual diagnostic and DMC tasks. If that displacement-tied MLP drives the frozen residual to the same plateau as PAMD and matches its downstream returns, the paper's claim that the positive-definite quadratic form specifically prevents distance-dominated solutions is falsified; a simpler displacement-tied head would suffice.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The same fixed-point targets used by DBC, MICo, and SimSR become reachable with higher final returns when the latent comparator is PAMD, so any bisimulation-style objective can adopt the module without re-deriving its target distance.
  • The frozen-encoder residual test offers a mechanistic explanation of why an unstructured distance head underperforms: low residual with a frozen encoder means the distance alone satisfies the objective, weakening gradient pressure on the encoder.
  • Removing trace normalization degrades performance, indicating that per-pair scale control is part of what prevents degenerate solutions in learned quadratic-form distances.
  • Diagonal and global Mahalanobis variants recover only part of PAMD's gains, suggesting that pair-dependent off-diagonal structure is an active ingredient, not a byproduct.
  • The contractivity, fixed-point, and value-function properties of the underlying behavioral distances are inherited unchanged, since PAMD modifies only the proxy latent comparator.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same frozen-encoder diagnostic could be used as a cheap architectural screening test for any similarity-based representation objective: measure how much residual an architecture can fit without changing the encoder, before committing to a full RL run.
  • If the mechanism is correct, part of the performance differences among existing bisimulation methods may be attributable to their latent comparators rather than their behavioral targets; re-running them under a common adaptive comparator would separate target choice from embedding geometry.
  • The dense Cholesky construction scales as O(d³) with latent dimension, so the observed gains may shrink in higher-dimensional latent spaces; low-rank or structured-sparse variants would test whether the essential benefit is the displacement-tied PD form or full per-pair anisotropy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces PAMD (Pairwise Adaptive Mahalanobis Distance), a plug-in latent dissimilarity for bisimulation-style representation learning in visual RL. PAMD replaces fixed global norms (e.g., ℓ1, cosine) with a pair-conditioned positive-definite quadratic form dθ(z,z') = sqrt((z−z')ᵀ G̃θ(z,z')(z−z') + ε), where G̃θ is a symmetrized, trace-normalized Cholesky-parameterized matrix. The authors argue that this structured adaptive geometry is more expressive than fixed norms while avoiding the degenerate solutions of an unconstrained pairwise MLP distance. They evaluate PAMD in two operator families: a deterministic-transition DBC variant and an independently-coupled MICo/SimSR style operator, reporting improved final returns on several pixel-based DMControl tasks. They also provide a residual-fitting diagnostic showing that a parameter-matched unstructured MLP distance can reduce the Bellman residual with a frozen encoder, whereas PAMD requires a trainable encoder, which they interpret as evidence that PAMD preserves learning pressure on the representation.

Significance. The core claim is consequential: if correct, the choice of latent distance geometry is a first-order design factor in bisimulation-based representation learning, not a minor implementation detail. The paper's strengths include hyperparameter-controlled comparisons, re-run baselines, a parameter-matched MLP-vs-PD diagnostic, and a clean conceptual framework connecting metric embedding to behavioral distances. The residual-fitting experiment is a genuinely useful tool. However, the current evidence is incomplete: the DBC-family comparison is confounded by the switch to a deterministic-transition variant, and the independently-coupled comparison suffers from ambiguity about which baseline pipeline the PAMD curve actually uses. These issues affect the two main empirical pillars of the paper, so the central claim is defensible but not yet fully demonstrated.

major comments (3)
  1. [§5.1, Table 1] The controlled DBC-family comparison is DBC-Det vs. DBC-Det+PAMD, not original DBC vs. DBC+PAMD, because PAMD cannot be inserted into the closed-form Gaussian Wasserstein term of probabilistic DBC. The paper bridges this with the assertion that DBC-Det and DBC are 'broadly comparable,' but the Hopper Hop row contradicts this: DBC-Det is 33±29 vs. DBC at 67±16, a factor-of-two gap, and Cheetah Run shows only overlapping error bars (350±60 vs. 310±32). No statistical test is provided. Since the DBC-family results are used to claim improvement over a standard bisimulation algorithm, this is a load-bearing confound. Please either (a) report DBC+PAMD using a PAMD-compatible transition-discrepancy estimator matched to the probabilistic baseline, or (b) provide a per-seed paired analysis and additional tasks demonstrating that DBC-Det is a faithful stand-in, and restrict the DBC claims accordin
  2. [§5.1, Fig. 2 bottom row and Table 3] The single curve 'Indep.-coupled (MICo/SimSR)+PAMD' cannot preserve the same training pipeline as both MICo and SimSR simultaneously, because Appendix D.5 states that MICo uses no learned transition model while SimSR uses an ensemble probabilistic transition model. If the PAMD curve is SimSR-based, the comparison to MICo is not controlled; if it is a third implementation, the exact pipeline is unspecified. This matters because the independently-coupled family is the best-controlled part of the evidence. Please report SimSR+PAMD and MICo+PAMD separately, each matched to its corresponding baseline pipeline, or clearly specify which baseline the 'Indep.-coupled+PAMD' curve extends.
  3. [§6 and Remark 4.2] The statement that PAMD 'preserves the contraction property of standard behavioral distance operators' is only valid for the fixed target distance, not for the learned proxy dθ. The training objective in Eq. (15) is a TD-style regression whose target contains dθ evaluated on next states; no convergence or contraction guarantee is proven for the learned comparator, and the residual diagnostic in §5.2 shows that the learned distance does not fully reach the fixed point under a frozen encoder. Please soften this claim to apply to the target operator only, or provide a formal statement and proof for the learned distance.
minor comments (5)
  1. [§4.2, Eq. (12)–(14)] The upper bound in Lemma 4.1 is dθ(z,z') ≤ sqrt(||Δ||₂² + ε), not dθ ≤ ||Δ||₂ as suggested by the prose 'dθ ≲ ||Δ||₂'. Please align the notation and the statement.
  2. [§4.2, Eq. (15)] The metric loss as written uses Tdθ in the target without indicating that the target is detached. The detachment is only described in Appendix D.1. Please state this in the main text to avoid apparent circularity.
  3. [Table 3] The DBC column for Cheetah Run reports 350±49, while Table 1 reports DBC (Prob.) = 310±32 and DBC-Det = 350±60. Please clarify whether Table 3 reports probabilistic DBC or DBC-Det, and reconcile the numbers.
  4. [§5.2 / Appendix E] The residual-fitting diagnostic would benefit from reporting the actual residual values after convergence for both frozen and trainable encoders, rather than only the learning curves, to support the claim that PAMD 'plateaus above a nonzero level' when the encoder is frozen.
  5. [Appendix D.1] The auxiliary anisotropy coefficient λ_aux = 1e−3 is introduced but not ablated. Since it directly encourages deviation from isotropy, a small sensitivity study (e.g., λ_aux ∈ {0, 1e−4, 1e−2}) would strengthen the claim that the benefits come from the structured form rather than this regularizer.

Circularity Check

0 steps flagged

No significant circularity: the metric objective is a standard Bellman-style bootstrap regression with stop-gradient targets, and the paper's theoretical and empirical claims do not reduce to their own inputs.

full rationale

The only potentially self-referential step is the metric loss in Eq. (15), Lmetric = E[(dθ(ϕω(s), ϕω(s′)) − T dθ(s, s′))²], where dθ appears both as the predicted distance and inside the target through γdθ on next states. This is the standard fixed-point/Bellman bootstrap used throughout bisimulation-based representation learning, not a circular reduction: the target branch is computed with a stop-gradient target copy (Appendix D.1: 'the target quantity ... is treated as a fixed target and detached from the computation graph'), so the current distance is not used as its own supervised label in an identity way. The residual-fitting diagnostic (Eq. 22) similarly uses an EMA target and is an intentional controlled probe, not a derivation. The theoretical claims are either proven in-paper from the definitions (Lemma 4.1) or inherited from external prior work (Ferns et al. 2004, Castro et al. 2021/2023, Zang et al. 2022, Zhang et al. 2021); there is no load-bearing self-citation chain. The DBC-Det vs. probabilistic-DBC baseline comparability issue noted by the reader is a legitimate experimental-design concern, but it is a potential confound, not a definitional or construction-level circularity. Hence there are no circular steps to report.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 2 invented entities

Free parameters are mostly small numerical stabilizers or inherited hyperparameters; the real assumptions are the contractivity of the target operators, the transferability of DBC-Det to DBC, and the stability of co-training encoder + distance. The invented entities are the metric network and the diagnostic, neither of which has external falsifiable support beyond task performance.

free parameters (7)
  • ridge lambda = small constant (exact value not given; used in Eq. 10)
    Added to ensure positive definiteness; the paper does not specify numeric value, so reproducibility depends on the code.
  • epsilon numerical stabilizers = small constants in Eqs. 11-12
    Used in trace normalization and square-root distance; numeric values not specified in the paper.
  • auxiliary anisotropy coefficient lambda_aux = 1e-3
    Auxiliary loss term that encourages deviation from isotropic metric; tuned by hand.
  • MICo loss coefficient alpha_MICo = 1e-5
    Taken from original MICo, but still a hand-chosen weight.
  • MICo angular weight beta = 0.1
    Taken from original MICo, hand-chosen.
  • MetricNet hidden width = 128
    Architecture choice, not justified by theory.
  • diagonal ReHU eta = 1.0
    Practical stabilization choice.
axioms (5)
  • standard math The independently-coupled Bellman operator T_M^pi (Eq. 6) is a gamma-contraction and has a unique fixed point.
    Standard result cited from MICo; needed for the target distances the paper regresses to.
  • standard math The bisimulation operator T_bis (Eq. 5) is a gamma-contraction in the sup norm and has a unique fixed point.
    Standard Ferns et al. result, cited; needed for DBC-family targets.
  • domain assumption Value functions are Lipschitz with respect to the behavioral distance.
    Classic bisimulation-metric property that links distance to control value; the paper inherits it in Table 2.
  • domain assumption The learned latent distance can be trained by TD-style regression to the fixed-point target computed with a stop-gradient/EMA target copy; the bootstrap does not destabilize the encoder.
    Empirically standard but unproven; the paper relies on it for all experiments and for the diagnostic.
  • ad hoc to paper DBC-Det (deterministic transition model) is behaviorally comparable to the original probabilistic DBC, so gains over DBC-Det transfer to gains over DBC.
    The paper does not establish this equivalence; it reports that final performances are 'broadly comparable' but with notable differences (Hopper Hop: 67±16 vs. 33±29).
invented entities (2)
  • PAMD pairwise-conditioned PD matrix field G_theta(z,z') no independent evidence
    purpose: Set the latent dissimilarity geometry adaptively per pair while remaining structured.
    No falsifiable external prediction; its only support is empirical control performance.
  • Residual-fitting diagnostic with frozen encoder no independent evidence
    purpose: Test whether a distance module can absorb the Bellman supervision without shaping the encoder.
    A diagnostic tool, not an entity with independent evidence; its interpretation is partly circular (it assumes small residual with frozen encoder is bad).

pith-pipeline@v1.3.0-alltime-deepseek · 18933 in / 8269 out tokens · 69355 ms · 2026-08-01T16:21:11.354510+00:00 · methodology

0 comments
read the original abstract

Many visual reinforcement learning (RL) algorithms learn representations by matching latent distances to a behavioral distance induced by reward and transition similarity. In practice, the choice of the latent distance can strongly affect performance: using a fixed, pre-specified global norms (e.g., $\ell_p$ norms or other hand-designed metrics) may be overly restrictive to capture the behavioral distance. In contrast, unconstrained pairwise distances may admit degenerate solutions that drive the metric loss down without improving the representation. To address this gap, we introduce **PAMD: Pairwise Adaptive Mahalanobis Distance**, which parameterizes a positive-definite, pair-conditioned metric for measuring latent state similarity. PAMD is a simple plug-in for existing bisimulation-based methods, offering a more expressive yet structured alternative to fixed, pre-specified latent distances. We empirically validate our method on visual MuJoCo continuous-control tasks, where final performance of several recent bisimulation-based RL algorithms is substantially improved when equipped with the distance we propose.

Figures

Figures reproduced from arXiv: 2607.18004 by Daegyeong Roh, Han-Lim Choi, Juho Bae.

Figure 1
Figure 1. Figure 1: Architecture summary. between latents. We therefore adopt a pairwise-conditioned PD quadratic-form distance defined on the displacement ∆ := z − z ′ , which guarantees nonnegativity and symmetry by construction, while allowing adaptive, input-dependent anisotropy. 4.2. Pairwise-Conditioned PD Metric Network We parameterize a pairwise-conditioned PD matrix with a lightweight MLP (MetricNet). Given z, z′ ∈ R… view at source ↗
Figure 2
Figure 2. Figure 2: Results comparing DBC-Det+PAMD to DBC-Det (top row) and the Independently-coupled (MICo/SimSR-style) operator equipped with PAMD to SimSR and MICo baselines (bottom row) across multiple visual MuJoCo tasks. Results show the mean average return over 5 seeds with 1 standard error shaded. For each seed, the average return is computed every 10,000 training steps, averaging over 10 episodes. The x-axis denotes … view at source ↗
Figure 3
Figure 3. Figure 3: Ablation results on DMC cheetah run. The left and right panels report the DBC and MICo/SimSR families, re￾spectively. Each panel compares PAMD with global and adaptive diagonal Mahalanobis variants. Shaded regions denote ±1 stan￾dard error over 5 seeds. correspond to a prescribed isotropic geometry, global diago￾nal Mahalanobis distances allow coordinate-wise anisotropy shared across the dataset, adaptive … view at source ↗
Figure 5
Figure 5. Figure 5: Top: DMC walker walk learning curves for SimSR, the SimSR pipeline equipped with PAMD, and SimSR with an unconstrained pairwise-MLP distance (5 seeds; mean ± 1 s.e.). Bottom: Residual-fitting diagnostic (4 seeds) under a frozen vs. trainable encoder, comparing the unconstrained MLP distance and our structured distance. a one-step fixed-point target using an EMA target network: Lres(θ; ϕ) = E h dθ(ϕ(s), ϕ(s… view at source ↗
Figure 6
Figure 6. Figure 6: reports results on two representative settings. In the left panel, we evaluate DBC and DBC with PAMD on the DMC finger spin task. In the right panel, we evaluate SimSR and the SimSR pipeline equipped with PAMD on the DMC walker run task. Across both algorithms, removing trace normalization consistently degrades performance. This behavior is consistent with scale degeneracy in the learned quadratic form, wh… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 8 linked inside Pith

  1. [1]

    Nature , volume=

    Human-level control through deep reinforcement learning , author=. Nature , volume=

  2. [2]

    Tassa, Yuval and Doron, Yotam and Muldal, Alistair and Erez, Tom and Li, Yazhe and Caspi, Itai and de Freitas, Nando and Silver, David , journal=

  3. [3]

    arXiv preprint arXiv:1312.6114 , year=

    Auto-Encoding Variational Bayes , author=. arXiv preprint arXiv:1312.6114 , year=

    Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:1803.10122 , year=

    World Models , author=. arXiv preprint arXiv:1803.10122 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:1807.03748 , year=

    Representation Learning with Contrastive Predictive Coding , author=. arXiv preprint arXiv:1807.03748 , year=

    Pith/arXiv arXiv

  6. [6]

    Conference on robot learning (pp

    Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation , author=. Conference on robot learning (pp. 651-673). PMLR , year=

  7. [7]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal Policy Optimization Algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  8. [8]

    Haarnoja, Tuomas and Zhou, Aurick and Abbeel, Pieter and Levine, Sergey , journal=

  9. [9]

    Scalable

    Castro, Pablo Samuel , booktitle =. Scalable. 2020 , pages =

    2020

  10. [10]

    Laskin, Michael and Lee, Kimin and Stooke, Adam and Pinto, Lerrel and Abbeel, Pieter and Srinivas, Aravind , booktitle =

  11. [11]

    Yarats, Denis and Fergus, Rob and Lazaric, Alessandro and Pinto, Lerrel , booktitle=

  12. [12]

    Hafner, Danijar and Lillicrap, Timothy and Ba, Jimmy and Norouzi, Mohammad , booktitle=

  13. [13]

    arXiv preprint arXiv:2301.04104 , year=

    Mastering Diverse Domains through World Models , author=. arXiv preprint arXiv:2301.04104 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:1611.05397 , year=

    Reinforcement Learning with Unsupervised Auxiliary Tasks , author=. arXiv preprint arXiv:1611.05397 , year=

    Pith/arXiv arXiv

  15. [15]

    Laskin, Michael and Srinivas, Aravind and Abbeel, Pieter , journal=

  16. [16]

    arXiv preprint arXiv:2007.05929 , year=

    Data-Efficient Reinforcement Learning with Self-Predictive Representations , author=. arXiv preprint arXiv:2007.05929 , year=

    Pith/arXiv arXiv 2007

  17. [17]

    Metrics for

    Ferns, Norman and Panangaden, Prakash and Precup, Doina , booktitle=. Metrics for

  18. [18]

    Kemertas, Mete and Aumentado-Armstrong, Tristan , booktitle =

  19. [19]

    SIAM Journal on Computing , volume=

    Bisimulation Metrics for Continuous Markov Decision Processes , author=. SIAM Journal on Computing , volume=

  20. [20]

    Tassa, Yuval and Doron, Yotam and Muldal, Alistair and Erez, Tom and Li, Yuval and de Las Casas, David and Budden, David and Heess, Nicolas and Bachrach, Yoram and Riedmiller, Martin , journal =

  21. [21]

    2021 , primaryClass =

    Amy Zhang and Rowan McAllister and Roberto Calandra and Yarin Gal and Sergey Levine , booktitle =. 2021 , primaryClass =

    2021

  22. [22]

    arXiv preprint arXiv:2101.00338 , year=

    Towards Representation Learning with Tractable Bisimulation Metrics , author=. arXiv preprint arXiv:2101.00338 , year=

    Pith/arXiv arXiv

  23. [23]

    IEEE Signal Processing Magazine , volume=

    Geometric Deep Learning: Going Beyond Euclidean Data , author=. IEEE Signal Processing Magazine , volume=

  24. [24]

    Journal of Machine Learning Research , volume=

    Geomstats: A Python Package for Riemannian Geometry in Machine Learning , author=. Journal of Machine Learning Research , volume=

  25. [25]

    Zang, Hongyu and Li, Xin and Wang, Mingzhong , booktitle=

  26. [26]

    Castro, Pablo Samuel and Kastner, Tyler and Panangaden, Prakash and Rowland, Mark , journal=

  27. [27]

    Kingma and Jimmy Ba , booktitle =

    Diederik P. Kingma and Jimmy Ba , booktitle =

  28. [28]

    International conference on machine learning , pages=

    Deepmdp: Learning continuous latent space models for representation learning , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  29. [29]

    Proceedings of the 37th International Conference on Machine Learning , pages=

    Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model , author=. Proceedings of the 37th International Conference on Machine Learning , pages=. 2020 , organization=

    2020

  30. [30]

    Castro, Pablo Samuel and Kastner, Tyler and Panangaden, Prakash and Rowland, Mark , journal=. A

  31. [31]

    Pathak, Deepak and Agrawal, Pulkit and Efros, Alexei A and Darrell, Trevor , booktitle=

  32. [32]

    Yarats, Denis and Zhang, Amy and Kostrikov, Ilya and Amos, Brandon and Pineau, Joelle and Fergus, Rob , booktitle=