Pith. sign in

REVIEW 2 major objections 5 minor 17 references

Action-conditioned latent prediction improves underwater goal reaching when cameras and DVL fail

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

AquaJEPA, a joint-embedding predictive model with action conditioning and sensor-validity masks, achieves the best closed-loop goal-reaching in 120 simulated underwater trials, with significant paired final-error gains over three action-conditioned baselines but not over state-only.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection Solid paired-simulation study of an action-conditioned JEPA for underwater control, but unequal checkpoint selection threatens the headline comparison. the 2 major comments →

arxiv 2607.29393 v1 pith:AFKON7L3 submitted 2026-07-31 cs.RO

AquaJEPA: Action-Conditioned Multimodal Predictive Representations for Underwater Robot Dynamics

Alan-Barsag Gazzaev , Alexey Gavrilov , Sergey Muravyov This is my paper
classification cs.RO
keywords joint-embedding predictive architectureunderwater roboticslatent dynamicspartial observabilitymultimodal fusionaction-conditioned predictionreceding-horizon planningsensor dropout
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces AquaJEPA, a dynamics model for underwater robots that predicts future abstract feature vectors conditioned on thruster commands, rather than reconstructing raw sensor images. It fuses camera, sonar, and proprioception with explicit validity masks, trains against a momentum-encoded future target, and feeds velocity and sonar predictions to a shared receding-horizon planner. In a preregistered 120-episode paired simulation study with poor visibility and scheduled Doppler velocity log loss, AquaJEPA reaches the highest goal count (74/120) and lowest mean final error (0.906 m), with paired final-error improvements over ordinary multimodal, supervised, and recurrent baselines that exclude zero. Its advantage over a state-only baseline is not statistically resolved. The paper argues the benefit comes from action-sensitive latent representation and sensor-loss exposure, not from extra sensors or a bigger transition model.

Core claim

AquaJEPA is a joint-embedding predictive architecture: instead of predicting the next camera or sonar frame, it predicts a future feature vector in a learned latent space, conditioned on the eight-thruster command sequence. A momentum-EMA encoder builds a stop-gradient target for that future latent; physical auxiliary heads predict future velocity change and a sonar range-intensity profile, and an action-margin loss insists that the executed command sequence produce a nearer target than inverse or zero-action counterfactuals. In closed-loop use, a three-seed ensemble scores a fixed library of receding-horizon candidate actions with a shared cost function. In 120 fresh paired simulation episo

What carries the argument

The load-bearing pieces are: a mask-aware multimodal encoder that multiplies camera, sonar, and proprioceptive embeddings by availability masks and appends the mask; an EMA target encoder with stop-gradient momentum updates that supplies the future latent; the action-margin loss that penalizes predictions for which an inverse or zero command sequence matches the future target as well as the executed sequence; modality dropout so the model trains without camera or sonar; and two physical readouts—predicted velocity change and a 32-bin sonar profile—that interface with a shared receding-horizon planner. The planner and action library are identical across all learned baselines, so closed-loop d

Load-bearing premise

The central claim rests on the assumption that AquaJEPA's closed-loop advantage, measured against baselines, is caused by its predictive objective and not by its privileged checkpoint-selection rule (composite action-aware criterion) that the baselines did not receive.

What would settle it

Run the preregistered 120-environment matrix with every model's checkpoint chosen by the same rule—either all by future-velocity MAE or all by the composite criterion. If AquaJEPA's paired final-error advantage over the three predictive baselines shrinks to include zero, the advantage is driven by checkpoint selection rather than the joint-embedding objective.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • In simulated underwater conditions with poor optical visibility and intermittent DVL lock, a latent-space predictive model can reach the goal more often and closer than supervised or recurrent-world-model baselines that share the same planner.
  • The action-margin loss—not lower future-velocity error—is what separates command sequences; recurrent models with the lowest velocity MAE had poor action ranking, pointing to the margin as the control-relevant training signal.
  • Training with modality dropout is the clearest driver of test-time robustness: with camera absent, models trained without modality dropout approximately double their velocity prediction error.
  • The aggregate closed-loop lead is spread across visibility and dynamics cells but is layout-dependent—AquaJEPA trails on the east-corridor layout, so the benefit is not uniform.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A key confound is checkpoint selection: AquaJEPA's checkpoints were chosen with a composite objective that rewards action sensitivity, while supervised and recurrent baselines were selected by future-velocity MAE alone; a closed-loop pass that varies the selection rule would reveal how much of the measured advantage is attributable to the model family rather than the selection criterion.
  • The paper's comparison is entirely in simulation; a real-robot transfer study with synchronized camera, sonar, IMU/DVL, and causal thruster telemetry would be the natural next test, and the authors list it as open work.
  • The action-margin idea is portable: any joint-embedding predictor used for control could add a counterfactual margin to keep representations action-sensitive, which might improve planning in other partially observable robotics settings.
  • One testable extension: give the recurrent and supervised baselines the same composite checkpoint-selection rule, or remove it from AquaJEPA, and rerun the 120-episode matrix; if the paired final-error intervals shift, the selection rule is a contributor.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces AquaJEPA, an action-conditioned joint-embedding predictive model for underwater robot dynamics. The architecture fuses RGB camera, forward-looking sonar, and proprioception with explicit sensor-validity masks, trains a predictor against an EMA target with auxiliary velocity and sonar heads plus an action-margin loss, and feeds the readouts into a shared receding-horizon planner. In a sealed 120-environment Stonefish evaluation with unseen maps, four visibility levels, nominal/shifted dynamics, and scheduled DVL loss, AquaJEPA reaches 74/120 goals with 0.906 m mean final error, versus 68/120 and 0.916 m for state-only and 68/120 and 1.012 m for the recurrent world model. Paired final-error confidence intervals exclude zero against ordinary multimodal prediction, supervised dynamics, and the recurrent world model; the state-only comparison is reported as statistically unresolved. The paper also reports prediction-table ablations of the EMA target, action margin, masks, and modality dropout.

Significance. If the central comparison can be taken at face value, the paper is a useful empirical demonstration that a control-sensitive latent objective can improve closed-loop goal-reaching under partial observability relative to supervised and recurrent alternatives. The strengths are substantial: the evaluation is preregistered and sealed, the paired bootstrap is cell-stratified and episode-level, the planner and action library are held fixed across methods, and the unresolved primary comparison against state-only is reported transparently. The work is relevant to underwater robot learning and to JEPA-style model-based control. However, the unequal checkpoint-selection protocol is a major unresolved confound, and the closed-loop results are not accompanied by closed-loop ablations that would support the paper's mechanistic attribution.

major comments (2)
  1. [Section V.A, Tables II and V] The central closed-loop comparison uses unequal checkpoint-selection rules. AquaJEPA/JEPA variants are selected by a composite of future-velocity MAE, executed-versus-zero-action ranking, and signed no-action gap, while supervised and recurrent baselines are selected by future-velocity MAE alone. Because the AquaJEPA selection criterion directly rewards action sensitivity—the property most relevant to the planner's candidate ranking—the reported paired final-error gains could reflect the selection rule rather than the JEPA objective. This is not a hypothetical concern: Table II shows the recurrent world model has better velocity MAE (0.0043 vs 0.0052) but much lower action rank (0.198 vs 0.436). Re-selecting it under the same composite could change its closed-loop performance, and no closed-loop ablation varies the selection rule. Please re-evaluate all methods under a common checkpoint-
  2. [Section VI.B, Section VII, Table IV] The paper attributes the closed-loop improvement to 'action-sensitive predictive structure' and to the predictive representation, but no closed-loop ablation isolates a component. The four one-factor ablations are reported only on held-out prediction metrics and are not re-run in the 120-episode closed-loop protocol. The ordinary multimodal baseline differs from AquaJEPA in several factors at once (latent target, action margin, masks, modality dropout), so the paired gain over that baseline cannot identify which factor matters. The paper's own limitation section acknowledges that the single-factor study does not independently validate every component, but the Discussion and Conclusion nevertheless make a stronger attribution. At minimum, closed-loop ablations removing the action margin and modality dropout would test the stated mechanism; without them, the attribution is not established.
minor comments (5)
  1. [Section V.A] The statement 'Existing state-only, ordinary multimodal, and AquaJEPA checkpoints are frozen before the new study' is ambiguous in light of the paragraph on validation checkpoint selection. Clarify the timeline and whether the composite selection rule was applied to the checkpoints used in the new closed-loop study.
  2. [Equation (7), Section VI.A] The action margin is said to be evaluated only for samples with 'non-negligible commands,' but the threshold is not defined. Also, the no-action ablation in Section VI.A is not defined; specify whether it is the same AquaJEPA with a zero action sequence or a separately trained model.
  3. [Table II and Section V.D] Please define 'action rank' precisely in terms of the test-set fraction, including whether ties are counted as rank successes, and report the rank construction (per-sample average vs fraction of samples with strict inequality). The caption says 'executed action has lower target error than zero action,' but the main text uses 'executed-versus-zero-action ranking' and 'matched counterfactual'; make the estimand explicit.
  4. [Figure 2] The left panel plots all 120 paired differences as dots with box plots; heavy overplotting is likely. Consider using jitter, transparency, or a violin/quantile summary so the per-episode heterogeneity is actually readable.
  5. [Section VIII] The Limitations section does not mention the small training corpus (eight 30 s episodes). Add this as a limitation: all learned models are trained on a small amount of simulated data, which bounds the strength of the generalization claim even within the simulation regime.

Circularity Check

0 steps flagged

No significant circularity: closed-loop outcomes are measured on a frozen paired matrix, not derived from fitted quantities; the checkpoint-selection asymmetry is a validity confound, not a circular step.

full rationale

The paper's central quantitative claims are closed-loop goal-reaching outcomes measured on a pre-registered, frozen 120-environment paired matrix (Tables IV and V), and the held-out prediction tables (Table II) are evaluated on test transitions that do not enter training. The training objective (Eq. 6) with the action margin (Eq. 7) is a design choice; showing that removing the margin changes action rank is an ablation, not a prediction from a fitted value. The only arguable concern is Section V.A's objective-aware checkpoint selection: JEPA checkpoints are chosen using future-velocity MAE plus executed-versus-zero-action ranking and signed no-action gap, while supervised/recurrent baselines are selected on future-velocity MAE alone. This is a real between-method protocol asymmetry that could confound the comparison, but it is not circularity under the stated standard: no headline number is equal by construction to a training target or selection metric, and the closed-loop final error is a downstream measured outcome, not an algebraic consequence of the selection criterion. The paper also relies on external JEPA/world-model references (e.g., [4]-[6], [8], [9]) rather than a self-citation chain, and no uniqueness theorem or ansatz is imported from the authors' own prior work. Accordingly, no load-bearing circular step can be quoted and exhibited as Eq. X = Eq. Y; the correct classification is a validity concern, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central result rests on several hand-chosen hyperparameters (loss weights, dropout rates, planner weights) and on domain assumptions about simulator fidelity and protocol fairness. No new physical entities are introduced; the 'validity mask' and 'action margin' are architectural constructs, not entities requiring independent physical evidence.

free parameters (6)
  • Loss weights in Eq. (6) = L_vel=1, L_sonar=0.5, L_cm=0.1, L_act=2.0
    Hand-set hyperparameters; no sensitivity analysis is reported.
  • Action-margin constant in Eq. (7) = 0.02
    Margin width for the counterfactual action loss; chosen on validation.
  • Modality dropout probabilities = 0.35 for camera/sonar; 0.35 for DVL
    Dropout rates are central to the robustness claim; fixed rather than swept.
  • EMA momentum = 0.99
    Standard JEPA target momentum, but a free constant.
  • Planner cost weights in Eq. (8) = lambda_v=1, lambda_c=0.05, lambda_e=0.001, lambda_s=0.005, lambda_q=0.05
    Frozen across methods but manually chosen; they shape which candidate actions are selected.
  • Residual correction parameters = alpha=0.25, clip=0.20 m/s
    Exponential residual update for DVL velocity bias; tuned values.
axioms (5)
  • domain assumption Stonefish BlueROV2 simulation with raw FLS, Jerlov attenuation, injected DVL loss, and thruster-efficiency shifts faithfully represents underwater robot perception and dynamics for this comparison.
    Sections II-D and VIII: all closed-loop evidence comes from this simulator, so the transferable claim depends on simulator fidelity.
  • domain assumption Holding the action library, planner cost, online residual correction, and episode realization fixed across methods isolates the predictive representation as the cause of closed-loop differences.
    Sections III-D and IV-D: this is the paper's central interpretive premise.
  • domain assumption Episode-level train/validation/test splits prevent leakage, and two held-out test episodes plus 120 fresh environments are representative enough for generalization.
    Section V-A: only eight training episodes are used, so representativeness is assumed rather than demonstrated.
  • standard math The cell-stratified paired bootstrap with 10,000 resamples of entire episodes gives valid 95% intervals for paired differences.
    Section V-D: standard paired-resampling procedure, reasonable for the stated estimand.
  • domain assumption Different validation checkpoint-selection rules across methods do not bias the comparison.
    Section V.A: AquaJEPA uses a composite objective while baselines use velocity MAE; the paper assumes this is fair but does not test it.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AquaJEPA: Action-Conditioned Multimodal Predictive Representations for Underwater Robot Dynamics." pith.science (2026). https://pith.science/paper/AFKON7L3

@misc{pith2026260729393,
  author       = {Pith},
  title        = {Pith review of: AquaJEPA: Action-Conditioned Multimodal Predictive Representations for Underwater Robot Dynamics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AFKON7L3}},
  note         = {Machine review of arXiv:2607.29393}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Underwater robots combine complementary sensors whose reliability changes abruptly with water visibility, viewpoint, and vehicle motion. We introduce AquaJEPA, an action-conditioned joint-embedding predictive model that fuses an RGB camera, forward-looking sonar, and proprioception with explicit sensor validity. It predicts a future latent target conditioned on eight-thruster commands and supplies velocity and sonar-profile predictions to a shared receding-horizon planner. We study the method in Stonefish against reactive, state-only, ordinary multimodal, supervised dynamics, and recurrent world-model baselines. We further isolate the EMA target, action margin, masks, and modality dropout. A preregistered 120-environment replication comprises five independent replicates of a grid crossing three unseen obstacle maps, four water-visibility coefficients, and nominal versus shifted dynamics, while intermittently removing DVL observations. In 120 fresh paired environments with scheduled DVL loss, AquaJEPA reaches 74 goals, versus 68 for both state-only and the recurrent world model, and attains the lowest mean final error (0.906 m). Paired final-error reductions relative to ordinary multimodal prediction, supervised dynamics, and the recurrent world model are 0.273 m (95% CI: 0.190-0.356), 0.364 m (0.260-0.468), and 0.106 m (0.025-0.187), respectively. AquaJEPA therefore achieves the best aggregate closed-loop performance and significantly outperforms three action-conditioned predictive baselines in paired final error; its advantage over state-only remains statistically unresolved.

Figures

Figures reproduced from arXiv: 2607.29393 by Alan-Barsag Gazzaev, Alexey Gavrilov, Sergey Muravyov.

Figure 1
Figure 1. Figure 1: AquaJEPA architecture and closed-loop use. Sensor-specific encoders produce a mask-aware fused state. A GRU encodes future thruster commands, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Closed-loop behavior under partial observability. Left: per-episode paired final-error differences between AquaJEPA and each learned baseline; [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 2 linked inside Pith

  1. [1]

    Recurrent world models facilitate policy evolution,

    D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” inAdvances in Neural Information Processing Systems, vol. 31. Curran Associates, Inc., 2018, pp. 2450–2462. [Online]. Available: https://proceedings.neurips.cc/paper/2018/hash/ 2de5d16682c3c35007e4e92982f1a2ba-Abstract.html

    2018

  2. [2]

    Learning latent dynamics for planning from pixels,

    D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, “Learning latent dynamics for planning from pixels,” inProceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 97. PMLR, 2019, pp. 2555–2565. [Online]. Available: https://proceedings.mlr.press/v97/hafner19a.html

    2019

  3. [3]

    Dream to control: Learning behaviors by latent imagination,

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learning behaviors by latent imagination,” inInternational Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=S1lOTC4tDS

    2020

  4. [4]

    A path towards autonomous machine intelligence,

    Y . LeCun, “A path towards autonomous machine intelligence,” 2022, openReview position paper, version 0.9.2. [Online]. Available: https://openreview.net/forum?id=BZ5a1r-kVsf

    2022

  5. [5]

    Self-supervised learning from images with a joint-embedding predictive architecture,

    M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y . LeCun, and N. Ballas, “Self-supervised learning from images with a joint-embedding predictive architecture,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2023, pp. 15 619–15 629

    2023

  6. [6]

    Revisiting feature prediction for learning visual representations from video,

    A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y . LeCun, M. Assran, and N. Ballas, “Revisiting feature prediction for learning visual representations from video,”Transactions on Machine Learning Research, 2024. [Online]. Available: https: //openreview.net/forum?id=QaCCuDfBk2

    2024

  7. [7]

    Deep reinforcement learning in a handful of trials using probabilistic dynamics models,

    K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” inAdvances in Neural Information Processing Systems, vol. 31. Curran Associates, Inc., 2018, pp. 4754–4765. [Online]. Available: https://proceedings.neurips.cc/paper/2018/hash/ 3de568f8597b94bda53149c7d7f5958c-Abstract.html

    2018

  8. [8]

    Mastering Atari, Go, chess and shogi by planning with a learned model,

    J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver, “Mastering Atari, Go, chess and shogi by planning with a learned model,”Nature, vol. 588, no. 7839, pp. 604–609, 2020

    2020

  9. [9]

    TD-MPC2: Scalable, robust world models for continuous control,

    N. Hansen, H. Su, and X. Wang, “TD-MPC2: Scalable, robust world models for continuous control,” inInternational Conference on Learning Representations, 2024. [Online]. Available: https: //openreview.net/forum?id=FzpfPa6unv

    2024

  10. [10]

    Multimodal machine learning: A survey and taxonomy,

    T. Baltruˇsaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, 2019

    2019

  11. [11]

    ModDrop: Adaptive multi-modal gesture recognition,

    N. Neverova, C. Wolf, G. W. Taylor, and F. Nebout, “ModDrop: Adaptive multi-modal gesture recognition,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 8, pp. 1692– 1706, 2016

    2016

  12. [12]

    A sonar-visual dataset for cross-modal underwater robot perception,

    W. Chen, P. Tinn, P. G. Auran, M. Ludvigsen, and P. H. Haro, “A sonar-visual dataset for cross-modal underwater robot perception,”arXiv preprint arXiv:2606.01398, 2026. [Online]. Available: https://arxiv.org/abs/2606.01398

    Pith/arXiv arXiv 2026

  13. [13]

    T. I. Fossen,Handbook of Marine Craft Hydrodynamics and Motion Control, 2nd ed. Wiley, 2021

    2021

  14. [14]

    UUV simulator: A gazebo-based package for underwater intervention and multi-robot simulation,

    M. M. M. Manh ˜aes, S. A. Scherer, M. V oss, L. R. Douat, and T. Rauschenbach, “UUV simulator: A gazebo-based package for underwater intervention and multi-robot simulation,” inOCEANS 2016 MTS/IEEE Monterey. IEEE, September 2016, pp. 1–8

    2016

  15. [15]

    Stonefish: An advanced open-source simulation tool designed for marine robotics, with a ROS interface,

    P. Cie ´slak, “Stonefish: An advanced open-source simulation tool designed for marine robotics, with a ROS interface,” inOCEANS 2019 - Marseille. IEEE, June 2019, pp. 1–6

    2019

  16. [17]

    Information-theoretic model predictive control: Theory and applications to autonomous driving,

    G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou, “Information-theoretic model predictive control: Theory and applications to autonomous driving,”IEEE Transactions on Robotics, vol. 34, no. 6, pp. 1603–1622, 2018

    2018

  17. [2025]

    Available: https://arxiv.org/abs/2510.07869

    [Online]. Available: https://arxiv.org/abs/2510.07869

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.