{"id":"464f8b99-fdcf-4291-b527-9fc607e11eac","arxiv_id":"2504.16591","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"JEPA can be adapted to pixel-based reinforcement learning, but it requires either RL gradients or a variance-based regularizer to avoid model collapse.","lead":"The authors show how to adapt Joint-Embedding Predictive Architectures (JEPA) to reinforcement learning from images, using a vision transformer to encode frames and a predictor to anticipate future latent embeddings. They report that on the Cart Pole task, adding a variance regularizer prevents the collapse of JEPA and allows the agent to learn, though JEPA works best when combined with reinforcement learning gradients.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: without blocking the JEPA loss but keeping variance regularization, the specific claim that JEPA (not the regularizer) learns informative representations is untested.","rationale":"The reader identified the narrow evidence base (single environment, five runs, no error bars) as the weakest assumption. I agree, but I see a more specific and potentially more damaging internal confound: the ablation omits the control that would separate the effect of the JEPA loss from the variance regularizer. This does not change the overall verdict of CONDITIONAL — the paper explicitly labels its results as exemplary and calls for further work — but it sharpens the condition: the central sub-claim that 'JEPA is able to learn informative state representations' without RL gradients remains unverified until this control is run. My recommended verdict is therefore unchanged, but the concrete test should be added to the revision requirements.","tokens_in":4032,"tokens_out":7903,"duration_ms":79103,"concrete_test":"Run the configuration 4 pipeline exactly, but remove the JEPA loss LJEPA from the total loss in Eq. 3, keeping Lreg and blocking actor/critic gradients to the encoder. Train for the same 100k environment steps with at least 5 seeds and report per-seed episodic returns. If this control achieves comparable returns to configuration 4, the JEPA objective is not the driver. If it collapses or fails to learn, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central sub-claim, stated in Section 3 ('This shows that JEPA is able to learn informative state representations without gradient propagation from the reinforcement learning task'), rests on configuration 4 (J, ∇̂, R). The ablation varies JEPA loss, gradient propagation, and regularization, but it omits the control (Ĵ, ∇̂, R) — no JEPA loss, no RL gradients, with the variance regularizer. The variance regularizer (Eq. 2) alone trains the encoder to maximize batch variance, which prevents collapse but does not enforce temporal predictive structure. In a low-dimensional task like Cart Pole, a high-variance but otherwise unstructured embedding might still support a policy learned by the actor-critic, especially with 64 dimensions. Without this control, the observed learning in configuration 4 cannot be attributed to the JEPA predictive objective; it could be driven entirely by the variance regularizer. This is a load-bearing gap because the strongest claim is specifically that JEPA learns informative representations without RL gradient propagation, and the ablation does not isolate the JEPA loss from the regularizer in that setting.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptation of Joint-Embedding Predictive Architectures (JEPA) to reinforcement learning from image observations. The authors use a vision transformer to encode three consecutive frames, a target encoder maintained as an exponential moving average, and a small predictor that receives the action; they train with a Euclidean latent prediction loss plus optional actor-critic gradients and a variance regularization loss. In a Cart Pole experiment with pixel observations and PPO, they compare four configurations (JEPA loss on/off, RL gradient propagation on/off, variance regularizer on/off) and report that the configuration with JEPA loss, no RL gradients, and variance regularization prevents collapse and learns. They conclude that JEPA can learn informative state representations without RL gradient propagation and are promising for image-based RL.","tokens_in":4223,"tokens_out":7369,"duration_ms":67224,"significance":"If the central claim were established, it would be a useful demonstration that a self-supervised latent prediction objective can support RL from images without direct reward-gradient flow through the encoder, potentially reducing sample complexity and improving transfer. The architecture and loss are clearly described, the collapse-prevention mechanism is concrete, and the three-way ablation is a sensible experimental strategy. However, the evidence is currently too thin to support the broad conclusion: one environment, five runs, no error bars, no comparison against standard representation baselines, and an ablation that does not isolate the JEPA loss from the variance regularizer. The contribution is more of a proof-of-concept than a validated method.","major_comments":[{"comment":"The central sub-claim in Section 3 ('This shows that JEPA is able to learn informative state representations without gradient propagation from the reinforcement learning task') rests on configuration 4 (J, no gradient propagation, R), but the ablation compares configuration 4 with configuration 3 (J, no gradient propagation, no R), which differs by two factors: the presence of the JEPA loss and the presence of the regularizer. Because Eq. (2) is present in configuration 4 and absent in configuration 3, the comparison cannot isolate the contribution of the JEPA loss from that of the variance regularizer. A control with no JEPA loss but with the same regularizer (J omitted, no gradient propagation, R) is required; without it, the observed learning could be driven entirely by the regularizer, which is known to prevent collapse and produce non-degenerate embeddings. This gap is load-bearing because the conclusion attributes the learning to JEPA rather than to variance regularization.","section":"Section 3, configuration 4; Eq. (2)"},{"comment":"The empirical evidence is too thin to support the paper's conclusions. Figure 3 shows only the averaged episodic return of five runs on Cart Pole, with no error bars, standard deviations, or seed-level analysis, and no quantitative comparison of final performance or sample efficiency. There is also no baseline against standard representation-learning methods for RL (e.g., a VAE or CURL), despite the introduction motivating JEPA as an alternative to autoencoders. The statement in Section 4 that 'JEPAs are promising candidates for reinforcement learning' therefore goes beyond what the presented data support.","section":"Section 3, Figure 3"},{"comment":"The introduction states that the method targets 'problems like Atari games' with low-dimensional state vectors, but the only experiment is Cart Pole. Since Cart Pole has a four-dimensional state and very simple visual dynamics, it is not representative of the harder image-based RL tasks the method is proposed for. The conclusion should be explicitly restricted to the demonstrated setting unless additional environments and comparisons are provided.","section":"Sections 1 and 4"}],"minor_comments":[{"comment":"The formula 'θt+1 = 0.99·θt + 0.01·θt+1' is self-referential; the intended exponential moving average should use distinct notation for the target and online encoder weights.","section":"Section 2, target encoder update"},{"comment":"The introduction mentions an additional latent variable z in the JEPA formulation, but z never appears in the described method or in Eq. (1); clarify whether z is omitted and why.","section":"Section 2, Figure 1"},{"comment":"The caption says 'accumulated results' and the text says 'running average,' but neither the aggregation window nor the accumulation procedure is specified.","section":"Section 3, Figure 3"},{"comment":"The term Var(sx)_i is not defined precisely; specify whether this is the sample variance over the batch for each embedding dimension and whether the batch is the same as that used for the JEPA loss.","section":"Section 2, Eq. (2)"},{"comment":"The phrase 'a very limited number of na neurons' appears to contain a typo, and PPO hyperparameters, vision transformer architecture, image resolution, optimizer, and batch size are not reported, which limits reproducibility.","section":"Section 2, reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The missing control (no JEPA loss but with the variance regularizer) is the key technical issue; it is fixable with one additional configuration. If the authors supply that control, report error bars, and either add at least one more environment or explicitly weaken the conclusion, the paper could be a useful short contribution. I would not recommend rejection on scope grounds, because the architecture description is sound and the question is timely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, clearly written paper that adapts JEPA to pixel-based RL with an action-conditioned temporal predictor, a classification token, and a variance regularizer to prevent collapse. The architecture is sensible and the collapse ablation is genuinely instructive. But the central claim — that JEPA alone learns informative representations without RL gradients — is under-supported, and one missing control makes that specific sentence shaky.\n\nWhat's new: the specific combination of a three-frame temporal context, an action-conditioned predictor, and a VICReg-style variance term for RL is not present in the cited I-JEPA/BYOL/VICReg/CURL literature. The design choices are explained clearly and mostly reasonable, especially keeping the predictor small and using token-level embeddings. The four-configuration ablation is a good idea and clearly demonstrates that pure JEPA without any collapse prevention collapses on Cart Pole, which is a useful warning for the community.\n\nWhere it's soft: one environment, five runs, no error bars or variance reporting. The paper targets Atari-class tasks but provides no Atari evidence, and the generalization claim in the conclusion goes well beyond the data. The bigger issue is the missing control the stress-test flags: configuration 4 is J, ∇hat, R, but there is no (Jhat, ∇hat, R). Since the variance regularizer alone is known to prevent collapse, you cannot tell whether the learned policy reflects temporal predictive structure from the JEPA loss or just a high-variance but otherwise unstructured embedding. On Cart Pole, a 64-dimensional embedding with high per-dimension variance might be enough for the actor-critic to find a decent policy. The sentence in Section 3, 'This shows that JEPA is able to learn informative state representations without gradient propagation from the reinforcement learning task,' is therefore too strong as written. The paper needs that control, plus per-run curves or at least means and standard deviations. Minor points: no code or full hyperparameters, and the claim that 'the same is true for all learnable tasks for which a short state description can be created' is a leap from a 4-dimensional toy. The references are appropriate and standard.\n\nWho benefits: people working on self-supervised representation learning for RL, especially at the workshop or preliminary-study level. This is a legitimate candidate for peer review — the question is worth asking and the paper is honest about being an investigation — but it needs the missing control, more environments, and basic statistics before the central claim can be accepted.\n\nRecommendation: do not desk reject. Send it to review with the expectation of heavy revision or re-review. If the authors add the control and a second environment, this becomes a useful short paper.","headline":"A sensible, clearly-written JEPA-for-RL adaptation with an instructive collapse ablation, but the central claim is under-supported by one narrow environment and a missing control.","tokens_in":4755,"tokens_out":2624,"would_cite":true,"duration_ms":26322,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that Joint-Embedding Predictive Architectures can be adapted to reinforcement learning, with a variance regularizer allowing them to learn state representations without reward-gradient feedback.","keywords":["joint-embedding predictive architecture","JEPA","reinforcement learning","self-supervised learning","model collapse","vision transformer","Cart Pole"],"falsifier":"Run the paper's regularized no-gradient configuration on an Atari game such as Pong with the same architecture and check whether the batch-wise embedding variance stays above $10^{-7}$ and the episodic return improves; if the agent fails or collapses despite the variance regularizer, the central claim is refuted.","tokens_in":3823,"feed_emoji":"🎮","tokens_out":6115,"duration_ms":58516,"temperature":0.7,"pith_summary":"Joint-Embedding Predictive Architectures (JEPA) are self-supervised models that predict embeddings in latent space rather than reconstructing pixels; this paper asks whether that machinery can supply the state representations an image-based reinforcement-learning agent needs. The authors build a JEPA around a vision transformer that encodes a short window of frames, with a small action-conditioned predictor forecasting the next frame's embedding, and test four training configurations on Cart Pole. Their central result is that when the JEPA loss is used alone and reinforcement-learning gradients are blocked from the encoder, the model collapses to constant embeddings unless a variance regularizer is added; with the regularizer, the agent learns. They also find that combining the JEPA loss with actor-critic gradient flow gives the fastest reward growth, and they conclude JEPAs are a viable route for representation learning in reinforcement learning from images.","feed_headline":"JEPA learns control states without RL gradients","feed_subtitle":"A variance regularizer stops collapse, and latent prediction alone trains a Cart Pole agent from pixels.","key_machinery":"The machinery is the JEPA pipeline: a vision-transformer x-encoder that takes the three frames $f_{t-2}, f_{t-1}, f_t$ with positional encodings for both patch location and frame index; a target y-encoder with weights updated as a moving average of the x-encoder, applied to frames $f_{t-1}, f_t, f_{t+1}$; a two-layer predictor that is fed the action to output a predicted embedding; and the squared Euclidean loss between predicted and target embeddings. Collapse prevention is the second piece of machinery: when the RL losses cannot reach the encoder, a variance regularizer encourages the batch-wise embedding variance to stay above zero, and the paper reports this restores learning.","core_discovery":"On the paper's own terms, the discovery is that a JEPA trained purely by latent prediction can carry representation learning for RL from images. The evidence is a four-way ablation on Cart Pole: the baseline encoder trained only by actor-critic gradients learns limited behavior; adding the JEPA loss improves reward and avoids plateau; stopping RL gradient flow while keeping the JEPA loss leads to model collapse, with batch-wise embedding variance dropping below $10^{-7}$; and adding a variance regularizer prevents that collapse so the actor can learn from the embeddings. The paper explicitly concludes that JEPA is able to learn informative state representations without gradient propagation from the reinforcement learning task, and that JEPAs are promising candidates for reinforcement learning.","pith_inferences":["A natural extension not tested in the paper: running the same four configurations on Atari games with strong visual distractors would test whether the regularized no-gradient variant still avoids collapse and learns, or whether harder perception demands RL-gradient feedback.","If JEPA representations are learned purely from next-frame latent prediction, they may transfer across reward functions defined on the same environment, a property the paper does not investigate.","Using a single classification token rather than all patch embeddings keeps the representation low-dimensional but may discard spatial details needed for fine-grained control; the paper does not measure this trade-off."],"forward_implications":["If JEPA plus a variance regularizer can replace reward-driven encoder training, pixel-based RL agents could learn reusable state representations before seeing any reward, cutting training cost.","Combining the JEPA loss with actor-critic gradient flow accelerates early reward growth and avoids the plateau seen in the baseline on Cart Pole.","Because the representation is a single 64-dimensional classification-token embedding, the approach is matched to low-dimensional-state tasks such as Atari games, as the authors intend.","Batch-wise embedding variance is a cheap, observable collapse indicator that can be monitored during RL training to detect representation failure early."],"supporting_citations":[{"why":"Defines the JEPA architecture of encoders plus a latent-space predictor that this paper adapts to reinforcement learning.","marker":"[7]"},{"why":"Supplies the momentum-based target-encoder update used to stabilize JEPA training and prevent collapse.","marker":"[8]"},{"why":"Supplies the variance-based regularization idea behind the paper's collapse-prevention term.","marker":"[10]"},{"why":"Provides a JEPA-style latent prediction objective for image data, the closest prior success this method extends to RL.","marker":"[9]"},{"why":"Provides the Cart Pole environment and image observations used in the experiments.","marker":"[2]"},{"why":"Provides the variational autoencoder baseline whose pixel-reconstruction limitation motivates moving to latent prediction.","marker":"[4]"}],"fun_headline_variants":["JEPA learns Cart Pole control with zero RL gradients","Latent prediction alone beats RL gradients on Cart Pole","JEPA without RL gradients masters pixel control","Variance regularizer stops JEPA collapse for RL","JEPA's latent prediction alone trains a control agent"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Cart Pole, tested with five runs and no error bars, stands in for all low-dimensional-state image-based RL tasks, so if Cart Pole is not representative of harder Atari-style environments, the paper's conclusion that JEPAs are promising for RL does not follow.","fun_headline_variants_meta":{"raw":{"variants":["JEPA learns Cart Pole control with zero RL gradients","Latent prediction alone beats RL gradients on Cart Pole","JEPA without RL gradients masters pixel control","Variance regularizer stops JEPA collapse for RL","JEPA's latent prediction alone trains a control agent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000686,"raw_usage":{"total_tokens":3013,"prompt_tokens":749,"completion_tokens":2264,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":365,"completion_tokens_details":{"reasoning_tokens":2190}},"tokens_in":365,"tokens_out":2264,"duration_ms":16186,"temperature":1.0,"reasoning_tokens":2190,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:59:25.362193+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's regularized no-gradient configuration on an Atari game such as Pong with the same architecture and check whether the batch-wise embedding variance stays above $10^{-7}$ and the episodic return improves; if the agent fails or collapses despite the variance regularizer, the central claim is refuted.","supporting_citations":[{"cited_title":"Bootstrap your own latent-a new approach to self-supervised learning","cited_arxiv_id":null,"evidence_quote":"Supplies the momentum-based target-encoder update used to stabilize JEPA training and prevent collapse."},{"cited_title":"Self-supervised learning from im- ages with a joint-embedding predictive architecture","cited_arxiv_id":null,"evidence_quote":"Provides a JEPA-style latent prediction objective for image data, the closest prior success this method extends to RL."},{"cited_title":"Auto-encoding variational bayes","cited_arxiv_id":null,"evidence_quote":"Provides the variational autoencoder baseline whose pixel-reconstruction limitation motivates moving to latent prediction."}],"review_version":1}