{"id":"57e24209-e63c-4300-8082-0b42365ef8b1","arxiv_id":"2502.05555","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"APE pretrains a ResNet18 encoder with adaptively selected augmentations and freezes its early layers during policy learning, improving sample efficiency of DreamerV3 and DrQ-v2 on several visual RL benchmarks.","lead":"This paper introduces APE, a method that pretrains a visual encoder for reinforcement learning using adaptive image augmentation, then freezes most of it while training the agent. It reports faster learning and higher scores on control, Atari, and memory maze tasks, sometimes matching agents that use full state information.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"APE's central claim that adaptive pretraining augmentation drives RL gains is not directly tested: no downstream comparison uses a fixed-augmentation MoCo v2 encoder on the same ImageNet-100 data.","rationale":"I agree with the reader that the conditional verdict is appropriate, but I see the load-bearing weakness as more direct than the linear-proxy concern. The paper's central novelty is adaptive pretraining augmentation; if a non-adaptive MoCo v2 encoder on the same data yields the same RL gains, then APE reduces to 'pretraining helps,' which is already established by RRL and PIE-G. The current experiments compare against random initialization and supervised pretraining, not against the immediate non-adaptive self-supervised competitor. Table 1's linear-probe numbers are close and not tied to RL performance. The missing control should therefore be a required revision. I keep the reader's CONDITIONAL verdict because the omission is fixable and the downstream results, while not isolating the adaptive mechanism, do suggest pretraining on ImageNet-100 can benefit these RL benchmarks. The reader's proposed conditions (code release, error bars, MVP/VIP baselines) remain useful but are secondary. My concrete test would settle the causal role of the adaptive schedule.","tokens_in":15356,"tokens_out":5770,"duration_ms":56067,"concrete_test":"Train a standard MoCo v2 ResNet18 on ImageNet-100 using a fixed augmentation recipe (e.g., the default MoCo v2 augmentations or the same 'main augmentation' applied statically, without the Eq. 5 updates), then insert it into the same DreamerV3 pipeline with identical hyperparameters and seeds on DMC Walker Walk, Finger Spin, and Cup Catch at 100K and 500K steps. Compare mean returns and run-to-run variance against APE. If the fixed-recipe encoder matches APE within error bars, the adaptive schedule is not the cause of the reported gains; if APE clearly exceeds it, the adaptive claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline contribution is the 'adaptively pretrained' encoder, but no RL experiment isolates the adaptive schedule. Every downstream comparison pits APE against random initialization, supervised ImageNet-1k pretraining (RRL, PIE-G), or the base RL algorithm with no pretraining. The only APE-vs-MoCo-v2 comparison is linear probe accuracy in Table 1, where the gap is ~0.8% (90.84 vs 91.7). No RL scores are reported for the non-adaptive MoCo v2 encoder pretrained on the same ImageNet-100 subset. Consequently, the observed gains on DMC, Atari, and Memory Maze cannot be attributed to the adaptive mechanism in Eq. 5; they could equally arise from any contrastive pretrained encoder on natural images. The ablations in Fig. 10 vary the 'main augmentation' inside APE, not APE versus a fixed recipe. Because Eq. 5 updates augmentation probabilities from pretext accuracy and Table 1 uses the same linear-probe metric for evaluation, the adaptive mechanism is only validated under the very proxy it is designed to optimize. Without an external RL test against a non-adaptive pretrained control, the central claim that adaptive pretraining is a transferable ingredient remains unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes APE, a visual encoder pretrained on ImageNet-100 with MoCo v2 and an adaptive augmentation schedule (Eq. 5) that updates augmentation sampling probabilities based on pretext-task accuracy. The pretrained ResNet18 encoder, with its first three layers frozen and the last layer trainable, is then integrated into DreamerV3 and DrQ-v2. Experiments on DMC, five Atari 100k games, and Memory Maze report improved sample efficiency and final performance over non-pretrained baselines and over RRL and PIE-G, approaching state-based SAC on some DMC tasks. The paper also claims that linear-probe accuracy can serve as a useful metric for assessing pretrained encoder quality without running RL environments.","tokens_in":15646,"tokens_out":8579,"duration_ms":73811,"significance":"If the central claim were fully supported, APE would provide a simple, transferable ingredient for visual RL: an adaptively pretrained encoder that improves sample efficiency across model-based and model-free algorithms. The paper's strengths include evaluating two RL backbones (DreamerV3 and DrQ-v2), reporting results on multiple domains, and including visualizations (LayerCAM, PCA, reconstructions) that illustrate representational differences. However, the claimed contribution of adaptivity is not isolated: no downstream RL experiment compares against a non-adaptive MoCo v2 encoder pretrained on the same ImageNet-100 data. The selective Atari evaluation, mixed per-task results at longer horizons, and absence of error bars further weaken the empirical case. The paper builds directly on AdDA (Zhang et al. 2023), and the incremental contribution would need the missing control experiment to be credible.","major_comments":[{"comment":"The paper's central claim is that the adaptive augmentation schedule in Eq. (5) improves downstream RL performance. Yet no experiment compares APE against a non-adaptive MoCo v2 encoder pretrained on the same ImageNet-100 data with the same architecture and then evaluated in the same RL policies. Table 1 reports only linear-probe accuracy, where APE (91.7) exceeds MoCo v2 (90.84) by less than one point and without variance; Fig. 8 and Table 1 of Appendix B compare APE only against RRL, PIE-G, random initialization, and the base algorithm. As a result, the observed DMC, Atari, and Memory Maze gains could be attributed to contrastive pretraining on natural images generally rather than to the adaptive mechanism, and the contribution named in the title is not isolated. Please add an RL experiment with the fixed-augmentation MoCo v2 baseline under identical conditions.","section":"Methodology – Adaptive Pretraining; Experiments – Comparison with Other Pretrained Algorithms"},{"comment":"The Atari 100k evaluation uses only 5 of the 26 games, selected after acknowledging a large domain gap (Appendix B). The standard Atari 100k protocol reports results on all 26 games, and the paper itself notes a performance decrease on Atari Boxing with ImageNet-pretrained encoders. Claiming 'state-of-the-art performance' (Abstract) based on this selective subset is not supported. Please either expand to the full 26-game benchmark or qualify the claim accordingly.","section":"Experiments – Results on Other Benchmarks"},{"comment":"No variability measures are reported despite the statement that 'Results reported are averaged over at least 3 runs' (Experiments). Figures 3, 6, 7, 9, 10, 11 and Tables in Appendix B show only point estimates. Several reported differences are small or task-dependent (e.g., at 500K DMC steps, APE(DreamerV3) is lower than DreamerV3 on Walker Walk, 943.8 vs 950.4; at 1M steps APE is lower on Cheetah Run, 688.6 vs 728.7). Without error bars, confidence intervals, or significance tests, the headline improvements cannot be assessed. Please report standard deviations or comparable statistics for all main results.","section":"Experiments – DMC Results; Appendix B, Tables 1–2"},{"comment":"Algorithm 1 is internally inconsistent with the formal update rule in Eq. (5). Line 2 computes sub-batch sizes as softmax(α p_i) × num_X, while Eq. (5) updates the sampling probability with softmax(α(1 − Acc_t)); the relationship between p_i used in the loss (Eq. (4)) and the sub-batch sizes is never defined. It is also unclear how the 'pretext task accuracy' Acc_t is computed (per sub-batch, using a linear probe, at which epoch). This ambiguity prevents exact reproduction of the central adaptive mechanism.","section":"Appendix C – Implementation Details, Algorithm 1; Methodology, Eq. (5)"},{"comment":"The paper claims that linear probes 'can serve as a useful metric to assess the quality of pretrained encoders quite effectively' (Introduction) and uses this to justify the adaptive update. However, the evidence is only the comparison of Table 1 with Figure 10 on DMC tasks for two augmentation choices (blur vs jitter); no quantitative correlation, no other domains, and no statistical test are provided. Given the paper's own finding that ImageNet-pretrained encoders can hurt on multi-object Atari observations (Appendix B), the claim that linear-probe accuracy predicts downstream RL performance is under-supported and should be either substantiated or softened.","section":"Experiments – Ablation Studies; Table 1; Figure 10"}],"minor_comments":[{"comment":"The sentence 'Following the common setup of Atari 100k, we set the environment steps to 40k' is inconsistent with the definition of Atari 100k (100k environment steps, 400k frames) stated in Appendix A; please correct the value.","section":"Experiments – Results on Other Benchmarks"},{"comment":"Several typos appear, e.g., 'high-demensional', 'Intersetingly', 'enviorments'; please copy-edit the manuscript.","section":"Introduction"},{"comment":"In the lower panel of Fig. 8, the labels 'Leber(DrQ-v2)' and 'Leber(DreamerV3)' appear where 'APE(DrQ-v2)' and 'APE(DreamerV3)' are expected; please fix the figure.","section":"Experiments – Comparison with Other Pretrained Algorithms, Fig. 8"},{"comment":"Memory Maze results are shown only as training curves in Fig. 7; providing a table of final scores (with variability) would allow quantitative comparison with DreamerV3.","section":"Experiments – Results on Other Benchmarks"},{"comment":"No information about code release is provided; sharing the implementation would substantially improve reproducibility of the adaptive schedule and the RL integrations.","section":"Appendix C – Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The paper's title and abstract promise an adaptive mechanism, but the missing non-adaptive control means the contribution could reduce to 'pretrained encoder helps RL,' which is not new. The selective Atari benchmark and lack of error bars are additional concerns for a high-profile venue. I would encourage the editor to request the control experiment and full benchmark results before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper’s headline mechanism is not actually tested. The authors claim adaptive pretraining augmentation drives the RL gains, but every downstream comparison pits APE against random initialization, supervised pretrained encoders (RRL, PIE-G), or no pretraining. There is no RL run with a fixed-augmentation MoCo v2 encoder pretrained on the same ImageNet-100 subset. Table 1 only shows linear probe accuracy, where APE Blur beats MoCo v2 by under a point. So the stress-test is on the mark: the observed RL gains could come from any contrastive pretrained encoder on natural images, not specifically from the adaptive schedule in Eq. 5.\n\nWhat is genuinely new: applying the AdDA-style adaptive feedback to RL, combined with freezing early layers and finetuning the last one, is a reasonable and underexplored combination. The paper demonstrates it across both model-based (DreamerV3) and model-free (DrQ-v2) RL, which is a useful extension. The 100K DMC scores are clearly better than RRL and PIE-G on the three tasks shown, and they report averages over at least 3 runs, even without error bars.\n\nThe soft spots are significant and in proportion. The missing control is load-bearing, since the adaptive mechanism is the contribution. Also: no code release, no error bars, only five Atari games with no stated selection criterion, and the abstract’s “state-of-the-art” claim exceeds the baseline coverage (no MVP, VIP, or other recent pretrained-visual-RL methods). The Atari section is thin: one task where random init does better is waved away as multi-item detection. Memory Maze gets just one figure with no table. The circularity concern about Eq. 5 and linear probes is real but secondary because the downstream RL results are externally evaluated; the missing fixed-augmentation control is the bigger problem.\n\nBottom line: worth a serious referee. The empirical setup is otherwise solid and the 100K DMC result looks reproducible, but the central claim about adaptive pretraining is unsupported. I would ask for the fixed-MoCo-v2 RL comparison, code, error bars, and a stricter protocol for Atari task selection before accepting.","headline":"A useful empirical recipe for contrastively pretrained visual RL, but the central claim about adaptive augmentation is untested because the paper never compares against a fixed-augmentation MoCo v2 encoder on the same data.","tokens_in":16170,"tokens_out":2470,"would_cite":false,"duration_ms":24537,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adaptively pretraining a ResNet18 encoder on natural images with a closed-loop augmentation schedule makes DreamerV3 and DrQ-v2 learn visual control tasks faster and reach higher scores, approaching state-based SAC.","keywords":["adaptive pretraining","contrastive learning","visual reinforcement learning","sample efficiency","DreamerV3","DrQ-v2","data augmentation","ImageNet-100"],"falsifier":"Pretrain an identical ResNet18 with a fixed augmentation schedule that always uses the best single composition (for example, gaussian blur applied with probability 0.5) and compare downstream DMC scores against APE. If the fixed schedule matches or exceeds APE's scores, the adaptive update in Eq. 5 is not the cause of the gains. A second test: construct a pretext task whose linear-probe accuracy orders encoders differently from their downstream DMC performance, which would break the claimed proxy relation.","tokens_in":15125,"feed_emoji":"🎮","tokens_out":6153,"duration_ms":50785,"temperature":0.7,"pith_summary":"The paper argues that the pretraining recipe of a visual encoder, not its depth, is what determines how well it transfers to reinforcement learning. APE pretrains a ResNet18 on ImageNet-100 with a contrastive objective (MoCo v2) while adaptively re-weighting five data augmentations: after each epoch, augmentations with lower pretext accuracy get sampled more often. When the first three layers of the resulting encoder are frozen and only the last layer is finetuned inside DreamerV3 or DrQ-v2, the agents learn faster and reach higher scores than with randomly initialized, supervised, or fixed-augmentation encoders. If the claim holds, adaptive pretraining augmentation becomes a simple and reusable ingredient for sample-efficient visual RL that needs no auxiliary tasks or extra sensors.","feed_headline":"Visual RL gets faster with adaptively pretrained encoders","feed_subtitle":"A ResNet18 pretrained with closed-loop augmentation lifts DreamerV3 and DrQ-v2 near state-based SAC performance.","key_machinery":"The load-bearing mechanism is the adaptive augmentation update of Eq. 5, $p^{t+1}_i = \\operatorname{softmax}(\\alpha(1 - \\mathrm{Acc}_t))$, which raises the sampling probability of augmentation compositions whose pretext accuracy is low. All compositions share the same query and key encoders, so the InfoNCE loss keeps the representations consistent across sub-batches. During policy learning, the first three ResNet18 layers are frozen and only the last layer is finetuned, preserving the pretrained generalization while allowing task-specific adaptation. Linear-probe accuracy on ImageNet-100 is used as the evaluation proxy for encoder quality.","core_discovery":"APE's central claim is that a closed-loop augmentation schedule during contrastive pretraining produces visual features that are more generalizable to downstream control tasks than any fixed augmentation recipe or deeper architecture. On DMC, APE raises DreamerV3's mean 100K-step score from 458.6 to 836.7 (SAC:state scores 816 on the same three tasks), and its 500K-step score from 749.1 to 882.8; DrQ-v2's mean score rises from 284.6 to 560.2 at 100K steps. The same encoder also lifts DreamerV3 on Atari 100k and Memory Maze with fixed hyperparameters. The paper also reports that a deeper ResNet50 pretrained the same way performs worse than ResNet18, and that linear-probe accuracy on ImageNet-100 ranks pretrained encoders in the same order as their downstream RL performance.","pith_inferences":["If the linear-probe proxy holds generally, pretraining research for visual RL can be conducted offline on static image datasets, which is far cheaper than environment interaction and would accelerate iteration.","The adaptive schedule is only tested during pretraining; an untested extension is to keep adapting augmentation probabilities while the agent finetunes, which could further narrow the domain gap on tasks like Atari where ImageNet features hurt.","The paper's Atari Boxing analysis suggests that single-object ImageNet features can blind the agent to multi-item scenes; a testable fix is to include multi-object or scene-level datasets in the pretraining distribution."],"forward_implications":["APE raises DreamerV3's mean DMC score from 458.6 to 836.7 at 100K steps and from 749.1 to 882.8 at 500K steps, bringing pixel-based control near the state-based SAC baseline on those tasks.","The benefit transfers across RL paradigms: the same pretrained encoder improves both model-based DreamerV3 and model-free DrQ-v2, so the gain lives in the representation rather than in a specific agent.","Because linear-probe accuracy on ImageNet-100 predicts downstream RL performance, cheap static-image evaluation can screen candidate pretrained encoders before any RL training is run.","Encoder depth is not the driver: ResNet50 pretrained with the same recipe underperforms ResNet18 on fine-grained control, so the pretraining strategy matters more than network size."],"supporting_citations":[{"why":"Provides the MoCo v2 momentum contrastive framework that APE's pretraining phase is built on.","marker":"(He et al. 2020)"},{"why":"Establishes the role of data augmentation in contrastive learning and the linear probe evaluation protocol.","marker":"(Chen et al. 2020a)"},{"why":"Supplies the closed-loop adaptive augmentation feedback structure that APE adapts to the RL encoder setting.","marker":"(Zhang, Zhu, and Yu 2023)"},{"why":"The DreamerV3 world-model agent that APE is integrated with and that provides the main model-based baseline.","marker":"(Hafner et al. 2023)"},{"why":"The DrQ-v2 model-free agent used to show APE's generality across RL paradigms.","marker":"(Yarats et al. 2021a)"},{"why":"The PIE-G supervised-pretrained encoder baseline that APE compares against for ResNet pretraining.","marker":"(Yuan et al. 2022)"},{"why":"The RRL baseline that first used a frozen ResNet as representation for RL, the setting APE extends.","marker":"(Shah and Kumar 2021)"},{"why":"Supports the motivation that pretraining augmentations are a key factor for robust downstream performance.","marker":"(Burns et al. 2023)"},{"why":"Defines the InfoNCE objective used in the contrastive pretraining loss.","marker":"(van den Oord, Li, and Vinyals 2018)"}],"fun_headline_variants":["Adaptive pretraining boosts visual RL sample efficiency","APE lifts DreamerV3 to SAC-level DMC scores","Closed-loop pretraining adapts encoders for RL speed","Adaptively pretrained encoder nears state-based RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim holds only if pretext-task accuracy on ImageNet-100 is a reliable proxy for how useful the learned features will be for downstream RL control; if that correlation is weak, the adaptive schedule has no principled reason to outperform a fixed augmentation recipe.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive pretraining boosts visual RL sample efficiency","APE lifts DreamerV3 to SAC-level DMC scores","Closed-loop pretraining adapts encoders for RL speed","Adaptively pretrained encoder nears state-based RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001117,"raw_usage":{"total_tokens":4661,"prompt_tokens":966,"completion_tokens":3695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":3630}},"tokens_in":582,"tokens_out":3695,"duration_ms":26391,"temperature":1.0,"reasoning_tokens":3630,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T18:51:37.644880+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pretrain an identical ResNet18 with a fixed augmentation schedule that always uses the best single composition (for example, gaussian blur applied with probability 0.5) and compare downstream DMC scores against APE. If the fixed schedule matches or exceeds APE's scores, the adaptive update in Eq. 5 is not the cause of the gains. A second test: construct a pretext task whose linear-probe accuracy orders encoders differently from their downstream DMC performance, which would break the claimed proxy relation.","supporting_citations":[{"cited_title":"Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning","cited_arxiv_id":"2212.08860","evidence_quote":"The PIE-G supervised-pretrained encoder baseline that APE compares against for ResNet pretraining."}],"review_version":1}