Pith. sign in

REVIEW 3 major objections 6 minor 63 references

Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read WSA, a learned weighted sum over frozen pre-trained visual encoders, matches or beats end-to-end trained RL agents on multiple Atari games.

desk verdict WSA is a solid empirical method paper; its Atari claim holds, but only with per-game pretrained features rather than generic prior knowledge. read the letter →

arxiv 2507.07197 v1 pith:O6IOMOKS submitted 2025-07-09 cs.LG cs.AI

Elia Piccoli , Malio Li , Giacomo Carfì , Vincenzo Lomonaco , Davide Bacciu This is my paper · ORCID
classification cs.LGcs.AI
keywords reinforcementlearningpre-trainedmodelsfeaturecombinationattentionmechanismAtarirepresentationweightsharingrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement-learning agents usually learn to see and act at the same time: a convolutional network is trained end-to-end with the policy, which is costly and can over-specialize. This paper proposes to instead freeze a set of pre-trained visual encoders and learn only how to combine them, using a module called Weight Sharing Attention (WSA). WSA assigns each encoder a normalized weight based on the current state, then forms the state representation as a weighted sum of embeddings. The paper claims that this enriched representation matches or beats end-to-end training on several Atari games, while making the agent robust to visual changes and able to add or remove encoders during training. The one caveat the paper documents is coverage: if the pretraining data does not include states the agent will later see, the frozen features are not enough and performance drops until the data is expanded.

What carries the argument

The central object is the Weight Sharing Attention (WSA) combination module. It takes embeddings $E_i$ computed by adapters $A_i$ from each frozen pre-trained model, forms a context $C$ from a separate state encoder, and uses a single shared MLP $f_\theta(C, E_i)$ to output a weight $w_i$ for every model. The weight vector is normalized by its $\ell^1$ norm and the enriched state is $R = \sum_i w_i E_i$. Because the MLP is shared and the adapters map any encoder output to a fixed dimension, the module works with any number of models and adds few trainable parameters; its job is to decide, for each state, which frozen view of the world the policy should trust.

What would settle it

Take a game whose state visually changes as play progresses, such as Space Invaders, train all encoders only on frames from random agents, then train WSA and an end-to-end agent with the same PPO budget; if WSA's evaluation score stays close to random while the end-to-end agent scores substantially higher, the parity claim is falsified for that game, because the failure is attributable to pretraining coverage rather than to the combination module.

Watch

Extended reading notes

Core claim

The paper's central discovery is that combined frozen features can carry the representational load of an RL agent. With WSA, each pre-trained model's output passes through a small trainable adapter to a common embedding space; a shared MLP takes the current state encoding $C$ and each embedding $E_i$ and predicts a weight $w_i$; the weights are L1-normalized and the final representation is $R = \sum_i w_i E_i$. On Pong and Ms. Pacman this representation matches or beats the end-to-end baseline, and on an extended nine-game evaluation WSA exceeds the end-to-end score on five games and is statistically comparable on Breakout. The paper also shows the weights are interpretable and adapt as the game changes, that WSA resists color and behavior modifications better than end-to-end nets, and that encoders can be inserted or removed mid-training; when the single most informative encoder is removed, performance collapses, while removing less informative ones does not.

Load-bearing premise

The load-bearing premise is that the frozen pre-trained encoders, trained on one million frames of random-agent play, encode features informative enough that they stay useful for every state the policy later encounters; if the pretraining data misses a part of the game, WSA's performance collapses regardless of how the embeddings are combined.

Editorial extensions

If this is right

  • On a nine-game Atari evaluation with tuned hyperparameters, WSA outscores the standard end-to-end agent on Asteroids, Enduro, MsPacman, Pong, and Seaquest, and overlaps statistically with it on Breakout, with mean capped human-normalized scores within 0.05.
  • During training, WSA reaches high rewards earlier on Pong and Ms. Pacman because the policy starts from meaningful visual features instead of raw pixels.
  • In modified Atari settings, WSA keeps roughly three times the end-to-end agent's score on Breakout color and behavior changes and remains positive in Pong's Lazy Enemy variation.
  • When the set of available encoders grows or shrinks while the agent is learning, WSA reallocates its weights without a performance drop, until the single most informative encoder is removed.
  • The learned weights are inspectable: on Breakout, WSA spreads attention across models early and shifts to the object-tracking encoder once the playfield opens up, giving a window into which visual prior is doing the work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same module could act as a lightweight router over heterogeneous frozen encoders, since adapters project any embedding into a common space; the open condition is that each encoder's pretraining data must cover the deployment states.
  • The Breakout failure suggests a cheap diagnostic for distribution shift during deployment: monitor the WSA weight distribution and flag sudden concentration on a single encoder, as the paper observes weights concentrating late in the game but does not propose this as a monitoring signal.
  • A direct testable extension is to pretrain the encoders on task-relevant expert frames, not random-agent frames, for the games where WSA lags (Beam Rider, Qbert, Space Invaders); the Breakout result predicts parity or better whenever the data covers the states the policy visits.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Weight Sharing Attention (WSA), a module that combines embeddings from multiple pre-trained vision models into a single state representation for deep RL agents. The module uses adapters to project each model's output into a shared space and a shared MLP, conditioned on a context encoding from an autoencoder, to compute L1-normalized weights per model; the final representation is a weighted sum of embeddings. The authors compare WSA against end-to-end PPO and several alternative combination modules (linear, fixed-linear, convolutional, reservoir, dot-product attention, averaging, single large pre-trained model, full training of the encoders) on three Atari games, then extend the comparison to nine Atari games with per-game hyperparameter optimization. Additional experiments evaluate robustness to HackAtari variations, dynamic addition/removal of pre-trained models, and a ManiSkill push-cube task. The central reported claim is that WSA achieves performance comparable to end-to-end models on multiple Atari games while adding modularity, explainability, and robustness.

Significance. If the central claim holds, WSA is a useful design pattern for injecting pre-trained visual knowledge into RL without fine-tuning large encoders, with a modest computational overhead and a dynamic, interpretable combination mechanism. The paper is an empirical method study with open code, clear architectural diagrams, and a wider breadth of combination baselines than is common in this literature. The reproducibility effort is a strength: the authors report pseudocode, hyperparameters, timing breakdowns, and per-seed evaluation protocols. However, the significance is bounded by two conditions: the pre-trained models are trained on 1M random-agent frames collected from the same game that is later used for RL, and the extractor/embedding-size selection is performed on the same games used for the headline comparison. These conditions substantially narrow the scope from 'prior knowledge about the world' to 'in-domain pretrained features,' and they weaken the head-to-head comparison against end-to-end baselines.

major comments (3)
  1. [Section 4.2 / Table 1] The central claim of comparable performance is conditional on environment-matched pretraining coverage, and the paper's own Breakout experiment demonstrates the failure mode. With random-agent pretraining data, WSA scores 99.58 ± 6.66 versus 404.46 ± 13.49 for E2E; performance only improves to 345.52 ± 6.47 after the pretraining data is augmented with expert demonstrations, still below E2E. As Section 4.3 notes, a similar distribution-shift problem is hypothesized for Space Invaders, meaning this is not an isolated artifact but a structural limitation. The abstract's framing that 'RL agents do not have any prior knowledge about the world' is undermined by the fact that the pretraining data is collected from the same game and the same environment, so the 'prior knowledge' is in-domain rather than general world knowledge. The authors should either re-scope the claims to explicitly say 'in-domain pretrained features' and report which games suffer from coverage gaps, or add experiments with encoders pretrained on other games or external image data to demonstrate true transfer.
  2. [Section 4 / 'Initial Experiments' and Appendix F] The selection of feature extractors and their embedding sizes is performed on the same three games that are later used for the main comparison (Pong, Ms. Pacman, Breakout). The paper states: 'we run a first round of experiments for the three games to select the three best performing extractors and their respective embedding size, which will be used for our empirical analysis.' This means the headline results in Table 1 reflect a test-set selection: the best combination module per game is chosen based on its performance on the evaluation games, and then that same module is reported as the winner on those games. The same issue applies to the extended experiments in Section 4.3, where CARBS sweeps are run per game to optimize hyperparameters including embedding size, and the best configuration is then evaluated on the same games. A fair comparison would require selecting configurations on development games and evaluating on held-out games, or at least documenting the selection protocol and its potential bias. As written, the 'comparable performance' over the alternative modules is not an unbiased estimate.
  3. [Section 4.3 / Table 2] The comparison with the end-to-end baseline is uneven because WSA receives per-game hyperparameter optimization while the E2E scores are taken from Open RL Benchmark without the same search budget. The paper reports that 'for each game, we run a set of sweeps consisting of 50/100 runs (2-3 days of compute time) to identify the optimal configurations,' whereas the E2E numbers are 'computed using the OpenAI/Baselines PPO results collected in Open RL Benchmark.' This confounds the contribution of the WSA module with the effect of per-game tuning. Moreover, even with this optimization, WSA is worse on three of nine games (Beam Rider, Qbert, Space Invaders), with Qbert showing a 40% drop in raw score. The honest summary is that WSA is comparable on roughly six of nine games after substantial per-game tuning, not that it matches E2E 'without any fine-tuning of hyperparameters.' The authors should either match the tuning budget for E2E or adjust the claim to explicitly distinguish the tuned WSA results from the untuned initial experiments.
minor comments (6)
  1. [Abstract / Introduction] There are several grammatical issues, e.g., 'pre-trained models have been a key components' and 'which presents themselves as the key challenges'; the manuscript would benefit from a careful proofread throughout.
  2. [Algorithm 1] Line 7 normalizes W by its L1 norm and reassigns W, but line 8 then writes 'R = sum_i w_i * E_i' without clarifying that w_i refers to the normalized weights; this is a minor notational ambiguity that could confuse readers implementing the algorithm.
  3. [Section 4.1] The sentence 'The poor performance of the ENS baseline further highlights the need for a smarter and more complex way to combine multiple embeddings' is slightly misleading because ENS is an intentionally simple baseline; a more precise statement would note that naive averaging is insufficient without overclaiming that complexity is the only remedy.
  4. [Appendix C.2 / Table 8] The comparison of WSA, InstructRL, and OpenVLA mixes training protocols: InstructRL is trained online for 7.5M steps, while OpenVLA is fine-tuned offline for 50K steps with LoRA. The table's 'Training Steps' column is therefore not directly comparable, and the conclusion 'WSA masters the task whereas the other methods struggle' should be qualified by this protocol mismatch even though wall-clock time is also reported.
  5. [Section 4.4 / Table 3] The table header 'Random E2E WSA Training Testing - variation original variation original variation' is confusing; it would be clearer to have separate column groups for the three methods (Random, E2E, WSA) with subcolumns for original and variation, as is done later in Appendix D.
  6. [Section 4.5] The 'WSA Explainability' paragraph is purely qualitative; reporting quantitative metrics for weight interpretability (e.g., agreement with oracle or ablation) would strengthen the claim, but as presented this is a minor presentation issue rather than a load-bearing flaw.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; WSA is an empirical combination module evaluated on held-out seeds, and the only self-citation (Cossu et al., 2024) is contextual, not load-bearing.

full rationale

The paper does not derive a predictive law from a fitted parameter. WSA's learned weights are optimized on PPO/DQN training episodes and evaluated on held-out seeds, so the reported rewards are not forced by construction. The pre-trained encoders are trained on 1M random-agent frames per game and, for Breakout, augmented with expert frames; this is an in-domain-data limitation, and the paper reports the failure explicitly (Section 4.2: WSA 99.58 vs E2E 404.46 before augmentation, 345.52 afterward). No equation reduces to its inputs: the combination module, adapters, and policy are all trained with ordinary RL losses and compared to external Open RL Benchmark E2E numbers (Huang et al., 2024). The only self-citation is Cossu et al. (2024), referenced in Section 4.2 as related work on updating pre-trained models; it does not justify WSA, does not forbid alternatives, and carries no load in the argument. Per-game hyperparameter sweeps (Section 4.3) and initial extractor selection are standard model selection, not circularity, though they may affect how the comparison generalizes.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method rests on the assumption that frozen, game-specific pre-trained features can be combined by a learned weighted sum into a policy-ready representation. That assumption is empirically validated in most tested games but demonstrably fragile when the pretraining data is not representative (Breakout). The free parameters are design choices disclosed in the paper.

free parameters (3)
  • Per-game embedding size = Pong: 1024; Ms.Pacman/Breakout: 256 (WSA)
    Chosen after initial experiments based on evaluator performance on the same games; affects capacity and results.
  • Number and choice of pre-trained models = 4 models: State Representation, Object Keypoints (CNN and KeyNet), Video Object Segmentation; plus a deep autoencoder…
    Arbitrary design choice; results may depend on this specific set.
  • Shared Weight Network depth = 1 hidden layer for Atari; 3 layers for ManiSkill
    Architecture adjusted per domain; the ManiSkill fix was a change, not a search.
assumptions (4)
  • domain assumption Frozen pre-trained features from random-agent rollouts are sufficiently informative for policy learning on the target game.
    The whole method relies on this; contradicted by the initial Breakout result (Section 4.2).
  • domain assumption A convex combination (L1-normalized weights) of embeddings is sufficient to represent the enriched state; no interaction terms are needed.
    WSA restricts to a weighted sum; the paper does not test multiplicative or higher-order combinations.
  • standard math PPO with default hyperparameters is an adequate base learner for the comparison.
    Treated as standard; later CARBS tuning changes this.
  • domain assumption The attention weights assigned by the shared MLP, given context, reflect the true usefulness of each model.
    This is the core inductive bias of WSA and is only evaluated indirectly through performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning." pith.science (2026). https://pith.science/paper/O6IOMOKS

@misc{pith2026250707197,
  author       = {Pith},
  title        = {Pith review of: Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O6IOMOKS}},
  note         = {Machine review of arXiv:2507.07197}
}
read the original abstract

The recent focus and release of pre-trained models have been a key components to several advancements in many fields (e.g. Natural Language Processing and Computer Vision), as a matter of fact, pre-trained models learn disparate latent embeddings sharing insightful representations. On the other hand, Reinforcement Learning (RL) focuses on maximizing the cumulative reward obtained via agent's interaction with the environment. RL agents do not have any prior knowledge about the world, and they either learn from scratch an end-to-end mapping between the observation and action spaces or, in more recent works, are paired with monolithic and computationally expensive Foundational Models. How to effectively combine and leverage the hidden information of different pre-trained models simultaneously in RL is still an open and understudied question. In this work, we propose Weight Sharing Attention (WSA), a new architecture to combine embeddings of multiple pre-trained models to shape an enriched state representation, balancing the tradeoff between efficiency and performance. We run an extensive comparison between several combination modes showing that WSA obtains comparable performance on multiple Atari games compared to end-to-end models. Furthermore, we study the generalization capabilities of this approach and analyze how scaling the number of models influences agents' performance during and after training.

Figures

Figures reproduced from arXiv: 2507.07197 by the authors.

Figure 1
Figure 1. A schematic representation of the main pipeline from observations to actions and WSA architecture. The last [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Cumulative reward during training of different agents using WSA and other combination modules on three Atari games. Each subfigure shows the mean score, with shaded areas indicating the standard deviations across multiple agents. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of WSA across different strategies on [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Figures illustrate the adding and removing experiments, red lines mark when a model is added or removed. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Average and standard deviation of the weights assigned by our model to each component, represented as bar [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Weights assigned by WSA to different pre-trained models during rollouts in the Breakout environment. The [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Cumulative reward during training of different agents, comparing WSA and end-to-end model using DQN. Each subfigure shows the mean score, shaded areas indicate the standard deviations across multiple agents. C.2 MANISKILL ADDITIONAL EXPERIMENTS In this experiment we co…
Figure 8
Figure 8. Figure 8: Training performance of WSA agent on Push-Cube-v1 environment, reporting episode reward and suc￾cess during training. Comparing performance of WSA using a single layer (simple) or three layer (deep) Shared Weight Network. (a) deep WSA mean reward on evaluation (b) deep…
Figure 9
Figure 9. Figure 9: Evaluation of deep WSA agent on Push-Cube-V1. Evaluation is performed over 16 episodes. The results, reported in [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Increasing MLP size [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison of CNN with PPO across various strategies in Breakout. Each subfigure displays the average score with the standard deviation shaded. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Performance comparison of FIX with PPO across various strategies in Breakout. Each subfigure displays the average score with the standard deviation shaded. Agent Reward Breakout CNN 65.98 ± 1.62 CNN (P) 118.71 ± 4.30 CNN (M) 62.21 ± 1.98 FIX 87.17 ± 6.87 FIX (P) 106.4…
Figure 13
Figure 13. Figure 13: Training curves of episodes reward for the agents used in the HackAtari evaluations on Pong and Breakout. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Graphical representation of different embedding combination modes that have been evaluated. The blocks [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Graphical representation of different embedding combination modes that have been evaluated. The blocks [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Initial analysis between different combination modules configurations in [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Initial analysis between different combination modules configurations in [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: Initial analysis between different combination modules configurations in [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 29 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Devon Hjelm

    Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc - Alexandre C \^ o t \' e , and R. Devon Hjelm. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp.\ 8766--8779...

    work page 2019

  3. [3]

    Agent57: Outperforming the atari human benchmark

    Adri \` a Puigdom \` e nech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, et al. Agent57: Outperforming the atari human benchmark. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research, pp.\ 507--517. PMLR , 202...

    work page 2020

  4. [4]

    Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling

    Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res., 47: 0 253--279, 2013. doi:10.1613/jair.3912. URL https://doi.org/10.1613/jair.3912

    doi:10.1613/jair.3912 2013

  5. [5]

    Dota 2 with large scale deep reinforcement learning

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, et al. Dota 2 with large scale deep reinforcement learning. CoRR, abs/1912.06680, 2019. URL http://arxiv.org/abs/1912.06680

    arXiv 1912

  6. [6]

    Experiment tracking with weights and biases, 2020

    Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb.com/. Software available from wandb.com

    work page 2020

  7. [7]

    Selective particle attention: Rapidly and flexibly selecting features for deep reinforcement learning

    Sam Blakeman and Denis Mareschal. Selective particle attention: Rapidly and flexibly selecting features for deep reinforcement learning. Neural Networks, 150: 0 408--421, 2022. doi:10.1016/j.neunet.2022.03.015. URL https://doi.org/10.1016/j.neunet.2022.03.015

    work page doi:10.1016/j.neunet.2022.03.015 2022

  8. [8]

    Lee, et al

    Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, et al. Robocat: A self-improving foundation agent for robotic manipulation. CoRR, abs/2306.11706, 2023. doi:10.48550/arXiv.2306.11706. URL https://doi.org/10.48550/arXiv.2306.11706

    arXiv doi:10.48550/arxiv.2306.11706 2023

Show all 63 references
  1. [9]

    Generalized attention-weighted reinforcement learning

    Lennart Bramlage and Aurelio Cortese. Generalized attention-weighted reinforcement learning. Neural Networks, 145: 0 10--21, 2022. doi:10.1016/j.neunet.2021.09.023. URL https://doi.org/10.1016/j.neunet.2021.09.023

    2022 doi

  2. [10]

    Passaro, Vincenzo Lomonaco, Tinne Tuytelaars, and Davide Bacciu

    Andrea Cossu, Antonio Carta, Lucia C. Passaro, Vincenzo Lomonaco, Tinne Tuytelaars, and Davide Bacciu. Continual pre-training mitigates forgetting in language and vision. Neural Networks, 179: 0 106492, 2024. doi:10.1016/J.NEUNET.2024.106492. URL https://doi.org/10.1016/j.neun...

    2024

  3. [11]

    Hackatari: Atari learning environments for robust and continual reinforcement learning

    Quentin Delfosse, Jannis Bl \"u ml, Bjarne Gregori, and Kristian Kersting. Hackatari: Atari learning environments for robust and continual reinforcement learning. arXiv preprint arXiv:2406.03997, 2024

    2024 arXiv

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, et al. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929

    2021 arXiv

  5. [13]

    Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Tom Griffiths, and Alexei A. Efros. Investigating human priors for playing video games. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July 10-15, 2018 ,...

    2018

  6. [14]

    Fetterman, Ellie Kitanidis, Joshua Albrecht, Zachary Polizzi, Bryden Fogelman, Maksis Knutins, et al

    Abraham J. Fetterman, Ellie Kitanidis, Joshua Albrecht, Zachary Polizzi, Bryden Fogelman, Maksis Knutins, et al. Tune as you scale: Hyperparameter optimization for compute efficient training. CoRR, abs/2306.08055, 2023. doi:10.48550/ARXIV.2306.08055. URL https://doi.org/10.485...

    2023 doi arXiv

  7. [15]

    Deep reservoir computing: A critical experimental analysis

    Claudio Gallicchio, Alessio Micheli, and Luca Pedrelli. Deep reservoir computing: A critical experimental analysis. Neurocomputing, 268: 0 87--99, 2017. doi:10.1016/J.NEUCOM.2016.12.089. URL https://doi.org/10.1016/j.neucom.2016.12.089

    2017 doi

  8. [16]

    Multimodal masked autoencoders learn transferable representations

    Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205.14204, 2022

    2022 arXiv

  9. [17]

    Unsupervised video object segmentation for deep reinforcement learning

    Vikash Goel, Jameson Weng, and Pascal Poupart. Unsupervised video object segmentation for deep reinforcement learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Mo...

    2018

  10. [18]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, et al

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, et al. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL http...

    2022

  11. [19]

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23 0 (274): 0 1--18, 202...

    2022

  12. [20]

    Open RL benchmark: Comprehensive tracked experiments for reinforcement learning

    Shengyi Huang, Quentin Gallou \' e dec, Florian Felten, Antonin Raffin, Rousslan Fernand Julien Dossa, et al. Open RL benchmark: Comprehensive tracked experiments for reinforcement learning. CoRR, abs/2402.03046, 2024. doi:10.48550/ARXIV.2402.03046. URL https://doi.org/10.4855...

    2024 doi arXiv

  13. [21]

    Unsupervised learning of object landmarks through conditional image generation

    Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, D...

    2018

  14. [22]

    Continual pre-training of language models

    Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openrevie...

    2023

  15. [23]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, et al. Openvla: An open-source vision-language-action model. In Conference on Robot Learning, 6-9 November 2024, Munich, Germany, volume 270 of Proceedings of Machine Learning Research, pp.\ 2679--2...

    2024

  16. [24]

    Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, et al

    Tejas D. Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, et al. Unsupervised learning of object keypoints for perception and control. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Syst...

    2019

  17. [25]

    Offline q-learning on diverse multi-task data both scales and generalizes

    Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview...

    2023

  18. [26]

    Bootstrapped representations in reinforcement learning

    Charline Le Lan, Stephen Tu, Mark Rowland, Anna Harutyunyan, Rishabh Agarwal, et al. Bootstrapped representations in reinforcement learning. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machin...

    2023

  19. [27]

    Instruction-following agents with jointly pre-trained vision-language models

    Hao Liu, Lisa Lee, Kimin Lee, and Pieter Abbeel. Instruction-following agents with jointly pre-trained vision-language models. CoRR, abs/2210.13431, 2022. doi:10.48550/ARXIV.2210.13431. URL https://doi.org/10.48550/arXiv.2210.13431

    2022 doi arXiv

  20. [28]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, et al. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pp.\ 9992--10002. IEEE , 2021. ...

    2021

  21. [29]

    Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Yecheng Jason Ma, Claire Chen, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing...

    2023

  22. [30]

    Playing atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, et al. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL http://arxiv.org/abs/1312.5602

    2013 arXiv

  23. [31]

    Rusu, Joel Veness, et al

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, et al. Human-level control through deep reinforcement learning. Nat., 518 0 (7540): 0 529--533, 2015. doi:10.1038/nature14236. URL https://doi.org/10.1038/nature14236

    2015 doi

  24. [32]

    Exploiting semantic segmentation to boost reinforcement learning in video game environments

    Javier Montalvo, \' A lvaro Garc \' a - Mart \' n, and Jes \' u s Besc \' o s. Exploiting semantic segmentation to boost reinforcement learning in video game environments. Multim. Tools Appl., 82 0 (7): 0 10961--10979, 2023. doi:10.1007/S11042-022-13695-1. URL https://doi.org/...

    2023 doi

  25. [33]

    R3m: A universal visual representation for robot manipulation, 2022

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation, 2022. URL https://arxiv.org/abs/2203.12601

    2022 arXiv

  26. [34]

    Mixtures of experts unlock parameter scaling for deep RL

    Johan Samir Obando - Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep RL . In Forty-first International Conference o...

    2024

  27. [35]

    Solving rubik's cube with a robot hand

    OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, et al. Solving rubik's cube with a robot hand. CoRR, abs/1910.07113, 2019. URL http://arxiv.org/abs/1910.07113

    1910 arXiv

  28. [36]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy Vo, Marc Szafraniec, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  29. [37]

    Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0

    Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 6892--6903. I...

    2024

  30. [38]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, et al. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020

    2021 arXiv

  31. [39]

    Rl baselines3 zoo

    Antonin Raffin. Rl baselines3 zoo. https://github.com/DLR-RM/rl-baselines3-zoo, 2020

    2020

  32. [40]

    Stable-baselines3: Reliable reinforcement learning implementations

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 0 (268): 0 1--8, 2021. URL http://jmlr.org/papers/v22/20-1364.html

    2021

  33. [41]

    The surprising ineffectiveness of pre-trained visual representations for model-based reinforcement learning

    Moritz Schneider, Robert Krug, Narunas Vaskevicius, Luigi Palmieri, and Joschka Boedecker. The surprising ineffectiveness of pre-trained visual representations for model-based reinforcement learning. In Advances in Neural Information Processing Systems 38: Annual Conference on...

    2024

  34. [42]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347

    2017 arXiv

  35. [43]

    Pretraining representations for data-efficient reinforcement learning

    Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, et al. Pretraining representations for data-efficient reinforcement learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Syst...

    2021

  36. [44]

    Shah and Vikash Kumar

    Rutav M. Shah and Vikash Kumar. RRL: resnet as representation for reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research, pp.\ 9465--947...

    2021

  37. [45]

    Maddison, Arthur Guez, Laurent Sifre, et al

    David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, et al. Mastering the game of go with deep neural networks and tree search. Nat., 529 0 (7587): 0 484--489, 2016. doi:10.1038/nature16961. URL https://doi.org/10.1038/nature16961

    2016 doi

  38. [46]

    Decoupling representation learning from reinforcement learning

    Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine ...

    2021

  39. [47]

    Pufferlib: Making reinforcement learning libraries and environments play nice, 2024

    Joseph Suarez. Pufferlib: Making reinforcement learning libraries and environments play nice, 2024. URL https://arxiv.org/abs/2406.12905

    2024 arXiv

  40. [48]

    Pufferlib 2.0: Reinforcement learning at 1m steps/s

    Joseph Suarez. Pufferlib 2.0: Reinforcement learning at 1m steps/s. In Reinforcement Learning Conference, 2025. URL https://openreview.net/forum?id=qRyteMTgn0

    2025

  41. [49]

    Maniskill3: A versatile, scalable, and flexible machine learning framework for medical imaging

    Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, et al. Maniskill3: A versatile, scalable, and flexible machine learning framework for medical imaging. arXiv preprint arXiv:2410.00425, 2024

    2024 arXiv

  42. [50]

    Scaling instructable agents across many simulated worlds

    SIMA Team, Maria Abi Raad, Arun Ahuja, Catarina Barros, Frederic Besse, et al. Scaling instructable agents across many simulated worlds. CoRR, abs/2404.10179, 2024. doi:10.48550/ARXIV.2404.10179. URL https://doi.org/10.48550/arXiv.2404.10179

    2024 doi arXiv

  43. [51]

    Building machines that learn and think like people

    Josh Tenenbaum. Building machines that learn and think like people. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 2018 , pp.\ 5. International Foundation for Autonomous Agents and Mul...

    2018

  44. [52]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  45. [53]

    Terry, Ariel Kwiatkowski, John U

    Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, et al. Gymnasium, March 2023. URL https://zenodo.org/record/8127025

    2023

  46. [54]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, et al. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pp.\ 5...

    2017

  47. [55]

    Czarnecki, Micha \" e l Mathieu, Andrew Dudzik, et al

    Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Micha \" e l Mathieu, Andrew Dudzik, et al. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nat., 575 0 (7782): 0 350--354, 2019. doi:10.1038/s41586-019-1724-z. URL https://doi.org/10.1038/s4158...

    2019 doi

  48. [56]

    A comprehensive survey of continual learning: Theory, method and application

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Trans. Pattern Anal. Mach. Intell. , 46 0 (8): 0 5362--5383, 2024. doi:10.1109/TPAMI.2024.3367329. URL https://doi.org/10.1109/TPAMI.2024.3367329

    2024

  49. [57]

    SAPIEN : A simulated part-based interactive environment

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, et al. SAPIEN : A simulated part-based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  50. [58]

    Masked visual pre-training for motor control

    Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. CoRR, abs/2203.06173, 2022. doi:10.48550/arXiv.2203.06173. URL https://doi.org/10.48550/arXiv.2203.06173

    2022 doi arXiv

  51. [59]

    Pre-trained image encoder for generalizable visual reinforcement learning

    Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, et al. Pre-trained image encoder for generalizable visual reinforcement learning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022,...

    2022

  52. [60]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 11975--11986, 2023

    2023

  53. [61]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  54. [62]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  55. [63]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.