Pith. sign in

REVIEW 4 major objections 6 minor 34 references

Energy-Based Transfer for Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that energy scores from a teacher's action logits measure how familiar a state is, and gating teacher advice on that score turns RL transfer from a source of bias into consistent positive transfer.

desk verdict A plausible and mostly well-executed transfer RL method whose 'theoretical justification' is circular; worth a referee, but the theory and hyperparameter reporting need real work. read the letter →

arxiv 2506.16590 v1 pith:MOFBSZKH submitted 2025-06-19 cs.LG cs.AI

Zeyun Deng , Jasorsi Ghosh , Fiona Xie , Yuzhe Lu , Katia Sycara , Joseph Campbell This is my paper
classification cs.LGcs.AI
keywords energy-basedtransferlearningreinforcementsampleefficiencyout-of-distributiondetectionteacher-studentstatevisitationdensitycovariateshiftselectiveactionadvising
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a pretrained teacher policy can accelerate a student's learning in a new task without biasing it, if the teacher intervenes only in states it has seen before. The proposed method, energy-based transfer learning (EBTL), uses the free energy of the teacher's action logits as a familiarity score and withholds advice when that score crosses a threshold computed from the teacher's own training states. The paper argues that under on-policy training this score is proportional to the logarithm of the teacher's state-visitation density, and experiments in navigation and cooking tasks with increasing covariate shift show that EBTL beats learning from scratch, always-on advising, fine-tuning, kickstarting, and jump-start transfer. The result matters because it suggests a threshold-based replacement for hand-designed advising schedules.

What carries the argument

The central object is the free energy of a state under the teacher policy, $E(s;\pi_T) = -T \log \sum_i e^{f_i(s)/T}$, where $f_i$ are the teacher's action logits; the energy score is its negative, $\phi(s) = -E(s;\pi_T)$. The free energy links the teacher's raw logits to a Boltzmann-style density $p(s) \propto e^{-E(s)}$, which is the machinery that lets the paper identify high score with in-training-distribution and low score with out-of-distribution. Around this score the algorithm builds a threshold, a decaying guidance probability, and an off-policy importance ratio that keeps mixed teacher/student actions valid for policy-gradient updates.

What would settle it

In a tabular MDP, train a teacher, then estimate empirical log visitation frequency from long rollouts and compute the energy score at each state. If there is a state with high visitation frequency but lower energy score than a rarely visited state, the monotone relation $\log d_\pi(s) \propto \phi(s)$ that the method relies on fails.

Watch

Extended reading notes

Core claim

The central claim is Proposition 4.1: for a teacher trained on-policy, the negative free energy $\phi(s) = -E(s)$ of a state under the teacher's network is proportional to $\log d_\pi(s)$, the log of the state-visitation density induced by the teacher policy. The argument follows from writing the teacher's model as an energy-based density $p(s) = e^{-E(s)}/Z$ and identifying the empirical training distribution with the visitation distribution. The paper then treats this score as an out-of-distribution detector: when the student visits a state with score above a threshold, the teacher suggests an action; otherwise the student acts alone. Energy regularization during teacher training, using random target-environment rollouts as out-of-distribution samples, sharpens the separation, and the paper's claim is that this selective gating converts transfer from a source of harmful bias into consistent positive transfer.

Load-bearing premise

The load-bearing premise is that the free energy computed from the teacher's policy logits is a calibrated proxy for the teacher's state-visitation density, so a single threshold can cleanly separate familiar from unfamiliar states in the target task.

Editorial extensions

If this is right

  • Time-based advising schedules, such as advising always or only in an early prefix, are the wrong gate; the correct gate is state familiarity, and gating by energy scores should preserve positive transfer even when the target task drifts.
  • In the tested navigation and cooking tasks, the threshold $q$ has a usable range around 0.3–0.7: too low admits harmful advice, while too high withholds helpful advice.
  • Energy regularization with random target-environment rollouts is what makes the in-distribution/out-of-distribution separation reliable under harder covariate shift, though even without it EBTL still matches the baselines.
  • A teacher trained on a source task can remain useful for a target task that is substantially different, provided the teacher is allowed to act only on the shared sub-task it already knows.
  • EBTL requires no task-specific reward shaping or policy initialization tricks, only the pretrained teacher's logits and a threshold from its own training states.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension is to run one energy detector per pretrained skill and use the highest familiarity score as a router, letting several teachers share a single student across a lifelong task sequence.
  • The hand-picked threshold suggests an adaptive rule: raise or lower $q$ based on the student's recent return, which would remove the one tunable hyperparameter.
  • Because the energy detector already scores every visited state, the same signal could flag which transitions to store in a replay buffer, making familiarity serve both exploration guidance and memory selection in continual learning.
  • The proportionality claim could be tested more sharply in a tabular MDP with exact visitation counts, which would separate the energy-detector idea from the neural approximation used in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Energy-Based Transfer for Reinforcement Learning (EBTL), a method for selective teacher-student transfer. The teacher policy is trained with an auxiliary energy-based loss; during student training, the teacher's free-energy score over its action logits is compared with a quantile threshold, and teacher actions are used only when the state is deemed in-distribution, subject to a decaying guidance schedule. The authors give Proposition 4.1 as a theoretical justification that the negative free energy is proportional to the log state-visitation density of the teacher, and they report experiments in MiniGrid (Alternating Goal Room, Locked Room) and Overcooked (Simple and Ring layouts, with three transfer configurations), claiming EBTL consistently outperforms all baselines in sample efficiency and final return.

Significance. Selective advising based on out-of-distribution detection is a plausible and practically motivated idea: if the teacher can reliably detect states where its advice is relevant, positive transfer can be obtained under covariate shift without the usual degradation from stale advice. The paper includes a useful experimental design with multiple difficulty levels, a comparison against several transfer baselines, and an ablation showing that the energy regularization helps. The empirical story is weakened, however, by a theoretical proposition that is circular, by thresholds selected using target-task performance, and by the use of target-environment random rollouts in teacher training. If the method is reframed as a heuristic and the evaluation protocol is made more rigorous, the contribution could still be useful, but the current manuscript overstates both the theory and the empirical claim.

major comments (4)
  1. [Section 4.1, Proposition 4.1] The proposition is a definitional identity rather than a derived statement. The proof defines p(s) = exp(-E(s;f))/Z and then asserts that p(s) equals the teacher's state-visitation distribution dπ(s) because 'training data is collected by sampling trajectories under the current policy.' This conflates two different objects: E(s;π_T) is computed from the action logits of the PPO policy network, i.e. an energy over actions conditioned on a state, not an energy-based model of states. Nothing in PPO training constrains the Boltzmann distribution induced by exp(-E(s;f))/Z to match the empirical state marginal of the teacher's rollouts. Moreover, the partition function Z is discarded, so even a monotone relationship would not give calibrated density values. Since the q-quantile threshold in Algorithm 1 and the selection rule in Eq. (2) require a monotone, roughly calibrated relationship between the energy score and visitation frequency, Proposition 4.1 does not provide the claimed theoretical justification. The empirical separation in Figures 3c and 4c may still hold as an inductive observation, but the paper should state this as a heuristic and provide direct evidence that the energy score ranks states by visitation frequency rather than by action confidence.
  2. [Section 5, Figures 3 and 4] The headline claim that EBTL 'consistently outperforms all baselines in both sample efficiency and final policy return across all scenarios' is not supported by the evaluation protocol as presented. The performance curves in Figures 3a and 4a show results for a range of q values, and the EBTL curves in Figures 3b and 4b use q=0.5 and q=0.7 respectively, i.e. thresholds selected per environment using the transfer-performance curves themselves. A method whose single hyperparameter is tuned on the test scenarios cannot be compared on equal footing with baselines whose hyperparameters are fixed without a validation protocol. The same issue applies to the Overcooked experiments, where q=0.5 and q=0.2 are chosen per setup. The paper should either report results with a fixed a-priori threshold, or describe a realistic validation procedure for choosing q (for example, a small labeled validation set of teacher-familiar states), and then compare all methods under that procedure.
  3. [Section 4.2, Eq. (3)] The off-policy correction in Eq. (3) is ambiguous and likely incorrect. When α_t=1, the action was sampled from the teacher policy π_T, so the behavior policy is π_T, not the student's previous policy π_{θ_old}. The importance ratio should be π_{θ_S}(a_t|s_t)/π_{θ_T}(a_t|s_t), which is what the paper writes, but the preceding definition says θ_old denotes the behavior policy used by π_S at data collection time; that definition does not cover the teacher-sampled case. More importantly, for on-policy PPO the entire trajectory distribution changes when actions come from the teacher, and using a per-action importance weight only corrects the action probability, not the state distribution. The paper should clarify what distribution the expectation in Eq. (3) is taken over and justify why this ratio yields a valid policy-gradient estimator for the student under mixed teacher/student rollouts.
  4. [Section 4.2 and Appendix D] The teacher is trained with out-of-distribution samples drawn from 'random policy rollouts in the target environment.' This means target-environment information is used during teacher training, which weakens the claim of transfer under covariate shift and creates an asymmetry with respect to the baselines. If the target environment is available for random rollouts, then the teacher has already seen a distribution of target states, albeit under a random policy. The paper should either remove this dependence on target-environment rollouts or explicitly frame the method as one that is allowed to collect unlabeled random transitions in the target environment before transfer, and discuss the implications for the transfer claim.
minor comments (6)
  1. [Section 6] The Limitations paragraph says the method is designed for 'covariance' shift; this should be 'covariate shift.'
  2. [Section 5] The statement that PPO is implemented with TorchRL cites reference [25], which is the Kickstarting paper; the appropriate citation for PPO is Schulman et al., 2017.
  3. [Figures 3 and 4] The reported results are averaged over 10 seeds, but no error bars or confidence intervals are shown in the return curves; adding shaded standard errors or interquartile ranges would make it easier to assess the 'consistently outperforms' claim.
  4. [Algorithm 1] The decay schedule δ(t)=max(0, δ0−κt) introduces parameters δ0 and κ that are not defined in the main text and are not discussed in the hyperparameter appendix; please specify their values and how they were chosen.
  5. [Section 4.2] The sentence 'the teacher is trained solely on trajectories from its own environment' is contradicted by the subsequent description of the energy regularization loss, which uses a 'fixed set of representative OOD states' from the target environment; please rephrase to avoid the inconsistency.
  6. [Appendix A.1] The teacher checkpoints are selected at different training steps for different environments (800K steps for Locked Room, 200K steps for Alternating Goal Room), but no justification is given for these choices; please explain why these checkpoints are representative.

Circularity Check

1 steps flagged · score 8.0 of 10

Proposition 4.1 is self-definitional: the state density is defined from the energy and then equated with the teacher's visitation distribution, so log dπ ∝ −E is assumed rather than derived.

  1. self definitional [Section 4.1, Proposition 4.1 (with Eq. 1 and the definition φ(s;π_T)=−E(s;π_T))]
    "Proposition 4.1. Under on-policy training, let dπ(s) denote the state-visitation distribution induced by policy π. Then the log of the visitation density is proportional to the score function ϕ(s) = −E(s): log dπ(s) ∝ ϕ(s). Proof. Given an energy-based model f, the density p(s) is defined in terms of its energy E(s): p(s;f) = e^{−E(s;f)/T}/Z ... Ignoring the normalizing constant Z and taking the logarithm ... log p(s) ∝ −E(s). In on-policy RL, training data is collected by sampling trajectories under the current policy π."

    E(s;π_T) is computed by Eq. (1) from the action logits of the teacher's PPO policy network. The proof then defines p(s) as the Boltzmann density of that same energy, exp(−E(s)/T)/Z, and equates this p(s) with dπ(s) solely because on-policy rollouts generate the teacher's states. That equation is the entire content of the proposition, and it is assumed, not derived: nothing in PPO or in the definition of free energy over action logits guarantees that the empirical state marginal of π_T matches exp(−E(s;π_T))/Z, even up to a monotone transformation. Consequently log dπ(s) ∝ φ(s) is true by construction of p(s), not by any property of the teacher policy. Since Algorithm 1's threshold τ = Quantile_q({φ(s) | s∈S_T}) and Eq.

full rationale

The paper's empirical transfer results (Figures 3–7) are external benchmarks and are not circular; the EBTL method is compared against standard baselines with fixed hyperparameters, and those comparisons have independent content. The circularity lies in the stated theoretical justification: Proposition 4.1 defines a state density from the energy and then identifies it with the teacher's visitation distribution, making the claimed proportionality true by definition rather than by derivation. The auxiliary energy regularization and the quantile threshold are fitting procedures, not predictions, so they are not separately circular. The self-citations (Campbell et al., Guo et al., Lu et al.) are used for baselines and context, not as load-bearing support for the energy-visitation identification. Score 8 reflects that the central theoretical claim reduces to a definitional assumption, while the experimental evaluation remains independently meaningful.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central transfer mechanism rests on four assumptions: energy-as-density, threshold separability, usefulness of in-distribution guidance, and availability of target-environment random rollouts for OOD training. The first is asserted rather than derived; Proposition 4.1 is a definitional identity. No code or data is provided, and threshold q is fitted per environment.

free parameters (5)
  • Energy threshold quantile q = q=0.5 Alternating Goal; q=0.7 Locked Room; q=0.5 or 0.2 Overcooked setups
    The q-quantile of teacher energy scores defines the advice gate. Main results use per-environment q values, and the paper shows performance peaks then falls as q grows.
  • Guidance decay schedule delta(t) = Linear decay over 20-80% of training, shown in Figure 3a
    Controls how often teacher advice is issued; Figure 3a shows transfer performance depends on the decay schedule.
  • Energy regularization margins m_in, m_out = m_in=10, m_out=15 for MiniGrid; m_in=12, m_out=14 for Overcooked
    Margins in the L_energy loss. The authors call the choices arbitrary and provide an appendix sensitivity analysis showing weak dependence.
  • Energy regularization weight lambda = Not reported
    Defined in L_total = L_RL + lambda * L_energy, but no value is stated anywhere in the paper.
  • Temperature T in free energy = Not reported
    Free energy in Equation 1 uses T > 0, but the temperature of the energy score is never specified.
assumptions (5)
  • ad hoc to paper A PPO policy network's free energy over action logits defines a state density equal to the teacher's state-visitation distribution.
    Proposition 4.1 proves log dπ ∝ -E by defining p(s)=e^{-E}/Z and identifying p(s) with dπ(s); no training objective enforces this equality.
  • domain assumption A single energy threshold separates in-distribution from out-of-distribution states in the target task.
    EBTL's gating rule in Equation 2 assumes separability; Locked Room results show weak separation, and q must be tuned per environment.
  • domain assumption Teacher actions are useful in all states deemed in-distribution for the target task.
    If a familiar state belongs to a sub-task with different optimal behavior in the target task, guidance would be harmful despite being in-distribution.
  • domain assumption Random rollouts in the target environment can be used to train the teacher's OOD detector.
    Energy regularization samples OOD states from random policy rollouts in the target environment before student training, requiring access to target-environment states.
  • standard math Importance ratio r_t = pi_S(a_t|s_t) / pi_T(a_t|s_t) for teacher-sampled actions gives valid PPO updates.
    Standard importance weighting, but can have high variance when teacher and student actions diverge; used in Equation 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Energy-Based Transfer for Reinforcement Learning." pith.science (2026). https://pith.science/paper/MOFBSZKH

@misc{pith2026250616590,
  author       = {Pith},
  title        = {Pith review of: Energy-Based Transfer for Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MOFBSZKH}},
  note         = {Machine review of arXiv:2506.16590}
}
read the original abstract

Reinforcement learning algorithms often suffer from poor sample efficiency, making them challenging to apply in multi-task or continual learning settings. Efficiency can be improved by transferring knowledge from a previously trained teacher policy to guide exploration in new but related tasks. However, if the new task sufficiently differs from the teacher's training task, the transferred guidance may be sub-optimal and bias exploration toward low-reward behaviors. We propose an energy-based transfer learning method that uses out-of-distribution detection to selectively issue guidance, enabling the teacher to intervene only in states within its training distribution. We theoretically show that energy scores reflect the teacher's state-visitation density and empirically demonstrate improved sample efficiency and performance across both single-task and multi-task settings.

Figures

Figures reproduced from arXiv: 2506.16590 by the authors.

Figure 1
Figure 1. Overview of energy-based transfer learning. As the student interacts with the environment, the teacher: 1) checks if each state is in-distribution or out-of-distribution by comparing the state’s energy score to a pre-defined energy threshold; 2) If the state is greater than the energy threshold, then it is considered in-distribution for the teacher and an expert action is suggested to the student. • We provide theor… view at source ↗
Figure 2
Figure 2. Environments used for empirical experiments. Refer to Section 5 for detailed descriptions. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Locked Room results (10 seeds). See [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: (a) Transfer performance with (solid) vs. without (dashed) energy regularization. (b) Heatmaps showing the average energy quantile of each state under the teacher policy for Alternating Goal Room. Left: source task where the goal is always in Room 1 (upper-left). Right…
Figure 6
Figure 6. Figure 6: Simple Room results. Result are averaged across 3 seeds. (Top) Empirical energy score distributions with respect to the teacher policy. The source task (blue) shows the teacher’s training distribution. The target task (orange + purple), measured during transfer, is bim…
Figure 7
Figure 7. Figure 7: Ring Room results (3 seeds). See [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Actor-Critic architectures used in our experiments. (a) MiniGrid. (b) Overcooked. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 17 canonical work pages

  1. [1]

    Hindsight experience replay.Advances in neural information processing systems, 30, 2017

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay.Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Deep reinforcement learning: A brief survey.IEEE Signal Processing Magazine, 34(6):26–38, 2017

    Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey.IEEE Signal Processing Magazine, 34(6):26–38, 2017

    2017

  3. [3]

    Torchrl: A data-driven decision-making library for pytorch, 2023

    Albert Bou, Matteo Bettini, Sebastian Dittert, Vikash Kumar, Shagun Sodhani, Xiaomeng Yang, Gianni De Fabritiis, and Vincent Moens. Torchrl: A data-driven decision-making library for pytorch, 2023

    2023

  4. [4]

    Policy transfer using reward shaping

    Tim Brys, Anna Harutyunyan, Matthew E Taylor, and Ann Nowé. Policy transfer using reward shaping. InAAMAS, pages 181–188, 2015

    work page 2015

  5. [5]

    Introspective action advising for interpretable transfer learning

    Joseph Campbell, Yue Guo, Fiona Xie, Simon Stepputtis, and Katia Sycara. Introspective action advising for interpretable transfer learning. InConference on Lifelong Learning Agents, pages 1072–1090. PMLR, 2023

    work page 2023

  6. [6]

    On the utility of learning about humans for human-ai coordination.Advances in neural information processing systems, 32, 2019

    Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination.Advances in neural information processing systems, 32, 2019

    2019

  7. [7]

    Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023

    2023

  8. [8]

    Explainable action advising for multi-agent reinforcement learning

    Yue Guo, Joseph Campbell, Simon Stepputtis, Ruiyu Li, Dana Hughes, Fei Fang, and Katia Sycara. Explainable action advising for multi-agent reinforcement learning. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5515–5521. IEEE, 2023

    work page 2023

Show all 34 references
  1. [9]

    Yue Guo, Xijia Zhang, Simon Stepputtis, Joseph Campbell, and Katia P. Sycara. Adaptive action advising with different rewards. In Vincenzo Lomonaco, Stefano Melacci, Tinne Tuytelaars, Sarath Chandar, and Razvan Pascanu, editors,Proceedings of The 3rd Conference on Lifelong Lea...

  2. [10]

    Reinforcement learning with deep energy-based policies

    Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. InInternational conference on machine learning, pages 1352–1361. PMLR, 2017

    2017

  3. [11]

    Expressing arbitrary reward functions as potential-based advice

    Anna Harutyunyan, Sam Devlin, Peter Vrancx, and Ann Nowé. Expressing arbitrary reward functions as potential-based advice. InProceedings of the AAAI conference on artificial intelligence, volume 29, 2015

    2015

  4. [12]

    A baseline for detecting misclassified and out-of-distribution examples in neural networks.arXiv preprint arXiv:1610.02136, 2016

    Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks.arXiv preprint arXiv:1610.02136, 2016

    2016 arXiv

  5. [13]

    Transferring policy of deep reinforcement learning from simulation to reality for robotics.Nature Machine Intelligence, 4(12):1077–1087, 2022

    Hao Ju, Rongshun Juan, Randy Gomez, Keisuke Nakamura, and Guangliang Li. Transferring policy of deep reinforcement learning from simulation to reality for robotics.Nature Machine Intelligence, 4(12):1077–1087, 2022

    2022

  6. [14]

    A tutorial on energy-based learning.Predicting structured data, 1(0), 2006

    Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy-based learning.Predicting structured data, 1(0), 2006

    2006

  7. [15]

    Deep reinforcement learning: An overview.arXiv preprint arXiv:1701.07274, 2017

    Yuxi Li. Deep reinforcement learning: An overview.arXiv preprint arXiv:1701.07274, 2017

    2017 arXiv

  8. [16]

    Energy-based out-of-distribution detection.Advances in neural information processing systems, 33:21464–21475, 2020

    Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection.Advances in neural information processing systems, 33:21464–21475, 2020. 10

    2020

  9. [17]

    Characterizing out-of-distribution error via optimal transport.Advances in Neural Information Processing Systems, 36:17602–17622, 2023

    Yuzhe Lu, Yilong Qin, Runtian Zhai, Andrew Shen, Ketong Chen, Zhenlin Wang, Soheil Kolouri, Simon Stepputtis, Joseph Campbell, and Katia Sycara. Characterizing out-of-distribution error via optimal transport.Advances in Neural Information Processing Systems, 36:17602–17622, 2023

    2023

  10. [18]

    University of Wisconsin-Madison

    Richard Maclin and Jude W Shavlik.Incorporating advice into agents that learn from reinforce- ments. University of Wisconsin-Madison. Computer Sciences Department, 1994

    1994

  11. [19]

    Policy invariance under reward transforma- tions: Theory and application to reward shaping

    Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. InIcml, volume 99, pages 278–287. Citeseer, 1999

    1999

  12. [20]

    Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

    Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 427–436, 2015

    2015

  13. [21]

    Transfer learning and fine- tuned transfer learning methods’ effectiveness analyse in the cnn-based deep learning models

    Celal Öztürk, Murat Ta¸ syürek, and Mehmet U˘gur Türkdamar. Transfer learning and fine- tuned transfer learning methods’ effectiveness analyse in the cnn-based deep learning models. Concurrency and computation: practice and experience, 35(4):e7542, 2023

    2023

  14. [22]

    Au- tomatic curriculum learning for deep rl: A short survey.arXiv preprint arXiv:2003.04664, 2020

    Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Au- tomatic curriculum learning for deep rl: A short survey.arXiv preprint arXiv:2003.04664, 2020

    2003 arXiv

  15. [23]

    Likelihood ratios for out-of-distribution detection.Advances in neural information processing systems, 32, 2019

    Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection.Advances in neural information processing systems, 32, 2019

    2019

  16. [24]

    Policy distillation.arXiv preprint arXiv:1511.06295, 2015

    Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, V olodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation.arXiv preprint arXiv:1511.06295, 2015

    2015 arXiv

  17. [25]

    Kickstarting deep reinforcement learning.arXiv preprint arXiv:1803.03835, 2018

    Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Woj- ciech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning.arXiv preprint arXiv:1803.03835, 2018

    2018 arXiv

  18. [26]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  19. [27]

    Transfer learning for reinforcement learning domains: A survey.Journal of Machine Learning Research, 10(7), 2009

    Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey.Journal of Machine Learning Research, 10(7), 2009

    2009

  20. [28]

    Teaching on a budget: Agents advising agents in reinforcement learning

    Lisa Torrey and Matthew Taylor. Teaching on a budget: Agents advising agents in reinforcement learning. InProceedings of the 2013 international conference on Autonomous agents and multi- agent systems, pages 1053–1060, 2013

    2013

  21. [29]

    Jump-start reinforcement learning

    Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. InInternational Conference on Machine Learning, pages 34556–34583. PMLR, 2023

    2023

  22. [30]

    A survey of transfer learning.Journal of Big data, 3:1–40, 2016

    Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning.Journal of Big data, 3:1–40, 2016

    2016

  23. [31]

    Generalized out-of-distribution detection: A survey.International Journal of Computer Vision, 132(12):5635–5662, 2024

    Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey.International Journal of Computer Vision, 132(12):5635–5662, 2024

    2024

  24. [32]

    How transferable are features in deep neural networks?Advances in neural information processing systems, 27, 2014

    Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks?Advances in neural information processing systems, 27, 2014

    2014

  25. [33]

    Transfer learning in deep reinforce- ment learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (11):13344–13362, 2023

    Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforce- ment learning: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (11):13344–13362, 2023. 11 Appendix A Training Details A.1 GridWorld Reward Structure and Ac...

    2023

  26. [267]

    URL https://proceedings.mlr.press/v274/guo25a

    PMLR, 29 Jul–01 Aug 2025. URL https://proceedings.mlr.press/v274/guo25a. html

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.