Pith. sign in

REVIEW 2 major objections 8 minor 63 references

Hybrid Cross-domain Robust Reinforcement Learning

T0 review · 2 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HYDRO keeps robust RL accurate on 10% of the offline data by borrowing only the right simulator samples.

desk verdict New hybrid source-target robust RL with real empirical gains; the theory is cleaner than the algorithm and the selection mechanism leans heavily on a 100K-transition model, so treat the robustness as promising but not fully nailed down. read the letter →

arxiv 2505.23003 v1 pith:XIGQ6FYY submitted 2025-05-29 cs.LG cs.AI

Linh Le Pham Van , Minh Hoang Nguyen , Hung Le , Hung The Tran , Sunil Gupta This is my paper
classification cs.LGcs.AI MSC 68T0590C40
keywords robustreinforcementlearningdistributionallyMDPofflinecross-domaintransfersimulatordataselectionuncertaintyfilteringprioritysamplingsampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HYDRO, a framework for robust reinforcement learning when offline data from the target environment is scarce but an imperfect online simulator is available. Its central claim is that by measuring the gap between the source simulator and the worst-case model within the target uncertainty set, and then filtering and prioritizing simulator samples by that gap, one can train a robust policy whose performance under target perturbations stays close to an oracle trained on the full offline dataset, even with only 10% of the target data. The paper supports the claim with a performance bound (Theorem 1) and a convergence guarantee for the combined robust and non-robust updates, and with continuous-control experiments where HYDRO outperforms the robust offline baseline, the hybrid cross-domain baseline, and non-robust methods.

What carries the argument

The load-bearing object is the worst-case gap functional, rewritten via the dual formulation of the robust Bellman operator into a computable score that uses the learned target dynamics model, the robust value function, and the dual network. This score defines the priority weight used in sampling, while an ensemble dynamics model provides both the target next-state predictions and the uncertainty quantifier that feeds the filtering threshold. This gap measure is what converts a source simulator from a liability into a data source that tightens the performance bound rather than loosening it.

What would settle it

Compute the exact worst-case gap for a sample of source transitions by solving the inner infimum over the total-variation uncertainty set (for instance by discretizing next-state returns), and compare it to HYDRO's learned approximation; if the learned score's ranking of source transitions disagrees substantially with the exact gap, the priority filter would be selecting for the wrong quantity and the claimed mechanism would fail on a new environment.

Watch

Extended reading notes

Core claim

HYDRO is the first hybrid cross-domain robust RL framework: it combines an online source simulator with a small offline target dataset in a distributionally robust MDP with a total-variation uncertainty set. The core discovery is that the harmful effect of dynamics mismatch can be controlled by a per-transition gap measure that approximates the difference between the source next-state value and the worst-case target value. Transitions whose ensemble-model uncertainty is below a threshold are kept, and kept transitions are sampled with probability proportional to the inverse gap, so that training concentrates on simulator data that is both reliable and close to the worst-case target model. The paper proves a performance bound showing target robust performance is controlled by this gap, and a convergence result for the combined update, and demonstrates empirically that the method retains robustness under perturbations with a tenth of the target data.

Load-bearing premise

The load-bearing premise is that the ensemble dynamics model trained on only 100K target transitions predicts the target next-state distribution accurately enough, even for source states that may lie outside the target data's coverage, so that the gap score and uncertainty filter correctly identify which source samples are safe.

Editorial extensions

If this is right

  • With 10% of the target dataset, HYDRO's robust returns under perturbation stay within a small margin of an oracle trained on the full offline dataset, improving over the robust offline baseline by up to roughly 36% in average return.
  • Naively merging source and target data degrades robustness, so HYDRO's filtering and priority sampling are what turn source data into an asset rather than a source of dynamics mismatch.
  • Removing either the uncertainty filter or the priority sampling visibly reduces robust performance, confirming that both components are load-bearing for the method's effect.
  • The provided performance bound and convergence result guarantee that the learned value function stays close to the optimal robust value function, with error governed by domain gaps and the combination weight between source and target updates.
  • HYDRO maintains consistently strong performance as the target dataset shrinks to 50K and 20K transitions, while the robust baseline degrades substantially.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same worst-case gap score could be reused as a curriculum signal in fully online robust RL, ordering simulator rollouts by distance to the target worst-case model even when no offline dataset is present.
  • The method's reliance on an estimated target transition model suggests a testable boundary: as the offline dataset shrinks, the ensemble model's error on out-of-distribution source states should eventually dominate, and measuring where HYDRO's advantage over naive mixing inverts would quantify that threshold.
  • If the gap functional were replaced by a nominal-model gap as in standard cross-domain transfer, performance degrades; this indicates the robust gap, not the nominal gap, is the right selection statistic, a claim that could be further tested by ablating the dual-network term alone.
  • The framework naturally extends to multiple source simulators with different morphology shifts, each weighted by its own worst-case gap, rather than a single combined source.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. This paper introduces HYDRO, a hybrid cross-domain robust reinforcement learning framework that couples a scarce offline dataset collected from the nominal (target) dynamics with abundant online rollouts from an imperfect source simulator, targeting sample-efficient offline robust RL under a total-variation RMDP uncertainty set. HYDRO estimates a per-transition gap between the source dynamics and the worst-case model in the uncertainty set using a learned target dynamics model, a dual network, and a robust Q-function (Section 4.3, Eqs. (13)-(15)), filters source samples by ensemble-model disagreement, and prioritizes survivors by a score derived from the gap. The authors prove a performance bound (Theorem 1) decomposing the target-source performance difference into an offline-estimation term A and a source-to-worst-case gap term B, and a convergence result for a hybrid fitted-Q iteration (Theorem 2). Experiments on HalfCheetah, Walker2d, and Hopper with 10% D4RL Medium target data, run over 30 seeds with ablations and statistical tests, show consistent improvements over RFQI, PQL, and H2O, approaching the performance of an Oracle trained on the full dataset.

Significance. If the results hold, HYDRO addresses a genuinely novel and practically relevant problem - the first hybrid cross-domain method for offline robust RL with TV uncertainty sets, to my knowledge - and the empirical work is a real strength: 30-seed runs, component-wise ablations, a naive-merge control, a substitution of the gap metric by a standard domain classifier, a data-scarcity sweep (100K/50K/20K), and supplementary code. The performance bound is a genuine lower bound with two interpretable error terms, and the algorithm's selection objective transparently targets term B, so the theory-algorithm link is a design principle rather than a circularity. The paper also states its main limitation explicitly in the Conclusion. The two substantive concerns are the mismatch between the convergence theorem's assumptions and the actual algorithm, and the reliance of the selection mechanism on a target model estimated from only 100K transitions; both are load-bearing for the paper's stated contributions, though neither, in my view, invalidates the empirical claims.

major comments (2)
  1. [Section 4.2, Theorem 2 (Eqs. (10)-(12); appendix Eqs. (32)-(43))] The convergence analysis treats the update Q_{k+1} = kappa*T^{sigma,g}Q_k + (1-kappa)*TQ_k under the assumption mu(s,a)=nu(s,a), but the algorithm in Section 4.3 (Eqs. (17)/(49)) uses priority sampling with an indicator weight w(s,a,s') = 1(psi(s,a)>psi_k%) and an uncertainty filter, none of which appear in the theorem, and no mechanism in Algorithm 1 enforces mu=nu. Source data in Algorithm 1 (lines 4-9) come from on-policy rollouts in the simulator, so the source state-action distribution is determined by the evolving policy and the source dynamics, not by the target dataset's mu. The text's assertion that the assumption 'can hold easily' because a simulator allows flexible starting points is not implemented in the pseudocode. Since Theorem 2 is one of the stated contributions and is repeatedly used to motivate the algorithm, the authors should either enforce the assumption explicitly (e.g., by initializing rollouts from target-dataset states and analyzing the induced distribution) or extend the theorem to the weighted, filtered, priority-sampled update that is actually executed.
  2. [Section 4.3, Eqs. (15) and (18), and Conclusion] The gap score Lambda_hat(s_src,a_src) and the dual-network update both depend on P_hat_o, the target transition model estimated from only 100K target transitions, and the ensemble disagreement filter is computed from the same P_hat_o ensemble. For source state-action pairs far from the target support, P_hat_o's next-state predictions can be inaccurate or overconfident; because P_hat_o-generated tuples are fed back into g_theta and then used to score subsequent source samples, a biased P_hat_o can systematically misrank the very samples the method selects. The authors acknowledge this in the Conclusion ('The limitation of our approach is its dependence on an estimated target transition model'), but the dependence is load-bearing: the paper's advantage over naive source-data merging is attributed precisely to the reliability of this selection, and the scarce-data regime is exactly where the estimated model is least trustworthy. I would like to see this addressed rather than merely stated, for example by an ablation that varies the P_hat_o training budget, or a variant that downweights the gap score by the ensemble's epistemic uncertainty, or a direct analysis of how selection quality degrades with source-state distance to the target support.
minor comments (8)
  1. [Section 5.1, baseline list] 'RQFI [41]' is a typo for 'RFQI'.
  2. [Section 4.2, Section 4.3, and appendix] The main text numbers the performance bound and the convergence result as Theorems 1 and 2, while the appendix numbers them Theorems 3 and 4, and the main text cross-references them as 'Theorem 3' and 'Theorem 4' (e.g., 'Theorem 3 provides the intuition that...'); the numbering should be unified.
  3. [Section 4.2] 'We focus on controlling the third term in Ineq (26)' is inaccurate, since Ineq (26) contains only two subtracted error terms, A and B; the algorithm targets term B, which is the second subtracted term.
  4. [Section 4.3, end] The text refers to 'our proposed method HYDRO in Algorithm 2,' but the main-text listing is Algorithm 1, and the appendix's Algorithm 2 includes additional components (VAE action sampling, policy update, target network updates) not present in the main-text sketch; the authors should clarify which version produced the reported numbers.
  5. [Appendix 5.1 and Section 5.1] The hypotheses of the z-test appear inverted. As stated (H0: R_baseline <= R_HYDRO versus Ha: R_baseline > R_HYDRO), small p-values would support the baseline being better than HYDRO, the opposite of the intended conclusion. Relatedly, the main-text claim that statistical testing 'confirms HYDRO significantly outperforms all baselines' is stronger than Table 4 warrants, since several cells have p>0.05 (e.g., HYDRO-m vs RFQI on Walker2d gravity, p=0.074, and HYDRO-s vs RFQI on Hopper foot joint stiffness, p=0.51).
  6. [Section 5.3 and Appendix 5.5] 'To further understand H2O's performance' should read 'HYDRO's performance,' since the paragraph discusses HYDRO.
  7. [Section 3.1, Eq. (6)] The robust Bellman operator's second term is written as 'inf_{P in U^sigma_rho(P^0_{s,a})} P V(s)'; the expectation symbol E_P is missing.
  8. [Section 4.3, Eqs. (13)-(15)] The jump from Lambda to Lambda_hat substitutes the learned dual network g_theta for the inf over eta and the learned Q_phi for V, but the approximation error of these substitutions is not discussed; since Q_phi and g_theta are themselves being trained, the priority score is non-stationary over the course of learning, and this should at least be noted.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular steps: HYDRO's gap-bound theory, priority sampling, and filtering form a self-contained derivation whose selection objective is an approximation of the theorem's gap term, not an identity.

full rationale

HYDRO's core derivation is not circular. Theorem 3 (Eq. 26) is a genuine lower bound proved in the appendix via the telescoping lemma; its gap term B is defined as |E_{Psrc}[V^{π,σ}_{P̂o}(s′)] − inf_{P∈Uσ(P̂o)} E_P[V^{π,σ}_{P̂o}(s′)]|. Equation (15) is a single-sample, neural approximation of exactly this term, using the learned Q, the dual network gθ, and the estimated dynamics P̂o; it is not equal to the true gap by construction, and the paper explicitly lists dependence on P̂o as a limitation in its Conclusion. The priority score ψ = 1/(1 + Λ̂) and the uncertainty filter select source transitions to tighten the bound, which is a design choice justified by the theorem, not a tautology. The training losses for Q (Eq. 17) and gθ (Eq. 18) follow the standard RFQI robust objective combined with a source Bellman backup; they rely on cited prior work [41] but do not reduce to the result being claimed. The only self-citation, [16], appears in a related-work sentence about existing cross-domain methods and is not load-bearing. The empirical comparison against RFQI, against naive source-data merging, and against ablations without priority sampling or the uncertainty filter provides independent support for the selection mechanism. The skeptic's concern that P̂o, trained on only 100K target transitions, may misrank out-of-distribution source states is a model-estimation limitation acknowledged by the authors, not a circular step: the claimed guarantee is conditional on the learned model being accurate, and the paper does not redefine success in terms of its own fitted values.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of a learned target dynamics model and on several domain assumptions. The method introduces two tunable hyperparameters beyond the RFQI backbone (β and top-k), both adjusted per task. The convergence theorem relies on a distribution-matching assumption that the actual algorithm violates. No new entities are postulated.

free parameters (4)
  • uncertainty set radius σ = 0.3 (HalfCheetah), 0.7 (Walker2d, Hopper)
    Tuned via grid search as in RFQI; controls the size of the TV uncertainty set around the nominal target model.
  • uncertainty filter coefficient β = 2.0 (HalfCheetah, Walker2d), 3.0 (Hopper)
    Controls the uncertainty threshold ε_u = max_D u(s,a) / (β h) with h=5; Appendix Tables 5 and 5.3 show performance varies with β.
  • top-k priority percentile = 75% (HalfCheetah), 50% (Walker2d), 25% (Hopper)
    Determines which source transitions pass the priority threshold ω(s,a,s')=1(ψ>ψ_k%); Appendix 5.2 shows optimal k varies across tasks.
  • source rollout length h = 5
    Number of simulator steps per rollout; set following prior model-based offline RL methods; affects uncertainty threshold via α=βh.
assumptions (4)
  • domain assumption Fail-state assumption: RMDP has a fail-state s_f with r(s_f,a)=0 and P(s_f|s_f,a)=1 for all a and all P in U^σ_TV(P^o)
    Assumption 1, needed for the dual reformulation of the robust Bellman operator and for bounding the dual variable η in [0, 2/(σ(1-γ))].
  • ad hoc to paper Source and target datasets share the same state-action distribution μ=ν
    Assumed in Theorem 2 (Section 4.2) for the convergence proof; the algorithm does not enforce this, and priority filtering actively changes the effective source distribution.
  • domain assumption The learned ensemble dynamics models P̂o_i(s'|s,a) = N(μ_φ(s,a), Σ_φ(s,a)) provide a reliable uncertainty estimate u(s,a)=max_{i,j}||μ_i-μ_j||_2
    The uncertainty filter uses max pairwise discrepancy of ensemble means as a proxy for model error; this is a standard heuristic from model-based offline RL.
  • domain assumption The estimated target model P̂o and dual network gθ yield a good approximation of the worst-case gap Λ in Eq (13)-(15)
    The priority score is computed from a single-sample estimate of Λ using Q_φ, g_θ and P̂o; the paper does not analyze the bias or variance of this estimate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid Cross-domain Robust Reinforcement Learning." pith.science (2026). https://pith.science/paper/XIGQ6FYY

@misc{pith2026250523003,
  author       = {Pith},
  title        = {Pith review of: Hybrid Cross-domain Robust Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XIGQ6FYY}},
  note         = {Machine review of arXiv:2505.23003}
}
read the original abstract

Robust reinforcement learning (RL) aims to learn policies that remain effective despite uncertainties in its environment, which frequently arise in real-world applications due to variations in environment dynamics. The robust RL methods learn a robust policy by maximizing value under the worst-case models within a predefined uncertainty set. Offline robust RL algorithms are particularly promising in scenarios where only a fixed dataset is available and new data cannot be collected. However, these approaches often require extensive offline data, and gathering such datasets for specific tasks in specific environments can be both costly and time-consuming. Using an imperfect simulator offers a faster, cheaper, and safer way to collect data for training, but it can suffer from dynamics mismatch. In this paper, we introduce HYDRO, the first Hybrid Cross-Domain Robust RL framework designed to address these challenges. HYDRO utilizes an online simulator to complement the limited amount of offline datasets in the non-trivial context of robust RL. By measuring and minimizing performance gaps between the simulator and the worst-case models in the uncertainty set, HYDRO employs novel uncertainty filtering and prioritized sampling to select the most relevant and reliable simulator samples. Our extensive experiments demonstrate HYDRO's superior performance over existing methods across various tasks, underscoring its potential to improve sample efficiency in offline robust RL.

Figures

Figures reproduced from arXiv: 2505.23003 by the authors.

Figure 1
Figure 1. Problem of existing offline robust RL model: Robustness performance [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Cumulative rewards of different methods in three Mujoco benchmarks [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. (a) Robust performance comparison between HYDRO, RFQI, and its vari [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Average priority scores of random and priority sampling. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Graphical illustration of all environments. The target domains (top) are [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 5
Figure 5. Figure 5: We use 1 million source data for all methods. Finally, we provide the [PITH_FULL_IMAGE:figures/full_fig_p027_5.png]
Figure 6
Figure 6. Figure 6: Parameter study for top k score data selection. The lines represent the average returns over 30 different seeded runs, and the shaded areas represent the standard deviation. 5.2 Top k Score Data Selection The optimal top k score for data selection is influenced by the …
Figure 7
Figure 7. Figure 7: Average source data remaining ratio after performing uncertainty filtering. [PITH_FULL_IMAGE:figures/full_fig_p032_7.png]
Figure 8
Figure 8. Figure 8: Robust performance comparison between HYDRO and RFQI under var [PITH_FULL_IMAGE:figures/full_fig_p033_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 55 canonical work pages

  1. [1]

    Human-level control through deep reinforcement learning,

    Mnih, V., et al., “Human-level control through deep reinforcement learning,”Na- ture, vol. 518, no. 7540, pp. 529–533, 2015. Hybrid Cross-domain Robust Reinforcement Learning 15

    work page 2015

  2. [2]

    Mastering atari, go, chess and shogi by planning with a learned model,

    Schrittwieser, J., et al., “Mastering atari, go, chess and shogi by planning with a learned model,” Nature, vol. 588, no. 7839, pp. 604–609, 2020

    work page 2020

  3. [3]

    The limits and potentials of deep learning for robotics,

    Sünderhauf, et al., “The limits and potentials of deep learning for robotics,”The International journal of robotics research, vol. 37, no. 4-5, pp. 405–420, 2018

    work page 2018

  4. [4]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

    Levine, S., et al., “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,”arXiv preprint arXiv:2005.01643, 2020

    arXiv 2005

  5. [5]

    Mildly conservative q-learning for offline reinforcement learning,

    Lyu, J., et al., “Mildly conservative q-learning for offline reinforcement learning,” NeurIPS, vol. 35, pp. 1711–1724, 2022

    work page 2022

  6. [6]

    Morel: Model-based offline reinforcement learning,

    Kidambi, R., et al., “Morel: Model-based offline reinforcement learning,”NeurIPS, vol. 33, pp. 21810–21823, 2020

    work page 2020

  7. [7]

    A Conservative Approach for Few-Shot Transfer in Off- Dynamics Reinforcement Learning,

    Daoudi, P., et al., “A Conservative Approach for Few-Shot Transfer in Off- Dynamics Reinforcement Learning,” inIJCAI, 2024

    work page 2024

  8. [8]

    A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agents,

    Niu, H., et al., “A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agents,” inIJCAI, 2024

    work page 2024

Show all 63 references
  1. [9]

    OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning,

    Wu, F., et al., “OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning,” inAAAI, vol. 38, pp. 15897–15905, 2024

    2024

  2. [10]

    When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning,

    Niu, H., Qiu, Y., Li, M., Zhou, G., Hu, J., Zhan, X., et al., “When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning,”Ad- vances in Neural Information Processing Systems, vol. 35, pp. 36599–36612, 2022

    2022

  3. [11]

    H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps,

    Niu, H., Ji, T., Liu, B., Zhao, H., Zhu, X., Zheng, J., Huang, P., Zhou, G., Hu, J., Zhan, X., “H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps,”arXiv preprint arXiv:2309.12716, 2023

    2023 arXiv

  4. [12]

    DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning,

    Liu, J., Hongyin, Z., Wang, D., “DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning,” inInternational Conference on Learning Rep- resentations, 2022

    2022

  5. [13]

    Beyond ood state actions: Supported cross-domain offline reinforcement learning,

    Liu, J., Zhang, Z., Wei, Z., Zhuang, Z., Kang, Y., Gai, S., Wang, D., “Beyond ood state actions: Supported cross-domain offline reinforcement learning,” inProceed- ings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 13945–13953, 2024

    2024

  6. [14]

    Contrastive Rep- resentation for Data Filtering in Cross-Domain Offline Reinforcement Learning,

    Wen, X., Bai, C., Xu, K., Yu, X., Zhang, Y., Li, X., Wang, Z., “Contrastive Rep- resentation for Data Filtering in Cross-Domain Offline Reinforcement Learning,” in Forty-first International Conference on Machine Learning, 2024

    2024

  7. [15]

    Off- Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers,

    Eysenbach, B., Chaudhari, S., Asawa, S., Levine, S., Salakhutdinov, R., “Off- Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers,” in International Conference on Learning Representations, 2021

    2021

  8. [16]

    Policy Learning for Off-Dynamics RL with Deficient Support,

    Le Pham Van, L., The Tran, H., Gupta, S., “Policy Learning for Off-Dynamics RL with Deficient Support,” in Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 1093–1100, 2024

    2024

  9. [17]

    Cross- domain policy adaptation via value-guided data filtering,

    Xu, K., Bai, C., Ma, X., Wang, D., Zhao, B., Wang, Z., Li, X., Li, W., “Cross- domain policy adaptation via value-guided data filtering,”Advances in Neural In- formation Processing Systems, vol. 36, 2024

    2024

  10. [18]

    Cross-Domain Policy Adaptation by Capturing Representation Mismatch,

    Lyu, J., Bai, C., Yang, J. W., Lu, Z., Li, X., “Cross-Domain Policy Adaptation by Capturing Representation Mismatch,” inForty-first International Conference on Machine Learning, 2024

    2024

  11. [19]

    Sim-to-real transfer of robotic control with dynamics randomization,

    Peng, X. B., Andrychowicz, M., Zaremba, W., Abbeel, P., “Sim-to-real transfer of robotic control with dynamics randomization,” in2018 IEEE international confer- ence on robotics and automation (ICRA), pp. 3803–3810, 2018

    2018

  12. [20]

    Domain randomization for transferring deep neural networks from simulation to the real world,

    Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P., “Domain randomization for transferring deep neural networks from simulation to the real world,” in 2017 IEEE/RSJ international conference on intelligent robots and sys- tems (IROS), pp. 23–30, 2017. 16 L. Le...

    2017

  13. [21]

    CAD2RL: Real Single-Image Flight Without a Single Real Image,

    Sadeghi, F., Levine, S., “CAD2RL: Real Single-Image Flight Without a Single Real Image,” Robotics: Science and Systems XIII, 2017

    2017

  14. [22]

    Neural networks for control and system identification,

    Werbos, P. J., “Neural networks for control and system identification,” inProceed- ings of the 28th IEEE Conference on Decision and Control,, pp. 260–265, 1989

    1989

  15. [23]

    Fast model identification via physics engines for data-efficient policy search,

    Zhu, S., Kimmel, A., Bekris, K. E., Boularias, A., “Fast model identification via physics engines for data-efficient policy search,” inProceedings of the 27th Inter- national Joint Conference on Artificial Intelligence, pp. 3249–3256, 2018

    2018

  16. [24]

    Closing the sim-to-real loop: Adapting simulation randomization with real world experience,

    Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N., Fox, D., “Closing the sim-to-real loop: Adapting simulation randomization with real world experience,” in2019 International Conference on Robotics and Automation (ICRA), pp. 8973–8979, 2019

    2019

  17. [25]

    Model-agnostic meta-learning for fast adaptation of deep networks,

    Finn, C., Abbeel, P., Levine, S., “Model-agnostic meta-learning for fast adaptation of deep networks,” inInternational conference on machine learning, pp. 1126–1135, 2017

    2017

  18. [26]

    Learning to Adapt in Dynamic, Real-World Environments through Meta- Reinforcement Learning,

    Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., Finn, C., “Learning to Adapt in Dynamic, Real-World Environments through Meta- Reinforcement Learning,” in International Conference on Learning Representa- tions, 2018

    2018

  19. [27]

    Zero-shot policy transfer with disentangled task representation of meta- reinforcement learning,

    Wu, Z., Xie, Y., Lian, W., Wang, C., Guo, Y., Chen, J., Schaal, S., Tomizuka, M., “Zero-shot policy transfer with disentangled task representation of meta- reinforcement learning,” in2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7169–7175, 2023

    2023

  20. [28]

    Provably good batch off- policy reinforcement learning without great exploration,

    Liu, Y., Swaminathan, A., Agarwal, A., Brunskill, E., “Provably good batch off- policy reinforcement learning without great exploration,”Advances in neural in- formation processing systems, vol. 33, pp. 1264–1274, 2020

    2020

  21. [29]

    Conservative q-learning for offline re- inforcement learning,

    Kumar, A., Zhou, A., Tucker, G., Levine, S., “Conservative q-learning for offline re- inforcement learning,”Advances in Neural Information Processing Systems, vol. 33, pp. 1179–1191, 2020

    2020

  22. [30]

    Rambo-rl: Robust adversarial model-based offline reinforcement learning,

    Rigter, M., Lacerda, B., Hawes, N., “Rambo-rl: Robust adversarial model-based offline reinforcement learning,”Advances in neural information processing systems, vol. 35, pp. 16082–16097, 2022

    2022

  23. [31]

    Robust dynamic programming,

    Iyengar, G. N., “Robust dynamic programming,”Mathematics of Operations Re- search, vol. 30, no. 2, pp. 257–280, 2005

    2005

  24. [32]

    Robust control of Markov decision processes with uncer- tain transition matrices,

    Nilim, A., El Ghaoui, L., “Robust control of Markov decision processes with uncer- tain transition matrices,”Operations Research, vol. 53, no. 5, pp. 780–798, 2005

    2005

  25. [33]

    Distributionally robust Markov decision processes,

    Xu, H., Mannor, S., “Distributionally robust Markov decision processes,”Advances in Neural Information Processing Systems, vol. 23, 2010

    2010

  26. [34]

    Policy gradient method for robust reinforcement learning,

    Wang, Y., Zou, S., “Policy gradient method for robust reinforcement learning,” in International conference on machine learning, pp. 23484–23526, 2022

    2022

  27. [35]

    On the convergence of policy gradient in robust mdps,

    Wang, Q., Ho, C. P., Petrik, M., “On the convergence of policy gradient in robust mdps,” arXiv preprint arXiv:2212.10439, vol. 5, 2022

    2022 arXiv

  28. [36]

    Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics,

    Yang, W., Zhang, L., Zhang, Z., “Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics,”The Annals of Statistics, vol. 50, no. 6, pp. 3223–3248, 2022

    2022

  29. [37]

    Improved sample complexity bounds for distri- butionally robust reinforcement learning,

    Xu, Z., Panaganti, K., Kalathil, D., “Improved sample complexity bounds for distri- butionally robust reinforcement learning,” inInternational Conference on Artificial Intelligence and Statistics, pp. 9728–9754, 2023

    2023

  30. [38]

    Online robust reinforcement learning with model uncertainty,

    Wang, Y., Zou, S., “Online robust reinforcement learning with model uncertainty,” Advances in Neural Information Processing Systems, vol. 34, pp. 7193–7206, 2021

    2021

  31. [39]

    Online policy optimization for robust mdp,

    Dong, J., Li, J., Wang, B., Zhang, J., “Online policy optimization for robust mdp,” arXiv preprint arXiv:2209.13841, 2022. Hybrid Cross-domain Robust Reinforcement Learning 17

    2022 arXiv

  32. [40]

    Finite-sample re- gret bound for distributionally robust offline tabular reinforcement learning,

    Zhou, Z., Zhou, Z., Bai, Q., Qiu, L., Blanchet, J., Glynn, P., “Finite-sample re- gret bound for distributionally robust offline tabular reinforcement learning,” in International Conference on Artificial Intelligence and Statistics, pp. 3331–3339, 2021

    2021

  33. [41]

    Robust reinforcement learning using offline data,

    Panaganti, K., Xu, Z., Kalathil, D., Ghavamzadeh, M., “Robust reinforcement learning using offline data,” Advances in neural information processing systems, vol. 35, pp. 32211–32224, 2022

    2022

  34. [42]

    Learning models with uniform performance via distri- butionally robust optimization,

    Duchi, J. C., Namkoong, H., “Learning models with uniform performance via distri- butionally robust optimization,”The Annals of Statistics, vol. 49, no. 3, pp. 1378– 1406, 2021

    2021

  35. [43]

    Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity,

    Shi, L., Chi, Y., “Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity,”arXiv preprint arXiv:2208.05767, 2022

    2022 arXiv

  36. [44]

    Distributionally robust offline reinforcement learning with linear function approx- imation,

    Ma, X., Liang, Z., Blanchet, J., Liu, M., Xia, L., Zhang, J., Zhao, Q., Zhou, Z., “Distributionally robust offline reinforcement learning with linear function approx- imation,” arXiv preprint arXiv:2209.06620, 2022

    2022 arXiv

  37. [45]

    Double pessimism is provably effi- cient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage,

    Blanchet, J., Lu, M., Zhang, T., Zhong, H., “Double pessimism is provably effi- cient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  38. [46]

    Using simulation and domain adaptation to improve efficiency of deep robotic grasping,

    Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., Downs, L., Ibarz, J., Pastor, P., Konolige, K., et al., “Using simulation and domain adaptation to improve efficiency of deep robotic grasping,” in2018 IEEE interna- tional conference on robotics an...

    2018

  39. [47]

    Darla: Improving zero-shot transfer in reinforcement learning,

    Higgins, I., Pal, A., Rusu, A., Matthey, L., Burgess, C., Pritzel, A., Botvinick, M., Blundell, C., Lerchner, A., “Darla: Improving zero-shot transfer in reinforcement learning,” inInternational Conference on Machine Learning, pp. 1480–1490, 2017

    2017

  40. [48]

    D4rl: Datasets for deep data-driven reinforcement learning,

    Fu, J., Kumar, A., Nachum, O., Tucker, G., Levine, S., “D4rl: Datasets for deep data-driven reinforcement learning,”arXiv preprint arXiv:2004.07219, 2020

    2004 arXiv

  41. [49]

    Mopo: Model-based offline policy optimization,

    Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., Ma, T., “Mopo: Model-based offline policy optimization,”Advances in Neural Information Processing Systems, vol. 33, pp. 14129–14142, 2020

    2020

  42. [50]

    Robust adversarial reinforce- ment learning,

    Pinto, L., Davidson, J., Sukthankar, R., Gupta, A., “Robust adversarial reinforce- ment learning,” in International conference on machine learning, pp. 2817–2826, 2017

    2017

  43. [51]

    Exponential bellman equation and im- proved regret bounds for risk-sensitive reinforcement learning,

    Fei, Y., Yang, Z., Chen, Y., Wang, Z., “Exponential bellman equation and im- proved regret bounds for risk-sensitive reinforcement learning,”Advances in neural information processing systems, vol. 34, pp. 20436–20446, 2021

    2021

  44. [52]

    One risk to rule them all: A risk-sensitive perspective on model-based offline reinforcement learning,

    Rigter, M., Lacerda, B., Hawes, N., “One risk to rule them all: A risk-sensitive perspective on model-based offline reinforcement learning,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  45. [53]

    Corruption-robust offline reinforcement learning,

    Zhang, X., Chen, Y., Zhu, X., Sun, W., “Corruption-robust offline reinforcement learning,” in International Conference on Artificial Intelligence and Statistics, pp. 5757–5773, 2022

    2022

  46. [54]

    Corruption-robust offline reinforcement learning with general function approximation,

    Ye, C., Yang, R., Gu, Q., Zhang, T., “Corruption-robust offline reinforcement learning with general function approximation,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  47. [55]

    Distributionally robust stochastic programming,

    Shapiro, A., “Distributionally robust stochastic programming,”SIAM Journal on Optimization, vol. 27, no. 4, pp. 2258–2275, 2017

    2017

  48. [56]

    Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees,

    Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., Ma, T., “Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees,” in International Conference on Learning Representations, 2018. 18 L. Le Pham Van et al

    2018

  49. [57]

    Deep reinforcement learning in a handful of trials using probabilistic dynamics models,

    Chua, K., Calandra, R., McAllister, R., Levine, S., “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,”Advances in neural information processing systems, vol. 31, 2018

    2018

  50. [58]

    Model-Bellman inconsistency for model-based offline reinforcement learning,

    Sun, Y., Zhang, J., Jia, C., Lin, H., Ye, J., Yu, Y., “Model-Bellman inconsistency for model-based offline reinforcement learning,” in International Conference on Machine Learning, pp. 33177–33194, 2023

    2023

  51. [59]

    Uncertainty-driven trajectory truncation for data augmentation in offline reinforcement learning,

    Zhang, J., Lyu, J., Ma, X., Yan, J., Yang, J., Wan, L., Li, X., “Uncertainty-driven trajectory truncation for data augmentation in offline reinforcement learning,” in ECAI 2023, pp. 3018–3025, 2023

    2023

  52. [60]

    Prioritized experience replay,

    Schaul, T., Quan, J., Antonoglou, I., Silver, D., “Prioritized experience replay,” arXiv preprint arXiv:1511.05952, 2015

    2015 arXiv

  53. [61]

    labml.ai Annotated Paper Implementations,

    Jayasiri, V., Wijerathne, N., “labml.ai Annotated Paper Implementations,” 2020

    2020

  54. [62]

    Revolver: Continuous evolutionary models for robot-to-robot policy transfer,

    Liu, X., Pathak, D., Kitani, K. M., “Revolver: Continuous evolutionary models for robot-to-robot policy transfer,”arXiv preprint arXiv:2202.05244, 2022. Hybrid Cross-domain Robust Reinforcement Learning 19 APPENDIX 1 Extended Related Works 1.1 Cross-domain Reinforcement Learni...

    2022 arXiv

  55. [63]

    -m": multi comp, “-s

    Our method only introduces two hyperparameters, which are the topk score selection and uncertainty filterβ. Both the value function and policy are updated at each training step. H2O We utilize the author’s official implementation from their GitHub repos- itory (https://github....

    2000

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.