Pith. sign in

REVIEW 2 major objections 8 minor 44 references

Federated Reinforcement Learning in Heterogeneous Environments

T0 review · 2 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Federated Q-learning hits worst-case optimum with a proven rate

desk verdict The robust-FRL idea is plausible, but Lemma 2's reversed learning-rate inequality breaks Theorem 1's convergence proof, leaving the paper's main claim unsupported. read the letter →

arxiv 2507.14487 v1 pith:Y7522FQR submitted 2025-07-19 cs.LG

Ukjo Hwang , Songnam Hong This is my paper
classification cs.LG
keywords federatedreinforcementlearningenvironmentheterogeneityrobustQ-learningworst-caseobjectiveexpectileregressiondeepcoveringset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated RL with heterogeneous local environments often averages updates, which works well only when local dynamics are similar. This paper claims that FedRQ, a tabular algorithm with a worst-case-aware local update, makes the global Q function converge, at an explicit rate, to the optimal policy of a robust objective defined over all local environments and their plausible perturbations. The robust objective is a worst-case value over a covering set of transition kernels, so the learned policy performs consistently even in environments not seen during training. The paper also extends the same principle to continuous state spaces using expectile loss, yielding deep variants that outperform averaging-only baselines in the reported experiments.

What carries the argument

The load-bearing object is the pair of robust Bellman operators: the local operator $T_k$ used in each agent's update and the global operator $\bar T$ whose fixed point is the target $Q^\star_R$. The identity $\bar T = \frac{1}{K}\sum_{k=1}^K T_k$, valid under Assumption 1, is what lets periodic averaging of local Q functions simulate one step of a global contraction mapping. The covering set $\mathcal{P}_\omega$, defined as mixtures of the average transition kernel $\bar P$ with arbitrary kernels supported on the shared neighbor set $\mathcal{N}^s$, specifies the ambiguity class over which worst-case performance is measured. In continuous spaces, the minimization inside $T_k$ is approximated by an expectile network trained with asymmetric squared loss, which learns the lower-tail minimum of the max-over-actions value across next states. The drift of local Q functions between synchronization rounds is bounded in Lemma 2, and that drift bound feeds directly into the final convergence rate.

What would settle it

Compute the gap $\|\bar Q_t - Q^\star_R\|_\infty$ for two tabular MDPs with identical state and action spaces but different supports, for instance where from state A agent 1 can transition only back to A while agent 2 can also transition to B. Under the paper's assumptions the gap should decay as $O(1/t)$; observing that the gap stagnates, or directly verifying that $\bar T \neq \frac{1}{K}\sum_k T_k$ for such an instance, would falsify the theorem.

Watch

Extended reading notes

Core claim

Under Assumption 1, which requires that every local environment share the same set of reachable next states from each state, FedRQ's global Q function, defined as $\bar Q_t = \frac{1}{K}\sum_{k=1}^K Q^k_t$, converges to $Q^\star_R$, the fixed point of the global robust Bellman operator $\bar T$, with the explicit bound $\|\bar Q_t - Q^\star_R\|_\infty \le \frac{16\gamma(E-1)}{(1-\gamma)^3(t+E)}$. The robust operator is built from a covering set $\mathcal{P}_\omega$ of plausible transition kernels, and the key identity $\bar T = \frac{1}{K}\sum_{k=1}^K T_k$ holds exactly under Assumption 1. Each agent runs a Q-learning update augmented with the extra term $\gamma\omega \min_{s'\in\mathcal{N}^s}\max_{a'} Q^k_t(s',a')$, which hedges against the worst next state in the local support; averaging the local Q functions at the server then implements one step of the global robust contraction. QAvg is recovered as the special case $\omega=0$. For continuous state spaces, the hard minimization over next states is replaced by an expectile-based degree network trained on local replay buffers, giving the deep algorithms FedRDQN and FedRDDPG.

Load-bearing premise

The theorem assumes that from every state, all K local environments can reach exactly the same set of next states; if any environment has a reachable state another lacks, the arithmetic-mean identity that makes FedRQ's global update a contraction no longer holds.

Editorial extensions

If this is right

  • For any robustness level $\omega$ satisfying the covering condition (20), the policy extracted from $\bar Q_t$ is asymptotically optimal for the worst case over $\mathcal{P}_\omega$, which includes all $K$ local environments and their allowed perturbations.
  • The explicit rate shows how the synchronization period $E$ trades off against convergence: the drift term grows with $E-1$, while the factor $(1-\gamma)^{-3}$ quantifies the sensitivity to discounting, so the result positions FedRQ against standard robust Q-learning guarantees.
  • With $\omega=0$, FedRQ reduces to QAvg, so the convergence theorem encompasses the known average-case federated algorithm as a special case rather than contradicting it.
  • Because only local Q functions, or their network parameters, are shared in the global update, the worst-case optimality guarantee is achieved under the federated constraint that raw trajectories remain private.
  • The deep extensions FedRDQN and FedRDDPG apply the same worst-case hedging principle to continuous state spaces, with the expectile network serving as the sample-based surrogate for the hard minimum over next states.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Assumption 1 fails, because some local environments can reach a state that others cannot, the identity $\bar T = \frac{1}{K}\sum_k T_k$ breaks and the averaging mechanism no longer implements a single contraction. A tabular experiment that varies the supports of the local transition kernels would directly reveal how the convergence gap degrades and would mark the boundary of the theorem.
  • The proof chooses $\omega$ from the true heterogeneity scale $\kappa(s,a)$, which is not known before training in practice. Adapting $\omega$ online from observed transition statistics is a natural extension, but its guarantee is not covered by the paper's analysis.
  • In continuous state spaces, replacing $\min_{s'\in\mathcal{N}^s}\max_a Q(s',a)$ with an expectile network introduces approximation error that Theorem 1 does not quantify. Comparing FedRDQN against an exact tabular FedRQ on a discretized version of the same environment would separate the algorithm's robust mechanism from the approximator's bias.
  • The worst-case objective is one point on a spectrum; a mixture of average and worst-case performance, or another risk-sensitive criterion, would sit naturally on the same Bellman-operator machinery and could be tested without changing the federated update structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. The paper studies federated reinforcement learning in which local environments have different transition dynamics. It defines a worst-case global objective over a covering set of transition probabilities, proposes a tabular algorithm FedRQ based on robust local Q-learning updates with periodic global averaging, and claims a finite-time convergence bound for the averaged Q-function to the optimal robust Q-function. The paper then extends the robust update to function approximation via expectile networks, yielding FedRDQN and FedRDDPG for discrete and continuous action spaces, and reports experiments against QAvg-based baselines on several Gym environments.

Significance. If Theorem 1 were fully established, the paper would provide an interesting first finite-time guarantee for worst-case federated RL over heterogeneous and perturbed environments, complementing the known suboptimality of QAvg. The structural observation that averaging the local robust Bellman operators yields the global robust operator under a support-consistency assumption is elegant and is a genuine strength. The empirical study is also reasonably broad, reporting both average and worst-case metrics across eight environments. However, the advertised theoretical guarantee is currently not proven as written: the proof of Lemma 2 contains a monotonicity error, and the tabular update analyzed in Theorem 1 is a model-based synchronous operator rather than a sample-based reinforcement learning algorithm. These issues are substantive but appear repairable.

major comments (2)
  1. [§3.3, Lemma 2] Step (c) of the proof of Lemma 2 asserts λ_{t'} ≤ λ_t for t' ∈ [t0, t−1]. Since λ_t = 2/((1−γ)(t+E)) is strictly decreasing in t, the inequality is reversed: for t' < t one has λ_{t'} ≥ λ_t. Consequently the bound on ∑_{t'=t0}^{t−1} λ_{t'} does not follow, and inequality (44) is not established as written. Because Lemma 2 supplies the drift term in the recursive inequality (36), Theorem 1's bound (30) is unsupported as written. The claim appears salvageable: a correct proof can use λ_{t'} ≤ λ_{t0}, then bound λ_{t0}/λ_t = (t+E)/(t0+E) ≤ (t+E)/(t+1) ≤ 2 for t ≥ E−1, with the finitely many small t cases handled separately. The proof still needs to be rewritten.
  2. [§3.2, Eqs. (25) and (34)] The robust local update in (25) and its operator form (34) use the full transition kernel P_k and the robust operator T_k; no transition sampling is introduced in the tabular section. Thus Theorem 1 proves convergence of a synchronous, model-based value-iteration scheme, not of a sample-based reinforcement learning algorithm. The deep extensions in Section 4 are heuristic and are not covered by the theorem. The paper should either present a stochastic tabular update and analyze it under standard stochastic-approximation conditions, or explicitly state that Theorem 1 is an analysis of an idealized federated robust value iteration, with the sample-based version deferred to future work.
minor comments (8)
  1. [§3.3, Lemma 2] In the proof, the bound 'Q^k_t(s,a) ≤ 1/(1−r)' should read 'Q^k_t(s,a) ≤ 1/(1−γ)', since rewards are bounded by 1 and the discount factor is γ.
  2. [§3.1, Eq. (17)] Inequality (17) is ambiguous as printed; it should be written as (ω P̄(s'|s,a) + P_k(s'|s,a) − P̄(s'|s,a)) / P̄(s'|s,a) ≥ 0, or equivalently with explicit parentheses.
  3. [§3.1, Eq. (18)] The quantity κ(s,a) in (18) is never formally defined; a precise definition is needed.
  4. [§3.3, Assumption 1] Assumption 1 is restrictive because it excludes environments with different sets of reachable next states from the same state. The paper should discuss whether the benchmark environments satisfy this assumption and what happens to identity (33) when only a subset of local supports agree.
  5. [§5, Table 2] The text states that FedRDDPG is compared against both DDPGAvg(1) and DDPGAvg(2), but Table 2 reports only DDPGAvg(2); the DDPGAvg(1) results should be included or the text revised.
  6. [Algorithms 1 and 2] In line 8 of both algorithms, the replay-buffer append operation is missing a closing parenthesis: it should read D_k ← D_k ∪ {(s, a, r, s')}.
  7. [§5.1] The moving-average filter used to smooth the learning curves is not specified; the window length should be reported for reproducibility.
  8. [Abstract and Theorem 1] The abstract describes 'asymptotic convergence', while Theorem 1 provides a non-asymptotic finite-time bound; these terminologies should be aligned.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FedRQ convergence proof is self-contained modulo a published external contraction result, and no fitted quantity is relabeled as a prediction.

full rationale

No circular step is present. The paper's main theoretical claim is that the FedRQ global Q-function converges to the fixed point of the averaged robust Bellman operator. That proof derives a recursion (36) from the explicit local and global update rules (34)-(35), aided by identity (33), which follows algebraically from Assumption 1. The only externally imported ingredient is the gamma-contraction of the robust Bellman operator, attributed to the authors' own prior work [39]. That citation is a published, parameter-free theorem with stated assumptions that do not include the federated convergence result, so under the stated rules it counts as independent evidence rather than a self-referential reduction. The robustness level omega is a hyperparameter required to satisfy condition (20), not a value fitted to the convergence target, and no empirical result is relabeled as a theoretical prediction. The experiments compare against external baselines (DQNAvg, DDPGAvg), so the empirical claims are not forced by construction. The proof display in Lemma 2 contains a monotonicity slip in the learning-rate comparison, but that is a correctness gap rather than a definitional equivalence between the theorem and its inputs, and it does not constitute circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central theoretical claim rests on the support-consistency assumption, the rectangularity of the covering set, the contraction property taken from the authors' own prior paper, and the empirically selected robustness level omega. The latter is a free parameter not anchored to a reported selection rule, which weakens the practical force of the robust guarantee.

free parameters (2)
  • Robustness level omega = not reported in text (shown only in figure legends)
    Controls the size of the covering set in Eqs. (13)-(14); the theory requires omega >= max kappa(s,a), but the paper gives no principled selection rule and appears to tune it per environment.
  • Expectile level tau = 0.01
    Hand-picked to emphasize the lower tail when approximating min_{s'} max_{a'} Q(s',a') with a regression network in continuous state spaces.
assumptions (5)
  • domain assumption Rewards are bounded in [0,1] and discount factor gamma in (0,1) (Section 2.1)
    Used to bound Q functions by 1/(1-gamma) throughout the convergence proof.
  • domain assumption Assumption 1: the sets of states reachable with non-zero probability from each s are the same across all K local environments (Section 3.3)
    Needed for identity (33), which identifies the global robust Bellman operator with the average of the local operators; if the supports differ, FedRQ's update does not target the stated global objective.
  • ad hoc to paper The robustness level omega is chosen so that all local transition probabilities lie in the covering set P_omega (conditions (16)-(20))
    The algorithm cannot verify this without knowing the true P_k's, so the guarantee applies only when the designer has prior knowledge of the heterogeneity.
  • standard math The covering set P_omega is rectangular, i.e., a product over state-action pairs (Eq. (14))
    Implicit but standard in robust MDPs; it lets the infimum over P be computed pointwise, which the algorithm relies on.
  • standard math The robust Bellman operator in (32) is a gamma-contraction in sup norm
    Quoted from the authors' prior work [39]; the present paper does not reprove this result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Federated Reinforcement Learning in Heterogeneous Environments." pith.science (2026). https://pith.science/paper/Y7522FQR

@misc{pith2026250714487,
  author       = {Pith},
  title        = {Pith review of: Federated Reinforcement Learning in Heterogeneous Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y7522FQR}},
  note         = {Machine review of arXiv:2507.14487}
}
read the original abstract

We investigate a Federated Reinforcement Learning with Environment Heterogeneity (FRL-EH) framework, where local environments exhibit statistical heterogeneity. Within this framework, agents collaboratively learn a global policy by aggregating their collective experiences while preserving the privacy of their local trajectories. To better reflect real-world scenarios, we introduce a robust FRL-EH framework by presenting a novel global objective function. This function is specifically designed to optimize a global policy that ensures robust performance across heterogeneous local environments and their plausible perturbations. We propose a tabular FRL algorithm named FedRQ and theoretically prove its asymptotic convergence to an optimal policy for the global objective function. Furthermore, we extend FedRQ to environments with continuous state space through the use of expectile loss, addressing the key challenge of minimizing a value function over a continuous subset of the state space. This advancement facilitates the seamless integration of the principles of FedRQ with various Deep Neural Network (DNN)-based RL algorithms. Extensive empirical evaluations validate the effectiveness and robustness of our FRL algorithms across diverse heterogeneous environments, consistently achieving superior performance over the existing state-of-the-art FRL algorithms.

Figures

Figures reproduced from arXiv: 2507.14487 by the authors.

Figure 1
Figure 1. Learning curves for DQNAvg and FedRDQN evaluated on the nominal environment. The perturbed model parameter for each local environment is indicated adjacent to the environment name. The value in parentheses following each algorithm name denotes the corresponding robustness level. (a) Ant-v4: Back Left Leg (b) Hopper-v4: Thigh (c) Walker2d-v4: Left Leg (d) HalfCheetah-v4: Front Foot (e) Ant-v4: Front Right Leg (f) Hop… view at source ↗
Figure 2
Figure 2. Learning curves for DDPGAvg and FedRDDPG evaluated on the nominal environment. The perturbed model parameter for each local environment is indicated adjacent to the environment name. The value in parentheses following FedRDDPG denotes the corresponding robustness level. After global aggregation, the server broadcasts the global parameters to all K agents, as defined by: (θk, θ′ k , ψk, ϕk, ϕ′ k ) ← ( ¯θ, ¯θ ′ , ψ, ¯… view at source ↗
Figure 3
Figure 3. Evaluation curves for DQNAvg and FedRDQN on perturbed environments. The x-axis represents the relative deviation of the model parameter from that of the nominal environment, with zero indicating the nominal configuration. The specific model parameter perturbed in each environment is annotated next to the environment name, and the value in parentheses following FedRDQN denotes its associated robustness level. (a) Ant… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Evaluation curves for DDPGAvg and FedRDDPG on perturbed environments. The x-axis represents the relative deviation of the model parameter from that of the nominal environment, with zero indicating the nominal configuration. The specific model parameter perturbed in eac…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 35 canonical work pages

  1. [1]

    Reinforcement learning: A tutorial survey and recent advances,

    A. Gosavi, “Reinforcement learning: A tutorial survey and recent advances,” INFORMS Journal on Computing, vol. 21, no. 2, pp. 178– 192, 2009

    work page 2009

  2. [2]

    Deep reinforcement learning: A brief survey,

    K. Arulkumaran, M. P . Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 26–38, 2017

    work page 2017

  3. [3]

    Applications of deep reinforcement learning in communications and networking: A survey,

    N. C. Luong, D. T. Hoang, S. Gong, D. Niyato, P . Wang, Y.-C. Liang, and D. I. Kim, “Applications of deep reinforcement learning in communications and networking: A survey,” IEEE communications surveys & tutorials, vol. 21, no. 4, pp. 3133–3174, 2019

    work page 2019

  4. [4]

    Federated reinforcement learning for fast personalization,

    C. Nadiger, A. Kumar, and S. Abdelhak, “Federated reinforcement learning for fast personalization,” in 2019 IEEE Second Interna- tional Conference on Artificial Intelligence and Knowledge Engineering (AIKE). IEEE, 2019, pp. 123–127

    work page 2019

  5. [5]

    Fed- erated reinforcement learning: Linear speedup under markovian sampling,

    S. Khodadadian, P . Sharma, G. Joshi, and S. T. Maguluri, “Fed- erated reinforcement learning: Linear speedup under markovian sampling,” in International Conference on Machine Learning. PMLR, 2022, pp. 10 997–11 057

    work page 2022

  6. [6]

    Federated reinforcement learning: Techniques, applications, and open challenges,

    J. Qi, Q. Zhou, L. Lei, and K. Zheng, “Federated reinforcement learning: Techniques, applications, and open challenges,” arXiv preprint arXiv:2108.11887, 2021

    arXiv 2021

  7. [7]

    Fedkl: Tackling data heterogeneity in fed- erated reinforcement learning by penalizing kl divergence,

    Z. Xie and S. Song, “Fedkl: Tackling data heterogeneity in fed- erated reinforcement learning by penalizing kl divergence,” IEEE Journal on Selected Areas in Communications, vol. 41, no. 4, pp. 1227– 1242, 2023

    work page 2023

  8. [8]

    Federated reinforcement learning acceleration method for precise control of multiple devices,

    H.-K. Lim, J.-B. Kim, I. Ullah, J.-S. Heo, and Y.-H. Han, “Federated reinforcement learning acceleration method for precise control of multiple devices,” IEEE Access, vol. 9, pp. 76 296–76 306, 2021

    work page 2021

Show all 44 references
  1. [9]

    Federated reinforcement learning with environment heterogeneity,

    H. Jin, Y. Peng, W. Yang, S. Wang, and Z. Zhang, “Federated reinforcement learning with environment heterogeneity,” in Inter- national Conference on Artificial Intelligence and Statistics . PMLR, 2022, pp. 18–37

    2022

  2. [10]

    Communication-efficient randomized algo- rithm for multi-kernel online federated learning,

    S. Hong and J. Chae, “Communication-efficient randomized algo- rithm for multi-kernel online federated learning,” IEEE Transac- tions on Pattern Analysis and Machine Intelligence , vol. 44, no. 12, pp. 9872–9886, 2021

    2021

  3. [11]

    Communication-efficient online federated learning framework for nonlinear regression,

    V . C. Gogineni, S. Werner, Y.-F. Huang, and A. Kuh, “Communication-efficient online federated learning framework for nonlinear regression,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 5228–5232

    2022

  4. [12]

    Tighter regret analysis and optimization of online federated learning,

    D. Kwon, J. Park, and S. Hong, “Tighter regret analysis and optimization of online federated learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 12, pp. 15 772– 15 789, 2023

    2023

  5. [13]

    Online multikernel learning method via online biconvex optimization,

    S. Hong, “Online multikernel learning method via online biconvex optimization,” IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 11, pp. 16 630–16 643, 2023

    2023

  6. [14]

    A communication- efficient adaptive algorithm for federated learning under cumula- tive regret,

    S. Salgia, T. Gabay, Q. Zhao, and K. Cohen, “A communication- efficient adaptive algorithm for federated learning under cumula- tive regret,” IEEE Transactions on Signal Processing, 2024

    2024

  7. [15]

    Learning to predict by the methods of temporal differences,

    R. S. Sutton, “Learning to predict by the methods of temporal differences,” Machine learning, vol. 3, pp. 9–44, 1988

    1988

  8. [16]

    Q-learning,

    C. J. Watkins and P . Dayan, “Q-learning,” Machine learning, vol. 8, pp. 279–292, 1992

    1992

  9. [17]

    Learning from delayed rewards,

    C. J. C. H. Watkins, “Learning from delayed rewards,” 1989

    1989

  10. [18]

    Playing atari with deep reinforcement learning,

    V . Mnih, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  11. [19]

    Policy gradient methods for reinforcement learning with function ap- proximation,

    R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function ap- proximation,” Advances in neural information processing systems , vol. 12, 1999

    1999

  12. [20]

    Deterministic policy gradient algorithms,

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Ried- miller, “Deterministic policy gradient algorithms,” in International conference on machine learning. Pmlr, 2014, pp. 387–395

    2014

  13. [21]

    Continuous control with deep reinforcement learn- ing,

    T. Lillicrap, “Continuous control with deep reinforcement learn- ing,” arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  14. [22]

    A survey on federated learning systems: Vision, hype and reality for data privacy and protection,

    Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y. Li, X. Liu, and B. He, “A survey on federated learning systems: Vision, hype and reality for data privacy and protection,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 4, pp. 3347–3366, 2021

    2021

  15. [23]

    Federated learning: Challenges, methods, and future directions,

    T. Li, A. K. Sahu, A. Talwalkar, and V . Smith, “Federated learning: Challenges, methods, and future directions,” IEEE signal processing magazine, vol. 37, no. 3, pp. 50–60, 2020

    2020

  16. [24]

    Federated machine learn- ing: Concept and applications,

    Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learn- ing: Concept and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019

    2019

  17. [25]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Ar- cas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics . PMLR, 2017, pp. 1273–1282

    2017

  18. [27]

    Lo- cal learning matters: Rethinking data heterogeneity in federated learning,

    M. Mendieta, T. Yang, P . Wang, M. Lee, Z. Ding, and C. Chen, “Lo- cal learning matters: Rethinking data heterogeneity in federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8397–8406

    2022

  19. [28]

    Tackling data heterogeneity in federated learning with class prototypes,

    Y. Dai, Z. Chen, J. Li, S. Heinecke, L. Sun, and R. Xu, “Tackling data heterogeneity in federated learning with class prototypes,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, no. 6, 2023, pp. 7314–7322

    2023

  20. [29]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine learning and systems, vol. 2, pp. 429–450, 2020

    2020

  21. [30]

    Scaffold: Stochastic controlled averaging for federated 14 learning,

    S. P . Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated 14 learning,” in International conference on machine learning . PMLR, 2020, pp. 5132–5143

    2020

  22. [31]

    A novel framework for the analysis and design of heterogeneous federated learning,

    J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V . Poor, “A novel framework for the analysis and design of heterogeneous federated learning,” IEEE Transactions on Signal Processing, vol. 69, pp. 5234– 5249, 2021

    2021

  23. [32]

    Federated deep reinforcement learning,

    H. H. Zhuo, W. Feng, Y. Lin, Q. Xu, and Q. Yang, “Federated deep reinforcement learning,” arXiv preprint arXiv:1901.08277, 2019

    1901 arXiv

  24. [33]

    Lifelong federated reinforcement learning: A learning architecture for navigation in cloud robotic systems,

    B. Liu, L. Wang, and M. Liu, “Lifelong federated reinforcement learning: A learning architecture for navigation in cloud robotic systems,” IEEE Robotics and Automation Letters , vol. 4, no. 4, pp. 4555–4562, 2019

    2019

  25. [34]

    Federated deep reinforcement learning for internet of things with decentral- ized cooperative edge caching,

    X. Wang, C. Wang, X. Li, V . C. Leung, and T. Taleb, “Federated deep reinforcement learning for internet of things with decentral- ized cooperative edge caching,” IEEE Internet of Things Journal , vol. 7, no. 10, pp. 9441–9455, 2020

    2020

  26. [35]

    A review of safe reinforcement learning: Methods, theories and applications,

    S. Gu, L. Yang, Y. Du, G. Chen, F. Walter, J. Wang, and A. Knoll, “A review of safe reinforcement learning: Methods, theories and applications,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  27. [36]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015

    2015

  28. [37]

    Federated learning- based computation offloading optimization in edge computing- supported internet of things,

    Y. Han, D. Li, H. Qi, J. Ren, and X. Wang, “Federated learning- based computation offloading optimization in edge computing- supported internet of things,” in Proceedings of the ACM Turing Celebration Conference-China, 2019, pp. 1–5

    2019

  29. [38]

    Addressing function approx- imation error in actor-critic methods,

    S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approx- imation error in actor-critic methods,” in International conference on machine learning. PMLR, 2018, pp. 1587–1596

    2018

  30. [39]

    On practical robust reinforcement learn- ing: Adjacent uncertainty set and double-agent algorithm,

    U. Hwang and S. Hong, “On practical robust reinforcement learn- ing: Adjacent uncertainty set and double-agent algorithm,” to appear in IEEE Transactions on Neural Networks and Learning Systems, 2024

    2024

  31. [40]

    Asymmetric least squares estima- tion and testing,

    W. K. Newey and J. L. Powell, “Asymmetric least squares estima- tion and testing,” Econometrica: Journal of the Econometric Society , pp. 819–847, 1987

    1987

  32. [41]

    Openai gym,

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schul- man, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  33. [42]

    Gymnasium: A standard interface for reinforcement learning environments,

    M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goul ˜ao, A. Kallinteris, M. Krimmel, A. KG et al. , “Gymnasium: A standard interface for reinforcement learning environments,” arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  34. [43]

    Stable-baselines3: Reliable reinforcement learning implementations,

    A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” Journal of Machine Learning Research , vol. 22, no. 268, pp. 1–8, 2021. [Online]. Available: http://jmlr.org/papers/v22/20-1364.html

    2021

  35. [44]

    Rl baselines3 zoo,

    A. Raffin, “Rl baselines3 zoo,” https://github.com/DLR-RM/rl- baselines3-zoo, 2020

    2020

  36. [45]

    Control- ling overestimation bias with truncated mixture of continuous dis- tributional quantile critics,

    A. Kuznetsov, P . Shvechikov, A. Grishin, and D. Vetrov, “Control- ling overestimation bias with truncated mixture of continuous dis- tributional quantile critics,” in International Conference on Machine Learning. PMLR, 2020, pp. 5556–5566

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.