On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations

Daniel Jiang; Guojun Xiong; Jian Li; Shufan Wang

REVIEW 4 major objections 4 minor 43 references

On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A personalized federated RL algorithm achieves linear speedup with more agents, under Markovian noise.

desk verdict First linear speedup claim for personalized FedRL with shared representations, but the rate rests on an unproven strong-monotonicity lemma that contradicts the paper's own non-uniqueness remark. read the letter →

arxiv 2411.15014 v2 pith:BQ4SSWEV submitted 2024-11-22 cs.LG math.OCstat.ML

Guojun Xiong , Shufan Wang , Daniel Jiang , Jian Li This is my paper · ORCID

classification cs.LGmath.OCstat.ML MSC 68Q3268T05

keywords personalizedfederatedreinforcementlearningsharedrepresentationlinearconvergencespeeduptwo-timescalestochasticapproximationMarkoviannoisetemporaldifferencefunctionheterogeneousenvironments

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a personalized federated reinforcement learning framework in which agents in heterogeneous environments share a common feature representation while keeping agent-specific weights. It proves that a temporal-difference instantiation, PFedTD-Rep, converges at a rate that improves linearly as the number of agents grows, despite the non-i.i.d., Markovian data that each agent collects. This is the claimed first linear speedup for personalized federated RL with shared representations. The framework also extends to control, with experiments on DQN-based variants showing faster learning in heterogeneous settings and better generalization to new environments.

What carries the argument

The central object is the weighted Lyapunov function $M(\{\theta^i_{t+1}, \forall i\}, \Phi_t) := \|\Phi_t - \Phi^*\|^2 + \frac{\beta_{t-1}}{\alpha_t}\frac{1}{N}\sum_{i=1}^N \|\theta^i_{t+1} - y_i(\Phi_t)\|^2$, which couples the slow feature iterate $\Phi_t$ and the fast local weight iterates $\theta^i_{t+1}$ through the mapping $y_i(\Phi)$ that gives the optimal weight for a fixed representation. Its drift is decomposed into separate terms for the representation update and the weight update, bound using Lipschitz continuity of the TD gradients, the mixing time of the Markov chains, and a strong monotonicity condition on the mean-field updates. Choosing the step sizes $\alpha_t = \alpha_0/(t+2)^{5/6}$ and $\beta_t = \beta_0/(t+2)$ makes the drift contractive and yields the $O(T^{-2/3})$ rate after telescoping.

What would settle it

Run PFedTD-Rep in a finite MDP with a linear representation where the mean-field feature update can be computed exactly, and measure the Lyapunov drift in (14) directly. If the empirical drift is not consistently negative across the parameter region, or if the convergence time in communication rounds does not decrease roughly as $1/N$ when duplicating environments, the claimed linear speedup would be contradicted.

Watch

Extended reading notes

Core claim

PFedTD-Rep is cast as a federated nonlinear two-timescale stochastic approximation with Markovian noise, where local weight vectors update on a fast timescale and the shared feature representation updates on a slow timescale. The paper's central bound, Theorem 4.13, shows the Lyapunov error $M$ decays as $\tilde{O}\big(N^{-2/3}(T+2)^{-2/3}\big)$ when $\alpha_t = \alpha_0/(t+2)^{5/6}$, $\beta_t = \beta_0/(t+2)$, $\beta_0 = o(N^{-2/3})$ and $T^2 > N$. Reaching $\epsilon$ accuracy then requires $T = O(N^{-1}\epsilon^{-3/2})$ communication rounds, which is a linear convergence speedup in the number of agents. The proof uses a weighted Lyapunov function that couples the representation error with the average local-weight error, and leverages Markov chain mixing time to control the bias from Markovian samples.

Load-bearing premise

The proof assumes there is a uniform positive constant $\omega$ such that the mean-field TD updates are strongly monotone in both the representation and the local weights; if that uniform coercivity does not actually hold for the feature update, the contractions that produce the $O(T^{-2/3})$ rate are not guaranteed.

Editorial extensions

If this is right

If correct, PFedTD-Rep is the first algorithm for personalized federated RL with shared representations that provably achieves linear convergence speedup in the number of agents under Markovian noise.
The framework generalizes beyond TD: the same two-timescale shared-representation design applies to Q-learning and DQN variants, and experiments indicate faster convergence and better generalization in control tasks.
The result implies that adding more agents proportionally reduces the number of communication rounds needed for a target accuracy, making massive parallelism beneficial in heterogeneous settings.
The convergence rate matches the best-known rate for nonlinear two-timescale stochastic approximation under i.i.d. noise, even though the federated Markovian setting is more challenging.
The analysis shows that multiple local weight updates do not hurt global convergence when learning rates are chosen appropriately, similar to the supervised federated learning setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The linear speedup result relies on the strong monotonicity assumption of Lemma 4.10; if that assumption fails for the feature update, the rate may degrade, so a useful extension would be to verify or replace the uniform coercivity condition with a weaker geometric condition.
The theory is proved for linear representations and TD; a natural testable extension is whether the same two-timescale Lyapunov argument can be carried to nonlinear function approximation or to Q-learning, where the update is not linear in the representation.
The empirical speedup experiment duplicates environments to keep cross-environment parameters fixed, so a tighter practical test would vary environment diversity while measuring the constant in the rate, to see when the speedup degrades.
The framework suggests a general recipe for personalized federated learning in Markovian settings: share a slow-timescale representation and personalize fast weights, which could be applied to policy-gradient methods with a similar drift analysis.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The genuinely new thing is the first finite-time linear speedup guarantee for personalized federated RL with shared representations under Markovian noise. That is a meaningful step for the FedRL subfield, and the machinery—casting PFedTD-Rep as a federated two-timescale stochastic approximation and using a Lyapunov drift with a mixing-time argument—is a sensible way to get there. The experiments, including the duplicated-environment speedup check in Appendix G.2, support the qualitative claims, and the limitations paragraph is honest about the control extension. No code, but for a theory paper that is not a blocker.

The soft spot is load-bearing. Lemma 4.10 asserts a uniform strong-monotonicity constant omega for the mean-field hbar and gbar. The proof in Appendix E.4 only shows the TD loss is convex in each argument separately, which gives monotonicity with omega=0, not the uniform coercivity needed to make the Lyapunov drift negative. Worse, the statement as written conflicts with Remark 4.1: if rotations give another equilibrium Phi', then hbar(y_i(Phi'),Phi') = 0, and the inequality becomes 0 <= -omega ||Phi'-Phi*_0||^2 < 0. No global uniform omega can exist. The negative drift in Lemma F.3 and the D1 <= omega beta_t step in F.1.3 use exactly this omega, so the O(T^{-2/3}) rate and Corollary 4.15 do not close as written. A local version near an isolated equilibrium might rescue the claim, but it is neither stated nor proved. This is a fixable gap—state omega as an explicit local assumption or prove it under a non-degeneracy condition—but it is not a cosmetic one.

Two smaller issues: Algorithm 2 normalizes Phi locally, but the analysis tracks the unnormalized update; and Lemma 4.8's proof asserts the Lipschitz constant Ly rather than deriving it. The second is minor; the first means the analyzed update does not exactly match the implemented one, which the authors should reconcile.

Who should read this: anyone working on federated RL theory or representation learning in RL. It deserves a serious referee because the claim is important and the gap is addressable. I would not personally cite the linear speedup result until the lemma is fixed, but I would bring the paper to our reading group to pick at the proof structure.

Referee Report

4 major / 4 minor

Summary. The paper proposes PFedRL-Rep, a personalized federated reinforcement learning framework in which agents learn a shared feature representation collaboratively and an agent-specific weight vector locally. The main instantiation, PFedTD-Rep, is analyzed for temporal-difference learning with linear representations; the authors model the coupled local-weight/global-feature updates as a federated two-timescale stochastic approximation under Markovian noise and claim a Lyapunov error bound of order O(1/(N^{2/3}(T+2)^{2/3})), which they interpret as the first linear convergence speedup in the number of agents for personalized FedRL with shared representations. The proof is a Lyapunov drift analysis with a detailed appendix, and the paper includes experiments on CliffWalking, CartPole, Acrobot, and Hopper, plus a direct speedup experiment with duplicated environments.

Significance. If the convergence claim were fully established, the paper would make a valuable contribution: it would provide the first linear speedup guarantee for personalized federated reinforcement learning with shared representations under Markovian noise, and the two-timescale Lyapunov analysis would be of independent interest. The empirical protocol used to verify the speedup, duplicating environments so that cross-environment constants are held fixed, is a thoughtful design. The paper also gives full proof details in the appendix and honestly states that the analysis does not cover control settings. However, the central convergence theorem rests on a strong-monotonicity assertion that is not proved, on a step-size order claim that appears false, and on an analysis that ignores the normalization step present in Algorithm 2. Because these issues are load-bearing for the linear speedup claim, the significance is currently conditional.

major comments (4)

[Section 4.2, Lemma 4.10 and Appendix E.4] Lemma 4.10 asserts a uniform omega > 0 such that <Phi - Phi*_0, hbar(y_i(Phi), Phi)> <= -omega ||Phi - Phi*_0||^2 for all Phi, and a corresponding inequality for gbar. The proof in Appendix E.4 only establishes that the TD loss is convex separately in Phi(s) with theta fixed and in theta with Phi fixed; convexity gives monotonicity with omega = 0, not the uniform coercivity used in Lemma F.3 and in the contraction step. Moreover, hbar(y_i(Phi), Phi) is not the gradient of a single convex function in Phi because y_i(Phi) itself depends on Phi, so separate convexity does not imply the displayed inequality. The statement also conflicts with Remark 4.1: if rotations produce another equilibrium (Phi', theta') with hbar(y_i(Phi'), Phi') = 0 and Phi' != Phi*_0, then the left-hand side is 0 while the right-hand side is -omega ||Phi' - Phi*_0||^2 < 0, so no uniform global omega can exist. A local version near an isolated equilibrium would require new assumptions and a basin-of-attraction argument; neither is supplied.
[Appendix F.1.3, Eqs. (52)-(54)] The proof states that D1 is o(beta_t) and D2 is o(alpha_{t+1}), so that D1 <= omega beta_t and D2 <= K omega alpha_{t+1}. But D1 contains the term beta_t L / alpha_t. With alpha_t = alpha_0/(t+2)^{5/6} and beta_t = beta_0/(t+2), we have beta_t L / alpha_t = (beta_0 L / alpha_0)(t+2)^{-1/6}, while beta_t = beta_0/(t+2). The ratio (beta_t L / alpha_t) / beta_t = (L/alpha_0)(t+2)^{5/6} tends to infinity, so D1 is not o(beta_t). Consequently the inequality D1 <= omega beta_t and the contraction M({theta^i_{t+2}}, Phi_{t+1}) <= (1 - omega beta_t) M({theta^i_{t+1}}, Phi_t) in (54) are not justified. The Lyapunov recursion therefore does not close at the claimed rate with the stated step sizes unless additional conditions or a different weighting are introduced.
[Algorithm 2 and Section 4.1, Eqs. (9)-(11)] Algorithm 2 normalizes each local feature update before averaging (line 11) and initializes feature rows as unit-norm vectors, but the theoretical update analyzed in Eqs. (9)-(11) is Phi_{t+1} = Phi_t + beta_t (1/N) sum_i h(theta^i_{t+1}, Phi_t, ...) with no normalization. The proof never accounts for the normalization step. Since normalization is a nonlinear map that changes the mean-field direction and can affect the Lipschitz and strong-monotonicity properties used in Lemmas 4.6-4.10, Theorem 4.13 as written applies to a different algorithm from the one implemented and simulated. The paper should either remove normalization from Algorithm 2, prove that the normalization is asymptotically negligible under the paper's assumptions, or incorporate a projection step into the analysis.
[Corollary 4.15 and Appendix F.2] Corollary 4.15 states only beta_0 = o(N^{-2/3}) and T^2 > N, but the proof in Appendix F.2 begins 'If alpha_0 = beta_0 = o(N^{-1/3} K^{-1/2})' and relies on that additional condition to suppress the alpha_0^2 K^2 terms in C1. Without an N-dependent condition on alpha_0 (or on the constants K, B, L), the term 2 alpha_0^2 (3 K^2 B^2 + 3 K^2 delta^2 + 2 L^2 K^2 B^2) (T+2)^{-2/3} in (16) is an O(T^{-2/3}) error that is independent of N and dominates, so the N^{-2/3} speedup does not follow. The hypotheses stated in Corollary 4.15 do not imply the claimed bound.

minor comments (4)

[Lemma 4.10] The second inequality in Lemma 4.10 has mismatched variables: the left-hand side uses theta^i_t and Phi_{t-1}, while the right-hand side uses arbitrary theta and y_i(Phi). The statement should be written uniformly for a single pair (theta, Phi).
[Definition 4.5 and Lemma 4.12] The mixing time tau_delta is defined before the Lipschitz constants L_g and L_h are introduced, and Lemma 4.12 later defines L1 in terms of L_g, L_h, and the steady-state gradients. Reordering these definitions would improve readability.
[Theorem 4.13, Eq. (16)] The theorem writes M({theta^i_{T+2}}, Phi_{T+1}) on the left but bounds it in terms of M({theta^i_1}, Phi_0); using a consistent time subscript, for example t and t+1, would avoid confusion.
[Appendix E.3, proof of Lemma 4.8] The proof first observes that the norm-scaling step gives ||y_i(Phi_1) - y_i(Phi_2)|| <= 2B and then asserts the existence of Ly such that the Lipschitz bound holds. This is only immediate on a bounded parameter domain; the compactness assumption or a direct argument from the implicit function theorem should be stated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the convergence proof is a Lyapunov-drift argument for two-timescale SA; the main weakness, Lemma 4.10, is an unsupported assumption rather than a reduction of the conclusion to its inputs.

full rationale

The derivation is self-contained in the sense required by the circularity pass: PFedTD-Rep's rate is not fitted from the data it predicts, and no equation in Theorem 4.13 or Corollary 4.15 is equivalent by construction to an input. The proof uses standard external machinery (Doan's two-timescale SA, Tsitsiklis & Van Roy's TD convergence, Chen et al.'s Markovian-noise bounds), which are legitimate independent supports. The authors' own previous work (Xiong et al. 2024, DepRL) is cited only for motivation and related work, not as the load-bearing basis of the linear-speedup claim. The genuine weakness is Lemma 4.10: its stated conclusion requires a uniform omega > 0 for the mean-field updates, but Appendix E.4 only proves separate convexity of the loss, which yields monotonicity with omega = 0; the paper's own Remark 4.1 about non-unique equilibria also makes a global uniform omega questionable. This is a gap in establishing a hypothesis of the proof, not a circularity: the theorem's output is not assumed or renamed as an input, and the authors do not fit parameters to make the rate true. The limitations section explicitly states that the analysis does not extend to control, which is consistent with the paper's claims and does not indicate circular dependence.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The analysis assumes geometric mixing of each agent's Markov chain (Assumption 4.3), unichain MDPs, and a uniform strong-monotonicity constant omega (Lemma 4.10) that is only supported by a convexity argument. The step sizes alpha0 and beta0, the clipping bound B, and the mixing slack delta are chosen analytically, not fitted to data. No new physical or mathematical entities are introduced.

free parameters (4)

alpha0
Step-size scale for local weight updates. Theorem 4.13 requires alpha0 <= 1/(2L sqrt(2(1+L^2))); Corollary 4.15 requires alpha0 = o(N^{-1/3} K^{-1/2}). It is chosen analytically to make the Lyapunov drift contractive, not fitted to data.
beta0
Step-size scale for feature representation updates. Requires beta0 <= omega/2 and beta0 = o(N^{-2/3}) for the linear speedup. It is an analytic choice, not a data fit.
B
Norm bound enforced by clipping theta after local updates. It enters Lipschitz constants Lg and Lh in Lemmas 4.6 and 4.7 and the error constants C1 and C2.
delta
Mixing-time slack in Definition 4.5. The theorem requires T >= 2 tau_delta and the final bound depends on delta through C1 and C2; delta can be taken as O(1/T) to balance the error terms.

assumptions (4)

domain assumption Each agent's Markov chain is irreducible, aperiodic, and geometrically mixing (Assumption 4.3).
Standard in Markovian stochastic approximation; gives the mixing-time bound in Lemma 4.12 and is used to bound biased gradients.
ad hoc to paper The mean fields gbar and hbar are strongly monotone with a uniform constant omega > 0 (Lemma 4.10).
This is the contractivity that makes the Lyapunov drift negative. The proof in Appendix E.4 only shows convexity in each variable separately, which yields monotonicity, not the uniform strong-monotonicity constant omega used in the theorem.
domain assumption The implicit mapping y_i(Phi) is Lipschitz with constant Ly (Lemma 4.8).
Used to couple the fast and slow timescales. The proof in Appendix E.3 bounds the difference by 2B and then asserts a Lipschitz constant; the uniform constant is not derived.
ad hoc to paper The normalization steps in Algorithm 2 do not change the convergence behavior.
The proof analyzes update (11) without the normalization in Algorithm 2 step 11; the paper does not analyze the projection or show it preserves the Lyapunov drift.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations." pith.science (2026). https://pith.science/paper/BQ4SSWEV

@misc{pith2026241115014,
  author       = {Pith},
  title        = {Pith review of: On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQ4SSWEV}},
  note         = {Machine review of arXiv:2411.15014}
}

read the original abstract

Federated reinforcement learning (FedRL) enables multiple agents to collaboratively learn a policy without sharing their local trajectories collected during agent-environment interactions. However, in practice, the environments faced by different agents are often heterogeneous, leading to poor performance by the single policy learned by existing FedRL algorithms on individual agents. In this paper, we take a further step and introduce a \emph{personalized} FedRL framework (PFedRL) by taking advantage of possibly shared common structure among agents in heterogeneous environments. Specifically, we develop a class of PFedRL algorithms named PFedRL-Rep that learns (1) a shared feature representation collaboratively among all agents, and (2) an agent-specific weight vector personalized to its local environment. We analyze the convergence of PFedTD-Rep, a particular instance of the framework with temporal difference (TD) learning and linear representations. To the best of our knowledge, we are the first to prove a linear convergence speedup with respect to the number of agents in the PFedRL setting. To achieve this, we show that PFedTD-Rep is an example of the federated two-timescale stochastic approximation with Markovian noise. Experimental results demonstrate that PFedTD-Rep, along with an extension to the control setting based on deep Q-networks (DQN), not only improve learning in heterogeneous settings, but also provide better generalization to new environments.

Figures

Figures reproduced from arXiv: 2411.15014 by the authors.

**Figure 1.** An illustrative example of PFEDRL-REP for 3 agents. (a) At the beginning of round t, each agent i = 1, 2, 3 has a local weight vector θ i t and a global feature representation Φt. (b) Using (Φt, θi t ), each agent i performs a K-step update to obtain θ i t+1 as in (3). Note that Φt remains unchanged at this step. (c) Agent i updates the feature representation by executing a one-step update to obtain Φ i t+1/2 as in … view at source ↗

**Figure 2.** Comparisons in a CliffWalking Environment with 3 agents. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Comparisons in control problems. sentation model is thus central to achieving stable and convergent learning. In Line 13 of Algorithm 4, the algorithm performs a scheduled update of the shared representation Φ of the main network’s parameters with the guidance of the target network. In Line 18 of Algorithm 4, every Ttarget steps, the algorithm performs a scheduled update of the target network’s parameters by copying… view at source ↗

Figures from the paper (13 more)

**Figure 4.** Figure 4: Schematic representation of FedRL, where [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

**Figure 5.** Figure 5: Our proposed PFEDRL-REP framework where N agents independently interact with their own environments and take actions according to their individual RL component parameterized by Φ and θ i . Agent i locally update weight vector θi while jointly updating the shared featur…

**Figure 6.** Figure 6: An illustrative example with three agents that demonstrates the conventional FedRL framework fails to work. Example of RL components that fit the proposed PFEDRL with shared representations. In the following, we aim to showcase examples of RL components that are compa…

**Figure 7.** Figure 7: An illustrative example for the proposed framework. Notice that both the DQN in (a) and policy gradient (PG) in (b) can be fitted into the proposed framework by treating the parameters of the feature extraction network as the shared feature representation Φ and the par…

**Figure 8.** Figure 8: Comparison of control by DQN, FedDQN and PF [PITH_FULL_IMAGE:figures/full_fig_p038_8.png]

**Figure 9.** Figure 9: Comparison of control by DQN, FedDQN and PF [PITH_FULL_IMAGE:figures/full_fig_p039_9.png]

**Figure 10.** Figure 10: Hopper environment. As shown in [PITH_FULL_IMAGE:figures/full_fig_p040_10.png]

**Figure 11.** Figure 11: Linear speedup for cartpole with duplicates of environments. [PITH_FULL_IMAGE:figures/full_fig_p040_11.png]

**Figure 12.** Figure 12: Computation versus worst case personalization error trade off. [PITH_FULL_IMAGE:figures/full_fig_p041_12.png]

**Figure 13.** Figure 13: Worst case personalization error with varying pole length discrepancy across environ [PITH_FULL_IMAGE:figures/full_fig_p041_13.png]

**Figure 14.** Figure 14: Heatmap of Cartpole environment. 1 2 3 4 5 6 7 8 9 10 Agent 1 2 3 4 5 6 7 8 9 10 Agent 1 0.2 0.0015 0.33 0.27 0.38 -0.074 0.18 0.17 0.15 0.2 1 -0.0061 0.28 0.24 0.23 -0.11 0.046 0.16 0.11 0.0015-0.0061 1 0.18 0.1 0.18 -0.14 0.056 0.021 0.12 0.33 0.28 0.18 1 0.26 0.52 …

**Figure 15.** Figure 15: Heatmap of Acrobot environment. G.6 MORE STATISTICS OF THE EMPIRICAL RESULTS We report the return average, variance average, return median and total running time for 10 environments for Cartpole and Acrobot environments. Among all algorithms, our PFedDQN-Rep achieves…

**Figure 16.** Figure 16: Shortened plot for cartpole and acrobot environment. [PITH_FULL_IMAGE:figures/full_fig_p043_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 27 canonical work pages

[1]

⟨ΦΦΦ∗ − ΦΦΦt, −1 N NX i=1 ¯h(θθθi t+1, ΦΦΦt)⟩ # | {z } Term 2 + 2βtE

≤ 2L1(∥θθθ − yi(ΦΦΦ∗)∥ + ∥ΦΦΦ − ΦΦΦ∗∥ + 1)dC1ρk 1, where the last inequality holds due to Assumption 4.3 with constants C1 > 0 and ρ1 ∈ (0, 1). To guarantee 2L1(∥θθθ − yi(ΦΦΦ∗)∥ + ∥ΦΦΦ − ΦΦΦ∗∥ + 1)dC1ρk 1 ≤ δ(∥θθθ − yi(ΦΦΦ∗)∥ + ∥ΦΦΦ − ΦΦΦ∗∥ + 1), we have τδ ≤ log(1/δ) + log(2L1C1d) log(1/ρ1) . (23) Using the same procedures we can show that ∥E[g(θθθ, ΦΦΦ,...

work page 2025
[2]

When the state and action spaces are large, it is com- putationally infeasible to store Qi,πi (s, a) for all state-action pairs

C A PPLICATION TO CONTROL TASKS IN RL The Q-function of agent i in environment Mi under policy πi is defined as Qi,πi (s, a) = Eπi P∞ k=0 γkRi(si k, ai k)|si 0 = s, ai 0 = a . When the state and action spaces are large, it is com- putationally infeasible to store Qi,πi (s, a) for all state-action pairs. One way to deal with is to approximate the Q-functio...

work page 1992
[3]

Openai gym

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540,

arXiv
[5]

(1 + βt−1/αt) (1 + 2βt−1/αt − 2αtKω ) + (12α2 t δ2K 2 + 2L2α3 t /βt−1 + 6K 2δ2α3 t /βt−1) 4β2 t−1L2 N ! + (1 + αt/βt−1) 4β2 t−1L4 N # · E h θθθi t − yi(ΦΦΦt−1) 2i +

(47) Proof. Term 6 = 2E "* θθθi t − yi(ΦΦΦt−1) + αt KX k=1 g(θθθi t,k−1, ΦΦΦt), yi(ΦΦΦt−1) − yi(ΦΦΦt) +# ≤ βt−1/αtE   θθθi t − yi(ΦΦΦt−1) + αt KX k=1 g(θθθi t,k−1 2  | {z } Term 4 + αt/βt−1E h yi(ΦΦΦt−1) − yi(ΦΦΦt) 2i | {z } Term 5 (48) Providing Term 4 in Lemma F.9, Term 5 in Lemma F.10, and Term 6 in Lemma F.11, we have the following result. Lemma F...

work page 2025
[6]

, Edo 2: Get the initial state of the environment; 3: for t = 0, 1, ..., T− 1 do 4: for i = 1,

In Figure 6a, 17 Published as a conference paper at ICLR 2025 Algorithm 4 PF EDDQN-R EP Initialize: The parameters (ΦΦΦ, θθθi) for each Q network Qi(s, a), the replay buffer Ri, and copy the same parameter from Q network to initialize the target Q network Qi,′(s, a) for agent i, ∀i ∈ [N ]; 1: for episode e = 1, . . . , Edo 2: Get the initial state of the ...

work page 2025
[7]

Nonlinear two-time-scale stochastic approximation: Convergence and finite-time performance

Thinh T Doan. Nonlinear two-time-scale stochastic approximation: Convergence and finite-time performance. arXiv preprint arXiv:2011.01868,

arXiv 2011
[10]

Federated learning for mobile keyboard prediction

Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Franc ¸oise Beaufays, Sean Augenstein, Hubert Eichner, Chlo ´e Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604,

arXiv
[11]

Federated learning for resource-constrained IoT devices: Panoramas and state of the art

11 Published as a conference paper at ICLR 2025 Ahmed Imteaj, Khandaker Mamun Ahmed, Urmish Thakker, Shiqiang Wang, Jian Li, and M Hadi Amini. Federated learning for resource-constrained IoT devices: Panoramas and state of the art. Federated and Transfer Learning, pp. 7–27,

work page 2025

Show all 43 references

[13]

Asynchronous federated reinforcement learning with policy gradient updates: Algorithm design and convergence analysis

Guangchen Lan, Dong-Jun Han, Abolfazl Hashemi, Vaneet Aggarwal, and Christopher G Brinton. Asynchronous federated reinforcement learning with policy gradient updates: Algorithm design and convergence analysis. arXiv preprint arXiv:2404.08003,

arXiv
[15]

Model-free representation learning and exploration in low-rank mdps

Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. Model-free representation learning and exploration in low-rank mdps. arXiv preprint arXiv:2102.07035 ,

arXiv
[16]

Federated reinforcement learning for fast personalization

Chetan Nadiger, Anil Kumar, and Sherine Abdelhak. Federated reinforcement learning for fast personalization. In 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pp. 123–127. IEEE,

2019
[17]

Federated reinforcement learning: Techniques, applications, and open challenges

Jiaju Qi, Qihao Zhou, Lei Lei, and Kan Zheng. Federated reinforcement learning: Techniques, applications, and open challenges. arXiv preprint arXiv:2108.11887,

arXiv
[18]

The sample-communication complexity trade-off in federated q- learning

12 Published as a conference paper at ICLR 2025 Sudeep Salgia and Yuejie Chi. The sample-communication complexity trade-off in federated q- learning. arXiv preprint arXiv:2408.16981,

2025 arXiv
[20]

Federated tem- poral difference learning with linear function approximation under environmental heterogeneity

Han Wang, Aritra Mitra, Hamed Hassani, George J Pappas, and James Anderson. Federated tem- poral difference learning with linear function approximation under environmental heterogeneity. arXiv preprint arXiv:2302.02212, 2023a. Sid Wang, Ashish Shenoy, Pierce Chuang, and John N...

arXiv
[21]

Applied federated learning: Improving google keyboard query suggestions

Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ra- mage, and Franc ¸oise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903,

arXiv
[22]

Federated reinforcement learning for generalizable motion planning

Zhenyuan Yuan, Siyuan Xu, and Minghui Zhu. Federated reinforcement learning for generalizable motion planning. In 2023 American Control Conference (ACC), pp. 78–83. IEEE,

2023
[23]

Finite-time analysis of on-policy heterogeneous federated reinforcement learning

13 Published as a conference paper at ICLR 2025 Chenyu Zhang, Han Wang, Aritra Mitra, and James Anderson. Finite-time analysis of on-policy heterogeneous federated reinforcement learning. arXiv preprint arXiv:2401.15273,

2025 arXiv
[24]

14 Published as a conference paper at ICLR 2025 A R ELATED WORK Single-agent reinforcement learning. RL is a machine learning paradigm that trains agents to make sequences of decisions by rewarding desired behaviors and/or penalizing undesired ones in a given environment (Sutt...

2025
[26]

leveraged the common representation among agents in heterogeneous environments to guarantee personalized models for federated supervised learning. Representation learning in MDP.Representation learning aims to transform high-dimensional ob- servation to low-dimensional embeddi...

2020
[27]

However, it is open in the context of leveraging representation learning in PFedFL

and block MDPs (Zhang et al., 2022c). However, it is open in the context of leveraging representation learning in PFedFL. In this work, we prove that representation augmented PFedFL forms a general framework as a federated two-timescale stochas- tic approximation with Markovia...

2021
[30]

It is important to note that both the DQN architecture in Figure 7a and the policy gradient (PG) approach in Figure 7b seamlessly integrate into our proposed framework. This integration is achieved by designating the parameters of the feature extraction network as the shared f...

2025
[31]

Hence, Li(ΦΦΦ(si k), θθθi) is convex on ΦΦΦ(si k) under a fixed θi

Positive semi-definiteness of the Hessian implies convexity. Hence, Li(ΦΦΦ(si k), θθθi) is convex on ΦΦΦ(si k) under a fixed θi. This property holds vice versa, i.e., Li(ΦΦΦ(si k), θθθi) is convex on θθθi under a fixed ΦΦΦ(si k). 21 Published as a conference paper at ICLR 2025...

2019
[33]

NX i=1 ∥θθθi t+1 − yi(ΦΦΦt)∥2 # . (29) Proof. We have Term 2 = 2βtE

We first bound Term 1 as follows. Lemma F.2. For any t ≥ τ, we have Term 1 ≤ 4β2 t (L2 + L4)E[∥ΦΦΦ∗ − ΦΦΦt∥2] + 4β2 t L2 N E " NX i=1 ∥θθθt+1 − yi(ΦΦΦt)∥2 # + 4β2 t δ2 (28) Proof. Note that Term 1 = β2 t N 2 E   NX i=1 h(θθθi t+1, ΦΦΦt) − NX i=1 h(yi(ΦΦΦt), ΦΦΦ∗) + NX i=1 h(...

2025
[34]

The proof is similar to that of Lemma 3 in Dal Fabbro et al

(31) Proof. The proof is similar to that of Lemma 3 in Dal Fabbro et al. (2023). Starting with ∥ΦΦΦ∗ − ΦΦΦt+1∥2 = ∥ΦΦΦ∗ − ΦΦΦt∥2 + β2 t N 2 NX i=1 h(θθθi t+1, ΦΦΦt) 2 − 2βt⟨ΦΦΦ∗ − ΦΦΦt, 1 N NX i=1 h(θθθi t+1, ΦΦΦt)⟩ ≤ (1 + βt/α0)∥ΦΦΦ∗ − ΦΦΦt∥2 + (βtα0 + β2 t ) N 2 NX i=1 hi t(...

2023
[35]

29 Published as a conference paper at ICLR 2025 Proof

+ (6βt/αt + 6βtαtδ2(1 + L2) + 4βtαtL2(3 + 4L2))E[∥ΦΦΦt − ΦΦΦ∗∥2] + 16βtαtL2 + 6βtαtδ2 N E " NX i=1 ∥θθθi,∗ − θθθt+1∥2 # + 11βtαtδ2. 29 Published as a conference paper at ICLR 2025 Proof. Substituting the bound of E[∥ΦΦΦt − ΦΦΦt−τ ∥2] in (31) into Term 3 in Lemma F.4 yield the ...

2025
[36]

* θθθi t − yi(ΦΦΦt−1), KX k=1 g(θθθi t,k−1, ΦΦΦt) +# ≤ E h θθθi t − yi(ΦΦΦt−1) 2i + 6α2 t δ2K 2E h ∥ΦΦΦt − ΦΦΦ∗∥2 i + 6α2 t δ2K 2(1 + B2) + 2α2 t K 2L2B2 + 2αtE

+ 4β2 t δ2 + 11βtαtδ2. This completes the proof. F.1.2 D RIFT OF θθθi t, ∀i. Next, we characterize the drift between θθθi t+1 and θθθi t. Lemma F.8. The drift between θθθi t+1 and θθθi t, ∀i is given by E[∥θθθi t+1 − yi(ΦΦΦt)∥2] = E   θθθi t − yi(ΦΦΦt−1) + αt KX k=1 g(θθθi t...

2025
[38]

(1 + βt/αt+1) (1 + 2βt/αt+1 − 2αt+1Kω ) + (12α2 t+1δ2K 2 + 2L2α3 t+1/βt + 6K 2δ2α3 t+1/βt) 4β2 t L2 N ! + (1 + αt+1/βt) 4β2 t L4 N # · 1 N E

+ 4β2 t δ2 + 11βtαtδ2 + βt αt+1 · " (1 + βt/αt+1) (1 + 2βt/αt+1 − 2αt+1Kω ) + (12α2 t+1δ2K 2 + 2L2α3 t+1/βt + 6K 2δ2α3 t+1/βt) 4β2 t L2 N ! + (1 + αt+1/βt) 4β2 t L4 N # · 1 N E " NX i=1 θθθi t+1 − yi(ΦΦΦt) 2 # + " (1 + βt/αt+1) (12α2 t+1δ2K 2 + 6K 2δ2α3 t+1/βt) + (12α2 t+1δ2K ...

2025
[39]

E[∥ΦΦΦt − ΦΦΦ∗∥2] + 1 N E

In addition, we have the following inequalities (t + 2)2 · αtβt ≤ α0β0(t + 2)1/3, (t + 2)2 · α2 t = α2 0(t + 2)2. Hence, multiplying both sides with (t + 2)2, we have (t + 2)2M ({θθθi t+2}, ΦΦΦt+1) ≤ (t + 1)2M ({θθθi t+1}, ΦΦΦt) + (144τ 2K 2L2δ2 + 4L2/N)α0β0(t + 2)1/3 " E[∥ΦΦΦ...

2022
[40]

PF EDDQN-R EP in Acrobot environment

38 Published as a conference paper at ICLR 2025 0 20 40 60 80 100 Episode -0.24 -10 -5 0 Return 102 0 20 40 60 80 100 Episode -0.18 -10 -5 0 Return 102 0 20 40 60 80 100 Episode -0.12 -10 -5 0 Return 102 0 20 40 60 80 100 Episode -0.06 -10 -5 0 Return 102 0 20 40 60 80 100 Epi...

2025
[41]

Because of this duplication, we know that the across- environment parameters (e.g., τδ, C, and others) remain constant. 39 Published as a conference paper at ICLR 2025 0 500 1000 1500 2000 Frame (a) Average return for 20 agents 0 0.2 0.4 0.6Return 103 0 500 1000 1500 2000 Fram...

2025
[42]

Figure 13: Worst case personalization error with varying pole length discrepancy across environ- ments

We notice that as the discrepancy increases, all algorithm encounter degradations in personalization quality (measured in terms of the worst case personalization error defined in (56)), but our proposed algorithm achieves the least degradation. Figure 13: Worst case personaliz...

2025
[43]

Table 3: Statistics for Cartpole environment. Algorithm Return average Variance average Return median Total running time(s) PFedDQN-Rep 143 43 154 466 DQN 135 54 127 3840 FedDQN 101 67 88 387 FedQ-K 112 34 107 490 LFRL 117 47 99 434 PerDQNAvg 127 48 131 520 FedAsynQ-ImAvg 119 ...

2025
[2013]

Personalized feder- ated learning with communication compression

El Houcine Bergou, Konstantin Burlachenko, Aritra Dutta, and Peter Richt´arik. Personalized feder- ated learning with communication compression. arXiv preprint arXiv:2209.05148,

arXiv
[2015]

Asynchronous methods for deep reinforcement learning

V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928–1937. PMLR,

1928
[2016]

Federated meta-learning with fast convergence and efficient communication

Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta-learning with fast convergence and efficient communication. arXiv preprint arXiv:1802.07876,

arXiv
[2017]

Federated reinforcement learning

and deep Deterministic Policy Gradients (DDPG) (Silver et al., 2014), leverage the Actor-Critic framework to provide more stable and robust ways to directly optimize the policy, overcoming challenges related to action space and variance. Federated reinforcement learning. Jin e...

2022
[2018]

Perfor- mance of q-learning with linear function approximation: Stability and finite-time analysis

Zaiwei Chen, Sheng Zhang, Thinh T Doan, Siva Theja Maguluri, and John-Paul Clarke. Perfor- mance of q-learning with linear function approximation: Stability and finite-time analysis. arXiv preprint arXiv:1905.11425, pp. 4,

1905 arXiv
[2019]

Exploiting shared repre- sentations for personalized federated learning

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In International conference on machine learning, pp. 2089–2099. PMLR,
[2020]

Finite-time convergence rates of nonlinear two-time-scale stochastic approximation under markovian noise

Thinh T Doan. Finite-time convergence rates of nonlinear two-time-scale stochastic approximation under markovian noise. arXiv preprint arXiv:2104.01627,

arXiv
[2021]

Personalized federated learning: A meta- learning approach

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint arXiv:2002.07948,

2002 arXiv
[2022]

Private learning with public features

Walid Krichene, Nicolas Mayoraz, Steffen Rendle, Shuang Song, Abhradeep Thakurta, and Li Zhang. Private learning with public features. arXiv preprint arXiv:2310.15454,

arXiv
[2023]

Fed- erated learning with personalization layers

Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Fed- erated learning with personalization layers. arXiv preprint arXiv:1912.00818,

1912 arXiv
[2024]

Proximal policy optimization algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

⟨ΦΦΦ∗ − ΦΦΦt, −1 N NX i=1 ¯h(θθθi t+1, ΦΦΦt)⟩ # | {z } Term 2 + 2βtE

≤ 2L1(∥θθθ − yi(ΦΦΦ∗)∥ + ∥ΦΦΦ − ΦΦΦ∗∥ + 1)dC1ρk 1, where the last inequality holds due to Assumption 4.3 with constants C1 > 0 and ρ1 ∈ (0, 1). To guarantee 2L1(∥θθθ − yi(ΦΦΦ∗)∥ + ∥ΦΦΦ − ΦΦΦ∗∥ + 1)dC1ρk 1 ≤ δ(∥θθθ − yi(ΦΦΦ∗)∥ + ∥ΦΦΦ − ΦΦΦ∗∥ + 1), we have τδ ≤ log(1/δ) + log(2L1C1d) log(1/ρ1) . (23) Using the same procedures we can show that ∥E[g(θθθ, ΦΦΦ,...

work page 2025

[2] [2]

When the state and action spaces are large, it is com- putationally infeasible to store Qi,πi (s, a) for all state-action pairs

C A PPLICATION TO CONTROL TASKS IN RL The Q-function of agent i in environment Mi under policy πi is defined as Qi,πi (s, a) = Eπi P∞ k=0 γkRi(si k, ai k)|si 0 = s, ai 0 = a . When the state and action spaces are large, it is com- putationally infeasible to store Qi,πi (s, a) for all state-action pairs. One way to deal with is to approximate the Q-functio...

work page 1992

[3] [3]

Openai gym

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540,

arXiv

[4] [5]

(1 + βt−1/αt) (1 + 2βt−1/αt − 2αtKω ) + (12α2 t δ2K 2 + 2L2α3 t /βt−1 + 6K 2δ2α3 t /βt−1) 4β2 t−1L2 N ! + (1 + αt/βt−1) 4β2 t−1L4 N # · E h θθθi t − yi(ΦΦΦt−1) 2i +

(47) Proof. Term 6 = 2E "* θθθi t − yi(ΦΦΦt−1) + αt KX k=1 g(θθθi t,k−1, ΦΦΦt), yi(ΦΦΦt−1) − yi(ΦΦΦt) +# ≤ βt−1/αtE   θθθi t − yi(ΦΦΦt−1) + αt KX k=1 g(θθθi t,k−1 2  | {z } Term 4 + αt/βt−1E h yi(ΦΦΦt−1) − yi(ΦΦΦt) 2i | {z } Term 5 (48) Providing Term 4 in Lemma F.9, Term 5 in Lemma F.10, and Term 6 in Lemma F.11, we have the following result. Lemma F...

work page 2025

[5] [6]

, Edo 2: Get the initial state of the environment; 3: for t = 0, 1, ..., T− 1 do 4: for i = 1,

In Figure 6a, 17 Published as a conference paper at ICLR 2025 Algorithm 4 PF EDDQN-R EP Initialize: The parameters (ΦΦΦ, θθθi) for each Q network Qi(s, a), the replay buffer Ri, and copy the same parameter from Q network to initialize the target Q network Qi,′(s, a) for agent i, ∀i ∈ [N ]; 1: for episode e = 1, . . . , Edo 2: Get the initial state of the ...

work page 2025

[6] [7]

Nonlinear two-time-scale stochastic approximation: Convergence and finite-time performance

Thinh T Doan. Nonlinear two-time-scale stochastic approximation: Convergence and finite-time performance. arXiv preprint arXiv:2011.01868,

arXiv 2011

[7] [10]

Federated learning for mobile keyboard prediction

Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Franc ¸oise Beaufays, Sean Augenstein, Hubert Eichner, Chlo ´e Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604,

arXiv

[8] [11]

Federated learning for resource-constrained IoT devices: Panoramas and state of the art

11 Published as a conference paper at ICLR 2025 Ahmed Imteaj, Khandaker Mamun Ahmed, Urmish Thakker, Shiqiang Wang, Jian Li, and M Hadi Amini. Federated learning for resource-constrained IoT devices: Panoramas and state of the art. Federated and Transfer Learning, pp. 7–27,

work page 2025