{"id":"3dec12e9-e656-4d35-8970-ee6eeb77f130","arxiv_id":"2412.14555","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"SFAC is a single-loop federated actor-critic method with convergence guarantees for agents in heterogeneous environments, achieving linear speedup when environment heterogeneity is zero.","lead":"This paper proposes a federated reinforcement learning algorithm, SFAC, where many agents train one shared policy while each interacts with its own different environment. It gives mathematical bounds showing the shared policy converges to a good solution, with speed that improves as more agents join, provided the environments are similar enough.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 1's averaged-transition construction is invalid: D* is not the stationary distribution of P*, so Assumption 4, the critic contraction in Proposition 1, and hence Theorem 1 are unsupported.","rationale":"The strongest claim requires both the critic error to contract and the actor's biased critic term to vanish. Proposition 1's contraction factor depends on λ from Assumption 4, and Lemmas 5–6 (used to control the drift of the critic fixed point across actor updates) are stated 'Suppose Assumption 2-4 hold' and cited from Chen, Sun, and Yin (2021). The paper never verifies Assumption 4 for the federated average of heterogeneous TD problems. The only bridge is Lemma 1, which is a two-line construction that mistakes an algebraic identity for a sampling process. A valid TD gradient requires the expectation to be taken under the stationary distribution of the transition kernel; the authors choose D* as the average of per-agent stationary distributions and P* as a D-weighted average of transition kernels, but there is no reason D* is invariant under P*. In fact D*P* = (1/N)ΣD_iP_i, which is generally not D*. Consequently the identity in Lemma 1 does not correspond to any Markov chain, and all downstream contraction and smoothness results are unsupported. The reader's weakest_assumption identifies precisely this gap, and I agree. I focus on this rather than the variance-scaling issue because the validity of the rate O(1/√NK) is secondary to whether the algorithm converges at all; if λ = 0, the critic recursion is not even a contraction and the bound in Theorem 1 is meaningless. The proposed numerical check directly tests whether Assumption 4 can hold for averaged heterogeneous TD problems; if it fails on generic two-state examples, the central claim should not be accepted as stated. I therefore leave the reader's REJECT verdict unchanged.","tokens_in":29093,"tokens_out":5486,"duration_ms":42709,"concrete_test":"Take N=2 two-state MDPs (states {1,2}, deterministic policy, transition matrices P1 and P2 irreducible with stationary distributions D1, D2). Compute D* = (D1+D2)/2 and P* = (D1+D2)^{-1}(D1P1 + D2P2). Test (i) whether D*P* = D*; (ii) with Φ = I_2 and γ = 0.9, compute M = Φ^T D*(γP* − I)Φ and the largest eigenvalue of (M+M^T)/2. If D*P* ≠ D*, Lemma 1's Markov-chain interpretation is invalid; if the largest eigenvalue of the symmetrized M is ≥ 0, Assumption 4 fails and Proposition 1's contraction does not hold for this instance. Repeat over random P1, P2 (e.g., 1000 draws) to confirm the failure is generic rather than a special case.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central proof of Theorem 1 rests on Proposition 1, whose contraction factor (1 − β υ λ /4) requires Assumption 4: ⟨g(ω), ω − ω*⟩ ≤ −λ/2‖ω − ω*‖². Assumption 4 is not among Assumptions 1–3 in the main text, and its only justification is Lemma 1 in the supplement, which claims that the averaged TD gradient g(ω) = (1/N)Σ_i g_i(ω) equals the TD gradient of a single Markov chain with transition P* = (Σ_i D_i)^{−1} Σ_i D_i P_i and reward R* = (Σ_i D_i)^{−1} Σ_i D_i R_i, where D_i is the stationary diagonal matrix of agent i's chain. For g(ω) to be the expected TD gradient of a Markov chain, the matrix D* = (1/N)Σ_i D_i would have to be the stationary distribution of P*. The authors assert only that P* is an irreducible transition matrix. But D*P* = (1/N)Σ_i D_i P_i, which equals D* only in the homogeneous case Σ_i D_i P_i = Σ_i D_i; in general it does not. Thus the constructed object is not a Markov chain with stationary distribution D*, the identity g(ω) = Φ^T D*R* + Φ^T D*(γP* − I)Φω does not follow from any sampling process, and Assumption 4 has no demonstrated basis. Since Lemmas 5 and 6 are imported from Chen et al. (2021) under exactly this assumption, the Lipschitz dependence of ω*(θ), the cancellation of the critic bias in the actor update, and the final O(1/√(NK)) rate all collapse if λ = 0. This is the single most load-bearing gap: it is not a looseness in constants but an unverified spectral condition on an averaged problem that does not reduce to a valid single-chain problem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SFAC, a single-loop federated actor-critic algorithm for learning a shared policy across heterogeneous environments. Federated critics (FedC) run local TD updates with multiple local iterations and are aggregated after each communication round; federated actors (FedA) use the aggregated critic to estimate policy gradients, which are then averaged by a server. The authors claim a finite-time bound on the average squared norm of the mixture-environment policy gradient, of order O(1/sqrt(NK)) plus terms depending on environment heterogeneity, and a linear speedup in the number of agents. The proofs rely on an average-MSPBE formulation of the critic, a contraction property of the averaged TD gradient, and Lipschitz continuity of the critic fixed point, and the analysis uses a five-term decomposition of the actor gradient bias. Experiments on Lunar Lander and Cartpole compare SFAC with A3C.","tokens_in":29532,"tokens_out":6839,"duration_ms":54162,"significance":"If the main theorem were correct, this would be the first finite-time convergence result for federated actor-critic with heterogeneous environments, Markovian sampling, and multiple local updates, and the claimed linear speedup would be both theoretically and practically notable. The algorithm is clearly described, and the proof strategy of decomposing the gradient bias into five terms in Eq. (12) is a reasonable way to address the two-level coupling. However, the central claim is not supported by the proof as written: the key condition on the averaged TD gradient (Assumption 4) is introduced only in the supplementary, and its only justification (Lemma 1) is invalid. Because Proposition 1, Theorem 1, and the smoothness lemmas all depend on this condition, the main convergence result remains unproven.","major_comments":[{"comment":"The construction of the averaged Markov chain in Lemma 1 is invalid. With D* = (1/N) sum_i D_i and P* = (sum_i D_i)^{-1} sum_i D_i P_i, one obtains D*P* = (1/N) sum_i D_i P_i, which is not equal to D* unless sum_i D_i P_i = sum_i D_i. Thus D* is not the stationary distribution of P* in general, so the identity g(omega) = Phi^T D* R* + Phi^T D* (gamma P* - I) Phi omega is not justified by any sampling process. Consequently, Assumption 4, which asserts that the averaged gradient satisfies <g(omega), omega - omega*> <= -(lambda/2)||omega - omega*||^2, is an unverified ad hoc condition. Since Proposition 1 and Theorem 1 depend on lambda from Assumption 4, this gap is load-bearing and not a matter of constant tuning.","section":"Supplementary Materials, Lemma 1 and Assumption 4"},{"comment":"Proposition 1 is stated under Assumptions 1 and 3 only, but the proof in the supplementary invokes Assumption 4 to obtain the contraction factor (1 - beta upsilon lambda / 4). Because Assumption 4 is not among the main-text assumptions and is not stated as an assumption of Proposition 1, the proposition is not derived from its stated hypotheses. The same issue affects Theorem 1, which lists only Assumptions 1-3 even though its proof relies on the same condition. Either the theorem must include the new condition explicitly, or a contraction must be proven without it.","section":"Proposition 1 and Theorem 1"},{"comment":"The Lipschitz continuity of the critic fixed point omega*(theta) and its Jacobian in Lemmas 5 and 6 of the supplementary are imported from Chen et al. (2021) under exactly the same Assumption 4. Because Assumption 4 has no valid derivation for the averaged heterogeneous TD problem, the bounds on J21, J22, and J3 in the proof of Theorem 1, which control the cancellation of the critic bias in the actor update, are not established. Thus the final O(1/sqrt(NK)) rate is unsupported.","section":"Proof of Theorem 1, Lemmas 5-6"},{"comment":"The supplementary proof of Proposition 1 ends with a recursion of the form E||omega_t - omega*||^2 <= (1 - lambda beta bar-upsilon / 4)^t E||omega_0 - omega*||^2 + C1 beta^3 + C2 chi^2 + C3 beta / N + C4 beta, whereas the main text states a one-step contraction with error terms C1 beta^4 + C2 beta^3 + C3 beta^2 / N + C4 beta^2. The powers of beta do not match, and no mapping from the supplementary constants to the stated C1-C4 is given. As a result, the supplementary material does not establish Proposition 1 as stated in the main text.","section":"Supplementary Materials, Proof of Proposition 1"}],"minor_comments":[{"comment":"The label 'decent direction' should be 'descent direction'.","section":"Eq. (13)"},{"comment":"The comparison baseline is described as 'A3C (Shen et al. 2023)', which is ambiguous because Shen et al. is an analysis of asynchronous advantage actor-critic rather than the original A3C algorithm; the experimental setup should clarify which implementation and variant is used.","section":"Experiments"},{"comment":"The figures would benefit from full axis labels and error bars; the current captions are minimal and do not state the number of seeds or the variance across runs.","section":"Figures 1-2"}],"recommendation":"reject","confidential_remarks":"The stress-test concern is valid: the averaged-chain construction in Lemma 1 is mathematically incorrect, so the central theorem rests on an unverified assumption. This is a reject-level issue rather than a revision-level one, because the proof of the core result cannot be repaired locally without either proving a genuinely new spectral condition for averaged heterogeneous TD problems or changing the algorithm and the claimed result. The paper may be worth resubmitting if the authors can supply such a proof, but the current manuscript does not provide it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper takes on a genuine gap: no one has a finite-time convergence analysis for federated single-loop actor-critic with heterogeneous environments and Markovian sampling. The algorithm design is thoughtful, especially the two-level federation and the attempt to cancel the critic bias in the actor update. The mixture-environment objective is a meaningful improvement over the virtual-environment construction used in prior FRL work. If the main theorem were correct, it would be the first result of its kind, and the error decomposition in the proof sketch is a sensible way to organize the argument.\n\nThe problem is in the critic contraction. Assumption 4, the strong-convexity-like condition on the averaged MSPBE gradient, appears only in the supplementary material and is not listed among the main-text assumptions. Lemma 1, which is supposed to justify it, is not correct as stated. The constructed P* does not have D* as its stationary distribution in general, so the averaged TD gradient is not literally the expected TD gradient of a single Markov chain with that transition matrix. The algebraic identity g(ω) = Φ^T D*R* + Φ^T D*(γP* - I)Φω is valid, but it does not by itself give the spectral condition Assumption 4. That condition may well be provable directly from per-agent spectral properties and positive definiteness of the average feature covariance, but the paper does not provide that argument. So the proof has a load-bearing gap, not just a cosmetic omission.\n\nThere are also smaller issues. The variance term in Theorem 1 does not shrink with N, so the linear-speedup claim applies only to the optimization term, not the total error; the abstract overstates this. The experiments have no code, no error bars, and only a loose comparison to A3C. These are minor next to the Assumption 4 problem.\n\nWho is this for? The theoretical federated RL community. The paper deserves a serious referee because the question is important and the flaw is likely fixable. I would not cite the main theorem as it stands, but I would want to see a revised version with a correct proof of Assumption 4.\n\nRecommendation: send it to review if you can find a referee willing to check the critic contraction carefully; expect major revision. If you want to make a quick decision, desk reject with an invitation to resubmit once Assumption 4 is properly established.","headline":"A real gap in the FRL literature tackled with a sensible algorithm, but the critic contraction proof rests on an unproven and likely mis-stated Lemma 1, so the main theorem is not yet supported.","tokens_in":30048,"tokens_out":6541,"would_cite":false,"duration_ms":54739,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Single-loop federated actor-critic provably converges to a near-stationary point of the average return, with a linear speedup in the number of agents and a residual error proportional to environment heterogeneity.","keywords":["federated reinforcement learning","actor-critic","heterogeneous environments","Markovian sampling","linear speedup","temporal difference learning","mixture environment","convergence analysis"],"falsifier":"Take a small two-agent tabular MDP with different transition kernels and features, form $\\phi(s)$, $D_i$, $P_i$, and test Assumption 4 by computing the eigenvalues of the averaged TD matrix $\\Phi^T D_*(\\gamma P_*-I)\\Phi$; if any eigenvalue has positive real part, the averaged gradient is not attracting and the recursion in Proposition 1 would not contract. Running FedC on that instance and observing the critic error fail to decrease at the predicted rate would falsify the theorem's reach as stated, unless the instance violates an assumed condition. A synthetic search over random heterogeneous MDPs for such an eigenvalue would settle whether Assumption 4 follows from Assumptions 1–3.","tokens_in":28910,"feed_emoji":"🤖","tokens_out":13934,"duration_ms":101692,"temperature":0.7,"pith_summary":"This paper proposes SFAC, an algorithm in which many agents, each interacting with a different environment, jointly learn one policy by federating both their value-function estimates and their policy-gradient estimates. The central claim is a finite-time convergence guarantee: after $K$ outer rounds, the average squared norm of the gradient of the average-return objective $J(\\theta) = \\frac{1}{N}\\sum_i J_i(\\theta)$ is bounded by $O(1/\\sqrt{NK})$ plus terms controlled by environment heterogeneity, actor sample size, and critic approximation error. When environments are identical, the leading term yields a linear speedup in the number of agents $N$. The authors present this as the first finite-time convergence result for single-loop federated actor-critic under heterogeneous environments with Markovian sampling and multiple local critic updates. The result matters because it separates what federation buys (a faster shared policy) from what heterogeneity costs (an irreducible bias), giving a concrete target for algorithm design.","feed_headline":"Federated actor-critic provably reaches a near-stationary point","feed_subtitle":"More agents mean faster learning: error drops as 1/√(NK), leaving only heterogeneity bias.","key_machinery":"Two objects carry the argument. The first is the average-MSPBE gradient identity. For linear value approximation with features $\\phi(s)$, the global critic objective is the average mean-squared projected Bellman error, and its gradient can be written as a single TD gradient $g(\\omega) = \\frac{1}{N}\\sum_i \\left[\\Phi^T D_i(\\gamma P_i-I)\\Phi\\omega + \\Phi^T D_i R_i\\right] = \\Phi^T D_*(\\gamma P_*-I)\\Phi\\omega + \\Phi^T D_* R_*$, with $D_*$ and $P_*$ weighted averages of the agents' stationary distributions and transition kernels (Lemma 1). This identity makes a biasing term in the inner-product decomposition (13) cancel, removing a non-vanishing bias that earlier federated TD analyses had to carry. The second is the five-term decomposition of the actor's gradient bias: lower-level critic error, local Markovian variance, gradient heterogeneity, smoothness of the drifting critic fixed point, and critic approximation error. These are assembled into the Lyapunov function $V_k = -J(\\theta_k) + \\|\\omega_k-\\omega^*_k\\|^2$, whose decrease over one outer round yields the theorem.","core_discovery":"The paper establishes that SFAC converges to a near-stationary point of the mixture-environment objective. Theorem 1 states that with step sizes $\\alpha_k, \\beta_k = O(\\sqrt{N/K})$, the average squared gradient norm satisfies $\\frac{1}{K}\\sum_{k=1}^K E\\|\\nabla J(\\theta_k)\\|^2 \\le O(1/\\sqrt{NK}) + O(H^2/M) + O(\\kappa^2) + O(\\xi_{\\mathrm{critic}})$, where $H^2/M$ is the variance from Markovian mini-batch gradient estimates, $\\kappa^2$ is the environment-heterogeneity constant from Assumption 1, and $\\xi_{\\mathrm{critic}}$ is the critics' approximation error. The critic's lower-level error is not removed by decoupling: the proof tracks the drift of the critic fixed point as the policy changes and shows the biased term cancels in the actor update through the global gradient of the average mean-squared projected Bellman error. Proposition 1 gives the per-round contraction of the federated critic, with a noise term scaled by $1/N$ that produces the linear speedup. The residual bias is asymptotically proportional to the heterogeneity constant $\\kappa^2$ and vanishes in identical environments.","pith_inferences":["Editorial inference: the same five-term bias decomposition should carry over to natural-actor-critic or entropy-regularized variants, because the cancellation of the lower-level bias uses only the average-MSPBE gradient identity, not the specific policy parameterization.","Editorial inference: client selection in federated reinforcement learning could be guided by the heterogeneity constant $\\kappa^2$; the theorem suggests that admitting an environment far from the current average directly raises the asymptotic bias, so distributional distance of transition and reward models should inform participation.","Editorial inference: a sharper version of the bound may exist in which $\\kappa^2$ is replaced by the sum of per-agent deviations from the mean TD operator, which would predict how the bias interpolates between the homogeneous case and the fully heterogeneous case.","Editorial inference: whether the $O(\\kappa^2)$ floor is tight can be tested numerically by comparing the theorem's heterogeneity term with the empirical eigenvalue spread of the averaged TD matrix on random MDPs; a loose $\\kappa^2$ would suggest the heterogeneity bias is an artifact of Assumption 1's coarse constant."],"forward_implications":["With identical environments ($\\kappa=0$), reaching gradient norm $\\epsilon$ needs $K \\approx 1/(N\\epsilon^2)$ outer rounds, so $N$ agents reduce the number of rounds by a factor $N$.","In heterogeneous environments, the asymptotic error floor $O(\\kappa^2)+O(\\xi_{\\mathrm{critic}})$ means that no amount of federation can drive the shared policy to an exact stationary point; the floor is set by environment differences and critic approximation power.","The bound holds under Markovian sampling directly: the $H^2/M$ variance term is controlled by the actor mini-batch size $M$, and the proof does not require independent and identically distributed samples.","The single-loop structure removes the double-loop requirement of an accurately converged critic at each policy; instead, the critic's memory carries over, and the bias from the moving critic target is bounded by the analysis."],"supporting_citations":[{"why":"Supplies the finite-time TD learning analysis and geometric-mixing assumptions that the critic's contraction bound builds on.","marker":"Bhandari, Russo, and Singal 2018"},{"why":"Provides the federated TD linear-speedup analysis under Markovian sampling whose per-round decomposition Proposition 1 adapts.","marker":"Khodadadian et al. 2022"},{"why":"Establishes the heterogeneous federated TD setting; the non-vanishing bias that SFAC cancels is the one left over in this work.","marker":"Wang et al. 2023"},{"why":"Supplies the bilevel actor-critic analysis and the smoothness lemmas for the critic fixed point (Lemmas 5 and 6) used in Theorem 1.","marker":"Chen, Sun, and Yin 2021"},{"why":"Supplies the single-loop actor-critic sample-complexity framework, Assumption 4's strong-attractivity condition, and Lemma 7's variance bound.","marker":"Xu, Wang, and Liang 2020"},{"why":"Provides the A3C baseline, the experimental comparison target, and the argument that actor and critic need separate Markov chains.","marker":"Shen et al. 2023"},{"why":"Supplies the Markovian bias-of-expectation estimates used as ingredients in the federated critic's error recursion.","marker":"Sun et al. 2020"},{"why":"Provides the finite-time linear stochastic approximation bounds used to control TD gradient noise.","marker":"Srikant and Ying 2019"}],"fun_headline_variants":["Federated actor-critic converges with linear speed-up","Heterogeneity limits federated actor-critic convergence","SFAC: near-stationary with federated speed-up","Federated RL: linear speed-up, but heterogeneity bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise, introduced only in the supplementary material as Assumption 4, is that the averaged critic objective satisfies $\\langle g(\\omega), \\omega-\\omega^*\\rangle \\le -\\frac{\\lambda}{2}\\|\\omega-\\omega^*\\|^2$ for some $\\lambda>0$—in plain terms, it attracts every iterate toward the shared optimum with a force proportional to the distance from it—so if the averaged heterogeneous TD problem lacks that attraction, the critic contraction in Proposition 1 and the smoothness lemmas used in Theorem 1 fail, and the convergence bound collapses.","fun_headline_variants_meta":{"raw":{"variants":["Federated actor-critic converges with linear speed-up","Heterogeneity limits federated actor-critic convergence","SFAC: near-stationary with federated speed-up","Federated RL: linear speed-up, but heterogeneity bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1425,"prompt_tokens":1018,"completion_tokens":407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":339}},"tokens_in":634,"tokens_out":407,"duration_ms":5417,"temperature":1.0,"reasoning_tokens":339,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:08:01.565866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small two-agent tabular MDP with different transition kernels and features, form $\\phi(s)$, $D_i$, $P_i$, and test Assumption 4 by computing the eigenvalues of the averaged TD matrix $\\Phi^T D_*(\\gamma P_*-I)\\Phi$; if any eigenvalue has positive real part, the averaged gradient is not attracting and the recursion in Proposition 1 would not contract. Running FedC on that instance and observing the critic error fail to decrease at the predicted rate would falsify the theorem's reach as stated, unless the instance violates an assumed condition. A synthetic search over random heterogeneous MDPs for such an eigenvalue would settle whether Assumption 4 follows from Assumptions 1–3.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the finite-time TD learning analysis and geometric-mixing assumptions that the critic's contraction bound builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the federated TD linear-speedup analysis under Markovian sampling whose per-round decomposition Proposition 1 adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the bilevel actor-critic analysis and the smoothness lemmas for the critic fixed point (Lemmas 5 and 6) used in Theorem 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the single-loop actor-critic sample-complexity framework, Assumption 4's strong-attractivity condition, and Lemma 7's variance bound."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the A3C baseline, the experimental comparison target, and the argument that actor and critic need separate Markov chains."},{"cited_title":"B.; Yang, Q.; and Yang, Z","cited_arxiv_id":null,"evidence_quote":"Supplies the Markovian bias-of-expectation estimates used as ingredients in the federated critic's error recursion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the finite-time linear stochastic approximation bounds used to control TD gradient noise."}],"review_version":1}