{"id":"6840a122-840e-4a0c-bccc-108baeeb16ad","arxiv_id":"2507.14487","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"FedRQ adds a robustness term to federated Q-learning and claims convergence to an optimal worst-case policy over heterogeneous local environments, but the proof has a reversed inequality.","lead":"Federated reinforcement learning agents usually assume all local environments are identical. This paper adds a worst-case robustness term to federated Q-learning, targeting policies that survive heterogeneous and perturbed environments, and it tests the idea on standard control benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 2's proof reverses the monotonicity of the learning rate, so the local-drift bound is unproven and Theorem 1's convergence guarantee does not follow as written.","rationale":"We read the paper as aiming to prove that FedRQ's global Q function converges to the optimal robust value for the covering set P_omega. The proof of Theorem 1 depends critically on Lemma 2's bound on the local disagreement term. We verified the algebra of the induction in (37)-(42) and found it internally consistent; the only broken link is Lemma 2's step (c), where the monotonicity of lambda_t is reversed. This is a genuine proof error that blocks the central theoretical claim, so the concern is load-bearing. We note that the claimed bound might still be true with a different argument, which is why we propose an analytical check rather than declaring the theorem false outright. However, as submitted, the proof does not establish the result. The reader's weakest_assumption, Assumption 1, is a real restriction on the framework, but it is not the most immediate blocker: the Lemma 2 error breaks the proof even when Assumption 1 holds. We therefore keep the REJECT verdict; the paper needs a corrected proof before the convergence guarantee can be accepted.","tokens_in":19455,"tokens_out":14565,"duration_ms":140445,"concrete_test":"Analytically recompute the local-drift sum in Lemma 2: fix E >= 2, gamma in [0.2,1), and any integer t with last sync t0 in [max(1,t-E+1), t-1]. Evaluate S = sum_{t'=t0}^{t-1} 2/((1-gamma)(t'+E)) and compare it with the claimed bound 2*lambda_t*(E-1) = 4(E-1)/((1-gamma)(t+E)). If any feasible (t, t0, E, gamma) has S exceeding the bound, then Lemma 2 is false and Theorem 1's rate (30) is not merely unproven but incorrect. If S never exceeds it, the bound is true and the proof can be repaired; in that case, re-check whether the corrected derivation still yields the exact constants in the induction (39)-(42). A numerical sweep over t in [1, 10^4] and E in [2, 10^2] would quickly indicate which outcome holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.3, Lemma 2 claims the local disagreement bound (44). In its proof, step (c) bounds sum_{t'=t0}^{t-1} lambda_{t'} <= 4*lambda_t*(E-1)/(1-gamma) because 'lambda_{t'} <= lambda_t for t' in [t0, t-1]'. This is backwards: lambda_t = 2/((1-gamma)(t+E)) is strictly decreasing in t, so for every t' < t we have lambda_{t'} >= lambda_t. Since t0 can be up to E-1 steps before t, the sum of learning rates can be larger than the claimed multiple of lambda_t by a factor roughly (t+E)/(t+1). Consequently, inequality (44) is not established, and the recursive inequality (36) in Theorem 1, which uses Lemma 2 through the term (gamma*lambda_t/K)*sum_k ||Q^k_t - bar Q_t||_infinity, does not follow. This is independent of Assumption 1: even when all local supports coincide so that identity (33) holds, the convergence proof is incomplete. A correct treatment would need to sum the actual decreasing sequence; a first-pass estimate gives sum_{t'=t0}^{t-1} lambda_{t'} <= (E-1)*lambda_{t0} = 2(E-1)/((1-gamma)(t0+E)), which is not obviously bounded by the claimed 2*lambda_t*(E-1). Thus the central advertised guarantee is unsupported as written.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies federated reinforcement learning in which local environments have different transition dynamics. It defines a worst-case global objective over a covering set of transition probabilities, proposes a tabular algorithm FedRQ based on robust local Q-learning updates with periodic global averaging, and claims a finite-time convergence bound for the averaged Q-function to the optimal robust Q-function. The paper then extends the robust update to function approximation via expectile networks, yielding FedRDQN and FedRDDPG for discrete and continuous action spaces, and reports experiments against QAvg-based baselines on several Gym environments.","tokens_in":19809,"tokens_out":19727,"duration_ms":178509,"significance":"If Theorem 1 were fully established, the paper would provide an interesting first finite-time guarantee for worst-case federated RL over heterogeneous and perturbed environments, complementing the known suboptimality of QAvg. The structural observation that averaging the local robust Bellman operators yields the global robust operator under a support-consistency assumption is elegant and is a genuine strength. The empirical study is also reasonably broad, reporting both average and worst-case metrics across eight environments. However, the advertised theoretical guarantee is currently not proven as written: the proof of Lemma 2 contains a monotonicity error, and the tabular update analyzed in Theorem 1 is a model-based synchronous operator rather than a sample-based reinforcement learning algorithm. These issues are substantive but appear repairable.","major_comments":[{"comment":"Step (c) of the proof of Lemma 2 asserts λ_{t'} ≤ λ_t for t' ∈ [t0, t−1]. Since λ_t = 2/((1−γ)(t+E)) is strictly decreasing in t, the inequality is reversed: for t' < t one has λ_{t'} ≥ λ_t. Consequently the bound on ∑_{t'=t0}^{t−1} λ_{t'} does not follow, and inequality (44) is not established as written. Because Lemma 2 supplies the drift term in the recursive inequality (36), Theorem 1's bound (30) is unsupported as written. The claim appears salvageable: a correct proof can use λ_{t'} ≤ λ_{t0}, then bound λ_{t0}/λ_t = (t+E)/(t0+E) ≤ (t+E)/(t+1) ≤ 2 for t ≥ E−1, with the finitely many small t cases handled separately. The proof still needs to be rewritten.","section":"§3.3, Lemma 2"},{"comment":"The robust local update in (25) and its operator form (34) use the full transition kernel P_k and the robust operator T_k; no transition sampling is introduced in the tabular section. Thus Theorem 1 proves convergence of a synchronous, model-based value-iteration scheme, not of a sample-based reinforcement learning algorithm. The deep extensions in Section 4 are heuristic and are not covered by the theorem. The paper should either present a stochastic tabular update and analyze it under standard stochastic-approximation conditions, or explicitly state that Theorem 1 is an analysis of an idealized federated robust value iteration, with the sample-based version deferred to future work.","section":"§3.2, Eqs. (25) and (34)"}],"minor_comments":[{"comment":"In the proof, the bound 'Q^k_t(s,a) ≤ 1/(1−r)' should read 'Q^k_t(s,a) ≤ 1/(1−γ)', since rewards are bounded by 1 and the discount factor is γ.","section":"§3.3, Lemma 2"},{"comment":"Inequality (17) is ambiguous as printed; it should be written as (ω P̄(s'|s,a) + P_k(s'|s,a) − P̄(s'|s,a)) / P̄(s'|s,a) ≥ 0, or equivalently with explicit parentheses.","section":"§3.1, Eq. (17)"},{"comment":"The quantity κ(s,a) in (18) is never formally defined; a precise definition is needed.","section":"§3.1, Eq. (18)"},{"comment":"Assumption 1 is restrictive because it excludes environments with different sets of reachable next states from the same state. The paper should discuss whether the benchmark environments satisfy this assumption and what happens to identity (33) when only a subset of local supports agree.","section":"§3.3, Assumption 1"},{"comment":"The text states that FedRDDPG is compared against both DDPGAvg(1) and DDPGAvg(2), but Table 2 reports only DDPGAvg(2); the DDPGAvg(1) results should be included or the text revised.","section":"§5, Table 2"},{"comment":"In line 8 of both algorithms, the replay-buffer append operation is missing a closing parenthesis: it should read D_k ← D_k ∪ {(s, a, r, s')}.","section":"Algorithms 1 and 2"},{"comment":"The moving-average filter used to smooth the learning curves is not specified; the window length should be reported for reproducibility.","section":"§5.1"},{"comment":"The abstract describes 'asymptotic convergence', while Theorem 1 provides a non-asymptotic finite-time bound; these terminologies should be aligned.","section":"Abstract and Theorem 1"}],"recommendation":"major_revision","confidential_remarks":"The proof gap in Lemma 2 is real but appears repairable without changing the claimed rate, so I do not recommend rejection on that ground alone. The larger concern is that the tabular algorithm analyzed in Theorem 1 is not a sample-based RL algorithm, which weakens the connection between the theory and the deep RL extensions. I would send the paper back for a major revision with the request to repair Lemma 2 and to state clearly what the theorem does and does not cover."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper combines robust RL with federated averaging, and the idea is not crazy. But the main theorem is broken as written. Lemma 2's proof uses λ_{t'} ≤ λ_t for t' ≤ t-1, which is backwards because λ_t decreases in t. So the drift bound doesn't follow, and without it the recursive inequality (36) collapses. This is not a minor typo: the advertised convergence guarantee is the paper's reason for existence.\n\nWhat it does well: The robust global objective (9) is a natural way to handle environment heterogeneity, and the expectile network trick for approximating the min over neighboring states is sensible for continuous state spaces. The experiments are reasonably extensive, comparing against QAvg-based baselines across eight discrete and eight continuous tasks, and the proposed methods consistently win on both average and worst-case reward. That is real evidence that something in the robust local update helps, independent of the theory.\n\nThe soft spots: Besides the proof error, Assumption 1 (identical transition supports) is restrictive and needed for identity (33); without it, the local robust operators don't average to the global robust Bellman operator. The authors also don't report the robustness levels used in the experiments, only showing them in figure legends, and no code is released. So the empirical section is incomplete by today's standards. Also, the covering set requires knowing the local transition kernels well enough to pick omega, which slightly undercuts the model-free claim.\n\nOverall: The central theoretical claim is unsupported, and the empirical work, while encouraging, doesn't substitute for a correct convergence proof. But the flaw is a specific inequality that might be repairable; I wouldn't desk reject this. I'd send it to reviewers with the expectation that the authors either fix the proof or substantially weaken the claim. If they can't, the paper should be rejected. The right audience is the FRL and robust-RL community, and a reading group could use it as a cautionary tale about checking monotonicity in learning-rate arguments.","headline":"The robust-FRL idea is plausible, but Lemma 2's reversed learning-rate inequality breaks Theorem 1's convergence proof, leaving the paper's main claim unsupported.","tokens_in":20299,"tokens_out":3493,"would_cite":false,"duration_ms":39110,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated Q-learning hits worst-case optimum with a proven rate","keywords":["federated reinforcement learning","environment heterogeneity","robust reinforcement learning","Q-learning","worst-case objective","expectile regression","deep reinforcement learning","covering set"],"falsifier":"Compute the gap $\\|\\bar Q_t - Q^\\star_R\\|_\\infty$ for two tabular MDPs with identical state and action spaces but different supports, for instance where from state A agent 1 can transition only back to A while agent 2 can also transition to B. Under the paper's assumptions the gap should decay as $O(1/t)$; observing that the gap stagnates, or directly verifying that $\\bar T \\neq \\frac{1}{K}\\sum_k T_k$ for such an instance, would falsify the theorem.","tokens_in":1848,"feed_emoji":"🤖","tokens_out":3178,"duration_ms":108456,"temperature":0.7,"pith_summary":"Federated RL with heterogeneous local environments often averages updates, which works well only when local dynamics are similar. This paper claims that FedRQ, a tabular algorithm with a worst-case-aware local update, makes the global Q function converge, at an explicit rate, to the optimal policy of a robust objective defined over all local environments and their plausible perturbations. The robust objective is a worst-case value over a covering set of transition kernels, so the learned policy performs consistently even in environments not seen during training. The paper also extends the same principle to continuous state spaces using expectile loss, yielding deep variants that outperform averaging-only baselines in the reported experiments.","feed_headline":"Federated Q-learning hits worst-case optimum with a proven rate","feed_subtitle":"FedRQ's worst-case local update yields a proven convergence rate for heterogeneous federated RL.","key_machinery":"The load-bearing object is the pair of robust Bellman operators: the local operator $T_k$ used in each agent's update and the global operator $\\bar T$ whose fixed point is the target $Q^\\star_R$. The identity $\\bar T = \\frac{1}{K}\\sum_{k=1}^K T_k$, valid under Assumption 1, is what lets periodic averaging of local Q functions simulate one step of a global contraction mapping. The covering set $\\mathcal{P}_\\omega$, defined as mixtures of the average transition kernel $\\bar P$ with arbitrary kernels supported on the shared neighbor set $\\mathcal{N}^s$, specifies the ambiguity class over which worst-case performance is measured. In continuous spaces, the minimization inside $T_k$ is approximated by an expectile network trained with asymmetric squared loss, which learns the lower-tail minimum of the max-over-actions value across next states. The drift of local Q functions between synchronization rounds is bounded in Lemma 2, and that drift bound feeds directly into the final convergence rate.","core_discovery":"Under Assumption 1, which requires that every local environment share the same set of reachable next states from each state, FedRQ's global Q function, defined as $\\bar Q_t = \\frac{1}{K}\\sum_{k=1}^K Q^k_t$, converges to $Q^\\star_R$, the fixed point of the global robust Bellman operator $\\bar T$, with the explicit bound $\\|\\bar Q_t - Q^\\star_R\\|_\\infty \\le \\frac{16\\gamma(E-1)}{(1-\\gamma)^3(t+E)}$. The robust operator is built from a covering set $\\mathcal{P}_\\omega$ of plausible transition kernels, and the key identity $\\bar T = \\frac{1}{K}\\sum_{k=1}^K T_k$ holds exactly under Assumption 1. Each agent runs a Q-learning update augmented with the extra term $\\gamma\\omega \\min_{s'\\in\\mathcal{N}^s}\\max_{a'} Q^k_t(s',a')$, which hedges against the worst next state in the local support; averaging the local Q functions at the server then implements one step of the global robust contraction. QAvg is recovered as the special case $\\omega=0$. For continuous state spaces, the hard minimization over next states is replaced by an expectile-based degree network trained on local replay buffers, giving the deep algorithms FedRDQN and FedRDDPG.","pith_inferences":["If Assumption 1 fails, because some local environments can reach a state that others cannot, the identity $\\bar T = \\frac{1}{K}\\sum_k T_k$ breaks and the averaging mechanism no longer implements a single contraction. A tabular experiment that varies the supports of the local transition kernels would directly reveal how the convergence gap degrades and would mark the boundary of the theorem.","The proof chooses $\\omega$ from the true heterogeneity scale $\\kappa(s,a)$, which is not known before training in practice. Adapting $\\omega$ online from observed transition statistics is a natural extension, but its guarantee is not covered by the paper's analysis.","In continuous state spaces, replacing $\\min_{s'\\in\\mathcal{N}^s}\\max_a Q(s',a)$ with an expectile network introduces approximation error that Theorem 1 does not quantify. Comparing FedRDQN against an exact tabular FedRQ on a discretized version of the same environment would separate the algorithm's robust mechanism from the approximator's bias.","The worst-case objective is one point on a spectrum; a mixture of average and worst-case performance, or another risk-sensitive criterion, would sit naturally on the same Bellman-operator machinery and could be tested without changing the federated update structure."],"forward_implications":["For any robustness level $\\omega$ satisfying the covering condition (20), the policy extracted from $\\bar Q_t$ is asymptotically optimal for the worst case over $\\mathcal{P}_\\omega$, which includes all $K$ local environments and their allowed perturbations.","The explicit rate shows how the synchronization period $E$ trades off against convergence: the drift term grows with $E-1$, while the factor $(1-\\gamma)^{-3}$ quantifies the sensitivity to discounting, so the result positions FedRQ against standard robust Q-learning guarantees.","With $\\omega=0$, FedRQ reduces to QAvg, so the convergence theorem encompasses the known average-case federated algorithm as a special case rather than contradicting it.","Because only local Q functions, or their network parameters, are shared in the global update, the worst-case optimality guarantee is achieved under the federated constraint that raw trajectories remain private.","The deep extensions FedRDQN and FedRDDPG apply the same worst-case hedging principle to continuous state spaces, with the expectile network serving as the sample-based surrogate for the hard minimum over next states."],"supporting_citations":[{"why":"introduces the FRL-EH framework and the QAvg baseline whose average objective FedRQ generalizes with a worst-case covering set.","marker":"[9]"},{"why":"supplies the robust Q-learning operator and the gamma-contraction property invoked for each local operator T_k and for the fixed point Q*_R.","marker":"[39]"},{"why":"provides the expectile asymmetric least-squares loss used to approximate the minimum over next states in continuous environments.","marker":"[40]"},{"why":"is the standard Q-learning update that the robust local update modifies with the worst-case next-state term.","marker":"[16]"},{"why":"is the DQN architecture extended into FedRDQN for discrete-action continuous-state environments.","marker":"[18]"},{"why":"is the DDPG architecture extended into FedRDDPG for continuous action spaces.","marker":"[21]"}],"fun_headline_variants":["Robust federated Q-learning provably converges to optimal","FedRQ: Federated RL that handles heterogeneous environments","Averaging local Q-tables wins in heterogeneous federated RL","Expectile loss enables deep federated RL for continuous states","Provable optimality for federated RL across diverse settings"],"cache_read_input_tokens":22400,"weakest_assumption_plain":"The theorem assumes that from every state, all K local environments can reach exactly the same set of next states; if any environment has a reachable state another lacks, the arithmetic-mean identity that makes FedRQ's global update a contraction no longer holds.","fun_headline_variants_meta":{"raw":{"variants":["Robust federated Q-learning provably converges to optimal","FedRQ: Federated RL that handles heterogeneous environments","Averaging local Q-tables wins in heterogeneous federated RL","Expectile loss enables deep federated RL for continuous states","Provable optimality for federated RL across diverse settings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1768,"prompt_tokens":1019,"completion_tokens":749,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":667}},"tokens_in":635,"tokens_out":749,"duration_ms":7926,"temperature":1.0,"reasoning_tokens":667,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:55:34.311820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the gap $\\|\\bar Q_t - Q^\\star_R\\|_\\infty$ for two tabular MDPs with identical state and action spaces but different supports, for instance where from state A agent 1 can transition only back to A while agent 2 can also transition to B. Under the paper's assumptions the gap should decay as $O(1/t)$; observing that the gap stagnates, or directly verifying that $\\bar T \\neq \\frac{1}{K}\\sum_k T_k$ for such an instance, would falsify the theorem.","supporting_citations":[{"cited_title":"Federated reinforcement learning with environment heterogeneity,","cited_arxiv_id":null,"evidence_quote":"introduces the FRL-EH framework and the QAvg baseline whose average objective FedRQ generalizes with a worst-case covering set."},{"cited_title":"On practical robust reinforcement learn- ing: Adjacent uncertainty set and double-agent algorithm,","cited_arxiv_id":null,"evidence_quote":"supplies the robust Q-learning operator and the gamma-contraction property invoked for each local operator T_k and for the fixed point Q*_R."},{"cited_title":"Asymmetric least squares estima- tion and testing,","cited_arxiv_id":null,"evidence_quote":"provides the expectile asymmetric least-squares loss used to approximate the minimum over next states in continuous environments."},{"cited_title":"Q-learning,","cited_arxiv_id":null,"evidence_quote":"is the standard Q-learning update that the robust local update modifies with the worst-case next-state term."}],"review_version":1}