{"id":"6b8cd915-168b-4984-be9e-56e98468125c","arxiv_id":"2412.20946","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"In a simplified net-zero microgrid, untuned federated TRPO learns useful battery policies, but tuned PPO gets much closer to the known optimal policy; personal encoding and feature grouping sometimes shrink the gap.","lead":"The paper trains two reinforcement learning algorithms, PPO and TRPO, to control home batteries in a simulated net-zero microgrid, with multiple buildings learning together while sending only model updates rather than raw data. A reader might care because battery control is a bottleneck for using rooftop solar at scale, and this work studies whether privacy-preserving collaborative training can improve it.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Tables 1 and 2 contradict the headline claim: in both environments the best TRPO evaluation reward (-0.47 and -0.46) is roughly three times worse than the best PPO (-0.14 and -0.15), so 'on par with tuned PPO' is not supported.","rationale":"The reader's verdict is REJECT, and I agree that the paper should be rejected or substantially revised. However, the most load-bearing concern is not the federated-equivalence assumption highlighted in the reader's weakest_assumption, although that is also a real issue. The decisive problem is that the paper's headline claim is contradicted by its own experimental data. The abstract says Federated TRPO is 'comparable with state-of-the-art federated RL methodologies,' and the conclusion says it 'performed on par with a tuned federated PPO.' Tables 1 and 2 show TRPO evaluation rewards around -0.47 and -0.46, while PPO achieves -0.14 and -0.15, i.e., roughly a threefold gap in a metric where zero is optimal. The paper itself admits in Section 4.2 that tuned PPO was superior. This is an internal inconsistency, not an external disagreement, and it directly undermines the central contribution. The concrete test of re-running the released code with the same seeds would settle whether the table values are reproducible; if they are, the abstract and conclusion must be rewritten to claim only that untuned TRPO learns a policy that sometimes beats a no-battery baseline, not that it matches tuned PPO. The federated-equivalence concern, while important for the title and method framing, would remain even if TRPO performed well, but it is secondary to the fact that the paper's own numbers fail to support its stated parity claim. Therefore, the reader's REJECT verdict stands, and no change to the final verdict is needed.","tokens_in":13430,"tokens_out":3023,"duration_ms":29288,"concrete_test":"Reproduce the two-building experiments from the released code for the best configurations (PPO pe gf and TRPO pe) using the same five seeds and evaluation protocol. Compute per-seed evaluation rewards and report the mean and standard deviation. If the reproduced mean gap remains approximately 0.33 in favor of PPO, the abstract and conclusion claims are unsupported; if the gap collapses or the confidence intervals overlap substantially, the original claim could be reconsidered after adding proper error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central advertised result is that untuned Federated TRPO is comparable with a tuned federated PPO. The paper's own quantitative results directly contradict this. In Table 1 (two-building environment), the best TRPO evaluation reward is -0.47064 (pe) while the best PPO evaluation reward is -0.14021 (pe gf), a gap of about 0.33 on a scale where the known optimal reward is 0. In Table 2 (five-building environment), the best TRPO evaluation reward is -0.45768 (base) versus -0.14662 (pe gf) for PPO, a similar gap. Furthermore, Section 4.2 explicitly states: 'the performance of the tuned PPO was superior to that of the TRPO, and the reward function value demonstrated it.' The conclusion's statement that 'Federated TRPO performed on par with a tuned federated PPO' is therefore internally inconsistent with both the tables and the paper's own narrative. No error bars or standard deviations are reported for these means, so there is no statistical basis for claiming parity even with high variance. The abstract also claims comparability with 'state-of-the-art federated RL methodologies,' but no comparison to any external federated RL baseline is provided; the only comparison is tuned PPO versus untuned TRPO, and the numbers show a large and consistent gap. This is not a matter of external consensus but an internal contradiction in the paper's central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies battery control in net-zero building microgrids using a customized CityLearn environment with synthetically generated data and a known optimal policy. It compares PPO and TRPO under several collaborative configurations (shallow network, personal encoding, grouped features, and shifted data) and reports that an untuned Federated TRPO is comparable with a tuned federated PPO. The authors also contribute a TorchRL wrapper for CityLearn and release code.","tokens_in":13736,"tokens_out":3696,"duration_ms":38776,"significance":"If the central claim were supported, the paper would offer a useful practical lesson: a hyperparameter-free TRPO could match a tuned PPO in a privacy-preserving shared-control setting for building energy management. The paper has concrete strengths: a reproducible synthetic benchmark with a known optimal policy, five-seed experiments, public code, and a modular environment wrapper. However, the headline claim is directly contradicted by the paper's own quantitative tables, and the experiments do not actually implement a federated protocol. The significance of the work as a federated RL contribution is therefore not established.","major_comments":[{"comment":"The central claim that Federated TRPO is comparable with tuned federated PPO is contradicted by the paper's own results. In Table 1, the best TRPO evaluation reward is -0.47064 (pe) while the best PPO evaluation reward is -0.14021 (pe gf); in Table 2, the corresponding values are -0.45768 (base) for TRPO and -0.14662 (pe gf) for PPO. Section 4.2 explicitly states that \"the performance of the tuned PPO was superior to that of the TRPO.\" No standard deviations or statistical tests are reported for these five-seed averages, so the conclusion \"Federated TRPO performed on par with a tuned federated PPO\" has no statistical or numerical basis in the presented data.","section":"Abstract, Section 4.2, Tables 1 and 2, Section 5"},{"comment":"The paper argues that \"passing the stacked observations to a single network has the same effect as doing a FedAvg,\" but this equivalence is not established and is generally false. FedAvg involves multiple local updates between communication rounds, client sampling, and aggregation of model weights; a single forward/backward pass over a concatenated batch of agent observations is a centralized mini-batch update, not a federated protocol with local training steps. Since no communication rounds, partial participation, or privacy mechanism are implemented or analyzed, the claims that the framework is \"privacy-preserving\" and that the results concern \"Federated TRPO\" are not supported by the experiments.","section":"Appendix A.2"},{"comment":"The abstract claims comparability with \"state-of-the-art federated RL methodologies,\" but the paper contains no comparison with any external federated RL baseline. All comparisons are among PPO and TRPO variants within the proposed setup. The conclusion's additional recommendation to warm-start with PPO and switch to TRPO is also not tested anywhere in the experiments.","section":"Abstract and Section 5"}],"minor_comments":[{"comment":"The tables report means over five seeds but no standard deviations or confidence intervals, even though the accompanying figures show substantial variance; please add dispersion measures to the tables.","section":"Tables 1 and 2"},{"comment":"The caption reads \"Values in the table represent the average for 5\" and appears to be missing the word \"seeds.\"","section":"Table 2 caption"},{"comment":"The sentence \"PPO overperforms TRPO significantly, but we highlight that we didn't spend time fine-tuning it\" is ambiguous about whether \"it\" refers to TRPO or PPO; Appendix C indicates PPO was tuned while TRPO was not, so please rephrase.","section":"Section 4.2"},{"comment":"The description of the selling-price margin as \"a margin of 40% of the lowest daily electricity pricing\" is unclear; specify exactly how the selling price is computed from the daily pricing series.","section":"Appendix B"},{"comment":"Figure 1 labels the Personal Encoding block as a local component, but Appendix A.2 states that \"we are not in the process of training a private part of the policy network\" and uses a one-hot encoder. Please clarify what the personal encoding block actually contains and how it relates to split learning.","section":"Section 3.2 and Appendix A.2"}],"recommendation":"reject","confidential_remarks":"The paper contains a useful synthetic benchmark and reproducible experimental scaffolding, but the main advertised result is internally inconsistent with the reported tables, and the federated/privacy framing is not implemented as claimed. A substantial rewrite that reframes the contribution as a centralized multi-agent RL study on a toy microgrid, adds real federated baselines, and aligns the conclusions with the numbers would be needed before this could be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, know this: the paper's advertised result does not survive contact with its own tables. The abstract and conclusion claim untuned Federated TRPO is comparable to tuned PPO, but Tables 1 and 2 show best TRPO eval rewards around -0.47 and -0.46 versus -0.14 and -0.15 for PPO. Section 4.2 even admits 'the performance of the tuned PPO was superior to that of the TRPO.' That is an internal contradiction, not a matter of interpretation.\n\nThe paper still has real value. The custom net-zero CityLearn scenario with a known optimal policy is a useful toy benchmark for studying optimality gaps without confounding from real-data messiness. The personal encoding and feature grouping are sensible split-learning tweaks, and the code release plus TorchRL wrapper make replication easy. The appendix is thorough, including hyperparameters and qualitative policy plots. The authors are honest about the difficulty of learning even 'simple' optimal policies, which is a fair observation.\n\nSoft spots beyond the headline: the federated framing is shaky. Appendix A.2 states that passing stacked observations to a single network 'has the same effect as doing a FedAvg.' That conflates centralized batch training with a federated protocol—no local updates, no communication rounds, no privacy analysis. The results are about collaborative training, not federated learning. Also, the main tables report only means over five seeds, with no standard deviations or significance tests, so 'comparable' has no statistical support. The PPO baseline is tuned while TRPO is untuned, which makes the 'on par' claim even harder to defend.\n\nThis is not a desk reject. It is a well-structured empirical study with a clear, reproducible core, but the claims need substantial revision. My recommendation: send it to peer review, but expect the referee to require either reframing the conclusions to match the data or adding a genuine federated protocol and error bars. The paper could become a solid negative result about untuned TRPO in this setting, which is worth publishing.","headline":"The headline claim contradicts the paper's own tables; the useful toy benchmark and honest experiments still merit a serious referee.","tokens_in":14252,"tokens_out":2722,"would_cite":false,"duration_ms":26419,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated TRPO without tuning is claimed to match tuned PPO for net-zero battery control.","keywords":["federated reinforcement learning","battery control","microgrids","net-zero buildings","PPO","TRPO","CityLearn","privacy-preserving control"],"falsifier":"Run the same experiment in a true federated setting: each building trains its own model on its own local data for several local epochs, then sends only model weights to a central server for averaging, with no direct sharing of observations across buildings, and compare the evaluation reward against the paper's stacked-gradient results; if true federated TRPO does not match a tuned PPO, the central claim is refuted.","tokens_in":13212,"feed_emoji":"⚡","tokens_out":3725,"duration_ms":37814,"temperature":0.7,"pith_summary":"The paper tries to show that a collaborative, privacy-preserving reinforcement learning framework can learn effective battery-charging policies for net-zero buildings. It argues that an untuned Trust Region Policy Optimization (TRPO) can perform on par with a tuned Proximal Policy Optimization (PPO) when trained across multiple buildings using a federated-style averaging scheme and split-learning personalization. The authors build a customized CityLearn environment where self-sustainability is achievable by design, so they know the optimal policy in advance and can measure the optimality gap. The central goal is to demonstrate that federated learning enables generalization across heterogeneous buildings while protecting occupant data, and that even simple net-zero policies are surprisingly hard to learn.","feed_headline":"Untuned federated TRPO claimed on par with tuned PPO for batteries","feed_subtitle":"A privacy-preserving collaborative RL framework learns net-zero battery policies that beat no-battery baselines in synthetic microgrids.","key_machinery":"The mechanism carrying the argument is a federated learning loop in which each building's agent samples its own trajectories and the gradients are averaged across agents, which the authors equate to FedAvg aggregation, plus a split-learning personalization block (an optional private encoding of household-specific features) that is not shared. TRPO's constrained, second-order policy updates provide stable, adaptive step sizes without manual tuning, while PPO's clipped surrogate serves as the tuned baseline. The environment is a stripped-down CityLearn scenario with synthetic data engineered so that solar generation plus battery storage can cover each home's load completely, making the optimal policy known (zero cost and zero emissions).","core_discovery":"The paper's central claim is that Federated TRPO, combined with FedAvg-style aggregation and a personal encoding block kept private to each building, is comparable with state-of-the-art federated RL methodologies without hyperparameter tuning. In the authors' experiments, a tuned PPO generally achieves higher rewards and lower cost and emissions, but TRPO converges reliably without tuning and consistently beats the no-battery baseline in both two-building and five-building microgrid scenarios. The paper further reports that grouping features and personal encoding help TRPO in the shifted (more heterogeneous) datasets. The authors conclude that even a simple optimal policy—charge when solar is available, discharge when needed—is hard to learn, and they suggest a pragmatic warm-start strategy: a few PPO iterations followed by TRPO.","pith_inferences":["A reader should note that the equivalence between the paper's stacked-observation setup and a true federated protocol (with local training steps and communication rounds) is assumed, not tested, so the 'federated' results may not transfer directly to real federated deployments with communication constraints or privacy noise.","The reported tables show a large reward gap between tuned PPO and untuned TRPO; the claim of parity may depend on the definition of 'comparable,' and a fair test would tune TRPO as well or hold both methods to the same tuning budget.","The synthetic scenario's design—where net-zero is achievable but difficult to learn—could be reused as a benchmark for diagnosing convergence and optimality of other RL algorithms, especially second-order methods.","If the stacked-gradient approximation is found to be inaccurate, the paper's conclusions about federated TRPO would need to be re-verified under a real federated averaging scheme, which is a straightforward extension."],"forward_implications":["If the paper is right, an untuned TRPO can be a drop-in replacement for tuned PPO in federated building-control applications, saving the effort of hyperparameter search.","The demonstrated generalization across buildings suggests that privacy-preserving federated learning can pool knowledge across heterogeneous sites and still approach net-zero operation.","The finding that simple optimal policies are hard to learn supports curriculum-style training that starts with well-controlled toy scenarios before scaling to real-world complexity.","Warm-starting with a few PPO iterations and then switching to TRPO could yield a practical hybrid algorithm for battery control.","The synthetic-data testbed with a known optimal policy offers a diagnostic for measuring optimality gaps in RL algorithms beyond this specific application."],"supporting_citations":[{"why":"Provides the CityLearn environment that the paper customizes for its microgrid scenarios.","marker":"Vázquez-Canteli et al., 2019"},{"why":"Defines FedAvg, the aggregation rule the paper approximates with stacked gradients.","marker":"McMahan et al., 2017"},{"why":"Supplies the PPO algorithm used as the tuned state-of-the-art baseline.","marker":"Schulman et al., 2017"},{"why":"Introduces split learning, which the paper uses for personalization by keeping a private encoding block.","marker":"Vepakomma et al., 2018"},{"why":"Supports the claim that split-learning personalization generalizes better in federated learning.","marker":"Han et al., 2022"},{"why":"The CityLearn 2022 challenge scenario provides the reference data and pricing structure from which the paper's synthetic data is derived.","marker":"Nweye et al., 2022b"}],"fun_headline_variants":["Federated TRPO beats no-battery baseline without tuning","Privacy-preserving federated RL for net-zero battery control","Untuned federated TRPO matches tuned RL for battery control","Personalised federated RL learns battery control without tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that training a single network on observations stacked from all buildings is equivalent to doing actual federated averaging of locally trained models, and the entire 'federated' claim falls if that equivalence does not hold in practice.","fun_headline_variants_meta":{"raw":{"variants":["Federated TRPO beats no-battery baseline without tuning","Privacy-preserving federated RL for net-zero battery control","Untuned federated TRPO matches tuned RL for battery control","Personalised federated RL learns battery control without tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00092,"raw_usage":{"total_tokens":3903,"prompt_tokens":860,"completion_tokens":3043,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":2975}},"tokens_in":476,"tokens_out":3043,"duration_ms":19277,"temperature":1.0,"reasoning_tokens":2975,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:06:02.279595+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same experiment in a true federated setting: each building trains its own model on its own local data for several local epochs, then sends only model weights to a central server for averaging, with no direct sharing of observations across buildings, and compare the evaluation reward against the paper's stacked-gradient results; if true federated TRPO does not match a tuned PPO, the central claim is refuted.","supporting_citations":[{"cited_title":"SplitGP: Achieving Both Generalization and Personalization in Federated Learning","cited_arxiv_id":"2212.08343","evidence_quote":"Supports the claim that split-learning personalization generalizes better in federated learning."}],"review_version":1}