{"id":"3abf09b5-f7bf-433a-b13e-7d990c6328ac","arxiv_id":"2505.08085","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A PySyft-backed federated Random Forest with weighted tree sampling keeps predictive accuracy within 9% of centralized training on two healthcare datasets.","lead":"Researchers built a privacy-preserving federated Random Forest framework on PySyft, letting hospitals train a shared model without pooling raw data. It combines trees from local sites using weighted sampling and showed accuracy within 9% of a centralized model on two medical datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Privacy half of the central claim is unsupported: the protocol exchanges plaintext serialized trees via PySyft, with no DP, secure aggregation, or threat model, while claiming to mitigate membership inference.","rationale":"The central claim is a conjunction: competitive accuracy within 9% and stringent privacy. The accuracy portion may be plausible, and the paper deserves credit for releasing reproducible code and for an honest engineering narrative. However, the privacy portion is load-bearing and lacks any mechanism. The actual protocol sends serialized random forest trees to a central coordinator in plaintext as described in Section III.C; the paper does not specify encryption, secure aggregation, or differential privacy, and the conclusion's future-work statement explicitly says DP would be explored later. Since serialized tree ensembles are known to leak information through split thresholds, leaf counts, and structure, the abstract's 'stringent privacy requirements' claim is contradicted by the paper's own description. This is not a matter of missing empirical evaluation alone; the claimed property is absent from the system as described. The reader correctly identifies this as the weakest assumption, but I weight it more heavily: a central conjunct of the advertised contribution cannot be sustained as submitted. A revised version that adds a concrete threat model and a privacy mechanism such as DP or secure aggregation, or that drops the 'stringent privacy' wording, could be reconsidered. For the current version, the central claim does not hold.","tokens_in":8980,"tokens_out":4745,"duration_ms":53627,"concrete_test":"Using the released fed_rf_mk package, train local forests with exactly the AIDS hyperparameters (4100 estimators) on an 80/20 split, serialize the client models, and transmit them to the coordinator exactly as in Section III.C. Give an independent auditor the serialized tree objects plus the public aggregate (number of trees and weights), and ask for (i) membership-inference AUC distinguishing the 20% held-out rows from training rows using leaf-path response, and (ii) reconstruction of feature-value intervals for individual training records by intersecting leaf thresholds. If membership AUC is materially above 0.5 or any record's feature ranges are narrowed to a small fraction of the domain, the 'stringent privacy' claim is false; a negative result would not prove privacy but would shift the burden.","verdict_should_be":"REJECT","load_bearing_attack":"Section III.C describes the protocol: each silo trains a Random Forest, then 'models are serialized and sent back to the coordinator' for weighted aggregation. No encryption, secure aggregation, or noise addition is specified. Section III.A nevertheless asserts that PySyft's remote-execution architecture 'mitigat[es] risks such as model inversion or membership inference attacks.' The conclusion's future-work sentence, 'explore differential privacy techniques ... to further ... enhance protection against inference attacks,' concedes that the current implementation does not add such protection. For tree ensembles, serialized trees are not neutral summary statistics: leaf thresholds, split features, leaf sample counts, and tree structure can be used to narrow feature-value ranges for training records and to perform membership inference. With 4100 trees on the AIDS dataset, the released model exposes substantial structure. Thus the conjunct 'while satisfying stringent privacy requirements' is not merely unevaluated; the described mechanism does not provide it. Because privacy is a stated deliverable of the system, not a secondary remark, the central claim fails as written. This is a correctness gap, not a disagreement with community norms.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents Fed-RF, a PySyft-based federated learning framework for Random Forest classifiers. The proposed protocol trains local forests at each data silo, serializes and transmits them to a central coordinator, and aggregates trees by weighted sampling; it also supports incremental warm-start rounds and local evaluation. Experiments on two healthcare datasets report accuracy within 0.26-8.17% and 2.23-9.05% of a centralized baseline, and the paper claims to satisfy 'stringent privacy requirements' by leveraging PySyft. The authors release the implementation as an open-source package.","tokens_in":9157,"tokens_out":3986,"duration_ms":39445,"significance":"If the accuracy results and privacy claims were rigorously established, the work would fill a real gap in tree-based federated learning and provide a useful open-source tool. The authors deserve credit for releasing the implementation, making the aggregation procedure explicit in Algorithm 1, and evaluating on two real healthcare datasets. However, the privacy guarantee is asserted rather than demonstrated, and the empirical evidence lacks the repeated trials, error bars, and statistical tests needed to support the stated accuracy margins. The contribution is therefore promising but not yet substantiated as written.","major_comments":[{"comment":"The central claim that the method 'satisf[ies] stringent privacy requirements' is not supported by the described protocol. Section III.C states that locally trained forests are 'serialized and sent back to the coordinator' with no encryption, secure aggregation, or noise addition; Section III.A asserts that PySyft's remote-execution architecture mitigates model inversion and membership inference, but exchanging plaintext decision trees does not provide such protection, as leaf thresholds, split features, and sample counts can leak information about training records. The conclusion's future-work sentence on differential privacy confirms that no such protection is currently included. Since privacy is half of the paper's stated contribution, this is a load-bearing gap that must be addressed by either adding a concrete privacy mechanism or substantially weakening the privacy claims.","section":"Section III.A and III.C; Conclusion"},{"comment":"The experimental evaluation does not support the reported accuracy margins. Each configuration appears to be a single run with no error bars, cross-validation, or significance tests; differences such as 0.26% (Table I, 3 Silos) and 2.23% (Table II, 3 Silos) are likely within sampling noise. Additionally, the centralized baseline is trained on the entire dataset while the federated silos train on the 80% training partition, giving the baseline an advantage and making the reported 'accuracy deviation' a biased measure. Repeated runs with variance reporting, or paired statistical tests on the same test partition, are needed before the competitive-accuracy claim can be evaluated.","section":"Section IV.B, Tables I and II"},{"comment":"The weighted sampling procedure as written does not guarantee that the global forest has the same total number of trees as a client model. Because k_i = floor(w'_i * N_i), rounding down can leave the sum of k_i smaller than N_i, and no renormalization or resampling step is described. Since Section III.C claims the global forest is 'composed of a mixture from all clients' with the same total number of trees, this ambiguity should be clarified and the implementation checked against the intended guarantee.","section":"Algorithm 1, lines 4-6"}],"minor_comments":[{"comment":"The text says N varied from 1 to 10, but Tables I and II report only 3, 5, and 10 silos; the Discussion's N=2 result (accuracy 0.7260) for the diabetic retinopathy dataset is not shown in any table, so the reader cannot verify the claimed peak above baseline.","section":"Section IV.C and Discussion"},{"comment":"There are several typographical issues: 'an novel' in the Introduction, 'fed rf' in the abstract's repository URL, and 'I NTRODUCTION' in the section heading; these should be corrected.","section":"Introduction, Abstract"},{"comment":"Reference [1] (Silva and Oliveira, 'BIT. UA at IDPP') appears unrelated to the present paper and is not cited in the body; please remove it or replace it with a directly relevant citation.","section":"References"},{"comment":"The sentence 'mitigating risks such as model inversion or membership inference attacks, this is a key feature that will enable our tool to provide privacy and confidentiality to data' is a run-on and should be split; moreover, the phrase 'will enable' is inconsistent with the paper's claim that privacy is already provided.","section":"Section III.A"}],"recommendation":"major_revision","confidential_remarks":"The paper's main deliverable is an open-source federated random forest with a privacy claim that is not backed by the described protocol. The empirical section is also too thin to support the accuracy margins. I recommend major revision rather than rejection because the code is released and the accuracy claims could be salvaged with proper evaluation; however, if the authors cannot add a concrete privacy mechanism or clearly reframe the privacy contribution, the central claim will need to be weakened substantially."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an honest engineering report about a PySyft-native federated random forest, but the headline privacy claim is not supported by the protocol, and the accuracy comparisons are too thin to support the claimed margins. The genuinely new part is the open-source implementation and the package, not the method: tree ensemble aggregation with weighted sampling is already in the paper's own references [13,26,31], and incremental warm-start training is a known technique. So novelty is modest.\n\nWhat the paper does well: the protocol is described plainly, the code is public, and the authors do not hide the accuracy degradation as silo count increases. A reader who wants to build a simple federated tree ensemble with PySyft would find the package useful and the write-up accessible.\n\nThe soft spots are real. First, the central claim says “satisfying stringent privacy requirements,” but the protocol exchanges serialized plaintext trees—no differential privacy, no secure aggregation, no threat model. Serialized trees can leak information through split thresholds and leaf statistics, so the privacy sentence in Section III.A is unsupported. The conclusion even concedes that differential privacy is future work, which undercuts the paper's own central claim. Second, the experiments have no error bars, no repeated runs, and no significance tests; a 0.26% difference is noise. Third, the centralized baseline is described as trained on the entire dataset, which would give it an unfair advantage if the test silo's 20% is included in its training data; the authors need to clarify. Fourth, the incremental learning results and the N=2 numbers are mentioned but not shown in the tables, which is easy to fix.\n\nThe citation pattern is fine: FedTree and Hauschild are cited, so the limited novelty is implicitly acknowledged, though it is not emphasized enough. The accuracy claim as stated is plausible but not demonstrated. The privacy claim is the load-bearing problem; as written, it is wrong.\n\nMy recommendation: this deserves referee time at an applied or workshop-oriented venue, because the code and the clear protocol are useful and the issues are addressable. A strict systems journal might desk reject on novelty. I would not cite it in my own work, but I would skim it for implementation details if I were building tree-based federated learning.","headline":"An honest but modest engineering report on a PySyft-based federated random forest whose central privacy claim does not survive close reading; the accuracy comparisons are also too thin to support the numeric margins.","tokens_in":9727,"tokens_out":2296,"would_cite":false,"duration_ms":24845,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims a federated Random Forest can match centralized accuracy within about 9% while keeping patient data in place.","keywords":["federated learning","Random Forest","tree aggregation","weighted sampling","healthcare privacy","distributed machine learning","incremental learning","local evaluation"],"falsifier":"Run a membership-inference or model-inversion attack against the serialized forest returned by a silo: if a specific patient record can be distinguished as having been in the training set, or its feature values recovered, from the tree splits alone, the privacy guarantee in the paper's central claim fails even if the accuracy numbers hold.","tokens_in":8734,"feed_emoji":"🩺","tokens_out":6707,"duration_ms":62389,"temperature":0.7,"pith_summary":"This paper argues that Random Forest classifiers, one of the most widely used and interpretable models in healthcare, can be adapted to federated learning without the accuracy collapse usually associated with decentralized tree training. It presents a framework in which each institution trains a local forest on data that never leaves its own server, then sends serialized trees to a central coordinator. The coordinator builds a global forest by sampling trees from each site, with sampling probability proportional to the site's data volume. On two healthcare benchmarks, the authors report that the federated model stays within a 9% accuracy margin of a centralized model, with larger margins appearing only when data is split across ten silos. The paper positions this as a practical route to privacy-preserving, interpretable collaborative learning in regulated settings.","feed_headline":"Federated Random Forests keep accuracy within 9% of centralized models","feed_subtitle":"Two healthcare benchmarks show tree-based federated learning keeps predictive power while data stays local.","key_machinery":"The load-bearing mechanism is weighted tree sampling for forest aggregation. After each silo trains a local Random Forest, the coordinator selects $k_i = \\lfloor w'_i \\times N_i \\rfloor$ trees from silo $i$, where $w'_i$ is the silo's normalized weight and $N_i$ is its local forest size, then assembles the selected trees into a single global forest. Uniform sampling serves as the default when no weights are given, and the scheme is linear in the total number of trees. The paper also uses warm-start incremental rounds, in which each silo adds new trees to an existing forest, allowing the global model to refine over multiple federated rounds. The secure remote-computation layer is what keeps local data on-site while training code is sent to it.","core_discovery":"The central claim is that federated Random Forest training can preserve most of the predictive performance of a centralized model while allowing each participating institution to keep raw data local. Experiments on two healthcare datasets show accuracy degradation of 0.26–8.17% and 2.23–9.05% depending on the number of silos. The paper attributes the stability to a weighted aggregation protocol in which the global forest contains the same total number of trees as a single client model, sampled from client forests proportionally to client weights. It also reports that one federated configuration slightly exceeded its centralized baseline, and that relative degradation patterns were consistent across the two datasets. The privacy guarantee, however, is asserted through the use of a secure remote-computation layer rather than measured; no differential privacy, secure aggregation, or threat-model analysis is presented.","pith_inferences":["The paper's privacy claim is stronger than its evidence: because no differential privacy or secure aggregation is applied, an outside observer with access to the serialized trees could attempt membership inference, and the paper's own text lists this as future work.","The weighted sampling rule could be extended to weight silos by estimated data quality or label balance rather than volume alone, which would give a direct test of whether the observed accuracy ceiling can be pushed lower on skewed datasets.","The reported degradation pattern suggests a design guideline the authors do not state explicitly: federations beyond roughly five partitions should expect accuracy loss in the 2–9% range, so deployments should weigh fragmentation costs against privacy benefits."],"forward_implications":["Healthcare institutions can collaboratively train a Random Forest without moving raw records, keeping final-model accuracy within roughly 9% of a centralized model when the number of participating sites is moderate.","Weighted aggregation lets larger or more representative datasets exert more influence on the global forest, providing a practical way to handle statistical heterogeneity across sites.","Incremental warm-start rounds allow new silos to join mid-training and let the global model improve over time without retraining from scratch.","Excessive fragmentation degrades performance by up to roughly 9 percentage points, setting a practical limit on how finely data can be split while preserving accuracy.","Local evaluation on each silo's own data gives sites a way to check how well the federated model generalizes to their specific population, supporting transparency in clinical contexts."],"supporting_citations":[{"why":"Defines the federated learning protocol and aggregation loop that the paper's architecture follows.","marker":"[3]"},{"why":"Defines the Random Forest ensemble method that the paper adapts to the federated setting.","marker":"[4]"},{"why":"Supplies the centralized reference accuracy on the AIDS dataset that the paper uses as a performance baseline.","marker":"[6]"},{"why":"Provides the tree-ensemble aggregation approach that the paper identifies as its primary design strategy.","marker":"[13]"},{"why":"Supplies the client-weighting idea behind the weighted sampling aggregation used in the global forest.","marker":"[31]"},{"why":"Provides the reference values for the diabetic retinopathy dataset used as the second benchmark.","marker":"[32]"},{"why":"Represents an existing tree-based federated learning system that the paper positions against.","marker":"[33]"},{"why":"Represents a general federated learning framework that motivates the need for a Random-Forest-specific implementation.","marker":"[34]"}],"fun_headline_variants":["Federated random forests match centralized accuracy within 9%","Privacy-preserving random forests: within 9% of centralized accuracy","Tree-based federated learning: accuracy within 9% of centralized","Federated random forests: secure training with 9% accuracy gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The privacy half of the central claim rests on the assumption that exchanging serialized decision trees between sites and the coordinator cannot be used to infer anything meaningful about the local training data; the paper states this but does not test it with a threat model, differential privacy, or secure aggregation.","fun_headline_variants_meta":{"raw":{"variants":["Federated random forests match centralized accuracy within 9%","Privacy-preserving random forests: within 9% of centralized accuracy","Tree-based federated learning: accuracy within 9% of centralized","Federated random forests: secure training with 9% accuracy gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000517,"raw_usage":{"total_tokens":2501,"prompt_tokens":933,"completion_tokens":1568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1493}},"tokens_in":549,"tokens_out":1568,"duration_ms":11499,"temperature":1.0,"reasoning_tokens":1493,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:03:32.188883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a membership-inference or model-inversion attack against the serialized forest returned by a silo: if a specific patient record can be distinguished as having been in the training set, or its feature values recovered, from the tree splits alone, the privacy guarantee in the paper's central claim fails even if the accuracy numbers hold.","supporting_citations":[{"cited_title":"Communication-Efficient Learning of Deep Networks from Decentralized Data,","cited_arxiv_id":null,"evidence_quote":"Defines the federated learning protocol and aggregation loop that the paper's architecture follows."},{"cited_title":"Random forests,","cited_arxiv_id":null,"evidence_quote":"Defines the Random Forest ensemble method that the paper adapts to the federated setting."},{"cited_title":"A Ma- chine Learning Approach to Aids Clinical Trials Group (Actg) Study,","cited_arxiv_id":null,"evidence_quote":"Supplies the centralized reference accuracy on the AIDS dataset that the paper uses as a performance baseline."},{"cited_title":"Transfer learning via random forests: A one-shot federated approach,","cited_arxiv_id":null,"evidence_quote":"Provides the tree-ensemble aggregation approach that the paper identifies as its primary design strategy."},{"cited_title":"Tackling system and statistical heterogeneity for federated learning with adaptive client sampling,","cited_arxiv_id":null,"evidence_quote":"Supplies the client-weighting idea behind the weighted sampling aggregation used in the global forest."},{"cited_title":"An effective inte- grated machine learning approach for detecting diabetic retinopathy,","cited_arxiv_id":null,"evidence_quote":"Provides the reference values for the diabetic retinopathy dataset used as the second benchmark."},{"cited_title":"FedTree: A Fast, Effective, and Secure Tree- based Federated Learning System,","cited_arxiv_id":null,"evidence_quote":"Represents an existing tree-based federated learning system that the paper positions against."}],"review_version":1}