{"id":"66977974-fe66-4a8b-a632-199ae2f00c18","arxiv_id":"2507.22790","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Federated learning across simulated client datasets improved prostate MRI segmentation and cancer detection compared with local models, and configuration tuning improved detection performance.","lead":"Researchers tested federated learning, a way to train AI on hospital data without moving the data, for prostate MRI analysis across several institutions. They found federated models beat locally trained models and that tuning training settings further improved cancer detection.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.02 csPCa tuning gain rests on a single-fold grid search; if selection is noisy, the improvement may not replicate.","rationale":"The paper's headline FL-over-local result is credible: the effect sizes are large (Dice +0.14, PI-CAI +0.09), evaluated on independent sets, and consistent with the broader federated learning literature. The novel and load-bearing increment is the configuration-tuning result for detection, which is small and selected from a one-fold grid. The single-fold selection is explicitly acknowledged in the Methods and Discussion, so the manuscript itself flags the weakness the reader identified. My concern adds a precise mechanism: because the candidate configurations are scored on a fixed combined validation set and not re-selected within each fold, the five-fold test scores are conditional on a noisy selection event. The overlapping confidence intervals and the unresolved permutation-unit question make the P≤0.01 value hard to interpret. A nested selection audit directly removes the selection bias by making the configuration choice internal to each test fold; if the gain survives, the concern is settled and the conditional acceptance is justified. I therefore agree with the reader's weakest assumption and see no reason to change the CONDITIONAL verdict.","tokens_in":13000,"tokens_out":7757,"duration_ms":91806,"concrete_test":"Run a nested five-fold selection audit for the detection task. Split each client's training data into five folds; within each fold, train every candidate E-R and aggregation configuration on the other four folds, select the best configuration on that fold's validation subset, and evaluate that fold's model on the in-house 199-patient test set. Average the five fold-selected test PI-CAI scores and compare with the same nested evaluation of the FL-baseline. If the nested selection gain is below 0.02 or is not significant, the reported tuning advantage is selection overfit rather than a real configuration effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Most of the strongest claim—FL-baseline over local models—is on solid ground: the gains are large, directionally consistent across internal and external test sets, and consistent with the broader FL literature. The fragile part is the additional tuning gain for detection (PI-CAI 0.72 to 0.74, P≤0.01). That gain is the output of a one-fold selection procedure: each candidate E-R combination and aggregation strategy is trained once, scored on a combined validation set assembled from the clients' small 20% validation splits, and the best candidate is then reported with five-fold cross-validated test models. Because the configuration is chosen before the test folds are trained and the choice is not repeated inside each fold, the reported five-fold performance inherits selection noise: on this particular combined validation set, E=5/R=200/FedAdagrad may simply be the lucky winner. The detection validation sets are small (roughly 70 to 160 patients per client), and the improvement is only 0.02, with widely overlapping 95% CIs (FL-baseline 0.66–0.79 vs FL-tuned 0.69–0.81). A secondary statistical point compounds this: the paper should state the permutation unit. With only five fold models, a paired model-level permutation test cannot produce P≤0.01 (minimum two-sided P is 1/16); if the test permutes at patient level, the P-value must still account for the five-model structure. The limitation section acknowledges the single-fold selection, but the conclusion that optimization further improves detection depends on exactly that unvalidated step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a simulation study of federated learning (FL) for MRI-based prostate segmentation and clinically significant prostate cancer (csPCa) detection. Using the Flower framework and nnU-Net, the authors simulate FL across four clients (1294 patients) for segmentation and three clients (1500 patients per the methods; 1440 per the abstract) for detection. They compare local, centralized, FL-baseline, and FL-tuned configurations, optimizing local epochs, federated rounds, and aggregation strategy on a combined validation set, then reporting five-fold cross-validated test performance. The main claims are that FL-baseline outperforms local models (segmentation Dice 0.73 to 0.87; detection PI-CAI 0.63 to 0.72) with P≤0.01, and that further configuration tuning improves detection (PI-CAI 0.72 to 0.74) but not segmentation.","tokens_in":13285,"tokens_out":2497,"duration_ms":29161,"significance":"If the results hold, the paper provides a useful practical demonstration that FL can approach centralized performance in prostate MRI tasks while preserving data privacy, and that aggregation strategy and epoch-round choices matter. The public code and data provenance are strengths, as is the use of independent test sets (PROMISE12 and the PI-CAI in-house test set). However, the central FL-versus-local comparison is much more strongly supported than the tuning claim; the latter rests on a single-fold model selection and reported significance that is not convincingly justified. The paper is therefore a reasonable contribution to the applied FL literature, but the statistical and selection issues need to be addressed before the tuning conclusion can be accepted.","major_comments":[{"comment":"The permutation test is not fully specified. The text says 'a permutation test on the performance metric over five differently trained models using 1000000 iterations.' With only five fold models, permuting model identity yields at most 32 permutations (16 for a two-sided test), so the minimum possible two-sided P-value is 1/16 ≈ 0.0625, not ≤0.01. If instead the permutation is over patient-level predictions, the procedure must be described and must account for the fact that only five fitted models contribute to the test. As written, the reported P≤0.01 values (e.g., abstract, Results sections, and Table 3/4 comparisons) are not reproducible from the stated method.","section":"Materials and Methods, Statistical Analysis"},{"comment":"The claim that fine-tuning further improves detection (PI-CAI 0.72 to 0.74, P≤0.01) is not well supported by the reported evidence. The 95% confidence intervals for FL-baseline (0.66–0.79) and FL-tuned (0.69–0.80) overlap substantially, and the difference is only 0.02. More importantly, the configuration (E=5, R=200, FedAdagrad) was selected by evaluating a single trained model on a combined validation set, as acknowledged in the Methods and Discussion. This single-fold selection can inflate the apparent performance of the chosen configuration when later evaluated on five-fold test models. The manuscript does not provide any quantification of selection instability (e.g., repeated validation splits, standard errors of validation scores, or a sensitivity analysis around the chosen configuration). Without such evidence, the conclusion that 'optimizing its configuration further improved lesion detection performance' overstates what the data can justify.","section":"Results, Prostate Lesion Detection and Table 4"},{"comment":"There is an inconsistency in the detection sample size: the abstract and key points state 1440 patients, while the Methods (client description: 350+800+350) and Results state 1500 bpMRI scans. This discrepancy must be corrected, as it affects the reported study population.","section":"Manuscript, Abstract and Methods"}],"minor_comments":[{"comment":"There are typographical errors, including 'Tabel 1' in the Results section and '0.88 ± 03' in the Results and Discussion (missing leading zero).","section":"Throughout"},{"comment":"The description of the local test sets says 'subsets of 10 patients were randomly selected from each client’s training set' for segmentation, but later the same sets are called 'test sets.' It would be clearer to specify whether these were separated before training and whether the random selection was stratified.","section":"Materials and Methods, Patients and Data"},{"comment":"Figure 4 reports AUC and AP improvements from FL-baseline to optimized FL models with P≤0.01; given the same permutation testing issue noted above, these P values should be re-derived or clarified.","section":"Results, Prostate Lesion Detection"},{"comment":"The discussion of adaptive aggregation strategies (FedAdam, FedYogi, FedAdagrad) is speculative and not grounded in a formal comparison of these methods beyond the chosen configuration; consider tempering the causal language about adaptive learning rates.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has already been accepted for publication in Radiology: Artificial Intelligence, but in the context of this review, the main blocker is the statistical validity of the tuning claim. The authors should be asked to (1) specify the corresponding author's preferred permutation unit and recompute P values accordingly, and (2) either provide evidence that the single-fold selection is stable or remove the strong causal conclusion that fine-tuning improved detection. The FL-versus-local comparison is solid and should not be downgraded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main take: this paper gives you a credible FL-over-local result on a real multi-client prostate MRI setup, and it is honest about its own limitations. The segmentation and detection gains are large (Dice 0.73 to 0.87, PI-CAI 0.63 to 0.72) on independent test sets, which is exactly the kind of evidence the medical FL field needs. The segmentation configuration result (FedMedian, E=1, R=300) is not in the prior literature, so that is a genuine addition. The detection optimization overlaps with their earlier conference paper, so treat that part as incremental.\n\nWhat the paper does well: it uses a standard nnU-Net/Flower stack, reports confidence intervals, uses external test sets (PROMISE12 and a PI-CAI in-house set), and openly states that hyperparameters were selected on a single fold. The FL-baseline beats local models even on clients with small data, which is the practically important message.\n\nSoft spots, in order of seriousness. First, the permutation test as described is suspect. The text says a permutation test on the performance metric over five differently trained models. If that means permuting model labels across the five folds, the smallest possible two-sided p-value is 1/16, so P<=0.01 is impossible. They either permuted at the patient level (and should say so) or the p-values are miscalibrated. Either way, the P<=0.01 claims need a clear statement of the permutation unit.\n\nSecond, the tuning gain for detection (0.72 to 0.74) rests on a single-fold grid search. The confidence intervals overlap (0.66-0.79 vs 0.69-0.81), and with only five fold models, the gain could easily be selection noise. The paper acknowledges this in the limitations, but the conclusion still leans on that step.\n\nThird, the local test sets are tiny (10 patients per segmentation client, 20 per detection client). That is not fatal because the independent test sets carry the argument, but it means the 'average local model' baseline is noisy.\n\nMinor: the GitHub link is mentioned but missing from the preprint; for reproducibility, that should be live.\n\nWho this is for: people working on FL in medical imaging, especially anyone deciding whether to bother tuning E, R, and aggregation strategy. The main result is a useful reference point. The tuning result is a cautionary tale. I would send it to review; it deserves a serious referee, but the authors need to fix the statistical reporting and soften the tuning claim.","headline":"Solid FL-over-local evidence for prostate MRI, but the permutation test needs clarification and the tuning gain is shakier than the headline suggests.","tokens_in":13878,"tokens_out":3163,"would_cite":true,"duration_ms":34973,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated learning lets separate institutions train prostate MRI models that beat each local model, and tuning the configuration further improves cancer detection.","keywords":["federated learning","prostate segmentation","prostate cancer detection","biparametric MRI","nnU-Net","PI-CAI score","aggregation strategy","simulation study"],"falsifier":"Run the same two-task protocol with, say, ten random initializations for each candidate configuration, keep the same independent test sets, and record the PI-CAI score for each run; if the tuned detection model does not beat the FL-baseline in most paired runs, the claimed tuning benefit is not reproducible.","tokens_in":12814,"feed_emoji":"🩻","tokens_out":9810,"duration_ms":106761,"temperature":0.7,"pith_summary":"Federated learning lets several institutions train a shared AI model without sharing patient images; this study asks whether that scheme works for prostate MRI, and whether the training configuration matters. Using public and in-house prostate MRI data split into separate clients, the authors trained deep-learning models for prostate gland segmentation and for clinically significant prostate cancer (csPCa) detection under various federated settings. They report that the federated models beat the average of the locally trained models on independent test sets, lifting segmentation Dice from 0.73 to 0.87–0.88 and the csPCa detection score from 0.63 to 0.72–0.74. Tuning local epochs, federated rounds, and the server-side aggregation rule produced a further significant gain in detection, while segmentation was already near its best with the baseline configuration. The conclusion is that institutions can collaborate on prostate MRI AI without centralizing data, gaining most of the benefit of centralized training and sometimes a bit more.","feed_headline":"Federated training lifts prostate MRI AI past locally trained models","feed_subtitle":"Tuning epochs, rounds, and aggregation adds a further significant gain for cancer detection.","key_machinery":"The mechanism is the federated learning loop parameterized by three knobs: local epochs ($E$), the number of passes each client takes over its own data before sharing parameters; federated rounds ($R$), the number of times the server aggregates client parameters into a global model; and the server-side aggregation rule, among FedAvg, FedAdagrad, FedAdam, FedYogi, and FedMedian. Keeping the product $E \\times R$ constant so that total computation is comparable, the authors searched over epoch-round combinations and aggregation rules, selecting the setting with the best score on a combined validation set. The chosen configurations were then evaluated with five-fold cross-validated models on independent test sets.","core_discovery":"The paper's central claim is that federated learning is a viable substitute for centralized data pooling in prostate MRI, and that the specific federated configuration carries real performance weight. On independent test sets, the optimized federated models improved prostate gland segmentation from an average local-model Dice score of $0.73 \\pm 0.06$ to $0.88 \\pm 0.03$ ($P \\le 0.01$) and improved csPCa detection from an average local-model PI-CAI score of $0.63 \\pm 0.07$ to $0.74 \\pm 0.06$ ($P \\le 0.01$). The best detection configuration used $E=5$ local epochs, $R=200$ federated rounds, and FedAdagrad aggregation, and its tuned gain over the FL baseline ($0.72 \\pm 0.06$ to $0.74 \\pm 0.06$, $P \\le 0.01$) was significant; the best segmentation configuration used $E=1$, $R=300$, and FedMedian, but its Dice gain over the FL baseline ($0.87$ to $0.88$) was not significant. These results are presented as the finding of a simulation study in which public and in-house datasets were partitioned into separate clients, with independent test sets used for final evaluation.","pith_inferences":["Editorial inference: Because the paper's selected configurations were chosen from a single trained model per setting, a natural next experiment is to run several random seeds per configuration; this would separate true tuning effects from selection noise.","Editorial inference: The pattern that FedMedian helped segmentation while FedAdagrad helped detection suggests that robust averaging preserves anatomical shape consistency, whereas adaptive server steps help when client prevalence and scanner properties differ; a direct comparison across data-heterogeneity levels would test this.","Editorial inference: If the results generalize beyond prostate MRI, the practical recipe of fixing the total compute budget, preferring few local epochs and many rounds, and picking the server optimizer by validation on a shared hold-out could become a default checklist for federated medical image analysis.","Editorial inference: Because the data were partitioned from existing datasets rather than collected through a live multi-site deployment, real-world federated learning with variable preprocessing and client compute may show different gains; running the same protocol prospectively across sites would be the natural next test."],"forward_implications":["Institutions with small or atypical local data gain the most from joining a federation; the weakest local segmentation model scored 0.47 Dice on the independent test set and jumped to 0.88 with the tuned federated model.","A hospital can obtain most of the benefit of centralized training without sending images to a central site, since the federated models matched or slightly exceeded the centralized baseline on the independent test sets.","Optimization effort in a federated deployment should be task-dependent: a simple FedAvg baseline sufficed for gland segmentation, while cancer detection required an adaptive aggregation strategy and a different epoch-round balance to gain.","Federated models trained this way generalize to data from institutions that did not contribute to training, as shown by the independent PROMISE12 and in-house test sets."],"supporting_citations":[{"why":"Supplies the nnU-Net architecture and self-configuring training pipeline used for both segmentation and detection.","marker":"[10]"},{"why":"One of the public datasets acting as a separate client in the prostate segmentation federation.","marker":"[16]"},{"why":"The Prostate-MRI-US-Biopsy dataset, another client in the segmentation federation.","marker":"[17-19]"},{"why":"The PROSTATEx dataset, the third public client in the segmentation federation.","marker":"[20]"},{"why":"The in-house dataset used as the fourth segmentation client and as the independent internal test set for detection.","marker":"[21]"},{"why":"The PROMISE12 challenge data used as the independent external test set for segmentation.","marker":"[23]"},{"why":"The PI-CAI challenge supplies the three-client detection data, the task definition, and the PI-CAI score metric.","marker":"[24]"},{"why":"Introduces FedAvg, the default aggregation strategy used for the FL baseline in both tasks.","marker":"[28]"},{"why":"Introduces the adaptive server optimizers FedAdagrad, FedAdam, and FedYogi evaluated in the aggregation-strategy search.","marker":"[29]"},{"why":"Introduces FedMedian, the aggregation rule that gave the best segmentation performance.","marker":"[30]"}],"fun_headline_variants":["Federated learning beats local models for prostate MRI cancer detection","Tuning federated config adds significant gain in prostate cancer detection","Optimized federated learning sharpens prostate MRI cancer detection","FL configuration tuning boosts prostate cancer detection, not segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on treating a configuration selected by a single trained model per setting, scored on a combined validation set, as stable enough that the five-fold test scores of that configuration measure the real improvement; if that single-fold selection is noisy, the small tuned detection gain (0.72 to 0.74) may not replicate.","fun_headline_variants_meta":{"raw":{"variants":["Federated learning beats local models for prostate MRI cancer detection","Tuning federated config adds significant gain in prostate cancer detection","Optimized federated learning sharpens prostate MRI cancer detection","FL configuration tuning boosts prostate cancer detection, not segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1656,"prompt_tokens":1134,"completion_tokens":522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":750,"completion_tokens_details":{"reasoning_tokens":454}},"tokens_in":750,"tokens_out":522,"duration_ms":6345,"temperature":1.0,"reasoning_tokens":454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:16:58.183599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same two-task protocol with, say, ten random initializations for each candidate configuration, keep the same independent test sets, and record the PI-CAI score for each run; if the tuned detection model does not beat the FL-baseline in most paired runs, the claimed tuning benefit is not reproducible.","supporting_citations":[{"cited_title":"PROSTATEx Challenges for computerized classification of prostate lesions from multiparametric magnetic resonance images","cited_arxiv_id":null,"evidence_quote":"The PROSTATEx dataset, the third public client in the segmentation federation."},{"cited_title":"Prostate158-An expert-annotated 3T MRI dataset and algorithm for prostate cancer detection","cited_arxiv_id":null,"evidence_quote":"One of the public datasets acting as a separate client in the prostate segmentation federation."},{"cited_title":"Multiparametric Prostate MRI in Biopsy-Naive Men: A Prospective Evaluation of Performance and Biopsy Strategies","cited_arxiv_id":null,"evidence_quote":"The in-house dataset used as the fourth segmentation client and as the independent internal test set for detection."},{"cited_title":"In: Medical Imaging with Deep Learning, short paper track: Apr 2023","cited_arxiv_id":null,"evidence_quote":"The PI-CAI challenge supplies the three-client detection data, the task definition, and the PI-CAI score metric."},{"cited_title":"In: Proc 9th Int Conf Learn Representations: 2020","cited_arxiv_id":null,"evidence_quote":"Introduces the adaptive server optimizers FedAdagrad, FedAdam, and FedYogi evaluated in the aggregation-strategy search."},{"cited_title":"Semisupervised Learning with Report-guided Pseudo Labels for Deep Learning-based Prostate Cancer Detection Using Biparametric MRI","cited_arxiv_id":null,"evidence_quote":"Introduces FedMedian, the aggregation rule that gave the best segmentation performance."}],"review_version":1}