{"id":"24fe20cd-0b3d-405f-bb0a-6a97b6217675","arxiv_id":"2607.08219","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"MobenFL is the broadest federated medical-imaging benchmark to date, pairing 20 algorithms with 22 multi-organ datasets and adding efficiency plus privacy metrics.","lead":"MobenFL is a large open benchmark that runs 20 federated-learning algorithms on 22 medical-image datasets spanning 12 organs. It measures accuracy, training cost and privacy so hospitals can pick methods that work under real multi-center constraints.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The fixed five-client Dirichlet skeleton undercuts the claim of clinical-scenario realism that the breadth of MobenFL is meant to deliver.","rationale":"The reader correctly isolates the fixed experimental skeleton as the weakest assumption. That skeleton is not a peripheral implementation detail; it is the sole generator of every quantitative result that underpins the paper’s claim of clinical utility. Because the paper never varies client count, participation or α, the breadth of organs and algorithms cannot by itself guarantee that the reported rankings transfer to realistic multi-center settings. The concrete test above is inexpensive relative to the original 20×22 matrix and would either confirm robustness or expose ranking instability. No stronger internal inconsistency is present, so the verdict remains CONDITIONAL rather than REJECT; the concern simply makes the conditionality more precise.","tokens_in":44730,"tokens_out":523,"duration_ms":7236,"concrete_test":"Re-run the full suite of 20 algorithms on the three largest datasets (ChestXray, TissueMNIST, NCH100K) under two alternative skeletons: (i) 20 clients with Dirichlet α=0.1 and (ii) 5 clients with participation rate 0.4 per round. If the top-3 ranking of algorithms (by accuracy or by the composite score used for Fig. 16) changes for any of the three datasets, the clinical-recommendation claim is materially weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that MobenFL supplies a “comprehensive and in-depth evaluation framework for the clinical application of FL” rests on the premise that its experimental skeleton fairly represents multi-center medical deployments. Section 4.1 freezes every run to ResNet-18 + Adam (lr=0.0001), batch size 128, exactly five clients, Dirichlet α=0.5 and early-stopping patience 10. Real hospital networks routinely involve 10–50+ sites with highly unbalanced sample sizes, device-specific batch-norm statistics and participation rates far from the uniform five-client Dirichlet draw. Because every accuracy, efficiency and privacy number reported in Sections 4.2–4.5 is generated under this single skeleton, the ranking of the 20 algorithms (and therefore the “recommended algorithms” of Fig. 16) may reverse once client count, participation fraction or α are varied. The multi-organ breadth therefore measures algorithmic behavior only inside a narrow, non-clinical regime; the claim of clinical relevance is consequently load-bearing on an untested assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript introduces MobenFL, a federated medical-image classification benchmark that unifies 20 recent FL algorithms (grouped into additional-network, regularization, aggregation-strategy and split-learning families) with 22 public 2-D imaging datasets spanning 12 organs. Under a fixed experimental skeleton (ResNet-18 + Adam, lr = 0.0001, batch 128, five clients, Dirichlet α = 0.5, early-stopping patience 10) the authors report accuracy under IID and non-IID partitions, theoretical and wall-clock efficiency, robustness to simple Gaussian differential-privacy noise, and three clinical-heterogeneity scenarios (cross-organ, cross-disease, cross-device). From these results they derive a recommendation map (Fig. 16) that ranks algorithms for different clinical priorities. Code is released.","tokens_in":45048,"tokens_out":985,"duration_ms":9439,"significance":"If the rankings prove stable, MobenFL would become the most extensive public reference point for FL medical-image classification, filling a clear gap left by FedCBD and Flamby. The multi-organ coverage, inclusion of split-learning and personalization methods, joint reporting of accuracy/efficiency/privacy, and open-source unified framework are genuine strengths that lower the barrier for subsequent comparative work. The contribution is therefore of clear practical value to the medical-AI community, provided the fixed experimental skeleton is shown not to reverse the reported orderings.","major_comments":[{"comment":"Section 4.1 freezes every experiment to exactly five clients, Dirichlet α = 0.5, ResNet-18 + Adam (lr = 0.0001), batch size 128 and early-stopping patience 10. All accuracy, efficiency and privacy numbers in Sections 4.2–4.5, and therefore the recommendation map of Fig. 16, are generated under this single skeleton. Real multi-center deployments routinely involve 10–50+ sites with highly unbalanced sample sizes and participation rates far from a uniform five-client Dirichlet draw. Because the central claim of a “comprehensive … evaluation framework for the clinical application of FL” rests on the premise that the skeleton fairly represents clinical conditions, at least a limited sensitivity study (varying client count and α on a representative subset of datasets) is required to establish that the reported rankings do not reverse.","section":null},{"comment":"Section 4.5.1 implements differential privacy by adding isotropic Gaussian noise scaled by a free factor ε to the transmitted parameters (Eq. 5). This is not a standard (ε,δ)-DP mechanism with clipping and calibrated noise; consequently the privacy-robustness claims and the special status accorded to SplitFed under “strict privacy” cannot be interpreted as formal DP guarantees. Either a proper DP-SGD implementation with reported privacy budgets should be used, or the language should be revised to “noise robustness” rather than “privacy protection capabilities.”","section":null}],"minor_comments":[{"comment":"Title and abstract contain the typographical error “Feredated”; correct throughout.","section":null},{"comment":"Table 2 lists TurboSVM under both Additional Network Structures and Regularization-Driven; clarify the intended category.","section":null},{"comment":"Section 3.2.2 states “FedProx, MOON, PGFed and TurboSVM all fall into this category” while TurboSVM is already placed in the additional-network group; remove the inconsistency.","section":null},{"comment":"Figure 4 caption and surrounding text refer to “p-value” without stating the statistical test used; specify the paired test and whether multiple-comparison correction was applied.","section":null},{"comment":"Several dataset URLs and accession numbers appear only as footnotes; a consolidated data-availability table would improve reproducibility.","section":null},{"comment":"The hyper-parameter study in Section 4.6 is performed only for FedAvg; a short note on whether the recommended (batch=16, local_iter=3) setting transfers to the other 19 algorithms would be useful.","section":null}],"recommendation":"major_revision","confidential_remarks":"The breadth of the benchmark is impressive and the open-source release is a clear plus. The two load-bearing issues (fixed five-client skeleton and non-standard DP) are fixable within a revision cycle; if the authors supply even a modest sensitivity study and either a proper DP implementation or corrected language, the paper would be a solid contribution. I see no novelty or citation-pattern concerns that would warrant rejection."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is the broadest medical FL classification benchmark we have so far: 20 recent algorithms (including personalization, split, and aggregation variants) run on 22 public datasets spanning 12 organs, with accuracy, wall-clock efficiency, and a simple DP-noise privacy axis, plus three clinical-scenario cuts (cross-disease, cross-device, cross-modality). Code is released. That combination is new relative to FedCBD and FLamby and is useful.\n\nWhat they did well is the engineering and the reporting. Everything sits in one framework with standardized ResNet-18 + Adam inputs, early stopping, and Dirichlet non-IID. They give p-values on the IID vs non-IID drops, ternary efficiency plots, and a practical recommendation map (Fig. 16). The multi-organ tables make it easy to see that some methods (HarmoFL, SplitFed, FedNP) hold up better than others once you leave the single-organ comfort zone. Self-citation is light and the baselines are the usual ones.\n\nThe soft spot the stress-test flags is real but proportional. Section 4.1 freezes five clients, batch 128, lr 1e-4, α=0.5 for every number that appears later. Real multi-center networks are larger, more unbalanced, and have device-specific BN quirks; rankings could shift. Privacy is just Gaussian noise on parameters, not a full DP-SGD or secure-aggregation stack. Hyper-parameters are not tuned per method, so some algorithms are probably under-sold. These are the usual benchmark caveats, not circular math or invented data. The paper still shows clear thinking and honest engagement with the literature.\n\nWho it is for: anyone building or reviewing medical FL methods who needs a ready multi-organ yardstick and open code. It will not rewrite clinical practice, but it will change how new algorithms get compared. I would bring it to reading group, cite the tables when I need a multi-organ reference, and send it to peer review. The fixed skeleton should be called out and ideally stress-tested in revision, but the artifact itself is solid enough to deserve referee time.","headline":"Solid multi-organ FL benchmark that actually ships code and multi-axis tables; the fixed five-client skeleton is a real but not fatal limit on the clinical-realism claim.","tokens_in":45584,"tokens_out":535,"would_cite":true,"duration_ms":7608,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"MobenFL is the broadest federated-learning benchmark for medical images, pairing 20 algorithms with 22 multi-organ datasets and scoring accuracy, speed, privacy and clinical heterogeneity together.","keywords":["Federated Learning","Medical Image Classification","Multi-organ data","Privacy robustness","Algorithmic efficiency","Cross-device evaluation","Non-IID medical data"],"falsifier":"Re-run the exact same twenty algorithms on a new multi-center collection whose client count, scanner mix and class imbalance differ sharply from the five-client Dirichlet-0.5 setup and check whether the recommended algorithm rankings reverse.","tokens_in":45674,"feed_emoji":"🏥","tokens_out":845,"duration_ms":13930,"temperature":0.7,"pith_summary":"Medical AI is blocked by privacy rules and by huge differences in organs, scanners and diseases across hospitals. Federated learning lets centers train together without sharing raw images, yet no shared yardstick exists to compare the flood of new algorithms under realistic clinical messiness. MobenFL fills that gap by running twenty recent federated methods on twenty-two public imaging collections that span twelve body organs and four modalities. Beyond plain accuracy it also times convergence, measures privacy-noise tolerance and tests three real-world mismatches: different diseases of the same organ, different devices imaging the same disease, and different modalities. The resulting scorecards let clinicians pick an algorithm that matches their data diversity, compute budget and privacy needs instead of relying on single-number leaderboards.","feed_headline":"20 FL algorithms ranked on 22 multi-organ medical sets","feed_subtitle":"Accuracy, speed and privacy scores now sit side-by-side so clinics can pick what fits their scanners and rules.","key_machinery":"MobenFL itself: a unified code framework that standardizes twenty algorithms (grouped as extra-network, regularization, aggregation and split-learning) and twenty-two datasets under fixed ResNet-18 training, Dirichlet non-IID splits, early stopping, and three specialized clinical-heterogeneity test suites.","core_discovery":"The paper claims that a single open benchmark covering twenty federated algorithms, twenty-two multi-organ datasets and four evaluation axes (accuracy, efficiency, privacy robustness and clinical-scenario transfer) is both feasible and necessary; without it, claims about which algorithm works in the clinic remain incomparable and untrustworthy.","pith_inferences":["The same multi-axis design could be ported to federated segmentation or detection, exposing whether the current accuracy–efficiency trade-offs reverse when pixel-level labels are required.","Because the privacy test injects noise only on the uplink, a follow-up that also privatizes the downlink or uses secure aggregation would test whether SplitFed’s advantage survives stronger threat models.","The observed organ-specific sensitivity (brain and knee drop hardest under non-IID) suggests that organ-aware client clustering or modality-specific personalization layers may be more fruitful than further generic regularizers."],"forward_implications":["Clinicians can now select algorithms by primary need: SplitFed or HarmoFL when accuracy under heterogeneity matters most, FedGH when wall-clock time is scarce.","Privacy-preserving deployments gain a concrete ranking: SplitFed keeps almost full accuracy even under strong differential-privacy noise while most others collapse.","Future algorithm papers must report efficiency and privacy numbers on the same multi-organ suite or risk being incomparable.","Benchmark users receive ready hyper-parameter defaults (batch 16, three local epochs) that generalize across the twelve organs."],"fun_headline_variants":["20 FL algorithms ranked across 22 multi-organ imaging datasets","MobenFL scores 20 federated methods on accuracy efficiency privacy","Unified benchmark for FL on 12 organs and clinical transfer cases","22 medical datasets pit 20 FL algorithms on four real-world axes","Open FL scorecard spans multi-organ scans devices and disease shifts"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"The claim that five clients, ResNet-18, a fixed learning rate and Dirichlet alpha 0.5 already capture the resource limits and data skews of real multi-hospital deployments.","fun_headline_variants_meta":{"raw":{"variants":["20 FL algorithms ranked across 22 multi-organ imaging datasets","MobenFL scores 20 federated methods on accuracy efficiency privacy","Unified benchmark for FL on 12 organs and clinical transfer cases","22 medical datasets pit 20 FL algorithms on four real-world axes","Open FL scorecard spans multi-organ scans devices and disease shifts"]},"model":"grok-4.5","effort":"low","cost_usd":0.005018,"raw_usage":{"total_tokens":1357,"prompt_tokens":781,"num_sources_used":0,"completion_tokens":92,"cost_in_usd_ticks":50180000,"prompt_tokens_details":{"text_tokens":781,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":484,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":781,"tokens_out":92,"duration_ms":4799,"temperature":1.0,"reasoning_tokens":484,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T11:07:00.015065+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the exact same twenty algorithms on a new multi-center collection whose client count, scanner mix and class imbalance differ sharply from the five-client Dirichlet-0.5 setup and check whether the recommended algorithm rankings reverse.","supporting_citations":[],"review_version":1}