{"id":"d7e85679-0d45-4247-b085-dfb9dbae0252","arxiv_id":"2412.19654","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A cross-silo federated learning method uses one-time foundation-model API queries on public data and asymmetric dual knowledge distillation to improve small, data-poor medical clients.","lead":"FedHelp is a federated learning framework that lets small medical clinics with little data borrow knowledge from bigger hospitals and from paid AI model APIs, without sharing patient data. It reports accuracy gains of up to 47% for the smallest clients on skin-lesion classification, while also cutting communication costs by replacing large client models with small proxies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run results without error bars leave the headline small-client gains indistinguishable from noise; repeated-seed evaluation is needed before the central claim can be accepted.","rationale":"The paper has real strengths: a clear motivation, a complete framework with ablations, multiple tasks (classification and 2D/3D segmentation), and released code. The ablation study usefully attributes the bulk of the improvement to the asymmetric distillation module rather than the foundation-model APIs, which weakens the reader's stated weakest_assumption. However, the central empirical claim is 'significant performance improvement,' and the evidence for that claim consists of single-run accuracy numbers on test sets as small as 12 to 45 images. For the flagship Client 6 result, the difference between FedHelp and the best baseline is roughly 6 correct images out of 45; with typical run-to-run variation in deep learning, such a gap is not reliable. The same concern applies to several other small-client rows across Tables 3 and 5. Since the paper's conclusion is explicitly framed around these numbers, the absence of repeated-seed evaluation is the most load-bearing issue. The correct remedy is straightforward: add multiple seeds, error bars, and a significance test, and then re-examine whether the claimed gains persist. This does not require rejecting the method; it requires establishing the claim empirically. Therefore the verdict should remain conditional on those additional experiments rather than moving to acceptance or rejection.","tokens_in":19953,"tokens_out":6324,"duration_ms":403566,"concrete_test":"Re-run the Fed-ISIC19 classification experiment (Table 1) with at least 10 random seeds for FedHelp, FedMD, FedGH, and FedHelp- using identical hyperparameters, the same public-data/API setup, and a fixed validation split for early stopping (never the test split). Report per-client mean accuracy, standard deviation, and 95% bootstrap confidence intervals, plus a paired significance test (e.g., McNemar on the 45-image Client 6 test set, or a Wilcoxon signed-rank test across seeds) for the FedHelp-versus-best-baseline difference on Client 6 and on average accuracy. If the 95% CI for the Client 6 difference includes 0, or if the average gain is not significant after correcting for multiple clients, the central claim of a 47.02% improvement should be substantially qualified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing weakness is evidentiary, not architectural: every headline number in Tables 1, 3, 5, and 6 is a single run, with no error bars, no seeds, and no significance test. This matters because the marquee result--Client 6 on Fed-ISIC19 improving from 0.2718 to 0.3996 (47.02%)--is computed on a 45-image test set. The raw difference is about 6 correct images; treating the two accuracies as independent binomial proportions gives a standard error of roughly 0.10, so the observed 0.128 gain is only about 1.3 standard errors and is not distinguishable from noise at the 5% level. Similar issues affect the small-client entries in Tables 3 and 5, where test sizes are as small as 12, 24, and 37 images. Without variance estimates or repeated runs, the abstract's phrase 'significant performance improvement' is unsupported, and the central claim rests on point estimates that could easily reorder across random seeds. Notably, the reader's weakest_assumption about cross-domain transfer of foundation-model knowledge is not the bottleneck: the paper's own ablation in Section 4.2.2 shows that removing the foundation-model APIs (FedHelp-) reduces average accuracy much less than the headline gain over baselines, locating the main contribution in the distillation module. The missing error bars, by contrast, directly undermine the quantitative claim of a 47.02% relative improvement and the 10.38% average gain.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes FedHelp, a cross-silo federated learning framework designed for settings where developed-region clients have large models and abundant data while underserved-region clients have small models and scarce data. FedHelp obtains logits from foundation-model APIs on public data and uses them, through a shared feature extractor, to train small-client surrogate models; it also trains large-client proxies and exchanges knowledge through a forward and a ranking-based backward distillation loss. The global model is obtained by aggregating the small surrogate/proxy models. The paper reports experiments on melanoma classification (Fed-ISIC19), pneumonia classification, 2D lung segmentation, and 3D brain segmentation, with consistent improvements over the included baselines.","tokens_in":20285,"tokens_out":7073,"duration_ms":56677,"significance":"If the reported improvements are reproducible, FedHelp is a practically motivated contribution: it addresses heterogeneous client capacities and asymmetric reciprocity with a communication-efficient aggregation of small models, and it provides ablations and a cost analysis. The paper also includes a source-code link and covers both classification and segmentation. However, the central empirical claim currently rests on single-run point estimates, the closest prior method (FedType) is not benchmarked, and a footnote about the foundation models is inaccurate. These issues make the current evidence conditional rather than conclusive.","major_comments":[{"comment":"All accuracy and Dice results are reported as single-run point estimates, with no standard deviations, seeds, or significance tests. This is load-bearing because the headline gain—Client 6 on Fed-ISIC19, from 0.2718 (FedMD) to 0.3996 (FedHelp), quoted as 47.02% relative improvement—is computed on a 45-image test set. Treating the two accuracies as independent binomial proportions gives a standard error of approximately 0.10, so the observed absolute gain of 0.128 is only about 1.3 standard errors and is not statistically significant at the 5% level. The analogous small-client entries in Tables 3 and 5 use test sets of 12, 24, and 37 images, and the segmentation small client in Table 6 has 7 or 15 test images. Repeating the experiments with multiple seeds and reporting means with confidence intervals (or a paired test across seeds) is necessary before the abstract's 'significant performance improvement' can be accepted.","section":"Tables 1, 3, 5, 6; Abstract"},{"comment":"FedType [43] is described in Section 2.2 as the most relevant work and as addressing the same asymmetrical-reciprocity problem between large and proxy models, but it is absent from the baseline list in Section 4.1.1 and therefore from all comparison tables. Without benchmarking FedType or explicitly justifying its exclusion, the claim that FedHelp outperforms state-of-the-art baselines is not established against the closest competitor.","section":"Section 2.2, Section 4.1.1"},{"comment":"The footnote in Section 4.1.2 states that the CLIP models selected as foundation-model APIs 'were pretrained on the CIFAR-100 dataset.' This is not correct for the CLIP models of Radford et al., which are pretrained on WebImageText. If the authors instead fine-tuned the CLIP models on CIFAR-100 (or on the medical public data, as described for NCT-CRC-HE-100K in Section 4.4), this must be stated precisely, because it changes how the knowledge acquired in Eq. (1) should be interpreted: API knowledge that comes from in-distribution public data is not evidence for cross-domain foundation-model transfer.","section":"Footnote 2, Section 3.2"},{"comment":"The ablation study concludes that the primary performance improvement stems from the asymmetric dual knowledge distillation rather than from the foundation-model APIs, based on the observation that FedHelp− (no APIs) drops less than the gain over baselines. Figure 3 provides no numerical values or variance information, and the comparison is again single-run. With the small test sizes noted above, this attribution of the improvement to the distillation module is not statistically supported and should either be backed by repeated runs or be stated more cautiously.","section":"Section 4.2.2"}],"minor_comments":[{"comment":"Typos: 'Comparision' in Section 4.2.1 and Table 5 heading, 'Abalation Study' in Section 4.2.2, 'Averge' in Figure 3 caption, and 'Switerzland' in the author affiliation.","section":"Throughout"},{"comment":"The paper says the framework has 'three key components' but then lists four (knowledge acquisition, small client training, large client training, and global model learning); renumber or rephrase.","section":"Section 1"},{"comment":"The sets Ω and Ω′ are not formally defined (Ω′ presumably denotes the complement of Ω in the class-index set), and Φ is only implicitly the softmax normalizer; define these symbols at first use.","section":"Equations (5) and (13)"},{"comment":"The KL-divergence distillation losses do not state whether softmax probabilities are used and whether a temperature parameter is applied; please specify the exact form and any temperature.","section":"Equations (4) and (12)"},{"comment":"The heterogeneous baselines are said to use the public data as part of model input, but Appendix C does not give the exact public-data protocol for FedGH, FedKEAF, and FCCL, so it is unclear whether the baselines are given the same access to D_p as FedHelp.","section":"Section 4.1.1 and Appendix C"},{"comment":"'Accuracy' should be labeled 'pixel accuracy' at first mention, and the averaging convention for Dice (per-image versus global) should be stated.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The omission of FedType is particularly important because it is the authors' own ICML 2024 work; resolving it is a matter of both empirical completeness and novelty disclosure. I would not recommend acceptance until the seeded reruns and the FedType comparison are provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, FedHelp is a real extension of FedType, not a rehash: the one-time API query on public data plus surrogate models sharing a feature extractor plus a ranking-based backward distillation is a concrete combination that isn't in the cited prior work. Second, the empirical core is currently weaker than the prose. Every number in Tables 1, 3, 5, and 6 is a single run. The marquee result—Client 6 on Fed-ISIC19 going from 0.2718 to 0.3996—is about 6 correct images on a 45-image test set; that difference is roughly 1.3 standard errors. The same issue hits the smallest clients in the pneumonia and segmentation tables. So the abstract's 'significant performance improvement' is not supported by the evidence as reported.\n\nWhat the paper does well: the framework is clearly motivated, the design choices are explained, and the ablation in Section 4.2.2 is honest. It shows that removing the foundation-model APIs costs less than the headline gain over baselines, locating the main contribution in the distillation module. That is exactly the kind of self-critical experiment you want. The resource analysis is useful, and the code is linked.\n\nThe soft spots, in proportion: missing error bars/seeds is the load-bearing one, and it is fixable. Missing FedType as a baseline is the second issue; the paper spends a paragraph differentiating from it, so the reader deserves the head-to-head. Third, the footnote claiming the CLIP models were 'pretrained on the CIFAR-100 dataset' is factually wrong—CLIP is trained on image-text pairs, not CIFAR-100. If the authors meant they fine-tuned on CIFAR-100, that is a different setup and needs to be stated correctly, because it affects how we read the public-data transfer result.\n\nThe central architectural claim is plausible, and the consistency across four medical tasks is a point in its favor, but the statistics need to be re-done. This is a revise-and-resubmit, not a reject: the method deserves referee time, and the authors have shown they can do an honest ablation. If they come back with repeated-seed results, a FedType comparison, and a corrected experimental description, the paper could be solid.","headline":"A sensible FedType follow-up with an honest ablation, but single-run numbers on tiny test sets make the headline small-client gains statistically unverified.","tokens_in":20784,"tokens_out":2550,"would_cite":false,"duration_ms":23459,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedHelp claims that in cross-silo medical federated learning, asymmetric dual knowledge distillation plus one-time foundation-model guidance lifts the smallest clients' accuracy by up to 47% over the best baseline.","keywords":["Federated learning","Healthcare disparity","Knowledge distillation","Model heterogeneity","Medical image classification","Medical image segmentation","Foundation model API","Asymmetric reciprocity"],"falsifier":"Run FedHelp on a small-client task whose private data domain is far from the public data, for example histopathology images with CIFAR-100 as the public set, and compare small-client accuracy against a variant with the foundation-model query term removed; equal accuracy would show the cross-domain transfer assumption contributes nothing.","tokens_in":1736,"feed_emoji":"🩺","tokens_out":1739,"duration_ms":66030,"temperature":0.7,"pith_summary":"FedHelp is a federated learning framework designed for collaborations between well-resourced \"large\" clients and data-scarce \"small\" clients, such as hospitals in developed versus underserved regions. The paper's central claim is that this asymmetry can be turned into an advantage: small clients' diagnostic accuracy rises substantially, by 47% relative on the smallest client in the melanoma benchmark, while large clients still benefit and pay far less in communication cost. The framework works by having small clients train small surrogate models guided by logits obtained once from foundation model APIs on public data, and large clients train small proxy models that exchange knowledge with their large models through a forward and a backward distillation step. All small models, surrogates and proxies, are then aggregated on the server, so the expensive large models never need to be uploaded. If the claim holds, federated learning can be reconfigured as a genuinely reciprocal arrangement in which data-rich institutions help lift data-poor ones without the poor ones dragging everyone down.","feed_headline":"FedHelp lifts its smallest medical client's accuracy by 47%","feed_subtitle":"Asymmetric distillation lets underserved clinics gain most from federated learning without uploading large models.","key_machinery":"The central mechanism is the asymmetric dual knowledge distillation, paired with a knowledge-guided surrogate for small clients. Forward distillation uses the Kullback-Leibler divergence to transfer the large client model's output distribution to a small proxy model; backward distillation uses only the top-ranked class indices from the proxy to raise the corresponding probabilities in the large model, a deliberately loose constraint that avoids polluting the large model with noisy small-model logits. Small clients train a surrogate model whose feature extractor is shared with the main small client model, jointly optimizing a private-data cross-entropy loss and a public-data loss that matches the surrogate's output to the one-time API logits from foundation models. The server aggregates only the small surrogate and proxy models with FedAvg, which is what makes the framework communication-efficient.","core_discovery":"The paper's central discovery claim is that the asymmetry between a few data-rich \"large\" clients and many data-poor \"small\" clients, normally a liability in federated learning, can be made into the engine of the whole system. On the Fed-ISIC19 melanoma benchmark, FedHelp reports an average client accuracy of 0.5091, a 10.38% relative improvement over the best baseline, and on the smallest client the accuracy jumps from 0.2718 (FedMD, the best heterogeneous baseline for that client) to 0.3996, a 47.02% relative gain. Similar gains appear in binary pneumonia classification and in 2D lung and 3D brain segmentation. The authors attribute the gains to two mechanisms acting together: knowledge-guided surrogate training that borrows foundation-model logits through a shared feature extractor, and an asymmetric dual knowledge distillation that lets large and small models exchange knowledge in both directions without forcing the large model to absorb low-quality small-model logits. Their own ablation, however, shows that removing the foundation-model API queries entirely produces a smaller drop than the gain they report, which they read as evidence that the distillation module carries most of the benefit.","pith_inferences":["A direct extension would be to replace the foundation-model API queries with other public-data teachers, such as an open pretrained model, to test whether the 47% effect persists; the paper's ablation suggests the distillation module, not the API choice, is the main driver.","The soft ranking-based backward distillation is a generic recipe that could transfer to any federated setting where a small aggregated model must influence heterogeneous large clients without degrading them.","Before deployment, the privacy properties of the shared feature extractor would need explicit accounting, since parameter sharing between clients still carries information even though raw data never leaves a client.","A natural stress test is to vary the data-size gap between large and small clients to map where the reported gains attenuate and whether the framework remains beneficial when clients are more balanced."],"forward_implications":["On Fed-ISIC19, FedHelp raises average client accuracy to 0.5091, a 10.38% relative improvement over the best baseline, with the smallest client's accuracy more than 47% higher than the best baseline's accuracy for that client.","Because only small surrogate and proxy models are uploaded, replacing a ResNet110 large model with a ResNet20 proxy cuts per-round communication cost by roughly 84%.","The framework extends beyond classification to 2D lung and 3D brain segmentation, with consistent gains across clients in both tasks.","Large clients also improve, so the arrangement is reciprocal rather than one-directional charity: the largest melanoma client reaches 0.5922 accuracy under FedHelp.","The ablation indicates that the asymmetric dual distillation, not the API query itself, is the main driver of the reported gains."],"supporting_citations":[{"why":"Supplies the FedAvg aggregation rule used on the server to combine the uploaded small surrogate and proxy models.","marker":"[23]"},{"why":"Provides the FedMD heterogeneous baseline whose 0.2718 accuracy on the smallest client is the comparison point for the 47.02% relative gain.","marker":"[15]"},{"why":"Supplies the CLIP pretraining behind the two foundation models queried as APIs to generate public-data logits.","marker":"[27]"},{"why":"Supplies the ViT-L/14 architecture used as one of the foundation models in the classification experiments.","marker":"[5]"},{"why":"Provides the FLamby data partitions for Fed-ISIC19 and Fed-IXI that define the large and small client splits.","marker":"[25]"},{"why":"Supplies the MedSAM foundation-model API used to guide small-client training in the segmentation experiments.","marker":"[20]"},{"why":"Is the most related prior work on asymmetrical reciprocity, and FedHelp positions its ranking-based loss against FedType's conformal uncertainty sets.","marker":"[43]"},{"why":"Supplies the basic knowledge-distillation formulation that the forward distillation step from large to proxy models builds on.","marker":"[7]"},{"why":"Provides the ProxyFL proxy-model sharing approach that the paper contrasts with its own private-versus-proxy distinction and server-side aggregation.","marker":"[12]"},{"why":"Provides the strongest homogeneous baseline (PFedBayes), used as a reference for the average accuracy improvements reported for FedHelp.","marker":"[52]"}],"fun_headline_variants":["Asymmetric FL boosts smallest clinic accuracy by 47%","47% accuracy leap for underserved clinics via FedHelp","Tiny clients gain most from asymmetric federated learning","FedHelp: asymmetric distillation lifts small-client accuracy"],"cache_read_input_tokens":22912,"weakest_assumption_plain":"The load-bearing premise is that knowledge distilled once from foundation-model APIs on public data transfers to each small client's private medical data through a shared feature extractor; if that cross-domain transfer is weak, the surrogate training contributes little, and the entire underserved-client benefit rests on the distillation module alone.","fun_headline_variants_meta":{"raw":{"variants":["Asymmetric FL boosts smallest clinic accuracy by 47%","47% accuracy leap for underserved clinics via FedHelp","Tiny clients gain most from asymmetric federated learning","FedHelp: asymmetric distillation lifts small-client accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1412,"prompt_tokens":998,"completion_tokens":414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":351}},"tokens_in":614,"tokens_out":414,"duration_ms":5109,"temperature":1.0,"reasoning_tokens":351,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:00:27.140444+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FedHelp on a small-client task whose private data domain is far from the public data, for example histopathology images with CIFAR-100 as the public set, and compare small-client accuracy against a variant with the foundation-model query term removed; equal accuracy would show the cross-domain transfer assumption contributes nothing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FLamby data partitions for Fed-ISIC19 and Fed-IXI that define the large and small client splits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the most related prior work on asymmetrical reciprocity, and FedHelp positions its ranking-based loss against FedType's conformal uncertainty sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ProxyFL proxy-model sharing approach that the paper contrasts with its own private-versus-proxy distinction and server-side aggregation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the strongest homogeneous baseline (PFedBayes), used as a reference for the average accuracy improvements reported for FedHelp."}],"review_version":1}