{"id":"364cd96c-428c-466a-9a20-d30bf0ee9900","arxiv_id":"2501.03292","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FedMME fuses vision-LLM-generated text features with image features and uses client voting to improve one-shot federated medical image classification.","lead":"This paper introduces FedMME, a one-shot federated learning method that combines medical images with text reports generated by a vision large language model to improve diagnostic accuracy. Across four medical datasets and several non-IID splits, it reports higher accuracy than existing one-shot federated learning baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main comparison is confounded: FedMME’s reported gains over one-shot baselines are measured with an extra frozen 11B vLLM+BERT text pathway that no baseline receives, so the claimed advantage is not isolated to the proposed federated multi-modal ensemble mechanism.","rationale":"Reader returned CONDITIONAL; I agree with that verdict. My pass did not find internal mathematical inconsistency in the pipeline (Eqs. 2–7), but it did find the headline claim is not yet isolated from a major confound: the vLLM+BERT feature pathway is available only to FedMME. The case study (§4.5) and the textual-feature-dimension ablation (§4.4.2) are genuine supporting evidence that the text signal matters inside FedMME, but they do not show that a comparably strong vision-only ensemble would not achieve the same numbers. Missing code, missing error bars, and the notation mismatch around Eq. (6) amplify the verification risk. Since the reported results are plausible and the method is coherent, the appropriate disposition remains CONDITIONAL; my concern strengthens the condition but does not move the verdict.","tokens_in":13675,"tokens_out":8321,"duration_ms":104084,"concrete_test":"On RSNA α=0.3 with the same 5-client Dirichlet split, run a matched-capacity ablation: (A) FedMME as reported; (B) FedEnsemble using the same ResNet-18 visual features but a frozen CLIP ViT-B/32 or DINOv2 image encoder in place of the vLLM+BERT pathway (same linear head, same voting); and (C) FedMME with vLLM reports replaced by a fixed string so BERT contributes no image-specific signal but the architecture is unchanged. If (B) or (C) reaches roughly 87.9%, the vLLM textual pathway is not the source of the headline gain and the comparison is confounded; if both fall below FedMME by a large margin, the multimodal text signal is load-bearing and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (abstract; §4.3) is that FedMME outperforms existing one-shot FL methods, e.g. 87.93% vs 70.39% (FedISCA) on RSNA at α=0.3. In every comparison in Table 1, FedMME is the only method that augments the vision model with Llama-3.2-11B-Vision-Instruct textual reports plus BERT embeddings (Eqs. 3–4) before fusion (Eq. 6). FedEnsemble is explicitly a 'single modal vision model' (§4.2), and the other one-shot baselines are not reported to use the vLLM/BERT pathway either. Thus the table contrasts two things at once: (i) the proposed one-shot multi-modal ensemble algorithm and (ii) access to a large pretrained vision-language model. The magnitude of the headline gap—17.5 points on RSNA α=0.3—is entirely consistent with the hypothesis that the gain comes from the extra pretrained model, not from the multi-modal fusion or the ensemble design. No ablation compares FedMME against a same-capacity vision-only baseline or against a version with text features ablated, so the contribution claim in §1 is underdetermined. A secondary concern is that Eq. (6) and Alg. 2 write fcombine = fvisual + f*textual while the text says 'concatenate'; if the implementation literally adds a 512-d visual vector to a 128-d reduced text vector, the dimensions are incompatible, and if it concatenates, the notation is wrong. Either way, implementation details needed to reproduce the claim are missing. No code or error bars are provided, so the three-trial averages in Table 1 cannot be checked.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes FedMME, a one-shot federated ensemble learning framework for medical image classification. Each client trains a local model on images using ResNet-18 visual features augmented by textual features obtained by prompting Llama-3.2-11B-Vision-Instruct to generate a report for each image and embedding that report with BERT; the textual features are dimensionality-reduced and combined with visual features, and the resulting per-client classifiers are aggregated on a central server by equal-weight voting. The paper reports held-out test accuracy on Blood, Derma, RSNA, and Diabetic Retinopathy datasets under Dirichlet-based non-IID partitions (alpha = 0.6, 0.3, 0.1), compares with FedAvg, DAFL, DENSE, FedISCA, and FedEnsemble, and claims consistent superiority, including a 17.5% accuracy improvement over one-shot baselines on RSNA at alpha = 0.3. Ablations examine the choice of vision LLM, the dimensionality of the textual features, and the number of local training epochs.","tokens_in":14003,"tokens_out":6402,"duration_ms":60012,"significance":"The motivating problem is real: one-shot FL reduces communication, and multimodal features can improve medical image classification. If the empirical claims were supported by a controlled comparison, FedMME would be a useful contribution, and the ablation of the vision LLM choice is a positive feature. The paper also reports honest held-out accuracies rather than circular metrics. However, the current evidence is underdetermined: the main comparison changes both the ensemble/fusion algorithm and the availability of a large external vLLM plus BERT feature extractor, so the claimed gains cannot be attributed to the proposed mechanism. The absence of error bars, code, and a validation-based selection protocol further limits the strength of the claims. The central idea is defensible, but the manuscript needs a substantially more controlled evaluation before the contribution can be assessed.","major_comments":[{"comment":"Every comparison in Table 1 gives FedMME an extra frozen 11B vision-language model plus BERT text pathway that the baselines do not receive: FedEnsemble is described in Sec. 4.2 as a single-modal vision model, and the remaining baselines are not reported to use any text branch. The comparison therefore changes two variables at once: (i) the proposed one-shot ensemble/fusion algorithm and (ii) access to a large pretrained multimodal feature extractor. The headline gap on RSNA at alpha=0.3 (87.93 versus 70.39 for FedISCA) is consistent with the hypothesis that the gain comes from the extra pretrained model rather than from the multi-modal fusion or ensemble design. Please add an ablation that gives a vision-only FedEnsemble baseline the same backbone, initialization, and training budget, or ablates the text branch in FedMME while keeping the vLLM, so that the contribution of the proposed mechanism is isolated.","section":"Sec. 4.2, Table 1, Eqs. (3)-(4)"},{"comment":"Equation (6) and Algorithm 2 define fcombine = fvisual + f*textual, while Sec. 3.2.2 and Fig. 2 say the features are concatenated. The described dimensions (512-d visual, 128-d reduced text, Sec. 4.4.2) make a literal vector addition impossible. Please specify the actual fusion: if it is concatenation, correct Eq. (6) to [fvisual; f*textual]; if it is addition after a linear projection, describe that projection. This is required to reproduce the method.","section":"Eq. (6) and Algorithm 2, step 3"},{"comment":"The textual feature dimension used in the main results appears to be selected from test-set performance: Fig. 4 reports test accuracy for several dimensions and 128 is chosen as optimal, but no validation split or selection rule is described. Because this selection is made on the same test partition used for the headline numbers in Table 1, the reported accuracies are at risk of optimistic bias from test-set model selection. Please either fix the dimension a priori, select it on a validation split, or report both the selection and test numbers separately.","section":"Sec. 4.1 and Sec. 4.4.2"},{"comment":"The FedISCA rows in Table 1 contain exact duplicate values across different partitions (Blood: 53.61 at alpha=0.6 and alpha=0.1; Derma: 53.86 at alpha=0.6 and alpha=0.1, while Derma at alpha=0.3 is 16.11). These duplicates are suspicious and directly affect the claimed improvements, since FedISCA is the strongest one-shot baseline on several rows. Please verify the baseline evaluation and report corrected numbers.","section":"Table 1"},{"comment":"The claim that the framework is suited to resource-constrained medical applications is not supported by the experimental setup in Sec. 4.1, where all clients are simulated on a single machine with 1TB RAM, a 256-core CPU, and an A100 40GB GPU, and where each client would need to run an 11B-parameter vision LLM to generate reports. No measurement of per-client memory, inference time, or communication payload is reported. Please either report these costs and validate on a more realistic client configuration or soften the applicability claim to an explicit assumption and limitation.","section":"Sec. 1 and Sec. 4.1"}],"minor_comments":[{"comment":"The phrase more than 17.5% in accuracy should read 17.5 percentage points, since it is a difference between two accuracy values.","section":"Abstract"},{"comment":"The citation for the 11B vision model is given as Chi et al. (2024), Llama Guard 3 Vision, which appears to be a different model from Llama-3.2-11B-Vision-Instruct; please cite the correct model release or repository.","section":"References"},{"comment":"The variable y is used both as the label in the dataset and as the loop index for vote accumulation; please use different names to avoid confusion.","section":"Algorithm 1"},{"comment":"The case study in Fig. 8 shows the vision LLM misclassifying the severity, yet the text claims the report contributes valuable insights; consider adding a quantitative analysis of how often the generated report is correct or useful rather than relying on a single example.","section":"Sec. 4.5"},{"comment":"No code, data splits, or random seeds are provided, which makes the three-trial averages in Tables 1 and 2 non-reproducible; a code and data availability statement would be helpful.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case: the central empirical claim is not yet demonstrated because the main comparison is confounded by the additional vLLM and BERT feature extractor, and the baseline table contains suspicious duplicate entries. The idea is promising and the manuscript is clearly written, but the evaluation needs a same-capacity vision-only baseline, corrected fusion notation, and error bars or per-trial values. I would be willing to review a revised version that addresses these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know the core idea is real: combining offline vLLM captioning, BERT features, and one-shot federated ensemble voting for medical images is not in the cited literature, and the authors are plausibly first to try it. The paper reads as an honest engineering attempt, not a shell game. Four datasets, multiple non-IID splits, a case study showing the vLLM gives useful textual descriptions even when it misclassifies—these are genuine assets. The reported accuracies are held-out numbers, so nothing is fabricated on the face of it.\n\nThe soft spot is load-bearing, not cosmetic. In every row of Table 1, FedMME is the only method that sees the frozen Llama-3.2-11B-Vision-Instruct plus BERT text pathway. FedEnsemble is explicitly single-modal vision. So the 17.5-point gap on RSNA at alpha=0.3 contrasts two things at once: the proposed ensemble/fusion algorithm and access to an enormous pretrained vision-language model. Without an ablation that removes the text pathway while keeping the same backbone, or a same-capacity vision-only baseline, the central claim in the abstract and Section 4.3 is underdetermined. This is not a minor cleanliness issue; it is the difference between demonstrating a new federated mechanism and demonstrating that a big frozen model helps.\n\nThe notation problem in Eq. (6) is also real: the text says concatenation, the equation says fvisual + f*textual. If implemented literally, a 512-d vector plus a 128-d reduced vector cannot work; if implemented as concatenation, the equation is wrong. Either way, reproduction is blocked. On top of that, three-trial averages come with no error bars, no code, and the textual feature dimension is picked using test-set ablations (Figure 4). Those are fixable, but they are exactly the kinds of details a reviewer should chase.\n\nA smaller honest note: on Diabetic, FedMME gets 31.93% at alpha=0.3. It beats the baselines, but the absolute number is weak. The contribution is a relative improvement over other one-shot FL methods, not a clinically strong diagnostic system.\n\nWho is this for? Researchers working on communication-efficient federated learning or multimodal fusion in medical imaging. They will get a usable recipe and a clear positioning statement, but they should not trust the headline numbers until the confound is resolved.\n\nMy recommendation: send to peer review with the expectation of major revision. Ask for the text-ablated control, the same-capacity vision baseline, corrected notation, error bars, and code. If those come back, this could be a solid subfield paper.","headline":"A genuinely new multi-modal recipe for one-shot federated learning, but the headline comparison doesn't isolate the method's contribution because FedMME is the only arm given the 11B vLLM+BERT pathway.","tokens_in":14578,"tokens_out":1560,"would_cite":false,"duration_ms":18125,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that one-shot federated medical image classifiers improve when each client generates a text report with a vision LLM, embeds it with BERT, and votes across models.","keywords":["one-shot federated learning","federated ensemble learning","multi-modal medical image analysis","vision large language model","BERT text features","non-IID data","medical image classification"],"falsifier":"Run the RSNA experiment at $\\alpha=0.3$ with the text report replaced by a fixed, meaningless string, while keeping every other setting identical; if accuracy stays at roughly 87.93% instead of dropping well below it, the paper's attribution of the gain to the textual modality is wrong.","tokens_in":13435,"feed_emoji":"🩺","tokens_out":11563,"duration_ms":96656,"temperature":0.7,"pith_summary":"FedMME is a one-shot federated ensemble method: each client trains a local model on its own medical images, sends the trained model to the server once, and the server combines the models by equal-weight voting. The local model is multimodal: a vision large language model writes a text report for each image, BERT embeds that report, the embedding is reduced in dimension, and the reduced text features are concatenated to ResNet-18 visual features before a fully connected classifier. The paper reports that this consistently beats one-shot federated baselines on Blood, Derma, RSNA, and Diabetic Retinopathy data under Dirichlet data splits with $\\alpha = 0.6$, $0.3$, and $0.1$, including an accuracy gain of more than 17.5 percentage points over the best one-shot baselines on RSNA at $\\alpha=0.3$. The claim matters because it suggests medical institutions can obtain multi-modal accuracy benefits in a single communication round without sharing raw images or reports.","feed_headline":"Vision LLM reports boost one-shot federated medical models","feed_subtitle":"FedMME adds BERT text features to client models and votes once, beating one-shot baselines on four medical datasets.","key_machinery":"The machinery is FedMME's two-phase client model. Phase one extracts features: ResNet-18 gives $f_{\\text{visual}}\\in\\mathbb{R}^{512}$; Llama-3.2-11B-Vision-Instruct writes a report $r$; BERT gives $f_{\\text{textual}}\\in\\mathbb{R}^{768}$. Phase two fuses: a dimensionality-reduction step $f^*_{\\text{textual}}=\\text{DR}(f_{\\text{textual}})$ shrinks the text embedding so it cannot dominate, and the combined vector $f_{\\text{combine}}=f_{\\text{visual}}+f^*_{\\text{textual}}$ (the paper writes $+$ for concatenation) goes into a fully connected layer. The server then applies equal-weight voting over the $N$ client models. The dimensionality reduction is doing load-bearing work: without it, text features at 512 dimensions degrade performance, and at 128 dimensions the gain is largest.","core_discovery":"The central claim is that unimodal one-shot federated ensembles leave diagnostic accuracy on the table, and the missing signal can be supplied locally by a vision LLM's free-form reports. In FedMME each client converts every image into a short radiology-style report, turns that report into a 768-dimensional BERT embedding, reduces the embedding to a smaller size, and concatenates it with the 512-dimensional visual feature vector. The authors report that this simple fusion, combined with voting, outperforms FedAvg, DENSE, DAFL, and FedISCA in the one-shot setting, and outperforms a unimodal FedEnsemble that uses only visual features. The reported margins are largest on RSNA (87.93% vs 70.42% for FedEnsemble at $\\alpha=0.3$) and Blood (80.12% vs 71.03%).","pith_inferences":["The paper implicitly treats the vision LLM as a shared prior that clients do not have to agree on; a natural test is whether a smaller, open-weight captioning model run on CPU-only clients preserves most of the gain.","Because the comparison is against image-only models, part of the gain may be a regularization or diversity effect from adding an auxiliary feature stream rather than true semantic understanding; an ablation with randomized text embeddings would separate these.","The privacy story is less clean than the communication story: although no raw report is shared, the 11-billion-parameter model's memory and compute footprint may be prohibitive on typical clinical hardware, so deployment value depends on the local hardware assumption.","If the mechanism transfers, one-shot federated systems outside medicine could adopt local captioning as a cheap auxiliary channel, but the cost-benefit will depend on how strongly the target images are describable in text."],"forward_implications":["If the reported gains hold, a hospital can join a one-shot federated study without any extra communication round; the only added cost is local inference of a vision LLM on its own images.","The number of clients helps: on the Blood dataset at $\\alpha=0.3$, FedMME accuracy rises from 80.12% with 5 clients to 86.23% with 20, indicating the ensemble benefits from diversity.","The text stream should be kept small; reducing BERT features to 128 dimensions gives the best accuracy, while letting them reach 512 dimensions hurts, so the textual modality is an auxiliary, not a replacement.","The method is not tied to one vision LLM: with prompts adjusted to describe rather than classify, Llama-3.2-11B-Vision-Instruct and ChatGPT-4o-Vision yield comparable results.","Even under the strongest non-IID split tested ($\\alpha=0.1$), FedMME remains ahead of every one-shot baseline on all four datasets, so the benefit is not limited to mildly heterogeneous data."],"supporting_citations":[{"why":"Defines the federated learning protocol; FedAvg is the first baseline FedMME is compared against.","marker":"McMahan et al., 2017"},{"why":"Introduces one-shot federated learning, the setting FedMME operates in.","marker":"Guha et al., 2019"},{"why":"DENSE is a data-free one-shot FL baseline using knowledge distillation.","marker":"Zhang et al., 2022"},{"why":"DAFL is a one-shot FL baseline also based on knowledge distillation.","marker":"Chen et al., 2019"},{"why":"FedISCA is the state-of-the-art one-shot medical FL baseline that FedMME claims to surpass.","marker":"Kang et al., 2023"},{"why":"BERT converts the vLLM-generated reports into the textual feature vectors used in fusion.","marker":"Devlin, 2018"},{"why":"Provides Llama-3.2-11B-Vision-Instruct, the vision LLM that writes the reports.","marker":"Chi et al., 2024"},{"why":"ResNet-18 is the visual backbone and defines the 512-dimensional visual feature space.","marker":"He et al., 2016"},{"why":"MedMNIST supplies the Blood and Derma datasets used in the main comparison.","marker":"Yang et al., 2023"},{"why":"RSNA pneumonia dataset is where the more than 17.5% accuracy improvement is reported.","marker":"Rsna, 2019"}],"fun_headline_variants":["FedMME: One-shot federated learning with vision LLM reports","Vision LLM text boosts one-shot federated medical accuracy","FedMME beats federated baselines using vision LLM+BERT fusion","Single-round federated ensemble gets boost from vision LLM reports","FedMME: Multimodal one-shot FL for medical image analysis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every client can run an 11-billion-parameter vision large language model locally on each medical image to produce a useful report, and that the resulting text features, after dimension reduction, add enough signal beyond a 512-dimensional visual feature to improve classification; the experiments were run on one large machine with an A100 GPU and do not test constrained hospital hardware.","fun_headline_variants_meta":{"raw":{"variants":["FedMME: One-shot federated learning with vision LLM reports","Vision LLM text boosts one-shot federated medical accuracy","FedMME beats federated baselines using vision LLM+BERT fusion","Single-round federated ensemble gets boost from vision LLM reports","FedMME: Multimodal one-shot FL for medical image analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000774,"raw_usage":{"total_tokens":3461,"prompt_tokens":1014,"completion_tokens":2447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2355}},"tokens_in":630,"tokens_out":2447,"duration_ms":16472,"temperature":1.0,"reasoning_tokens":2355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:01:26.128240+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the RSNA experiment at $\\alpha=0.3$ with the text report replaced by a fixed, meaningless string, while keeping every other setting identical; if accuracy stays at roughly 87.93% instead of dropping well below it, the paper's attribution of the gain to the textual modality is wrong.","supporting_citations":[],"review_version":1}