{"id":"de373fba-8a64-4e70-b581-5a1815acae8d","arxiv_id":"2506.09638","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FedVLMBench systematically benchmarks federated fine-tuning of vision-language models and finds that a 2-layer MLP connector with joint connector-LLM training is optimal for encoder-based models, while vision-centric tasks are more sensitive to non-IID data.","lead":"FedVLMBench is a new benchmark for training vision-language models across distributed, privacy-protected datasets. It compares two model architectures, four fine-tuning strategies, five federated learning algorithms, and six datasets, and reports practical guidance such as using a 2-layer MLP connector trained together with the language model for encoder-based models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Vision-vs-text heterogeneity comparison is confounded by unmatched client counts and partition severity; matched-heterogeneity check needed before Takeaway 5 is accepted.","rationale":"My concern targets the causal interpretation of the benchmark's headline finding. Even if the models are representative, the experimental design does not distinguish 'vision-centric tasks suffer more under heterogeneity' from 'vision-centric datasets were given stronger non-IID partitions.' The most direct fix is a matched-heterogeneity ablation. I did not make seed selection the primary concern because the paper explicitly argues the 2-layer MLP is preferred for stability, but the absence of error bars across Tables 3-6 remains a secondary weakness and should be addressed in the same reruns. The reader's weakest assumption about model representativeness is real but secondary; once the heterogeneity confound is settled, testing a second architecture per family would further support generalization.","tokens_in":12505,"tokens_out":5399,"duration_ms":59957,"concrete_test":"Run a matched-heterogeneity ablation on all four single-task datasets: fix the same number of clients (e.g., K=10) and the same Dirichlet alpha (e.g., 0.1 and 0.5) for Fed-SLAKE, Fed-ScienceCap, Fed-FGVC, and Fed-RadGenome, and report mean plus/minus std over at least 3 seeds. Compute normalized drop (IID minus non-IID)/IID per task. If the vision-centric drop remains significantly larger under equal alpha and K, the heterogeneity-sensitivity claim is supported; if not, the difference in Tables 4 and 5 is attributable to unequal partition severity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Takeaway 5 claim that vision-centric tasks are significantly more sensitive to data heterogeneity than text-centric tasks across both VLM architectures. The evidence in Tables 4 and 5 does not hold heterogeneity fixed across the task types being compared. Fed-SLAKE has 3 clients partitioned by modality; Fed-ScienceCap uses up to 27 category-based clients; Fed-FGVC uses up to 30 manufacturer-based clients plus additional Dirichlet partitions; Fed-RadGenome has 3 organ-based clients. Client count and partition rule directly determine non-IID severity, so the larger relative drops on Fed-FGVC and Fed-RadGenome (e.g., encoder-free FedAvg: 0.721 to 0.493 on FGVC and 0.604 to 0.485 on RadGenome) could reflect a stronger partition, not a property of vision-centric tasks. The paper reports no common heterogeneity metric (e.g., a shared Dirichlet alpha or label-distribution divergence) across datasets, so the contrast is confounded. This matters because Takeaways 4 and 5 and the abstract's central claim are used to motivate new vision-centric heterogeneity algorithms.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FedVLMBench, a benchmark for federated fine-tuning of vision-language models (VLMs) that integrates two VLM architectures (encoder-based LLaVA-1.5 with CLIP ViT-B/32 and LLaMA3.2-3B; encoder-free Show-O), four fine-tuning strategies, five FL algorithms, and six datasets covering four single-task and two multitask scenarios. The main empirical claims are: (1) a 2-layer MLP connector with concurrent connector and LLM tuning is the optimal configuration for encoder-based VLMs in FL; (2) text-centric tasks benefit from LLM tuning while vision-centric tasks benefit from connector tuning; (3) vision-centric tasks are significantly more sensitive to data heterogeneity than text-centric tasks across both architectures; and (4) federated multitask training achieves near-ceiling performance comparable to centralized training. The authors also state that conventional FL optimizers are largely ineffective for vision-centric heterogeneity and call for new solutions.","tokens_in":12739,"tokens_out":5418,"duration_ms":57120,"significance":"If the findings hold, FedVLMBench is a useful resource: it provides public datasets and code, covers task types that existing FL-VLM benchmarks omit (report generation, visual grounding, detection), and includes both encoder-based and encoder-free architectures, which is a real gap in the literature. The paper also gives actionable guidance for practitioners, e.g., concurrent connector-LLM tuning and task-dependent tuning priorities. However, the strength of the contributions is currently diminished by the lack of statistical support and by a controlled-heterogeneity design that does not yet match the strength of the headline claims. The dataset and benchmark infrastructure are valuable, but the stated conclusions—especially the vision-versus-text sensitivity contrast and the optimal-connector claim—require additional experiments to be considered established.","major_comments":[{"comment":"The claim that vision-centric tasks are significantly more sensitive to data heterogeneity than text-centric tasks is confounded by unmatched partition rules and client counts across the compared datasets. Fed-SLAKE uses 3 clients partitioned by modality; Fed-ScienceCap uses up to 27 category-based clients; Fed-FGVC uses up to 30 manufacturer-based clients plus additional Dirichlet partitions; Fed-RadGenome uses 3 organ-based clients. Client number and partition rule directly control non-IID severity, so the larger relative drops on Fed-FGVC and Fed-RadGenome (e.g., encoder-free FedAvg accuracy 0.721 to 0.493 on FGVC and 0.604 to 0.485 on RadGenome) could reflect stronger partition-induced heterogeneity rather than a property of vision-centric tasks. To support Takeaway 5, the authors should compare tasks under matched heterogeneity, e.g., a common Dirichlet alpha with equal client counts, or at least report a quantitative heterogeneity metric (such as label-distribution divergence) for each split and demonstrate that the vision-centric drop is larger after controlling for that metric.","section":"Section 5.3 / Takeaway 5, with Tables 4 and 5"},{"comment":"The comparison that selects the linear connector as suboptimal is biased by the authors' own admission that the linear connector results were 'derived from optimal hyperparameter tuning, including the selection of the most favorable random seeds,' while the MLP results are not described as receiving the same favorable-seed treatment. The text also states that the linear connector is 'highly susceptible to parameter initialization' and exhibits 'significant fluctuations,' but no variance or seed distribution is reported. Consequently, Table 3 does not provide a symmetric comparison and cannot support the conclusion that the 2-layer MLP is more effective and more stable. Please report mean and standard deviation across multiple seeds for all connector types, or apply the same seed-selection rule to every configuration, so that the comparison is fair.","section":"Section 5.2 / Table 3 and Takeaway 1"},{"comment":"The 'near-ceiling' conclusion for federated multitask learning is based on point estimates without error bars or significance tests. Several FedAvg results actually exceed the MT-Central baseline (e.g., encoder-free Fed-Nature VQA accuracy 0.781 vs. 0.752 and CIDER 0.930 vs. 0.912), which is possible but should be addressed explicitly. Without repeated runs and variance reporting, the reader cannot judge whether 'near-ceiling' is a real phenomenon or an artifact of a single seed. Please provide mean and standard deviation over at least three seeds, or a paired significance test, for the multitask results in Table 6.","section":"Section 5.4 / Takeaway 6 and Table 6"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: 'LLAV A 1.5' in Section 5.1, 'as denoted as denoted as F-CL' in Section 5.2, 'Fed-RadGnome' in the Table 5 header, and inconsistent capitalization of 'LLaMA'/'LLAMA'.","section":"Throughout"},{"comment":"References [34] and [35] appear to refer to the same paper (FedMLLM) with different years; please disambiguate or merge them.","section":"References"},{"comment":"The text states that Fed-FGVC includes 'four heterogeneous partitions are generated using varying Dirichlet coefficients,' but the main tables report only a single IID/non-IID column, and the Dirichlet results are referenced as a figure in the supplementary material. Since the supplementary file is not part of this manuscript, please make the Dirichlet results (or a summary) available in the main text or an accessible appendix.","section":"Section 3 and Section 5.2"},{"comment":"The 'Central' baseline is not fully defined in the main text; it is described as training on aggregated client data, but it is unclear whether it uses the same local-update schedule, LoRA rank, and number of communication rounds as the FL methods, and whether it is evaluated under IID or non-IID conditions. Please specify this in the experimental setup.","section":"Tables 4 and 5"},{"comment":"The notation 'Mlp2x' and 'Mlp6x' is not defined in the main text; please define the number of layers and hidden sizes in the caption or in Section 5.1.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a potentially useful benchmark contribution, but the headline claims currently exceed the evidence. The most important issues are the confounded heterogeneity comparison and the seed-selection bias in the connector comparison; both are fixable with additional controlled experiments and variance reporting. The paper's self-citation rate is noticeable (several references from the same group), but the citations themselves are relevant to the FL background and are not a problem per se. If the authors can add matched-heterogeneity experiments and repeated-seed statistics, a revised version could become a solid empirical benchmark paper; as it stands, I would not recommend acceptance without those changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: FedVLMBench is a genuinely useful benchmark, and the two new multi-task datasets (Fed-Nature, Fed-Med) fill a real gap. But the headline claim that vision-centric tasks are more heterogeneity-sensitive than text-centric tasks is not established by the tables as presented, because heterogeneity severity is not held fixed across datasets. The paper also admits that its linear-connector result came from selecting the most favorable seeds, which softens the connector comparison.\n\nWhat's new: the Fed-Nature and Fed-Med datasets are real contributions—task-per-client multi-task VLM data is missing from prior benchmarks like FedMLLM. The systematic matrix (two architectures, four tuning strategies, five FL algorithms, six datasets) is broader than what's out there and gives practical guidance. The 2-layer MLP connector finding is plausible, and the paper is honest about linear's seed sensitivity. The observation that encoder-free VLMs lose more on vision tasks under non-IID is consistent with the idea that a trainable connector helps, even if the evidence isn't airtight.\n\nSoft spots: the main one is the confound noted above. Fed-SLAKE has 3 clients partitioned by modality, Fed-ScienceCap up to 27 category-based clients, Fed-FGVC up to 30 manufacturer-based clients plus Dirichlet partitions, Fed-RadGenome 3 organ-based clients. Client count and partition rule directly determine non-IID severity, so the larger drops on FGVC and RadGenome could simply reflect a stronger partition rather than a vision-specific property. There is no shared heterogeneity metric (e.g., a common Dirichlet alpha) across the datasets. The paper mentions Dirichlet experiments in the supplement, but the main-text tables don't control for this. The abstract and Takeaway 5 should be toned down until a matched-heterogeneity comparison is shown.\n\nSecond, there are no error bars anywhere, and the code/data are promised but not linked in the manuscript. For a benchmark paper, that's a practical issue: the benchmark's value depends on others being able to run it. Third, the seed-selection procedure for the linear connector is an explicit limitation, and it means the claim that a 2-layer MLP is strictly better than linear should be framed as \"more stable\" rather than \"higher accuracy\" in the seed-optimal case.\n\nWho this is for: people working on federated fine-tuning of multimodal models, especially in privacy-sensitive domains. They'll get a solid testbed and actionable starting points. It deserves a serious referee, but the referee should ask for matched-heterogeneity analysis, variance reporting, and a working code/data link before the central claims are accepted.","headline":"Useful new benchmark with two genuinely new multi-task federated datasets, but the vision-vs-text heterogeneity claim is confounded by unmatched partition severity and the paper needs variance reporting.","tokens_in":13236,"tokens_out":4072,"would_cite":true,"duration_ms":39206,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedVLMBench establishes that a 2-layer MLP connector with joint connector and LLM tuning is the best federated fine-tuning configuration for encoder-based vision-language models, and that vision-centric tasks are far more sensitive to…","keywords":["federated learning","vision-language models","fine-tuning benchmark","non-IID data heterogeneity","multimodal multitask learning","parameter-efficient fine-tuning","vision-language connector","LoRA"],"falsifier":"Run the same four fine-tuning strategies and three connector depths on a held-out encoder-based VLM with a different pretrained visual encoder and a larger LLM, and on a real-world non-IID partition with more than three clients; the central claim would be falsified if the 2-layer MLP joint-tuning advantage disappears or if text-centric tasks show non-IID drops comparable to vision-centric ones.","tokens_in":12358,"feed_emoji":"📊","tokens_out":5542,"duration_ms":52772,"temperature":0.7,"pith_summary":"The paper builds FedVLMBench, a systematic benchmark for federated fine-tuning of vision-language models that spans two architectures, four fine-tuning strategies, five federated algorithms, and six datasets. Using it, the authors try to establish practical rules: for encoder-based VLMs, a 2-layer MLP connector outperforms both a linear layer and a 6-layer MLP, and tuning the connector and LLM together beats tuning the connector first. They also claim that text-centric tasks such as VQA and captioning remain largely robust to non-IID client data, while vision-centric tasks such as classification and detection degrade sharply, and that existing FL optimizers do not fix this. In multi-task federated settings, however, both architectures reach near-centralized performance. If right, these results give practitioners concrete defaults and expose where new FL algorithms are needed.","feed_headline":"2-layer MLP connector wins federated VLM tuning","feed_subtitle":"Benchmark across 6 datasets, 5 FL algorithms finds vision tasks most hurt by non-IID data.","key_machinery":"The load-bearing object is the vision-language connector, the trainable projection (linear, 2-layer MLP, or 6-layer MLP) that maps visual encoder features into the token space consumed by the LLM. The benchmark systematically varies connector depth, tuning order (connector-only, LLM-only, joint, two-stage), and data partition, making the connector the variable that carries the main takeaways. A second mechanism is the comparison between encoder-based and encoder-free architectures, which lets the paper attribute heterogeneity sensitivity to the presence or absence of a trainable alignment module.","core_discovery":"The paper's central discovery is that the connector between visual encoder and LLM is the decisive design point for federated fine-tuning of encoder-based VLMs. Among the tested configurations, a 2-layer MLP connector combined with simultaneous connector and LoRA-based LLM tuning yields the best balance of accuracy, stability, and computational cost. The paper further finds that vision-centric tasks suffer significantly larger performance drops under non-IID data than text-centric tasks, across both encoder-based and encoder-free VLM architectures, and that standard FL optimizers such as FedProx and FedYogi fail to recover this loss. In contrast, when each client is assigned a distinct task in a multitask setup, federated training approaches centralized-training quality, regardless of architecture.","pith_inferences":["A testable extension the paper does not run: varying the visual encoder and LLM scale would show whether the 2-layer MLP recommendation is tied to the specific LLaVA-1.5/LLaMA3.2-3B pairing or is a general property of encoder-based VLMs.","The pronounced vision-centric sensitivity could be caused by the connector being trained against inconsistent per-client visual distributions; a follow-up could test whether regularizing connector updates or aggregating connector parameters separately from LLM LoRA parameters reduces the drop.","The multi-task recovery result suggests that task-dedicated clients act like natural personalization; a follow-up could compare multi-task FL against personalized FL baselines on the same datasets to see whether the near-ceiling performance comes from task separation rather than from the FL algorithm."],"forward_implications":["Federated deployments of encoder-based VLMs should default to a 2-layer MLP connector and simultaneous connector plus LoRA LLM tuning, since this combination balances accuracy, stability, and cost in the paper's experiments.","When a federated task is text-centric, most of the tuning budget should go to the LLM; when it is vision-centric, connector tuning matters at least as much as LLM tuning.","Existing FL optimizers such as FedProx and FedYogi will not recover the vision-centric accuracy lost under non-IID partitions; the paper's results call for new FL methods designed around the connector and visual features.","In multi-task deployments where each client owns one task, federated fine-tuning can approach centralized-training quality, so privacy-preserving multimodal systems are viable despite non-IID data."],"supporting_citations":[{"why":"Supplies the encoder-based VLM architecture (LLaVA) used for all encoder-based experiments.","marker":"[20]"},{"why":"Supplies the encoder-free VLM (Show-O) used as the architecture contrast.","marker":"[32]"},{"why":"Supplies LoRA, the parameter-efficient tuning method used in every fine-tuning strategy.","marker":"[10]"},{"why":"Supplies FedAvg, the base FL aggregation algorithm and the training loop the benchmark builds on.","marker":"[23]"},{"why":"Supplies FedProx, one of the heterogeneity-targeted FL baselines the paper evaluates.","marker":"[16]"},{"why":"Supplies FedAdam and FedYogi, the adaptive FL optimizers compared across all settings.","marker":"[28]"},{"why":"Supplies SLAKE, the data source for Fed-SLAKE and the VQA component of Fed-Med.","marker":"[19]"},{"why":"Supplies FGVC-Aircraft, the data source for the vision-centric classification dataset Fed-FGVC.","marker":"[22]"},{"why":"Supplies RadGenome-Chest CT, the data source for the detection tasks in Fed-RadGenome and Fed-Med.","marker":"[42]"},{"why":"Supplies RefCOCO, the data source for visual grounding and caption generation tasks in Fed-Nature.","marker":"[13]"}],"fun_headline_variants":["2-layer MLP connector is key for federated VLM tuning","Vision tasks most hurt by non-IID data in federated VLM","Benchmark shows 2-layer MLP connector wins for federated VLM","Federated multitask VLM tuning matches centralized quality","First systematic benchmark for federated VLM fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on LLaVA-1.5 with CLIP ViT-B/32 plus LLaMA3.2-3B and Show-O being representative of encoder-based and encoder-free VLMs, and on the paper's synthetic and simulated non-IID partitions standing in for real federated deployments.","fun_headline_variants_meta":{"raw":{"variants":["2-layer MLP connector is key for federated VLM tuning","Vision tasks most hurt by non-IID data in federated VLM","Benchmark shows 2-layer MLP connector wins for federated VLM","Federated multitask VLM tuning matches centralized quality","First systematic benchmark for federated VLM fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000896,"raw_usage":{"total_tokens":3879,"prompt_tokens":981,"completion_tokens":2898,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2809}},"tokens_in":597,"tokens_out":2898,"duration_ms":20904,"temperature":1.0,"reasoning_tokens":2809,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:42:33.040201+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four fine-tuning strategies and three connector depths on a held-out encoder-based VLM with a different pretrained visual encoder and a larger LLM, and on a real-world non-IID partition with more than three clients; the central claim would be falsified if the 2-layer MLP joint-tuning advantage disappears or if text-centric tasks show non-IID drops comparable to vision-centric ones.","supporting_citations":[{"cited_title":"Visual instruction tuning.NIPS, 36:34892– 34916, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the encoder-based VLM architecture (LLaVA) used for all encoder-based experiments."},{"cited_title":"Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas","cited_arxiv_id":null,"evidence_quote":"Supplies FedAvg, the base FL aggregation algorithm and the training loop the benchmark builds on."},{"cited_title":"Federated optimization in heterogeneous networks, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies FedProx, one of the heterogeneity-targeted FL baselines the paper evaluates."},{"cited_title":"Brendan McMahan","cited_arxiv_id":null,"evidence_quote":"Supplies FedAdam and FedYogi, the adaptive FL optimizers compared across all settings."},{"cited_title":"Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies SLAKE, the data source for Fed-SLAKE and the VQA component of Fed-Med."},{"cited_title":"Fine-grained visual classification of aircraft, 2013","cited_arxiv_id":null,"evidence_quote":"Supplies FGVC-Aircraft, the data source for the vision-centric classification dataset Fed-FGVC."},{"cited_title":"Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies RadGenome-Chest CT, the data source for the detection tasks in Fed-RadGenome and Fed-Med."},{"cited_title":"ReferItGame: Referring to objects in photographs of natural scenes","cited_arxiv_id":null,"evidence_quote":"Supplies RefCOCO, the data source for visual grounding and caption generation tasks in Fed-Nature."}],"review_version":1}