{"id":"3d774cb3-2f6b-41eb-bf81-28e806438cfd","arxiv_id":"2501.13985","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Pilot is a federated multimodal instruction tuning framework that combines task-specific and client-specific adapters with a cross-task mixture-of-adapters module and Euclidean-distance-based text adapter aggregation.","lead":"This paper introduces Pilot, a framework for federated multimodal instruction tuning that lets distributed devices fine-tune vision-language models on different tasks without sharing data. The method uses a two-stage adapter-on-adapter design with cross-task mixture-of-adapters and distance-based aggregation of text adapters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adaptive text-adapter aggregation (Eqs. 11–12) assumes Euclidean distance in parameter space identifies beneficial client merges, yet no evidence or theory supports this; the largest ablation gains hinge on this assumption.","rationale":"I agree with the reader that the Euclidean-distance assumption in Eqs. 11–12 is the most load-bearing weakness. The paper's central claim is that Pilot enables cross-task collaboration 'without being affected by task heterogeneity,' and the ATA module is one of the two key mechanisms (along with CT-MoA) that purportedly delivers this. The ablation evidence shows ATA contributes materially, but the selection rule itself is never stress-tested: Table 5 only sweeps M and compares against same-task/all-client aggregation, which cannot establish that parameter distance correlates with beneficial merging. The concern is not that Euclidean distance is obviously wrong; it is that no evidence rules out the alternative that any Top-M rule, or random selection, would give similar results. Since the paper reports a single split, no seeds or error bars, and no code, the reported superiority of M=6 could be an artifact. This does not force rejection: the framework is coherent, the experiments are internally consistent, and the reader's CONDITIONAL verdict already reflects the need for additional validation. My read does not move the verdict; it sharpens the specific test that should be required before the aggregation claim is accepted.","tokens_in":13602,"tokens_out":5751,"duration_ms":57383,"concrete_test":"Run the FL-oriented visual-understanding scenario with 5 random seeds and compare the ATA variant (Eqs. 11–12, M=6) against three controls: (a) random selection of M neighbor adapters, (b) cosine-similarity-based selection, and (c) an oracle that selects the M adapters whose pairwise merge yields the highest local validation gain per client. If random or cosine selection matches Pilot's average scores within noise, or if the oracle substantially outperforms Pilot, then Euclidean distance is not the operative mechanism and the aggregation claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central algorithmic novelty that most distinguishes Pilot from FedAvg/FedProx is the adaptive text-adapter aggregation: for each client, Eq. 11 computes Euclidean distances between LoRA text-adapters and Eq. 12 selects the Top-M closest clients for inverse-distance-weighted averaging. This assumes parameter-space proximity is a reliable proxy for 'merging this client's adapter with mine will help my local task.' The paper offers no theoretical justification and no direct empirical validation of this proxy. Table 5 compares 'same task client', 'all clients', and M=5/6/7, but it never tests whether Euclidean distance actually selects the most beneficial partners; any fixed similarity metric or even random Top-M could plausibly behave similarly on this single data split. The ablation in Table 3 shows that removing ATA drops GQA from 50.4 to 49.3, COCO from 124.0 to 114.5, and RefCOCO from 51.0 to 47.5, so the claim that Pilot 'benefits from positive effects while reducing negative effects' rests substantially on this unvalidated selection mechanism. Because only one random split, no error bars, and no code are reported, we cannot tell whether the M=6 advantage is signal or noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Federated Multimodal Instruction Tuning (FedMIT) task and proposes Pilot, a framework for collaboratively fine-tuning multimodal large language models (MLLMs) on heterogeneous visual instruction tasks across clients. Pilot integrates a two-stage 'adapter-on-adapter' design into the vision-LLM connector: stage 1 trains a task-specific adapter and a client-specific adapter with an orthogonality loss, and stage 2 builds a Cross-task Mixture-of-Adapters (CT-MoA) module with cross-task adapters and a router. On the server side, task-specific visual adapters are aggregated per task, while text adapters (LoRA) are aggregated via an adaptive scheme that selects the Top-M closest clients by Euclidean distance (Eqs. 11-12). Experiments on two 9-client, 3-task scenarios (GQA/COCO/RefCOCO and ScienceQA/GQA/OCRVQA) report that Pilot outperforms FedAvg, FedProx, FedAdam, Shepherd, and FedDPA, with ablations supporting the contribution of each module.","tokens_in":1509,"tokens_out":2195,"duration_ms":40861,"significance":"If the reported results hold, the paper makes a useful contribution by identifying and formalizing the FedMIT problem and by demonstrating that a mix of task-specific adapters, cross-task MoA, and distance-based selective aggregation can mitigate task heterogeneity in federated MLLM instruction tuning. The two-stage connector design is intuitive, and the empirical gains over strong baselines (e.g., Tables 1-2) are consistent across both scenarios. The paper also provides a new evaluation scenario for cross-task federated learning with multimodal data. However, the novelty is incremental in its components (adapter-on-adapter, MoA, distance-weighted aggregation are known ideas), and the central claims rest on an experimental validation whose statistical robustness is not established.","major_comments":[{"comment":"The results in Tables 1-5 are reported without error bars, repeated seeds, or statistical significance tests. The improvements over the best baselines are often small (e.g., GQA Client 1: Pilot 51.4 vs FedAdam 50.8 in Table 1; Pilot 49.2 vs FedDPA 47.3 in Table 2). Given a single random data split and no variance information, the claim that Pilot 'achieves state-of-the-art results' is not yet supported. I recommend reporting mean and standard deviation over at least three seeds and, if feasible, a paired significance test across clients.","section":"Experimental Results; Ablation Studies"},{"comment":"The central novelty of the aggregation strategy is the assumption that Euclidean distance in LoRA parameter space identifies clients whose text-adapters are beneficial to merge. The paper offers no theoretical justification, no analysis of the parameter geometry, and no direct validation of this proxy. The ablation in Table 3 shows that removing ATA degrades COCO from 124.0 to 114.5 and RefCOCO from 51.0 to 47.5, so the method's main gain relies on this assumption. Table 5 compares M values and same/all-client aggregation, but it does not include a control such as random Top-M selection, cosine similarity, or an oracle that selects partners by measured task complementarity. Without such a control, the observed gains could be due to selecting any subset of clients rather than to the distance criterion.","section":"Adaptive Text-adapter Aggregation, Eqs. 11-12"},{"comment":"The hyperparameter Top-M is set to 6 in Implementation Details and is then evaluated at M=5,6,7 in Table 5, apparently on the same test sets used for the final results. There is no held-out validation split for hyperparameter selection, so the reported numbers may be optimistically biased. Similarly, the coefficients lambda0, lambda1, lambda2 and other hyperparameters are fixed without sensitivity analysis. The paper should either use a validation split or explicitly acknowledge that the reported configuration was selected on the test data, and provide robustness analysis for the key hyperparameters.","section":"Implementation Details; Further Remarks, Table 5"},{"comment":"The abstract and conclusion state that Pilot learns 'without being affected by the task heterogeneity during instruction tuning.' This is too strong a claim given the experimental scope: only 9 clients and 3 tasks per scenario, 3 communication rounds, and no evaluation under different numbers of clients, tasks, or partial participation. The method deliberately preserves task-specific adapters, so it does not remove heterogeneity effects; it mitigates them. I recommend softening the claim and adding discussion of conditions under which the method might fail (e.g., highly imbalanced data, non-IID label distributions, or more tasks than adapters).","section":"Introduction; Methodology; Conclusion"}],"minor_comments":[{"comment":"The phrase 'without being affected by the task heterogeneity' is repeated several times and overstates the mitigation; consider rewording to 'reduces the impact of task heterogeneity' or 'robust to task heterogeneity in the tested settings.'","section":"Abstract and Introduction"},{"comment":"In Table 1 and Table 2, the baseline is spelled 'FedA VG' and 'FedA VG' with a space and small caps; elsewhere it is 'FedAvg.' Please unify the notation.","section":"Experiment, Baselines"},{"comment":"The evaluation metric is 'loU' (IoU) in the text; use 'IoU' consistently.","section":"Experimental Results"},{"comment":"The total loss formula has an awkward fraction layout: the inner sum over n_k and the outer weighting by n_k/n make the expression redundant; simplify it to a weighted sum over clients.","section":"Equation (2)"},{"comment":"The orthogonality constraint in Eq. (3) is applied to the output features x_t and x_s, but the text says the client-specific adapter produces 'more refined' features. The connection between the Frobenius-norm condition and the semantic goal is not explained; please clarify what property of the features is being enforced.","section":"Stage 1: Task-specific Feature Mining"},{"comment":"The acronym is inconsistently written as CT-MOA and CT-MoA; choose one and use it throughout.","section":"Stage 2: Cross-task Visual Interaction"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the proposed framework is plausible, but the experimental evidence is not yet convincing enough for acceptance: no error bars, no code, and a key aggregation mechanism is only tested indirectly. The authors should be asked to provide multi-seed results, a random-selection baseline for Top-M, and a validation-based hyperparameter selection. If those are supplied, the paper could be acceptable; in its current form it overstates the strength of the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the problem definition. FedMIT—federated instruction tuning where different clients own different multimodal tasks—is a natural next step for the FedIT line, and I don't know of prior work that frames it this way. The architecture is also more interesting than the results: task-specific and client-specific adapters on the vision connector, a difference loss to keep them orthogonal, then a cross-task mixture-of-adapters with task-specific adapters plus cross-task adapters initialized from the local one. That's a coherent design, and the ablation in Table 3 shows each piece adds something. The paper is clearly written and the two scenarios are reasonable testbeds.\n\nThe soft spots are real, and they line up with your reader's concerns. No error bars, no repeated seeds, one random split, no code. Top-M is tuned on the test set (Table 5 shows M=5/6/7 and they pick 6). The adaptive text-adapter aggregation is the load-bearing novelty that separates Pilot from FedAvg and FedProx, and the paper gives no theoretical or empirical reason to believe Euclidean distance in parameter space picks partners whose adapters merge well. Table 3 says ATA alone is worth roughly +0.6 GQA, +5 COCO, +9 RefCOCO over the no-ATA variant, so this unvalidated mechanism is carrying a lot. The stress-test note is fair: a random Top-M or a different similarity metric could behave similarly on this single split, and we can't tell.\n\nThe abstract's claim that the method works \"without being affected by task heterogeneity\" is overstated. Pilot still trails centralized training by a clear margin in every scenario, and on some clients local training is competitive. That is not fatal—the method does beat all five baselines in both settings—but the claim should be softened to something like \"reduces the negative impact of task heterogeneity.\"\n\nI'd also note only three communication rounds, which is a very small regime. The authors mention all baselines use their best settings, but with no seeds we can't verify the differences are beyond noise.\n\nFor a first cut this is a reasonable paper. The task alone justifies attention. But it needs multiple seeds, error bars, a proper test of the aggregation assumption (random Top-M, cosine similarity, or a small synthetic study), and a more measured headline claim. I'd send it to review—a serious referee could push it into shape—but it is not close to acceptance as is.","headline":"Pilot defines a genuinely new federated multimodal instruction-tuning task and a sensible two-stage adapter architecture, but the main aggregation mechanism is unvalidated and the experiments are too thin to support the strong claims.","tokens_in":14413,"tokens_out":1823,"would_cite":false,"duration_ms":18998,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pilot lets devices with different visual tasks share knowledge without sharing data.","keywords":["federated learning","multimodal instruction tuning","parameter-efficient fine-tuning","mixture-of-adapters","task heterogeneity","cross-task knowledge transfer","visual question answering","image captioning"],"falsifier":"Run Pilot on the same two scenarios but replace the Euclidean-distance selection of the Top-M text adapters with a random selection of M clients; if the random selection matches the distance-based selection in average task accuracy, the distance assumption is not driving the reported gains.","tokens_in":13374,"feed_emoji":"🤖","tokens_out":5045,"duration_ms":40980,"temperature":0.7,"pith_summary":"Federated learning lets devices train a shared model without uploading private data, but existing federated instruction-tuning methods assume all clients work on similar language tasks. This paper introduces Federated Multimodal Instruction Tuning (FedMIT), where each client holds a different visual-language task such as captioning, question answering, or grounding, and shows that standard federated baselines degrade when tasks differ. To solve this, the authors propose Pilot, which inserts two stacked stages of lightweight adapters into the visual connector. The first stage separates task-specific from client-specific visual features; the second stage routes each client's input through a mixture of adapters initialized from all tasks, letting one client borrow knowledge from others. An adaptive aggregation step then merges each client's text adapters only with the most similar peers, and the authors report state-of-the-art results across two nine-client, three-task scenarios.","feed_headline":"Pilot beats federated baselines across mixed visual tasks","feed_subtitle":"Two-stage adapter-on-adapter connector lets each device borrow knowledge from other tasks without sharing data.","key_machinery":"The load-bearing components are (1) the two-stage 'adapter-on-adapter' connector: stage one's task-specific adapter $\\psi_t$ and client-specific adapter $\\psi_s$ with difference loss $L_d = \\|\\mathbf{x}_t^\\top \\mathbf{x}_s\\|_F^2$, and stage two's Cross-task Mixture-of-Adapters (CT-MoA), where each non-local task-specific adapter has a cross-task adapter $\\psi^c$ initialized from the local adapter, and a router $\\phi$ with load-balancing loss $L_b$ and router z-loss $L_z$; and (2) the adaptive text-adapter aggregation, which computes Euclidean distances $d_{k,i}$ between LoRA text adapters, keeps the Top-M closest, and weights them by inverse distance. The CT-MoA lets a client activate knowledge from other tasks, while the adaptive aggregation prevents negative transfer by not averaging incompatible text adapters.","core_discovery":"Pilot solves the FedMIT task, defined as collaboratively instruction-tuning an MLLM on K clients whose data cover T different multimodal task types, by making the visual connector itself the site of cross-task communication. The connector is trained in two stages: stage one uses a task-specific adapter and a client-specific adapter with a soft subspace orthogonality loss, so that shared task knowledge and private data patterns do not collide; stage two assembles a Cross-task Mixture-of-Adapters (CT-MoA) with one adapter per task, adds a cross-task adapter on top of each non-local task adapter to smooth the heterogeneity gap, and uses a router with load-balancing and z-losses. On the text side, each client trains a LoRA adapter, and the server aggregates every client's text adapter by selecting the M nearest peers in Euclidean parameter distance and inverse-distance weighting, instead of averaging all clients. Evaluated with LLaVA 1.5 on two scenarios (GQA/COCO/RefCOCO and ScienceQA/GQA/OCRVQA), Pilot outperforms FedAvg, FedProx, FedAdam, FedDPA, and Shepherd, and surpasses local training, which the ablations show is mainly due to the adaptive aggregation and the CT-MoA cross-task adapters.","pith_inferences":["The distance-based aggregation rule is a heuristic that could be probed further: if parameter distance does not track merge compatibility, the method would be selecting partners almost arbitrarily, so a direct study of when parameter distance predicts successful merging would settle the mechanism.","The CT-MoA design assumes one task per client; extending it to clients holding multiple tasks would require routing at the sample level rather than the client level.","Because all task-specific adapters are broadcast to every client, communication cost grows linearly with the number of tasks; hierarchical or clustered aggregation could keep the cross-task benefit while scaling to more tasks.","The same connector-level two-stage adapter idea could be tested on other modality pairs, such as audio-text or video-text, in federated settings."],"forward_implications":["Federated multimodal instruction tuning can work across different visual task types without centralized data collection, as long as the connector is made the locus of cross-task adaptation.","The two-stage connector gives a concrete recipe for separating personalized from shared visual knowledge under task heterogeneity.","Adaptive, distance-based aggregation of text adapters provides a way to reduce negative transfer in heterogeneous federated fine-tuning.","Pilot's state-of-the-art results in both evaluated scenarios suggest that task heterogeneity is not an obstacle to collaborative MLLM tuning, provided the right architecture is used."],"supporting_citations":[{"why":"Defines FedAvg, the base federated optimization that Pilot's aggregation strategy is designed to improve.","marker":"McMahan et al. 2017"},{"why":"Introduces Shepherd, the representative FedIT method that Pilot adapts and compares against in preliminary studies.","marker":"Zhang et al. 2024b"},{"why":"Provides FedProx, a non-IID federated baseline that Pilot must beat to show the benefit of its aggregation.","marker":"Li et al. 2020"},{"why":"Supplies LLaVA 1.5, the MLLM backbone used in all experiments.","marker":"Liu et al. 2024"},{"why":"Supplies LoRA, the parameter-efficient text adapter that is aggregated adaptively in Pilot.","marker":"Hu et al. 2021"},{"why":"Contributes the load-balancing and router z-losses used to stabilize the CT-MoA routing.","marker":"Zoph et al. 2022"},{"why":"Provides the domain-separation soft subspace orthogonality constraint used as the difference loss.","marker":"Bousmalis et al. 2016"}],"fun_headline_variants":["Federated tuning for multimodal LLMs via adapter stacks","Multi-task adapters let devices share knowledge without data","Pilot: cross-task adapters beat baseline federated tuning","Adaptive aggregation and cross-task adapters boost federated tuning","Federated multimodal instruction tuning with cross-task adapters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that if two clients' text adapter parameters are close points in Euclidean parameter space, then merging their adapters will help both, and that the closest M are the best partners; the paper does not prove this correlation and tests the selection size M in only one scenario.","fun_headline_variants_meta":{"raw":{"variants":["Federated tuning for multimodal LLMs via adapter stacks","Multi-task adapters let devices share knowledge without data","Pilot: cross-task adapters beat baseline federated tuning","Adaptive aggregation and cross-task adapters boost federated tuning","Federated multimodal instruction tuning with cross-task adapters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1290,"prompt_tokens":1020,"completion_tokens":270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":188}},"tokens_in":636,"tokens_out":270,"duration_ms":2695,"temperature":1.0,"reasoning_tokens":188,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:56:28.928577+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Pilot on the same two scenarios but replace the Euclidean-distance selection of the Top-M text adapters with a random selection of M clients; if the random selection matches the distance-based selection in average task accuracy, the distance assumption is not driving the reported gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the domain-separation soft subspace orthogonality constraint used as the difference loss."}],"review_version":1}