{"id":"ce66a7ff-47bc-4863-8606-ad1f102a0d50","arxiv_id":"2506.14824","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"FedNano centralizes the frozen LLM on the server, trains lightweight NanoAdapters on clients, and reports higher federated VQA accuracy than FedAvg, FedProx, and FedDPA-F on ScienceQA and IconQA.","lead":"FedNano is a federated learning framework that keeps the large language model on a central server and trains only small low-rank adapters on client devices. The paper reports accuracy gains over federated baselines on two visual question answering benchmarks, along with large claimed reductions in client storage and communication.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.01% communication figure counts only adapter uploads and omits per-step embedding and gradient transfers between client and server, which can dominate total communication.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw: FedNano's communication accounting ignores the per-step embedding and gradient traffic required by server-side LLM execution. I confirm this concern independently from the architecture and Algorithm 1. Because the paper's central contribution is communication efficiency, an unaccounted traffic source of comparable or larger size than the baselines' uploads undermines the main claim. The paper is a plausible engineering composition of split learning and LoRA, but as presented the efficiency and privacy claims are unsupported. The accuracy experiments show only small, single-run gains over FedAvg, and the relevant split-learning baselines are absent, but the communication accounting issue alone is sufficient to reject the paper's central claim. The verdict remains REJECT; no adjustment is needed.","tokens_in":12900,"tokens_out":3206,"duration_ms":37331,"concrete_test":"Measure or simulate end-to-end per-client communication per round for the LLaVA-1.5-7B setup in Table 1, counting (a) client-to-server embeddings for every sample in each local step, (b) server-to-client gradients for every sample, and (c) the 1.05M-parameter NanoAdapter upload. Sum over 10 rounds and one local epoch per round, then compare with FedDPA-F's 180.89M-parameter upload per round (≈723 MB in fp32). If FedNano's total bytes are higher, the 'over 99% communication reduction' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency claim (Abstract, Section 1, Table 1) is that FedNano limits communication to 0.01% of model parameters by transmitting only NanoAdapter updates. This accounting is incomplete. In FedNano's architecture (Figure 2, Algorithm 1), the frozen LLM runs on the server, so every local training step requires the client to send adapted visual/text embeddings to the server for the forward pass and receive gradients back for backpropagation. These per-step transfers are not parameter updates and are excluded from Table 1's 'Server Uploads' column and from the 99% reduction claim. Their size is not negligible: for LLaVA-1.5-7B, the client's projected visual embeddings are 576 tokens × 4096 dimensions, about 9.4 MB per sample in fp32, and the gradients returned are similarly sized. For a batch of 8, one local step already moves roughly 150 MB in each direction. Across 10 communication rounds and one local epoch per round—with typical VQA training sets of hundreds to thousands of images per client—the cumulative embedding/gradient traffic can exceed FedDPA-F's 180.89M-parameter upload (≈723 MB per round) by orders of magnitude. The claim 'only compact NanoAdapter updates are exchanged' (Section 3.2) is therefore false unless embeddings and gradients are also compact, which the paper does not establish. This invalidates the paper's headline communication-efficiency result and the associated conclusion that FedNano is suitable for bandwidth-constrained environments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedNano, a federated tuning framework for multimodal LLMs in which the frozen LLM is hosted on the server and clients train only a lightweight 'NanoEdge' module consisting of frozen modality encoders, a connector, and low-rank adapters (AI for images, AT for text). Aggregation uses a Fisher-information-weighted average (Eq. 1). The authors claim a 95.7% reduction in client-side storage, a 99.4% reduction in transmitted parameters relative to FedDPA-F, and accuracy gains over FedAvg/FedProx/FedDPA-F on ScienceQA and IconQA with MiniGPT-4 and LLaVA-1.5. The paper also reports ablations on data heterogeneity level, client count, cross-task distributions, adapter rank, and communication frequency.","tokens_in":13147,"tokens_out":5669,"duration_ms":61205,"significance":"The architectural idea of centralizing the large backbone on the server and using external, client-side adapters so that clients never store or execute the LLM is a reasonable and potentially useful direction for MLLM federated learning, provided the communication cost is honestly measured. The empirical evaluation uses external benchmarks (ScienceQA, IconQA) and standard FL baselines, so the claimed gains are not circular. However, the headline efficiency result is based on an incomplete measure of communication, and the accuracy comparisons are too weak, with small margins, no error bars, and very small scale, to support the paper's conclusions. With those caveats, the contribution is not yet established; the paper also provides no code or detailed system measurements to support the resource-efficiency claims.","major_comments":[{"comment":"The headline communication-efficiency claim is based on an incomplete accounting. The 0.01% figure and the statement that 'only compact NanoAdapter updates are exchanged' count solely the NanoAdapter parameter deltas (1.05M parameters per round in Table 1) and omit the per-local-step transfers that the architecture in Figure 2 and Algorithm 1 requires: each client must upload adapted visual and text embeddings to the server for the frozen LLM forward pass and receive gradients back for backpropagation through the adapters. For LLaVA-1.5-7B, the projected visual embedding alone is 576 tokens times 4096 dimensions, which is about 9.4 MB per sample in fp32, so with batch size 8 the per-step bidirectional traffic is already on the order of 150 MB; over 10 rounds with hundreds to thousands of local samples this can dominate or exceed the roughly 723 MB per round that FedDPA-F transmits as parameter updates. The paper's own Limitation section repeats the claim that FedNano transmits 'only lightweight adapter updates,' which is contradicted by the architecture. Since the central contribution is suitability for bandwidth-constrained environments, this accounting error invalidates the main efficiency result as stated.","section":"Abstract; Section 3.2; Table 1; Figure 2"},{"comment":"The accuracy comparisons do not support the claim that FedNano 'outperforms' baselines. All numbers are single-seed runs with 5 clients, 10 rounds, and 1 local epoch per round, and the margins are small: on LLaVA-1.5, FedNano exceeds FedAvg by 0.65 points on ScienceQA and 0.32 points on IconQA (Table 2), and at alpha = 5 it leads FedAvg by 0.03 points (Table 3). Without error bars, repeated seeds, or a significance test, these differences are within run-to-run noise for this setting, so the cross-method ranking in Tables 2-5 is not established.","section":"Section 4.2; Tables 2 and 3"},{"comment":"Equation (1) as written does not do dataset-size weighting: the factor |D_k|/sum|D_k| appears in both the numerator and the denominator and cancels, leaving theta_global = (sum F_k theta_k)/(sum F_k). The surrounding text describes the method as combining posterior estimates with dataset-size weighting; the formula should be corrected, and the claimed benefit of client-data-volume weighting should be verified or removed. This does not by itself invalidate the aggregation idea, but it makes the description in Section 3.4 inaccurate.","section":"Section 3.4, Eq. (1)"}],"minor_comments":[{"comment":"The FIM computation is specified only as 'Compute FIM F_k'; Section 3.4 mentions a diagonal approximation from squared gradients, but the exact computation, including which parameters are included and how many samples are used, should be stated for reproducibility.","section":"Algorithm 1"},{"comment":"Figure 2 shows 'Client to Server' and 'Server to Client' arrows that appear to represent per-step activation and gradient transfers, but the caption only describes adapter uploads; the communication protocol should be described precisely in the caption or text.","section":"Figure 2"},{"comment":"The paper does not state the number of random seeds or show variance for any experiment; adding this information is necessary to support the reported rankings.","section":"Section 4.2"},{"comment":"Several typos and spacing errors remain, such as 'F edNano' and 'LLaV A-1.5' in the text; these should be cleaned up in revision.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The communication-accounting issue is load-bearing and not fixable by a local revision: the architecture inherently requires per-step embedding and gradient transfers, so the central efficiency claim, as stated, is false. The accuracy evidence is also too weak to support the empirical conclusions. The paper would need a substantial redesign and re-evaluation to address these concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper in one line: FedNano keeps the frozen LLM on the server and puts lightweight adapters on clients, which is a sensible answer to a real deployment problem, but the headline \"0.01% communication\" figure is misleading because it counts only adapter uploads and ignores the per-step embedding and gradient transfers the architecture requires.\n\nWhat's genuinely useful here: centralizing the LLM is a clean way to sidestep the client-capacity limit, and the split into vision and text NanoAdapters is a natural fit for MLLMs. Applying Fisher merging to adapter aggregation is also a reasonable, if not deeply novel, choice. The authors clearly state the problem and the architecture is easy to follow.\n\nThe soft spots are proportionate to how they matter. The communication accounting is the big one. Figure 2 and Algorithm 1 show the client sending adapted embeddings to the server for every forward pass and receiving gradients back. That traffic scales with batch size, sequence length, and number of local steps, not with adapter size. For LLaVA-1.5-7B, a single sample's embeddings are already tens of megabytes. The paper's 99% reduction claim only works if you ignore this, and the text even says \"only compact NanoAdapter updates are exchanged,\" which is false on the paper's own architecture.\n\nThe experiments are also thinner than the claims. Five clients, ten rounds, one seed, and accuracy gaps around one point over FedAvg. That is not enough to support \"outperforms\" in a strong sense, and there is no split-learning baseline, which is the closest existing approach. The \"first\" claim in the abstract is also overreaching; split learning with client-side encoders has been around for years.\n\nThat said, the core direction is not junk. With corrected communication accounting, a split-learning baseline, and multi-seed runs, the paper could be a credible contribution. As written, the central efficiency and privacy claims do not hold up, and the empirical evidence is too weak to compensate. The right move is to send it to review anyway, because the problem is important and a good referee can force the authors to fix the accounting and comparisons. I would not cite the current version, but I would want to see the revised one.","headline":"The central communication-efficiency claim is undercut by the architecture's per-step embedding/gradient traffic, but the server-hosted-LLM idea is a real direction worth a serious look.","tokens_in":13714,"tokens_out":1505,"would_cite":false,"duration_ms":19798,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedNano claims federated tuning of multimodal LLMs works with the frozen LLM on the server and only small client-side adapters, cutting client storage by 95% and communication to 0.01% of parameters while matching or beating federated…","keywords":["federated learning","multimodal large language models","parameter-efficient fine-tuning","low-rank adaptation","server-side deployment","non-IID data","visual question answering","Fisher merging"],"falsifier":"Run FedNano and measure total bytes exchanged per round, including the embedding tensors clients must send for the server's frozen LLM forward pass and the gradient tensors returned, then compare that per-round byte count with FedDPA-F's transmitted adapter updates. If the embedding and gradient transfer exceeds the compared method's uploads, the 0.01% communication claim does not hold.","tokens_in":12671,"feed_emoji":"⚙️","tokens_out":7603,"duration_ms":79759,"temperature":0.7,"pith_summary":"FedNano tries to make federated learning practical for multimodal large language models, which are too large to run on client devices. It claims that the LLM can be frozen and kept on the server while each client trains only a small module, NanoEdge, made of frozen modality encoders and trainable low-rank adapters. The paper reports that this cuts client-side storage by more than 95% and the number of transmitted parameters to 0.01% of the model, and that the approach matches or beats existing PEFT-based federated baselines on ScienceQA and IconQA. A reader should care because, if correct, it removes the main obstacle, client-side model size, to federated tuning of billion-parameter multimodal models.","feed_headline":"Clients skip the LLM in federated tuning, cutting load by 95%","feed_subtitle":"With the language model frozen on the server, clients trade tiny adapter updates and still beat federated baselines.","key_machinery":"The load-bearing object is NanoEdge: a client-side module composed of frozen modality-specific encoders (a vision transformer and a text embedding layer), a frozen modality connector, and trainable NanoAdapters, one for vision and one for text, implemented as LoRA-style low-rank down- and up-projections external to the LLM. These adapters sit at the boundary between the connector and the server-hosted LLM, so clients never execute the LLM. Aggregation is carried by Fisher merging, which weights each client's adapter updates by a diagonal Fisher Information Matrix approximated from squared gradients, serving as the precision matrix in a Laplace approximation of the client posterior and reducing the cost from $O(|\\theta|^2)$ to $O(|\\theta|)$.","core_discovery":"The authors' claim is that adaptation can be decoupled from the LLM itself: instead of inserting adapters into the model and running it on each client, FedNano places the frozen LLM on the server and attaches lightweight NanoAdapters at the connector interface on the client. Only these adapter updates are aggregated, using Fisher-weighted averaging to handle non-IID data. On the paper's own experiments, FedNano reaches 77.05% average on ScienceQA and 76.42% on IconQA with MiniGPT-4, and 81.41% and 78.04% with LLaVA-1.5, exceeding FedAvg, FedProx, and FedDPA-F in each case, while using far fewer client resources.","pith_inferences":["The paper does not measure the per-step embedding and gradient traffic that the server-hosted LLM requires; adding that traffic would be needed to validate the 0.01% communication claim.","Because clients send adapted embeddings to the server, the privacy guarantee is structural rather than cryptographic, and stronger protection would likely require differential privacy or encryption.","The adapter interface is modular enough that the same scheme could extend to other modality pairs or to server-hosted models that expose only an embedding API, since the adapters never touch internal LLM weights.","The FedNano-EF variant indicates most of the accuracy gains survive without the extra FIM computation, suggesting a cost-accuracy trade-off that the main paper does not emphasize."],"forward_implications":["Federated tuning of MLLMs no longer requires shipping or executing the LLM on clients, so devices with limited storage and compute can participate.","Per-round communication drops to the size of the adapter updates (0.01% of model parameters), making low-bandwidth federated settings more plausible.","Under non-IID splits, including strongly heterogeneous Dirichlet partitions and a cross-task four-client setup, the Fisher-guided aggregation reports better average accuracy than FedAvg, FedProx, and FedDPA-F.","The dual-modality design matters: using either the visual or textual adapter alone yields clearly worse results than using both.","Accuracy improves with higher adapter rank and more frequent communication rounds, at the cost of more transmitted data."],"supporting_citations":[{"why":"Supplies the LoRA low-rank decomposition used by the NanoAdapters.","marker":"Hu et al., 2021"},{"why":"Source of Fisher-weighted averaging adopted for aggregation of client updates.","marker":"Matena and Raffel, 2022"},{"why":"Defines the FedAvg baseline and the federated learning setting FedNano compares against.","marker":"McMahan et al., 2017"},{"why":"FedDPA-F, the PEFT-FL baseline used in the 95% storage and 99% communication reduction comparison.","marker":"Yang et al., 2024"},{"why":"Provides the MiniGPT-4 backbone and connector design that NanoEdge extends.","marker":"Zhu et al., 2023"},{"why":"Provides the LLaVA-1.5 backbone and visual instruction tuning baseline.","marker":"Liu et al., 2024b"},{"why":"Supplies the ScienceQA benchmark used for federated evaluation.","marker":"Lu et al., 2022"},{"why":"Supplies the IconQA benchmark used for federated evaluation.","marker":"Lu et al., 2021"},{"why":"Provides the diagonal Fisher approximation that keeps the aggregation cost linear in the number of parameters.","marker":"Kirkpatrick et al., 2017"}],"fun_headline_variants":["Federated LLM tuning without the LLM on clients","Server keeps LLM, clients send tiny adapters","95% less client storage via server-side LLM","Federated tuning: tiny adapters beat full model baselines","Client-side adapters only, server holds LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central efficiency claim rests on counting only the tiny adapter updates as communication, while ignoring the per-step embeddings and gradients that must travel between client and server at every training step, and that unmeasured traffic could dwarf the advertised 0.01%.","fun_headline_variants_meta":{"raw":{"variants":["Federated LLM tuning without the LLM on clients","Server keeps LLM, clients send tiny adapters","95% less client storage via server-side LLM","Federated tuning: tiny adapters beat full model baselines","Client-side adapters only, server holds LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000926,"raw_usage":{"total_tokens":3970,"prompt_tokens":946,"completion_tokens":3024,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2943}},"tokens_in":562,"tokens_out":3024,"duration_ms":20972,"temperature":1.0,"reasoning_tokens":2943,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:13:22.424257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FedNano and measure total bytes exchanged per round, including the embedding tensors clients must send for the server's frozen LLM forward pass and the gradient tensors returned, then compare that per-round byte count with FedDPA-F's transmitted adapter updates. If the embedding and gradient transfer exceeds the compared method's uploads, the 0.01% communication claim does not hold.","supporting_citations":[],"review_version":1}