{"id":"f367ff24-3880-4209-aba3-a5fff86173ce","arxiv_id":"2608.08138","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A server-side foundation model can be fine-tuned in a federated way through bidirectional knowledge distillation from lightweight client proxies onto LoRA adapters, improving average top-1 accuracy over FedPromo by 3.9% across five fine-grained domains.","lead":"EFFEKT is a federated learning method where small phone-sized models on user devices work with a large server-side vision model, and the server learns new categories by distilling knowledge from the small models into lightweight adapters. The approach reports top-1 accuracy gains of about 4 percent over prior art across five fine-grained image domains while keeping client computation light.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 3.9%/2.7% gains are conditional on matched public pretraining data; without a controlled similarity sweep, the source-pair selection in Table 1 is the load-bearing favorable condition.","rationale":"The reader's weakest_assumption identifies the same load-bearing premise: server-side distillation on task-specific public data only helps the private target classes if those public data are sufficiently aligned with the private data. I agree with that assessment. The paper itself provides the strongest evidence for the concern in Section A.5 and in the Out-of-Domain experiments of Table 8 and Table A.6, where removing the alignment makes EFFEKT's advantage over FedPromo disappear or reverse. The ablation studies and on-device deployment are genuinely supportive within the matched-source setting, and the framework is not internally inconsistent. However, the central claim as stated in the Introduction is broader than what the evidence supports: the 3.9% top-1 and 2.7% top-5 improvements are averages over five hand-picked, largely aligned source-target pairs, and no experiment measures performance at intermediate similarity levels. Thus the verdict should remain CONDITIONAL rather than being upgraded to ACCEPT, but the concern is already reflected in the reader's verdict, so no change is needed.","tokens_in":23652,"tokens_out":8087,"duration_ms":85700,"concrete_test":"For each target domain, run EFFEKT and FedPromo with source datasets spanning a similarity gradient: the matched source, an unrelated fine-grained source (e.g., CUB200 for MilitaryAircraft, StanfordCars for NABirds), and ImageNet-1k. Plot EFFEKT minus FedPromo top-1 and top-5 accuracy against the DINO/CLIP-I similarities from Figure A.9. If the gain crosses zero at similarity values comparable to the reported 0.67 average, the central claim must be explicitly restricted to scenarios with aligned public pretraining data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on C2S/JA distillation on D^di_p (Eqs. 1 and 2) transferring client knowledge to the server-side FM for private target classes. Because the source and target class sets are disjoint, client heads can only provide meaningful logits on public images when the source distribution is aligned with the target distribution. Section A.5 quantifies this alignment with DINO/CLIP-I similarities averaging about 0.67, with MilitaryAircraft markedly lower. The Out-of-Domain experiments (Table 8 and Table A.6) show that when this alignment is removed by using ImageNet-1k, EFFEKT's advantage over FedPromo disappears (28.3 vs 28.5 top-1 averaged), and per-domain EFFEKT trails FedPromo on NABirds and OxfordPets and trails FedAvg+EMA on Military top-5. The Impact Statement's admission that EFFEKT 'requires pretraining data with a reasonable alignment' is therefore not a peripheral caveat: it is the very condition under which the headline gains are obtained. No experiment interpolates between the matched source-target pairs and the ImageNet extreme, so the 3.9% top-1 and 2.7% top-5 averages are established only for favorably selected source domains, not as a general property of the method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EFFEKT, a multi-domain federated learning framework in which lightweight client-side proxy models (MobileNetV3-small) train only a small classification head, while a server-side DINOv2 foundation model is updated via domain-specific LoRA adapters. Two new server-side distillation objectives, Clients-to-Server (C2S, Eq. 1) and Joint Alignment (JA, Eq. 2), transfer client knowledge to the foundation model and realign the proxy feature space using public task-specific pretraining datasets. The evaluation compares EFFEKT with six baselines (FedAvg, FedAvg+EMA, FedProx, MOON, FedHEAL, FedPromo) on five fine-grained domain pairs, reports a multi-domain inference setting, on-device deployment on Raspberry Pi and Jetson devices, and extensive ablations.","tokens_in":23897,"tokens_out":8751,"duration_ms":82249,"significance":"If the headline result is taken at face value, EFFEKT is a practically useful contribution: it improves server-side accuracy of a frozen foundation model under strict client compute constraints, with public code, six-baseline comparisons, multi-seed statistics (Table A.9, Figures A.5 and A.6), a component ablation (Table 5), real-device validation with energy and bandwidth measurements, and an honest out-of-domain failure analysis (Table 8 and Table A.6). The paper also clearly separates simulated and on-device training. Its main weakness is that the claimed 3.9%/2.7% average gains are obtained only for source-target pairs that are semantically aligned, and the manuscript does not quantify how the gain degrades with domain similarity; the Introduction does not qualify the central claim accordingly.","major_comments":[{"comment":"The central claim in the Introduction, an average increase of 3.9% top-1 and 2.7% top-5 over the state of the art across five fine-grained domains, is supported only by Table 1, where each target domain is paired with a semantically matched public pretraining set. When pretraining is switched to ImageNet-1k, Table 8 shows EFFEKT at 28.3 top-1 versus 28.5 for FedPromo, and Table A.6 shows that EFFEKT trails FedPromo on NABirds (16.5 vs 18.5), OxfordPets (89.0 vs 89.5), and Military Aircraft top-5 (24.6 vs 24.8). The Appendix A.5 domain-shift analysis reports DINO similarities averaging about 0.67, so the favorable source-target selection is the very condition under which the headline gains appear. Because no experiment interpolates between the matched pairs and the ImageNet extreme, the reader cannot determine at what level of alignment the advantage disappears. The Impact Statement already concedes that the method requires pretraining data with reasonable alignment; the main text should state the 3.9%/2.7% claim as conditional and add a controlled similarity sweep (for example, varying source-set mixing or using intermediate source datasets) to quantify the boundary.","section":"Section 7 (Out-of-Domain Pretraining Results, Tables 8 and A.6)"},{"comment":"The two most important hyperparameters, lambda_KD and the LoRA rank, are selected by ablations run on CompCars (Tables 6 and 7), and CompCars is also one of the five test domains in the central comparison (Table 1). This makes the CompCars row of the headline table optimistically biased, and it is unclear how much of the 7.2% top-1 gain over FedPromo on CompCars is attributable to tuning on that same domain. The authors should either fix these hyperparameters before observing the test domains, tune them on a validation task not included in the five evaluation domains, or report the full Table 1 for rank 8 and rank 32 and for neighboring lambda_KD values so the reader can see the sensitivity on all domains.","section":"Section 5 and Section 7 (Tables 6 and 7)"}],"minor_comments":[{"comment":"The paragraph on convergence explicitly disclaims a formal proof and then describes C2S/JA as a proximal alternating minimization strategy; since the required smoothness and bounded-drift assumptions are not stated, this should be labeled as intuition and empirical evidence rather than a convergence argument.","section":"Section 4.1 (Discussion on Convergence)"},{"comment":"Figure A.6 is described in the text as Top-5 accuracy, but the y-axis label in the figure caption reads \"Top-1 accuracy\"; please correct the label.","section":"Section A.3 (Figures A.5 and A.6)"},{"comment":"The sentence \"with n_r = 1000 rounds the top-1 accuracy is is42.1%\" contains a duplicated \"is\"; please fix the typo.","section":"Section A.1"},{"comment":"The phrase \"distillation on the task-specific pretraining datasets alone is meaningless... effectively tackle a data-free Unsupervised Domain Adaptation task\" is potentially misleading, because C2S and JA do use the public pretraining data D^di_p; if the intended meaning is \"without private client data,\" the wording should be changed accordingly.","section":"Section 7 (Component Analysis)"},{"comment":"The loss L_KD^L(P,T) is introduced in the text as the reverse of the standard KD orientation, but in Eq. (1) both entries are outputs of the same client head on server and client features; a brief clarification of which side plays the teacher and which the student in this reverse setting would improve readability.","section":"Equation (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is built directly on FedPromo by overlapping authors, but FedPromo is included as an explicit baseline and the new components are ablated against it, so I do not see a circularity problem. The main issue is scope: the paper's headline numbers are conditional on matched source-target pretraining pairs, and the current experiments only characterize two extremes. This is fixable with a qualification and a similarity sweep, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe short version: EFFEKT is a well-executed incremental step over FedPromo, and its headline numbers are credible as far as they go. But the 3.9% top-1 / 2.7% top-5 advantage is conditional on using public pretraining data that is semantically aligned with the private target domain. When that alignment is removed (ImageNet-1k pretraining), EFFEKT's edge over FedPromo disappears (28.3 vs 28.5 top-1 averaged, and trailing on several per-domain numbers). The paper does mention this requirement in the Impact Statement, but it shows up far from the headline claim.\n\nWhat's genuinely new is the server-side distillation scheme: Clients-to-Server (C2S) distills client-head logits into domain-specific LoRA adapters on the frozen foundation model, and Joint Alignment (JA) re-aligns the client proxy with the updated server. That's a clean idea, and the component ablation in Table 5 supports it: each piece helps, and together they account for ~7 points over FedAvg. The multi-seed stats (Table A.9), the on-device deployment match, and the honest \"no convergence proof, here's why it behaves\" discussion all raise my confidence. The authors also report that performance degrades gracefully with fewer active clients and different Dirichlet α, which is more than most FL papers do.\n\nThe soft spots are real but not fatal. The hyperparameters (λ_KD, LoRA rank) are tuned on CompCars, which is one of the five test domains; that's standard in FL papers but it inflates confidence in that particular result. There's no comparison with federated LoRA baselines like SLoRA or pFedLoRA, even though they're cited. And the domain-similarity condition is the load-bearing assumption—the paper's own out-of-domain experiment shows that the method becomes just another baseline when the public data is mismatched. I'd have liked a sweep between matched and ImageNet extremes to see where the benefit kicks in.\n\nWho this is for: people working on federated fine-tuning of foundation models with lightweight clients, and anyone building on FedPromo. It deserves a serious referee, not a desk reject. I'd push for a revision that either narrows the claims or adds the similarity sensitivity experiment.\n\nBest,\n[You]","headline":"EFFEKT's reported gains are real under matched source-target pairs, but the paper should state that condition up front; it's a solid incremental contribution, not a general-purpose fix.","tokens_in":24469,"tokens_out":2828,"would_cite":true,"duration_ms":24897,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EFFEKT replaces server-side weight averaging with two distillation steps, letting a DINOv2 foundation model absorb new fine-grained domains from lightweight clients and gain 3.9 points of top-1 accuracy over prior art.","keywords":["federated learning","foundation models","knowledge distillation","LoRA adapters","server-side aggregation","fine-grained classification","edge devices"],"falsifier":"Run the released code on the StanfordCars-to-CompCars configuration with the stated hyperparameters; if the server-side top-1 accuracy does not reach approximately 43% (and thus exceeds FedPromo's 35.8 by the reported 7.2 points), the central claim of improved server-side accuracy is falsified.","tokens_in":23390,"feed_emoji":"🧠","tokens_out":19846,"duration_ms":163718,"temperature":0.7,"pith_summary":"The paper proposes EFFEKT, a federated learning framework in which clients never train or transmit the large model. Each client runs a small proxy encoder (MobileNetV3-small) pretrained to align its feature space with a server-side DINOv2 foundation model, and locally optimizes only a small classification head on private data. The server aggregates these heads and then updates a low-rank (LoRA) adapter for the foundation model using two distillation steps on public, task-relevant images: Clients-to-Server distillation transfers what the clients learned, and Joint Alignment re-synchronizes the proxy and server feature spaces. Across five fine-grained recognition domains, EFFEKT reports an average top-1 improvement of 3.9 percentage points and top-5 of 2.7 percentage points over the strongest prior method, FedPromo, while on-device experiments show the client load stays within a few watts and a few megabytes per round. If these results hold, foundation models can keep absorbing new domains from user data without being deployed or fine-tuned on the devices themselves.","feed_headline":"Server foundation model gains 3.9% from tiny client models","feed_subtitle":"Clients train a small head; the server foundation model learns new domains by distillation, not weight averaging.","key_machinery":"The load-bearing mechanism is the bi-directional cross-distillation pair formed by Clients-to-Server (C2S) and Joint Alignment (JA), applied around a shared representation space between two heterogeneous encoders. The proxy MobileNetV3 encoder is pretrained to align its features with DINOv2's on a public task-specific dataset; during federated rounds the client updates only a head, and the server updates a per-domain LoRA adapter (targeting Q and V projections with rank 16, totalling 1.57M parameters) using the distillation losses of Equations 1 and 2. C2S transfers the clients' logit-level knowledge into the LoRA, and JA re-aligns the two feature spaces with a combination of cross-entropy, feature-space distances, and bidirectional logit distillation, which the paper argues regularizes the alternating-minimization loop into a stable fixed point.","core_discovery":"The central claim is that server-side accuracy of a foundation model can be improved under federated constraints by replacing weight-space averaging with two complementary distillation objectives. In Clients-to-Server (C2S) distillation, the updated client heads are held frozen and are used to score features from the server model on public pretraining images; a reversed knowledge-distillation loss (KL divergence from the client-head predictions to the server model's predictions) is minimized by updating only a LoRA adapter, so the server model learns the new private-domain concepts without ever seeing private data. In Joint Alignment (JA) distillation, the aggregated client model and the LoRA-adapted server model are jointly optimized on the same public images with cross-entropy, bidirectional feature-space (L1, L2, cosine) and logit-level losses, restoring the feature-space compatibility that the LoRA update disturbed. The paper reports that this pair of updates yields an average improvement of 3.9 percentage points (top-1) and 2.7 percentage points (top-5) over the previous state of the art across five fine-grained domains, while client-side training remains limited to a small head and the LoRA adapters represent fewer than 1% of the server model's parameters.","pith_inferences":["A natural extension the paper does not explore is replacing the public task-specific images with synthetic or text-driven proxies; since the distillation losses operate on logits and features rather than raw pixels, class-conditional image generation or paired captions might supply the required alignment without any public dataset.","The reversed orientation of the C2S loss (KL from the client-head distribution to the server model's distribution) makes the server match the client heads' entropy; this may act as a regularizer that prevents overfitting to the public pretraining set, a role the paper does not directly test.","The reported 3.9-point average gain is concentrated in two of the five domains (NABirds and CompCars); a broader multi-domain benchmark would clarify whether the mechanism systematically helps fine-grained domains or mainly those with the largest public-private similarity."],"forward_implications":["Server-side accuracy of the foundation model improves by an average of 3.9 points (top-1) and 2.7 points (top-5) over the previous state of the art across five fine-grained domains, without any client ever training the large model.","Client devices remain lightweight: they train only a small classification head on private data, and real deployments on low-power edge boards consume under 7 W and under 5 MB/s of network traffic.","Per-domain LoRA adapters and heads let the server add new domains without retraining old ones; a prototype-based discriminator selects the correct adapter when the domain is not specified.","Ablations show both distillation steps are needed: disabling C2S or JA individually costs about 2.5 points of top-1 accuracy, and the best results require both together with a rank-16 LoRA.","The gains depend on task-specific public pretraining: when the public set is replaced by ImageNet, the advantage over FedPromo disappears (28.3 vs. 28.5 top-1), so the method targets scenarios with semantically aligned public data."],"supporting_citations":[{"why":"Defines the CA-FKT task, the dataset couples, and the ICP client regularization that EFFEKT builds on; also serves as the strongest baseline.","marker":"Caligiuri et al. 2025"},{"why":"Provides the FedAvg aggregation used for the client heads and the principal baseline for comparison.","marker":"McMahan et al. 2017"},{"why":"Supplies the LoRA adapters that keep server-side updates to 1.57M parameters.","marker":"Hu et al. 2022"},{"why":"The DINOv2 foundation model whose feature space is aligned with the proxy client encoder.","marker":"Oquab et al. 2023"},{"why":"The knowledge-distillation formulation underlying both C2S and JA logit losses.","marker":"Hinton 2015"},{"why":"Provides the Dirichlet-based non-IID client data partitioning used in all federated experiments.","marker":"Hsu et al. 2019"},{"why":"The MobileNetV3-small client encoder with its linear feature translator.","marker":"Howard et al. 2019"},{"why":"Supplies the feature-space KD objective (L1, L2, cosine) used in the Joint Alignment loss.","marker":"Barbato et al. 2024"}],"fun_headline_variants":["Federated distillation lifts foundation models by 3.9%","Client proxies teach server FM via LoRA, not averaging","EFFEKT: Tiny clients, big server accuracy gain","Lightweight client heads add 3.9% to federated FM","Distill client knowledge, skip weight averaging, get +3.9%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The public task-relevant images used for server-side distillation must be semantically close to the private client images, because the server only ever learns from those public images; when the public set is generic (ImageNet), the paper's own results show the advantage over FedPromo disappears.","fun_headline_variants_meta":{"raw":{"variants":["Federated distillation lifts foundation models by 3.9%","Client proxies teach server FM via LoRA, not averaging","EFFEKT: Tiny clients, big server accuracy gain","Lightweight client heads add 3.9% to federated FM","Distill client knowledge, skip weight averaging, get +3.9%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1374,"prompt_tokens":921,"completion_tokens":453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":362}},"tokens_in":537,"tokens_out":453,"duration_ms":5374,"temperature":1.0,"reasoning_tokens":362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:21:35.851428+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released code on the StanfordCars-to-CompCars configuration with the stated hyperparameters; if the server-side top-1 accuracy does not reach approximately 43% (and thus exceeds FedPromo's 35.8 by the reported 7.2 points), the central claim of improved server-side accuracy is falsified.","supporting_citations":[{"cited_title":"Communication-efficient learning of deep networks from decentralized data","cited_arxiv_id":null,"evidence_quote":"Provides the FedAvg aggregation used for the client heads and the principal baseline for comparison."},{"cited_title":"Searching for mobilenetv3","cited_arxiv_id":null,"evidence_quote":"The MobileNetV3-small client encoder with its linear feature translator."},{"cited_title":"Cross-architecture auxiliary feature space translation for efficient few-shot personalized object detection","cited_arxiv_id":null,"evidence_quote":"Supplies the feature-space KD objective (L1, L2, cosine) used in the Joint Alignment loss."}],"review_version":1}