REVIEW 3 major objections 6 minor 26 references
On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Low-rank adaptation (LoRA) offers the most favorable accuracy–communication trade-off when fine-tuning a vision-language model federated on remote sensing data.
desk verdict A useful first systematic comparison of VLM adaptation strategies in federated remote sensing, with a plausible LoRA recommendation, but the communication-efficiency claim needs round counts and variance to be fully convincing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the choice of trainable parameter subset $\theta_{\text{train}} \subseteq \{\theta_v, \theta_t\}$ in a CLIP model, since this subset determines both what is transmitted to the server each federated round and how flexibly the model can specialize. The argument is carried by a systematic comparison under a fixed non-IID country-based partition of BigEarthNet-S2: for each strategy the paper measures in-distribution micro/macro-mAP, out-of-distribution accuracy on EuroSAT, RESISC45, and ImageNet, plus FLOPs and trainable-parameter counts. LoRA (low-rank adaptation injecting small trainable matrices into the attention query and value projections) is the mechanism that occupies the favorable middle: it updates both visual and textual representations while keeping the backbone frozen, which the paper finds preserves pretrained features and keeps communication near 1 million parameters.
What would settle it
Repeat the BigEarthNet-S2 federated experiment with per-strategy hyperparameter tuning (especially a learning-rate grid for CoOp and LoRA) and a fixed, reported number of communication rounds; if prompt learning then matches LoRA's in-distribution accuracy at comparable communication cost, or if full fine-tuning's catastrophic forgetting disappears under a longer or differently scheduled training run, the paper's guideline would be overturned.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that the choice of trainable parameter subset $\theta_{\text{train}} \subseteq \{\theta_v, \theta_t\}$ for federated CLIP adaptation largely determines a three-way trade-off, and that this trade-off has a clear winner for typical remote-sensing deployments. LoRA, which injects low-rank matrices into the query and value projections of the transformer attention blocks, reaches 79.1 micro-mAP on BigEarthNet-S2 versus 78.3 for full fine-tuning, while exchanging only 1.08M trainable parameters per round compared with 427.62M for FFT. At the same time, LoRA preserves 75.1% top-1 accuracy on ImageNet, within 0.5 points of zero-shot CLIP, whereas FFT collapses to 7.8%. The paper therefore concludes that adapting a VLM in a federated setting beats both a from-scratch ResNet-50 baseline and zero-shot CLIP, and that LoRA provides the most favorable performance–communication trade-off; it also finds that text-encoder fine-tuning is the best fine-tuned choice for cross-dataset transfer, and that PAINT-style weight interpolation can control forgetting only at the price of full fine-tuning's high communication cost.
Load-bearing premise
The load-bearing premise is that one shared learning rate and an unreported common number of communication rounds give every adaptation strategy a fair comparison, so the observed ranking could reflect hyperparameter mismatch rather than intrinsic properties of the strategies.
Editorial extensions
If this is right
- A practitioner who needs both strong task accuracy and low bandwidth should prefer LoRA over full fine-tuning, since it matches in-distribution performance while reducing communicated parameters by roughly a factor of 400.
- Text-encoder fine-tuning is the best fine-tuned strategy when the goal is transfer to related remote-sensing datasets, as it achieves the highest EuroSAT and RESISC45 accuracy among that group.
- Prompt learning should be chosen only when communication is the dominant constraint, because it sends just 46.85K parameters but gives the weakest accuracy and worst generalization.
- Full fine-tuning should be avoided when the model must retain general visual knowledge, because it drops ImageNet top-1 accuracy from 75.6% to 7.8%.
- Weight interpolation between zero-shot and fine-tuned CLIP weights (PAINT) can control forgetting, but only with full fine-tuning's high communication overhead.
Reading between the lines
- The headline ranking may partly reflect the paper's decision to use the same learning rate and training procedure for all strategies; separate tuning could move CoOp or LoRA up or down.
- The communication comparison counts raw trainable-parameter counts; with quantization or compression of updates, the practical bandwidth gap between full fine-tuning and LoRA could narrow.
- Catastrophic forgetting is probed only with ImageNet; other held-out distributions, or multispectral inputs, might show a different forgetting pattern.
- The conclusions rest on a general-purpose CLIP backbone; if remote-sensing-specific foundation models were used, the trade-off could shift, as the paper itself notes as future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a comparative study of CLIP adaptation strategies (full fine-tuning, image/text encoder fine-tuning, CoOp prompt learning, and LoRA) in federated learning for remote sensing image classification. Experiments use BigEarthNet-S2 as the federated training set, with EuroSAT, RESISC45, and ImageNet for out-of-distribution evaluation. The paper reports in-distribution and OOD accuracy, local training FLOPs/MACs, and per-round communication cost, and derives a guideline recommending LoRA as the best performance/communication trade-off, text encoder fine-tuning for cross-dataset generalization, prompt learning only under severe bandwidth constraints, and FFT with PAINT interpolation for controlled adaptation. The code is publicly available.
Significance. This is a timely and practically useful comparison for federated remote sensing; it appears to be the first systematic study of these VLM adaptation strategies in this setting. Strengths include the realistic non-IID partition by country, evaluation of catastrophic forgetting on a large natural image benchmark, detailed FLOPs/MACs and parameter counts, and public code. However, the headline ranking is not fully supported by the current experimental reporting: the number of communication rounds and hence total communication volume are missing, all strategies share a single learning rate without sensitivity analysis, and no variance estimates are provided. If these issues are addressed, the paper would provide a solid empirical guideline.
major comments (3)
- [§4.2, §5.5, Table 2] The paper states in Section 4.2 that 'the same FL training procedure was adopted across all adaptation strategies' but never reports the number of communication rounds used for the results in Table 1 or for the curves in Figure 1. Table 2 defines communication cost as the number of trainable parameters exchanged per round, and the conclusion in Section 6 claims that LoRA reduces communicated parameters by more than two orders of magnitude compared to FFT. In federated learning, total communication volume is per-round parameters multiplied by the number of rounds. If LoRA requires more rounds than FFT to reach the reported performance, the per-round ratio of about 396x could shrink substantially, possibly undermining the central recommendation. Please report the round count, convergence behavior, and total communication volume for each strategy, or justify that all strategies are compared at a common, converged round count.
- [§4.2] All strategies are trained with AdamW at a single learning rate of 1e-5, with no per-strategy learning-rate search or sensitivity analysis. CoOp, LoRA, and full fine-tuning typically operate in different optimization regimes, so the large gap between CoOp (66.5 micro-mAP) and LoRA/FFT (79.1/78.3) may reflect hyperparameter mismatch rather than intrinsic capacity differences. To support the guideline that prompt learning is only suitable under bandwidth constraints and that LoRA is the best trade-off, please provide a per-strategy hyperparameter sweep or at least a small sensitivity analysis over learning rates and show that the ranking is stable across reasonable choices.
- [§5.1, Table 1] No error bars, confidence intervals, or multiple seeds are reported. The in-distribution difference between LoRA and FFT is small (micro-mAP 79.1 vs 78.3; macro-mAP 58.7 vs 61.8), and the OOD differences are also computed from single runs. Without variance estimates, the claim that LoRA achieves competitive in-distribution performance is not statistically substantiated. Please run at least three independent federated trials with different random seeds and report mean and standard deviation (or confidence intervals) for the main results.
minor comments (6)
- [§4.1] The number of clients is implied by the eight listed countries, but K is never explicitly stated; please report K and per-client sample counts to allow reproduction.
- [§5.3, Figure 1] Figure 1's caption mentions fine-tuning duration defined by E and C, but no concrete values are given; please report the E and C used for each curve and label the interpolation coefficient axis.
- [§5.2] The phrase 'around 14% below zero-shot' for RESISC45 should be 'approximately 14.3 percentage points' to avoid confusion between relative and absolute differences.
- [§5.4] The FLOPs/MACs table reports per-forward-pass costs; please clarify whether these counts include the backward pass and whether they are measured or computed analytically, since the no-caching rows for FFT and encoder-specific fine-tuning are identical by construction.
- [§5.3, §6] The PAINT analysis is presented qualitatively and Figure 1 does not provide numerical results; since PAINT is listed as a recommended strategy in the conclusion, please report the quantitative trade-off (e.g., the best interpolation coefficient and the resulting accuracies on BigEarthNet-S2 and ImageNet) that supports that recommendation.
- [References] Reference [12] is listed as 'Hou et al.' without full author names; please ensure all references are complete and consistently formatted.
Circularity Check
No significant circularity: the LoRA recommendation is an empirical conclusion from measured accuracy and per-round communication counts, not a fitted or self-referential derivation.
full rationale
The paper's central claims are empirical findings from a comparative experiment, not derivations from fitted inputs. LoRA's reported advantage combines Table 1 (79.1 micro-mAP vs. 78.3 for FFT) with Table 2 (1.08M trainable parameters per round vs. 427.62M for FFT); this is direct arithmetic from the authors' own measurements and definitions, so nothing is predicted from a fit or renamed as a prediction. FLOP/MAC and communication counts are defined and measured rather than claimed as theory. Self-citations (e.g., refs. [1], [3], [19]) support background statements about federated learning and the BigEarthNet dataset; they do not carry the LoRA-versus-FFT conclusion, which rests on the reproduced experiments and the publicly available code. The unreported number of communication rounds and the use of a single learning rate are legitimate experimental-reporting concerns, but they make the ranking potentially confounded, not circular: the conclusion is not equivalent to its inputs by construction. No step reduces one equation to another, imports a uniqueness theorem from prior self-cited work, or smuggles in an ansatz via citation. Accordingly, the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Shared learning rate =
1e-5
- LoRA rank =
8
- Number of communication rounds =
not reported
assumptions (4)
- domain assumption Pretrained OpenAI CLIP ViT-L/14 weights are a suitable initialization for RS image classification
- domain assumption Country-based partitioning of BigEarthNet-S2 produces a representative non-IID FL setting for RS
- domain assumption ImageNet top-1 accuracy is a valid proxy for catastrophic forgetting of pretrained representations
- ad hoc to paper A single learning rate and identical training protocol ensure a fair comparison across strategies
Cite this review
Pith. "Pith review of On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing." pith.science (2026). https://pith.science/paper/EALHBOEF
@misc{pith2026260804791,
author = {Pith},
title = {Pith review of: On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing},
year = {2026},
howpublished = {\url{https://pith.science/paper/EALHBOEF}},
note = {Machine review of arXiv:2608.04791}
}
read the original abstract
Federated learning (FL) enables collaborative training of deep learning models across decentralized image archives without requiring data centralization. This paradigm is particularly relevant in remote sensing (RS), where legal regulations, privacy concerns, and bandwidth constraints restrict data sharing. However, the presence of training data heterogeneity across clients (known as non-IID data) can impede convergence and limit the generalization capability of the aggregated global model. To mitigate the adverse effects of training data heterogeneity, vision-language models (VLMs) can be leveraged in FL due to their transferable representations, which have demonstrated robustness under distribution shifts. However, their large parameter size may substantially increase communication overhead and local computational complexity in federated settings. Therefore, it is crucial to select an appropriate VLM adaptation strategy that balances the generalization ability with the communication and computational constraints. To address this issue, in this paper, we present the first comparative study of VLM adaptation strategies for FL in the context of RS image classification. We investigate full fine-tuning, encoder-specific fine-tuning, prompt learning, and low-rank adaptation (LoRA) tuning, and analyze them with respect to three criteria: 1) generalization capability under non-IID data, 2) communication overhead, and 3) local computational complexity. Experiments on BigEarthNet-S2, EuroSAT, RESISC45, and ImageNet reveal distinct trade-offs between task specialization, cross-domain generalization, and efficiency. Based on our findings, we derive a guideline for the selection of an appropriate VLM adaptation strategy in FL for RS image classification under different operational constraints. The code of this work is publicly available at https://git.tu-berlin.de/rsim/FL-RS-VLM.
Figures
Reference graph
Works this paper leans on
-
[1]
B. B¨ uy¨ ukta¸ s, G. Sumbul, and B. Demir, “Federated learning across decentralized and unshared archives for remote sensing image classification: A review,”IEEE Geosci. Remote Sens. Mag.12(3), 64–80 (2024)
work page 2024
-
[2]
Federated learning meets remote sensing,
S. Moreno-Alvarez, M. E. Paoletti, A. J. Sanchez-Fernandez, J. A. Rico-Gallego, L. Han, and J. M. Haut, “Federated learning meets remote sensing,”Expert Syst. Appl.255, 124583 (2024)
work page 2024
-
[3]
A multi-modal federated learning framework for remote sensing image classification,
B. B¨ uy¨ ukta¸ s, G. Sumbul, and B. Demir, “A multi-modal federated learning framework for remote sensing image classification,”IEEE Trans. Geosci. Remote Sens.63, 1–11 (2025)
work page 2025
-
[4]
D. M. Jimenez G., D. Solans, M. Heikkila, A. Vitaletti, N. Kourtellis, A. Anagnostopoulos, and I. Chatzi- giannakis, “Non-IID data in federated learning: A survey with taxonomy, metrics, methods, frameworks and future directions,”arXiv preprint arXiv:2411.12377(2024)
arXiv 2024
-
[5]
FedKL: Tackling data heterogeneity in federated reinforcement learning by penalizing KL divergence,
Z. Xie and S. Song, “FedKL: Tackling data heterogeneity in federated reinforcement learning by penalizing KL divergence,”IEEE J. Sel. Areas Commun.41(4), 1227–1242 (2023)
work page 2023
-
[6]
Learning transferable visual models from natural language supervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,”Proc. Int. Conf. Mach. Learn., 8748–8763 (2021)
work page 2021
-
[7]
T. Guo, S. Guo, J. Wang, X. Tang, and W. Xu, “PromptFL: Let federated participants cooperatively learn prompts instead of models–federated learning in age of foundation model,”IEEE Trans. Mob. Comput.23(5), 5179–5194 (2023)
work page 2023
-
[8]
Federated learning from vision-language foundation models: Theoretical analysis and method,
B. Pan, W. Huang, and Y. Shi, “Federated learning from vision-language foundation models: Theoretical analysis and method,”Adv. Neural Inform. Process. Syst.37, 30590–30623 (2024)
work page 2024
Show all 26 references
-
[9]
Global and local prompts cooperation via optimal transport for federated learning,
H. Li, W. Huang, J. Wang, and Y. Shi, “Global and local prompts cooperation via optimal transport for federated learning,”Proc. IEEE Conf. Comput. Vis. Pattern Recog., 12151–12161 (2024)
2024
-
[10]
FedRSCLIP: Federated learning for remote sensing scene classification using vision-language models,
H. Lin, C. Zhang, D. Hong, K. Dong, and C. Wen, “FedRSCLIP: Federated learning for remote sensing scene classification using vision-language models,”IEEE Geosci. Remote Sens. Mag.13(3), 260–275 (2025)
2025
-
[11]
Learning to prompt for vision-language models,
K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,”Int. J. Comput. Vis.130(9), 2337–2348 (2022)
2022
-
[12]
CAPT: Class-aware prompt tuning for federated long-tailed learning with vision-language model,
Hou et al., “CAPT: Class-aware prompt tuning for federated long-tailed learning with vision-language model,”arXiv preprint arXiv:2503.06993(2025)
2025 arXiv
-
[13]
Harmonizing generalization and personalization in federated prompt learning,
T. Cui, H. Li, J. Wang, and Y. Shi, “Harmonizing generalization and personalization in federated prompt learning,”Proc. Int. Conf. Mach. Learn., 381 (2024)
2024
-
[14]
FedCLIP: Fast generalization and personalization for CLIP in federated learning,
W. Lu, X. Hu, J. Wang, and X. Xie, “FedCLIP: Fast generalization and personalization for CLIP in federated learning,”IEEE Data Eng. Bull.46(1), 52–66 (2023)
2023
-
[15]
FedVLM: Scalable personalized vision-language models through federated learning,
A. Mitra, A. Anjum, P. Agbaje, M. Pes´ e, and H. Olufowobi, “FedVLM: Scalable personalized vision-language models through federated learning,”Proc. Eur. Conf. Artif. Intell.413, 4249–4256 (2025)
2025
-
[16]
F AA-CLIP: Federated adversarial adaptation of CLIP,
Y. Wu, A. Chaddad, C. Desrosiers, T. Daqqaq, and R. Kateb, “F AA-CLIP: Federated adversarial adaptation of CLIP,”IEEE Internet Things J.12(12), 21091–21102 (2025)
2025
-
[17]
CLIP-guided federated learning on heterogeneity and long-tailed data,
J. Shi, S. Zheng, X. Yin, Y. Lu, Y. Xie, and Y. Qu, “CLIP-guided federated learning on heterogeneity and long-tailed data,”Proc. AAAI Conf. Artif. Intell.38(13), 14955–14963 (2024)
2024
-
[18]
LoRA: Low-rank adaptation of large language models,
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, and others, “LoRA: Low-rank adaptation of large language models,”Proc. Int. Conf. Learn. Represent.1(2), 3 (2022)
2022
-
[19]
reBEN: Refined BigEarthNet dataset for remote sensing image analysis,
K. N. Clasen, L. Hackel, T. Burgert, G. Sumbul, B. Demir, and V. Markl, “reBEN: Refined BigEarthNet dataset for remote sensing image analysis,”Proc. Int. Geosci. Remote Sens. Symp., 1264–1268 (2025)
2025
-
[20]
EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification,
P. Helber, B. Bischke, A. Dengel, and D. Borth, “EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification,”IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.12(7) (2019)
2019
-
[21]
Remote sensing image scene classification: Benchmark and state of the art,
G. Cheng, J. Han, and X. Lu, “Remote sensing image scene classification: Benchmark and state of the art,” Proc. IEEE105(10), 1865–1883 (2017)
2017
-
[22]
ImageNet: A large-scale hierarchical image database,
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,”Proc. IEEE Conf. Comput. Vis. Pattern Recog., 248–255 (2009)
2009
-
[23]
ImageNet large scale visual recognition challenge,
Russakovsky et al., “ImageNet large scale visual recognition challenge,”Int. J. Comput. Vis.115(3), 211–253 (2015)
2015
-
[24]
Finetune like you pretrain: Improved finetuning of zero-shot vision models,
S. Goyal, A. Kumar, S. Garg, Z. Kolter, and A. Raghunathan, “Finetune like you pretrain: Improved finetuning of zero-shot vision models,”Proc. IEEE Conf. Comput. Vis. Pattern Recog., 19338–19347 (2023)
2023
-
[25]
Communication-efficient learning of deep networks from decentralized data,
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,”Proc. Int. Conf. Artif. Intell. Stat., 1273–1282 (2017)
2017
-
[26]
Patching open-vocabulary models by interpolating weights,
G. Ilharco, M. Wortsman, S. Y. Gadre, S. Song, H. Hajishirzi, S. Kornblith, A. Farhadi, and L. Schmidt, “Patching open-vocabulary models by interpolating weights,”Adv. Neural Inform. Process. Syst.35, 29262– 29277 (2022)
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.