Pith. sign in

REVIEW 1 major objections 1 minor 20 references

Decoupling image and text encoders with server-client alignment and two-stage local fine-tuning balances global task adaptation and generalization in federated vision-language models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 13:33 UTC pith:ELLRM245

load-bearing objection FedDTL decouples image and text encoders with server-client alignment plus a two-stage local SFT-then-RL fine-tune to handle FL heterogeneity in VLMs; the combination is reasonable but incremental. the 1 major comments →

arxiv 2605.27900 v1 pith:ELLRM245 submitted 2026-05-27 cs.CV

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning

Yuting Ma , Lechao Cheng , Xiaohua Xu This is my paper
classification cs.CV
keywords federated learningvision-language modelsdecoupled trainingmodality alignmentreinforcement learningfine-tuningtask adaptationgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Federated learning with pre-trained vision-language models often amplifies inconsistent updates across clients and over-specialization within clients when data distributions differ. The paper proposes FedDTL to address this by separating image and text encoder training, using server-client modality alignment to keep global semantics coherent. A subsequent two-stage local process starts with supervised fine-tuning for a stable beginning and switches to reinforcement learning to improve how well the model generalizes beyond local data. A sympathetic reader would care because this setup aims to make federated training practical for heterogeneous real-world deployments in both limited and complete data cases.

Core claim

FedDTL decouples the image encoder and text encoder across clients and the server. Through decoupled encoder training with server-client modality alignment, it promotes coherent global semantic update and reduces inter-client optimization inconsistency, improving global task adaptation. To further mitigate intra-client over-specialization, it introduces a two-stage local fine-tuning where a supervised fine-tuning stage enables rapid and reliable warm-start, followed by a reinforcement learning stage that enhances generalization. Experiments on label skew and feature shift benchmarks show it achieves an effective balance between global task adaptation and generalization under various FL data

What carries the argument

Decoupled encoder training with server-client modality alignment plus two-stage local fine-tuning (supervised warm-start followed by reinforcement learning).

Load-bearing premise

That separating image and text encoder training plus inserting a reinforcement learning stage will reduce both inter-client inconsistency and intra-client over-specialization without creating offsetting problems in full heterogeneous data settings.

What would settle it

A controlled run on a full-data heterogeneous federated benchmark in which FedDTL fails to improve global adaptation or generalization over standard parameter aggregation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Reduces inter-client optimization inconsistency through modality alignment.
  • Improves global task adaptation by maintaining coherent semantic updates.
  • Enhances generalization via the reinforcement learning stage after supervised warm-start.
  • Maintains the adaptation-generalization balance across label skew and feature shift distributions.
  • Operates effectively in both few-shot and full-data federated regimes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The decoupling pattern could transfer to other multimodal federated settings that pair separate modality encoders.
  • The reinforcement learning stage might be adjusted to target different generalization metrics without changing the supervised warm-start.
  • Server-client alignment could help stabilize training when client count or data heterogeneity increases beyond the tested cases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper introduces FedDTL, a federated learning framework for Vision-Language Models. It decouples the image encoder and text encoder training, uses server-client modality alignment to reduce inter-client optimization inconsistency, and applies a two-stage local fine-tuning (supervised warm-start followed by reinforcement learning) to mitigate intra-client over-specialization. The authors claim this balances global task adaptation and generalization, with extensive experiments on benchmarks under label skew, feature shift, few-shot, and full-data regimes.

Significance. If the results hold, this could be a significant contribution to federated VLM adaptation by providing a method to handle heterogeneity without sacrificing generalization. The use of reinforcement learning in the local fine-tuning stage is a distinctive element that may enhance robustness.

major comments (1)
  1. [Abstract] Abstract: The abstract asserts that 'Extensive experiments on multiple benchmarks... demonstrate that FedDTL achieves an effective balance...' but provides no quantitative results, error bars, baselines, or ablation details; the central empirical claim cannot be evaluated from the available text.
minor comments (1)
  1. The manuscript would benefit from explicit definitions or examples of 'inter-client optimization inconsistency' and 'intra-client over-specialization' in the introduction to improve clarity.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive comment on the abstract. We address it point-by-point below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The abstract asserts that 'Extensive experiments on multiple benchmarks... demonstrate that FedDTL achieves an effective balance...' but provides no quantitative results, error bars, baselines, or ablation details; the central empirical claim cannot be evaluated from the available text.

    Authors: We agree that the abstract would benefit from including key quantitative highlights to make the central empirical claims more concrete and evaluable at a glance. In the revised version we will add specific results such as average accuracy improvements over strong baselines (e.g., +X% on label-skew and +Y% on feature-shift benchmarks), mention of error bars across runs, and reference to the main ablation findings, while keeping the abstract within length limits. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper is a method proposal describing FedDTL, a federated VLM framework using decoupled image/text encoders, server-client modality alignment, and two-stage local fine-tuning (supervised then reinforcement). No equations, fitted parameters, or derivations are presented that reduce to their own inputs by construction. Claims rest on experimental results across regimes rather than self-referential logic or load-bearing self-citations. The derivation chain is self-contained as an engineering approach without mathematical circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The central claim rests on standard federated learning assumptions about data heterogeneity and the untested premise that the proposed decoupling and RL stages will produce the stated benefits.

axioms (2)
  • domain assumption Heterogeneous data distributions across clients cause optimization inconsistency and over-specialization in VLM fine-tuning.
    Invoked in the problem statement of the abstract.
  • domain assumption Server-client modality alignment can produce coherent global semantic updates.
    Stated as the mechanism for reducing inconsistency.
invented entities (1)
  • FedDTL framework no independent evidence
    purpose: Decoupled encoder training plus two-stage local RL fine-tuning for federated VLMs
    New named method introduced to solve the stated problems; no independent evidence supplied in abstract.

pith-pipeline@v0.9.1-grok · 5738 in / 1380 out tokens · 43505 ms · 2026-06-29T13:33:59.284048+00:00 · methodology

0 comments
read the original abstract

Federated Learning (FL) with pre-trained Vision-Language Models (VLMs) has emerged as a promising paradigm for various downstream tasks. By leveraging its strong representations, recent studies improve task adaptation under insufficient local data while preserving generalization. However, these methods emphasize fully local optimization with simple parameter aggregation,which can amplify inter-client optimization inconsistency and intra-client over-specialization under heterogeneous and full-data FL settings, making it difficult to balance global task adaptation and generalization. To address these challenges, we propose FedDTL, a novel federated VLM framework that decouples the image encoder and text encoder across clients and the server. Through decoupled encoder training with server-client modality alignment, FedDTL promotes coherent global semantic update and reduces inter-client optimization inconsistency, improving global task adaptation.To further mitigate intra-client over-specialization,we introduce a two-stage local fine-tuning, where a supervised fine-tuning stage enables rapid and reliable warm-start, followed by a reinforcement learning stage that enhances generalization. Extensive experiments on multiple benchmarks, including label skew and feature shift, demonstrate that FedDTL achieves an effective balance between global task adaptation and generalization under various FL data distributions in both few-shot and full-data regimes.

Figures

Figures reproduced from arXiv: 2605.27900 by Lechao Cheng, Xiaohua Xu, Yuting Ma.

Figure 1
Figure 1. Figure 1: The framework of FedDTL. Each client k performs a two-stage local fine-tuning with local image encoders: a supervised fine-tuning stage for rapid task adaptation followed by a reinforcement learning stage for generalization enhancement. The server trains a global text encoder via supervised fine-tuning and performs parameter aggregation. Clients and the server transmit latent embeddings and visual LoRA par… view at source ↗
Figure 2
Figure 2. Figure 2: The average accuracy (%) of ablation study on local fine-tuning strategy and reference model choice over seven datasets under communication rounds. “SFT” and “RL” refer to our SFT stage and GRPO-inspired RL stage. Per-dataset results in Appendix Section C.3. under feature-level distribution shifts, even in heterogeneous data settings. 4.3. Ablation Study We perform ablation studies on seven datasets (CIFAR… view at source ↗
Figure 3
Figure 3. Figure 3: The average accuracy (%) of ablation study on dif￾ferent image embedding upload ratios over seven benchmarks. Per-dataset results in Appendix Section C.3. impact of communication cost on model performance by varying the image embedding upload ratio from {0.2, 0.4, 0.6, 0.8, 1.0}. In [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The average accuracy comparison results (%) across five datasets (OxfordPet, Flower102, CIFAR100, Caltech101, Caltech256) under communication rounds. our decoupled encoder training and two-stage local fine-tuning can benefit the balance between global task adaptation and generalization in federated VLMs. We also observe that pFedDC achieves strong local accuracy in Non-IID settings because it focuses on lo… view at source ↗
Figure 5
Figure 5. Figure 5: The average test accuracy (%) of different training accuracy threshold εacc across five datasets (OxfordPet, Flower102, CIFAR100, Caltech101, and Caltech256) under four data settings. Per-dataset results in [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The average accuracy (%) of different LoRA ranks r across five datasets under four data settings. Per-dataset results in [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The average accuracy (%) of different LoRA starting layers l across five datasets under four data settings. Per-dataset results in [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The average accuracy (%) of different sampling counts G in the RL stage across five datasets under four data settings. Per-dataset results in [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The average accuracy (%) of different noise scales σ in the RL stage across five datasets under four data settings. Per-dataset results in [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The average accuracy (%) of different coefficients β in RL loss across five datasets under four data settings. Per-dataset results in [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The per-dataset accuracy of ablation study on local fine-tuning strategies and reference model choices under few-shot with IID data setting. 10 20 77.5 80.0 82.5 Base Acc (%) CIFAR100 10 20 72.5 75.0 77.5 Tiny_ImageNet 10 20 88 90 92 Food101 10 20 90 95 OxfordPet 10 20 80 90 Flower102 10 20 90 95 Caltech101 10 20 92 94 Caltech256 10 20 Communication Round 72.5 75.0 77.5 Novel Acc (%) 10 20 Communication R… view at source ↗
Figure 12
Figure 12. Figure 12: The per-dataset accuracy of ablation study on local fine-tuning strategies and reference model choices under few-shot with Non-IID data setting. 10 20 80 85 Base Acc (%) CIFAR100 10 20 77.5 80.0 82.5 Tiny_ImageNet 10 20 91 92 93 Food101 10 20 90.0 92.5 95.0 OxfordPet 10 20 70 80 90 Flower102 10 20 90 95 Caltech101 10 20 93 94 95 Caltech256 10 20 Communication Round 70 75 Novel Acc (%) 10 20 Communication … view at source ↗
Figure 13
Figure 13. Figure 13: The per-dataset accuracy of ablation study on local fine-tuning strategies and reference model choices under full-data with IID data setting. 10 20 75 80 85 Base Acc (%) CIFAR100 10 20 70 75 Tiny_ImageNet 10 20 87.5 90.0 Food101 10 20 90.0 92.5 95.0 OxfordPet 10 20 80 90 Flower102 10 20 90 95 Caltech101 10 20 92 94 Caltech256 10 20 Communication Round 60 70 Novel Acc (%) 10 20 Communication Round 50 60 10… view at source ↗
Figure 14
Figure 14. Figure 14: The per-dataset accuracy of ablation study on local fine-tuning strategies and reference model choices under full-data with Non-IID data setting. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 11 canonical work pages · 3 internal anchors

  1. [1]

    Bossard, L., Guillaumin, M., and Gool, L. V . Food-101– mining discriminative components with random forests. InComputer Vision–ECCV 2014: 13th European Con- ference, Zurich, Switzerland, September 6-12, 2014, Pro- ceedings, Part VI, pp. 446–461. Springer,

    2014

  2. [2]

    Heterogeneity-aware personalized federated learning via adaptive dual-agent reinforcement learning.arXiv preprint arXiv:2501.16966,

    Chen, X., Li, Q., Cai, H., and Wang, T. Heterogeneity-aware personalized federated learning via adaptive dual-agent reinforcement learning.arXiv preprint arXiv:2501.16966,

    work page arXiv

  3. [3]

    Geodesic flow kernel for unsupervised domain adaptation

    Gong, B., Shi, Y ., Sha, F., and Grauman, K. Geodesic flow kernel for unsupervised domain adaptation. In2012 IEEE conference on Computer Vision and Pattern Recognition, pp. 2066–2073,

    2066

  4. [4]

    pfedprompt: Learning per- sonalized prompt for vision-language models in federated learning

    Guo, T., Guo, S., and Wang, J. pfedprompt: Learning per- sonalized prompt for vision-language models in federated learning. InProceedings of the ACM Web Conference 2023, pp. 1364–1374. ACM,

    2023

  5. [5]

    Dcp: Dual-cue pruning for efficient large vision-language models

    Jiang, L., Zhang, Z., Zeng, Y ., Xie, C., Liu, T., Li, Z., Cheng, L., and Xu, X. Dcp: Dual-cue pruning for efficient large vision-language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 21202–21215,

    2025

  6. [6]

    Federated reinforcement learning with constraint heterogeneity.arXiv preprint arXiv:2405.03236,

    Jin, H., Zhang, L., and Zhang, Z. Federated reinforcement learning with constraint heterogeneity.arXiv preprint arXiv:2405.03236,

    work page arXiv

  7. [7]

    P., Munoz, J

    Nguyen, D. P., Munoz, J. P., and Jannesari, A. Flora: En- hancing vision-language models with parameter-efficient federated learning.arXiv preprint arXiv:2404.15182,

    work page arXiv

  8. [8]

    Proximal Policy Optimization Algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y . K., Wu, Y ., and Guo, D. Deepseek- math: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo, X., YuYue, Dai, W., Fan, T., Liu, G., Liu, J., Liu, L., Liu, X., Lin, H., Lin, Z., Ma, B., Sheng, G., Tong, Y ., Zhang, C., Zhang, M., Zhang, R., Zhang, W., Zhu, H., Zhu, J., Chen, J., Chen, J., Wang, C., Yu, H., Song, Y ., Wei, X., et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv pr...

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Person- alized federated learning via dual-prompt optimization and cross fusion.arXiv preprint arXiv:2506.21144,

    Zhang, Y ., Guo, K., Lu, Z., Wang, Y ., and Liang, J. Person- alized federated learning via dual-prompt optimization and cross fusion.arXiv preprint arXiv:2506.21144,

    work page arXiv

  12. [12]

    37 3680 1785 3669 1 Flower102 (Nilsback & Zisserman, 2008)102 7169 2983 1020 1 Caltech101 (Fei-Fei et al.,

    2008

  13. [13]

    10 1969 1969 413 4 DomainNet (Peng et al.,

    1969

  14. [14]

    Ablation Study.We conduct ablation studies on seven datasets (CIFAR100, Tiny-ImageNet, OxfordPet, Flower102, Caltech101, Caltech256, Food101)

    shot=4 Non-IID Dirichlet(0.1) Dirichlet(0.3) Dirichlet(0.5) IID Local Base Novel Local Base Novel Local Base Novel Local Base Novel Local Base Novel CLIP 80.22 79.62 81.99 79.44 79.76 82.12 79.83 79.76 82.12 79.77 79.76 82.12 79.76 79.76 82.12 pFedDC 95.19 47.66 61.53 87.73 73.28 72.91 85.68 81.11 75.78 86.49 84.72 77.04 86.83 86.83 76.64 FedPGP 80.42 79....

    2083

  15. [15]

    We further introduce label skew (IID, Dirichlet(0.1), Dirichlet(0.3), Dirichlet(0.5)) on top of the feature shift

    More Performance Comparison.We further perform new experiments on the DomainNet dataset to investigate model performance in the feature shift with label skew setting under different data distributions. We further introduce label skew (IID, Dirichlet(0.1), Dirichlet(0.3), Dirichlet(0.5)) on top of the feature shift. Data from each domain are distributed to...

    work page arXiv 2097

  16. [16]

    As a result, the simple accuracy-based signal is sufficient for stage transition

    We observe that varyingεacc can cause moderate changes in model performance on base and novel classes across all settings, indicating that the proposed transition criterion is not sensitive to the training accuracy threshold. As a result, the simple accuracy-based signal is sufficient for stage transition. Notably, the sample size, data distribution, and ...

    2025

  17. [17]

    For numerical stability, both the product and clipping operations are implemented in log space

    RL Algorithm FewIID FewNon-IID FullIID FullNon-IID Base Novel HM Base Novel HM Base Novel HM Base Novel HM GRPO 94.11 84.04 88.43 92.38 85.03 88.31 94.78 83.26 88.29 93.10 82.96 87.43 DRGRPO 94.2282.87 87.72 92.18 83.76 87.41 94.55 82.35 87.67 93.2081.59 86.67 GMPO 94.14 83.27 87.99 92.25 84.79 88.16 94.5383.2788.24 93.00 81.90 86.67 DAPO 94.06 84.10 88.4...

    work page arXiv 2081

  18. [18]

    Per-dataset results in Table

    TE Backbone FewIID FewNon-IID FullIID FullNon-IID Base Novel HM Base Novel HM Base Novel HM Base Novel HM ViT-B/32 94.6556.37 69.33 91.02 47.00 60.95 94.8752.81 66.27 93.1946.57 60.90 ViT-L/14 94.36 54.20 67.84 90.44 47.64 61.51 94.82 54.26 67.65 93.01 46.73 61.63 ViT-B/16 94.11 84.04 88.43 92.38 85.03 88.31 94.78 83.26 88.29 93.10 82.96 87.43 Table 12.Th...

    1946

  19. [19]

    However, the accuracy on novel classes with mismatched backbones (i.e., ViT-L/14 and ViT-B/32) declines significantly, demonstrating generalization degradation

    Client Number FewIID FewNon-IID FullIID FullNon-IID Base Novel HM Base Novel HM Base Novel HM Base Novel HM K=5 93.64 80.80 86.41 91.6682.2086.43 94.42 79.75 86.18 92.62 79.56 85.34 K=10 94.56 80.70 86.72 92.04 81.47 86.14 94.73 79.93 86.29 91.81 79.65 84.86 K=15 94.70 81.20 87.14 93.0381.6086.58 95.11 80.71 86.98 94.90 79.95 86.41 ViT-L/14 in IID setting...

    work page arXiv 2086

  20. [20]

    one”) and the feature shift with label skew (Dirichlet(α=0.1)) setting (“Dir(0.1)

    23 Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning E. Full Experiment Results Table 13.The per-dataset accuracy comparison results (%) under 16-shot with IID and Non-IID data distributions. IID Average CIFAR10 CIFAR100 EuroSAT Tiny-ImageNet Local Base Novel Local Base Novel Local Base Novel Local Base Novel Local Base Novel C...

    work page arXiv 2090