Pith. sign in

REVIEW 3 major objections 6 minor 18 references

FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Federated prompt tuning can be hardened against adversarial attacks by generating visual prompts from text prompts under a global label beacon; FedAPT reports an 11.49% PGD-100 gain over existing methods.

desk verdict FedAPT has a solid empirical sweep and a genuinely new combination, but the beacon—its main conceptual contribution—is mis-specified in the equations and needs a rewrite. read the letter →

arxiv 2509.06992 v1 pith:YGGHA4W7 submitted 2025-09-03 cs.CV

Kun Zhai , Siheng Chen , Xingjun Ma , Yu-Gang Jiang This is my paper
classification cs.CV
keywords federatedlearningadversarialrobustnessprompttuningvision-languagemodelsnon-IIDdatatrainingCLIPclassinformationgap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper takes on a gap in federated prompt tuning for vision-language models: defenses trained on non-IID data are brittle because each client only sees a handful of classes, so its adversarial examples never exercise the global decision boundary. The proposed method, FedAPT, introduces a class-aware prompt generator that converts text prompts into visual prompts, guided by a global label embedding called a beacon, which aggregates label information across clients without sharing raw data. This lets local training generate attacks and defenses that reflect the whole class distribution, not just the client's slice. The paper reports that FedAPT outperforms five federated prompt-tuning baselines, with an 11.49% average PGD-100 robustness improvement in few-shot settings and consistent gains in zero-shot transfer to unseen datasets, while keeping clean accuracy higher than baselines. If this works, it would make federated fine-tuning of large multimodal models safer in realistic, heterogeneous deployments.

What carries the argument

The carrying mechanism is the beacon-guided class-aware prompt generator G. For a text prompt P, the generator computes W_P = P W_K and V_P = P W_V, concatenates the global beacon B with a learned query Q, and applies cross-attention followed by a projection h_φ: G(P) = h_φ(cross-attention(B ⊕ Q, W_P, V_P)). The beacon itself evolves as B_t = βB_{t−1} + (1−β)(1/E)Σ_i W^i_t[−m:], aggregating the last m token embeddings from participating clients' word embeddings. This single generator is shared across transformer layers (cross-layer generator sharing), so it coordinates prompts across the network while keeping parameters small. The work it does is to inject global class distribution into each

What would settle it

A controlled comparison that would settle the mechanism: train FedAPT with three beacons—(1) the proposed momentum-aggregated tokens, (2) a random fixed vector of the same dimension, and (3) a fixed average of clean class-name embeddings from a public template—under the same non-IID split. If PGD-100 robustness with the random beacon does not drop below the proposed beacon, the class-information-gap story is not validated. A second check: as clients gain more classes, the paper itself predicts the gap closes; robustness under an IID split should be similar to FedAPT, confirming the mechanism.

Watch

Extended reading notes

Core claim

The central claim is that the vulnerability of federated prompt tuning to adversarial attacks in non-IID settings is an information problem, not just an optimization problem. Each client generates adversarial perturbations using only its local labels; the global model must then defend against attacks aimed at all global labels, leaving most of the decision boundary unhardened. FedAPT closes this class information gap with two components. A class-aware prompt generator—a lightweight cross-attention network—maps text prompts to visual prompts, and it is guided by a beacon, a momentum-aggregated global label embedding built from clients' token embeddings. The generated visual prompts therefore

Load-bearing premise

The whole mechanism hinges on the beacon actually encoding cross-client label information: B_t averages the last m token embeddings of clients' word embeddings, which in this design are the learned prompt tokens rather than the class-name tokens, and the paper assumes those aggregated tokens carry global class semantics; if they do not, the central explanation for FedAPT's gains loses its foundation.

Editorial extensions

If this is right

  • In non-IID federated settings, adversarial robustness can be improved by sharing aggregated label semantics (the beacon) rather than raw data or per-client labels.
  • The defense transfers: models fine-tuned on ImageNet keep higher zero-shot robustness on 14 unseen datasets under PGD-100, DI, and CW attacks.
  • Robustness gains grow as heterogeneity increases (e.g., +8.75% at 10 classes per client), and the gap to centralized training narrows when all 1000 classes are available, consistent with the class-information-gap explanation.
  • Sharing one generator across layers improves robustness over independent per-layer generators while using fewer parameters, so the design is both efficiency and defense-oriented.
  • Clean accuracy is not sacrificed; FedAPT reports higher clean accuracy than competing defense methods in few-shot experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether the beacon's benefit comes specifically from its global label content or just from added capacity; replacing B_t with a random or fixed public-template embedding would isolate that.
  • If the mechanism generalizes, a similar label-semantics beacon could improve not just adversarial robustness but also out-of-distribution generalization in federated prompt tuning, since both depend on aligning local views with a global label distribution.
  • A harder, untested scenario is overlapping but skewed label distributions across clients, which may change how the beacon should be aggregated; the paper only evaluates mutually exclusive client classes.
  • The beacon is built from learned prompt tokens rather than class-name embeddings, so one could test a variant where the server publishes clean global class-name embeddings only, which would further reduce privacy exposure while preserving global class information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. FedAPT proposes a federated adversarial prompt-tuning method for CLIP. To mitigate what the authors call the class information gap in non-IID federated learning, it introduces a 'class-aware prompt generator' that converts text prompts into visual prompts under the guidance of a 'beacon' (global label embedding), along with a cross-layer generator-sharing strategy. The paper evaluates FedAPT against five FL-adapted baselines on 15 datasets and reports large gains under PGD-100, DI, and CW attacks, including an 11.49% PGD-100 improvement in few-shot settings and a 6.60% average zero-shot transfer improvement over the best baseline.

Significance. The problem studied is timely and relevant: adversarial robustness of federated prompt tuning for VLMs is underexplored, and the paper proposes a concrete method with a broad evaluation across many datasets and attack types. If the central mechanism worked as stated, this would be a solid contribution. However, the paper's key conceptual claim—that the beacon encodes global label information—is not what the equations actually implement, and the empirical evaluation lacks error bars, multiple seeds, and statistical tests. The breadth of the experimental setup and the use of standard attack protocols are strengths, but they do not by themselves establish the proposed mechanism. The absence of code also limits reproducibility.

major comments (3)
  1. [§3.2.1, Eq. (5); §3.2.4, Step 2] The beacon is not a 'global label embedding' as claimed. Eq. (5) defines B_t as a momentum average of W_i^t[-m:], and the accompanying text defines W_i^t=[W,P], so the last m tokens are the learned prompt tokens P, not the class-label token embeddings W. This is confirmed by the local beacon update B_i^{t+1}=[W,P_i^{t+1}][-m:] in Step 2. Thus the beacon is essentially an exponential moving average of the federated prompt, not an aggregation of class-label information. The central mechanism for bridging the class information gap is therefore mis-specified; the w/ vs w/o Beacon ablation in Fig. 4 could be explained by prompt ensembling or momentum regularization rather than by global label information. The paper must either redefine the beacon to actually aggregate class-label token embeddings or substantially reframe the claimed contribution.
  2. [§4.1, Table 1, Figs. 3–5] No error bars, multiple-seed results, or statistical tests are reported. Several headline per-dataset gains are extremely small: in Table 1, ImageNet-Sketch improves by only 0.02% (PGD) and 0.04% (DI), and EuroSAT by 0.1% (PGD) and 0.87% (DI); in Table 2, several entries are below 0.3%. Without variance estimates, these numbers are indistinguishable from noise, and the claim of consistent per-dataset improvements is not supported. The authors should report mean and standard deviation over at least three seeds and, where possible, paired significance tests, especially for small-gain datasets.
  3. [§4.3, Fig. 4] The 'with vs. without Beacon' ablation is underspecified. The text does not state how the 'w/o Beacon' condition is constructed: is the generator removed, is B replaced by a random or zero vector, or is the cross-attention query set to Q alone? This matters because, with the current Eq. (5), removing B is tantamount to removing a momentum-averaged prompt, not removing global label information. The ablation therefore cannot isolate the mechanism the paper claims to validate. The authors should describe the control precisely and, ideally, also ablate an alternative beacon that truly aggregates class-label embeddings.
minor comments (6)
  1. [§3.2.1, Eq. (5)] The notation W_i^t = [W_i^t, P] reuses W_i^t on both sides; this is confusing. Use a different symbol for the concatenated token sequence, e.g., S_i^t=[W,P_i^t].
  2. [Figure 2, §3.2.2] Figure 2 lists B inside the client parameters θ_i={φ,Q,W_K,W_V,B}, but the text defines θ={φ,Q,W_K,W_V} and treats B as a separately updated global state. Clarify whether B is part of the generator parameters or a distinct global variable.
  3. [Table 1] The table header 'Source Target' is confusing: the first column appears to be ImageNet itself, not a downstream target. Please restructure the header and caption so that the source and target columns are unambiguous.
  4. [Figure 5 caption] Typo: 'corss-dataset' should be 'cross-dataset'. Also, the caption should state that each point is a single run unless multiple seeds are reported.
  5. [General] The paper states experiments last 'over 30 rounds' without the exact number; use a fixed number (e.g., 50 or 100) and report all hyperparameters needed to reproduce the baseline and FedAPT runs.
  6. [§3.1 and Figure 3] Minor wording issues remain, e.g., 'a obtained' in §3.2.1, and Figure 3's caption says 'The top figure displays clean accuracy' but the figure contains multiple panels; clarify the layout.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'global label embedding' beacon is defined as the last prompt tokens, so it is an EMA of the prompt it guides; the claimed class-information bridge is self-referential by construction.

  1. self definitional [Section 3.2.1, Eq. (5); Section 3.2.4, Step 2 and Eqs. (10)-(11)]
    "The beacon is computed by aggregating the last m token embeddings from each layer of all clients’ word embeddings. ... Bt = βBt−1 + (1− β) 1/E Σ_i W_i^t[−m:] ... where ... m represents the number of prompt tokens appended to each transformer layer. ... Concurrently, the client updates its Beacon as B_i^{t+1} = [W, P_i^{t+1}][−m:]."

    In the notation of the paper, [W, P] is the concatenation of base class-label embeddings W with the trainable prompt P. Therefore the last m tokens of W_i^t = [W, P] are exactly the prompt tokens P, not class-label embeddings. Step 2 confirms this by setting B_i^{t+1} = [W, P_i^{t+1}][−m:] = P_i^{t+1}. Combining Eq. (10) (P_{t+1} = 1/E Σ P_i^{t+1}) with Eq. (11) (B_{t+1} = βB_t + (1−β) 1/E Σ B_i^{t+1}) shows the beacon is an exponential moving average of the global text prompt. Thus the 'global label embedding' is defined in terms of the very prompt it is claimed to guide; no cross-client class-label information enters the generator. The class-information-bridging mechanism is therefore self-referential by construction, even though the external benchmark results remain empirical measuremen

full rationale

The only significant circularity is in the conceptual mechanism: the 'Global Label Embedding' beacon, which is presented as encoding cross-client class information, is actually defined as the last m tokens of [W, P] — i.e., the trainable prompt tokens. Since the server aggregates prompts via FedAvg and the beacon via a momentum average of those same local prompt tokens, the beacon is an EMA of the prompt it is supposed to guide. This makes the claimed class-information-bridging effect self-referential by construction. No load-bearing self-citations, imported uniqueness theorems, or ansatz-smuggling citations were found. The experimental comparisons against APT, FedTPG, and other baselines are standard external benchmark evaluations, so the reported robustness improvements are not themselves fabricated by the circular step. However, because the central novelty claim ('beacon encodes global class distributions') reduces by construction to a moving average of the prompt, the score is set to 6 rather than a lower non-circularity score.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The paper introduces several hand-chosen hyperparameters and a central invented entity, the beacon. The beacon's definition in Eq. (5) does not support the claim that it encodes global label information, since it aggregates prompt tokens rather than class label embeddings. This undermines the proposed mechanism, though the empirical method may still function as a prompt generator.

free parameters (6)
  • prompt tokens per layer (m) = 2
    Chosen by hand, balances parameter count and capacity.
  • transformer depth for prompts (J) = 8
    Chosen to keep prompt count consistent.
  • beacon momentum (beta) = 0.9
    Momentum for beacon update, chosen by hand.
  • training perturbation budget (epsilon) = 1/255
    Small l-infinity budget used in APT; may not reflect stronger attacks.
  • SGD learning rate = 0.0035
    Set without reported tuning.
  • number of communication rounds = 30+
    Stated but not justified.
assumptions (5)
  • standard math FedAvg aggregation (Eq. 10) preserves convergence in a non-IID setting.
    Standard federated averaging used as the aggregation rule.
  • domain assumption CLIP zero-shot classification via cosine similarity (Eq. 1) is a valid scoring function.
    Basis for all classification and adversarial losses.
  • domain assumption Non-IID setting with mutually exclusive classes per client (from FedTPG) represents real federated scenarios.
    The paper adopts this setup for all experiments.
  • ad hoc to paper The beacon encodes global label information (Eq. 5).
    The equation aggregates prompt tokens, not label embeddings; this is the paper's central unproven premise.
  • ad hoc to paper 3-step PGD with epsilon=1/255 during training produces adversarial samples that improve robustness to PGD-100, DI, CW.
    Standard adversarial training assumption, not verified for this setting.
invented entities (1)
  • Beacon (Global Label Embedding)
    purpose: To provide clients with global class distribution information for generating adversarial examples.
    Defined in Eq. (5) as a momentum average of the last m token embeddings of clients' word embeddings, which are prompt tokens, not label embeddings. No mechanism shows it carries label information.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models." pith.science (2026). https://pith.science/paper/YGGHA4W7

@misc{pith2026250906992,
  author       = {Pith},
  title        = {Pith review of: FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGGHA4W7}},
  note         = {Machine review of arXiv:2509.06992}
}
read the original abstract

Federated Prompt Tuning (FPT) is an efficient method for cross-client collaborative fine-tuning of large Vision-Language Models (VLMs). However, models tuned using FPT are vulnerable to adversarial attacks, leading to misclassification in downstream tasks. In this work, we introduce Federated Adversarial Prompt Tuning (\textbf{FedAPT}), a novel method designed to enhance the adversarial robustness of FPT. We identify a key issue in FedAPT under non-independent and identically distributed (non-IID) settings: a \textit{class information gap} between clients and the global model. Clients rely solely on limited local label information to generate adversarial samples for training, while the global model must defend against adversarial attacks from global labels. To address this issue, we propose a \textbf{class-aware prompt generator} that generates visual prompts from text prompts. This generator is guided by a \emph{Global Label Embedding} (serving as a ``beacon") which encodes cross-client label information to create more globally-aligned visual prompts. Additionally, we propose a \textbf{cross-layer generator sharing} strategy to enhance prompt coupling across different layers of the model, further boosting adversarial robustness. Extensive experiments on multiple image classification datasets demonstrate the superiority of FedAPT in improving adversarial robustness, outperforming existing methods by a large margin. FedAPT also exhibits exceptional generalization in cross-domain and cross-dataset scenarios, indicating its effectiveness in real-world applications.

Figures

Figures reproduced from arXiv: 2509.06992 by the authors.

Figure 1
Figure 1. Overview of FedAPT: In this approach, only the text prompt P and the prompt generator G are tuned, while the rest of the model remains frozen. FedAPT generates vision prompts P˜ from text prompts P using the prompt generator G, which is guided by B. The B token encapsulates the textual label information for all classes. 3.2.1. THE BEACON In FedAPT, the beacon (global label embedding) is a ob￾tained by aggregating la… view at source ↗
Figure 2
Figure 2. Our proposed FedAPT collaboratively learns a unified text adversarial prompt P and a prompt generator G across multiple clients with diverse classification datasets. During local training, the beacon allows each client to re￾fine its understanding of class semantics by integrating both its local data distribution and the aggregated global view. Specifically, the word embeddings of client i in communica￾tion round t … view at source ↗
Figure 3
Figure 3. Clean accuracy (%) and PGD-100 adversarial robustness (%) across eleven datasets with varying shot numbers [1, 2, 4, 8, 16]. The top figure displays clean accuracy, while the bottom figure shows adversarial robustness. All defense methods (APT, APT-V, APT-VLI, APT-VLJ, FedTPG, and FedAPT) were trained on the eleven datasets using a training perturbation budget of ϵ = 1/255 and α = 2/3ϵ. block contains 2 prompt token… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of PGD-100 adversarial robustness (%) for FedAPT with and without Beacon guidance. (a) Results on training and testing within the same dataset across 11 datasets; (b) Zero-shot adversarial robustness (%) from ImageNet to cross-dataset datasets; (c) Zero-shot…
Figure 5
Figure 5. Figure 5: Zero-shot PGD-100 adversarial robustness (%) from ImageNet to corss-dataset datasets , under perturbation budgets of ϵ = 1/255, 2/255, and 3/255. 5 10 20 50 100 1000 Class Per Client 15 18 21 24 27 30 33 36 Robust Accuracy (%) APT APT-V APT-VLI APT-VLJ FedTPG FedAPT […
Figure 6
Figure 6. Figure 6: PGD-100 adversarial robustness (%) with varying num￾bers of classes per client ([5, 10, 20, 50, 100, 1000]) on ImageNet. cross-layer generator sharing not only reduces the model’s parameter size but also enhances adversarial robustness by fostering stronger inter-layer…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 9 canonical work pages

  1. [5]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee,

    2009

  2. [6]

    Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories

    Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop, pp. 178–178. IEEE,

    2004

  3. [7]

    Torchattacks: A pytorch repository for adversarial attacks

    Kim, H. Torchattacks: A pytorch repository for adversarial attacks. arXiv preprint arXiv:2010.01950,

    arXiv 2010

  4. [8]

    Towards deep learning models resistant to adversarial attacks

    Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 ,

    arXiv

  5. [9]

    Fine-grained visual classification of aircraft

    Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151,

    arXiv

  6. [10]

    and Zisserman, A

    Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722–729. IEEE,

    work page 2008

  7. [15]

    Learning to Prompt Your Domain for Vision-Language Models

    Wei, G., Wang, F., Shah, A., and Chellappa, R. Dual prompt tuning for domain-aware federated learning. arXiv preprint arXiv:2310.03103,

    work page Pith review arXiv

  8. [18]

    Few-Shot Adversarial Prompt Learning on Vision-Language Models

    Zhou, Y ., Xia, X., Lin, Z., Han, B., and Liu, T. Few-shot adversarial prompt learning on vision-language models. arXiv preprint arXiv:2403.14774,

    work page Pith review arXiv

Show all 18 references
  1. [2008]

    M., Vedaldi, A., Zisserman, A., and Jawahar, C

    Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition , pp. 3498–3505. IEEE,

    2012

  2. [2012]

    Intriguing properties of neural networks

    Szegedy, C. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199,

    arXiv

  3. [2014]

    and Wagner, D

    Carlini, N. and Wagner, D. Towards evaluating the robust- ness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39–57. Ieee,

    2017

  4. [2017]

    S., Dai, H., and Dou, D

    Che, T., Liu, J., Zhou, Y ., Ren, J., Zhou, J., Sheng, V . S., Dai, H., and Dou, D. Federated learning of large lan- guage models with parameter-efficient prompt tuning and adaptive optimization. arXiv preprint arXiv:2310.15080,

    arXiv

  5. [2019]

    Adver- sarial training in communication constrained federated learning

    Shah, D., Dube, P., Chakraborty, S., and Verma, A. Adver- sarial training in communication constrained federated learning. arXiv preprint arXiv:2103.01319,

    arXiv

  6. [2021]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Soomro, K. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402,

    arXiv

  7. [2022]

    A., Oliva, A., and Torralba, A

    Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485–3492. IEEE,

    2010

  8. [2023]

    Vi- sual prompting for adversarial robustness

    Chen, A., Lorenz, P., Yao, Y ., Chen, P.-Y ., and Liu, S. Vi- sual prompting for adversarial robustness. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE,

    2023

  9. [2024]

    Food-101– mining discriminative components with random forests

    Bossard, L., Guillaumin, M., and Van Gool, L. Food-101– mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European con- ference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part VI 13, pp. 446–461. Springer,

    2014

  10. [2025]

    Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning

    Zhao, H., Du, W., Li, F., Li, P., and Liu, G. Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pp. 1–5. IEEE,

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.