Pith. sign in

REVIEW 3 major objections 4 minor 19 references

Generalized and Personalized Federated Learning with Black-Box Foundation Models via Orthogonal Transformations

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FEDOT proposes that client-specific orthogonal transformations of frozen foundation-model embeddings, combined with a shared global classifier, balance generalization and personalization in federated learning while keeping the foundation…

desk verdict Solid empirical framework for black-box FL personalization, but Theorem 1 is a tautology and should be replaced; the experiments stand on their own. read the letter →

arxiv 2505.19888 v3 pith:MXYTD5P3 submitted 2025-05-26 cs.LG

Eun Gyung Kong , Je Won Yeom , Yonghoon Jeon , Taesup Kim This is my paper · ORCID
classification cs.LG
keywords federatedlearningpersonalizationblack-boxfoundationmodelsorthogonaltransformationsgradientconflictCayleytransformnon-IIDdatadualprivacy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FEDOT is a way to run federated learning when the shared feature extractor is a proprietary foundation model that clients may only query, not modify. The paper's central claim is that personalization can be handled entirely outside the black box: each client learns its own orthogonal transformation of the frozen output embeddings, while all clients contribute to one small shared classifier. The authors argue that orthogonal maps have condition number $\kappa=1$, which by their Theorem 1 gives the smallest possible upper bound on gradient disagreement between clients, and that this is what lets training stay stable under non-IID data while the classifier keeps generalizing. If that is right, it matters because it gives a gradient-based, communication-light path to federated learning with dual privacy—protecting both client data and the model provider's intellectual property—without backpropagating through or exposing the foundation model. The reported experiments on five domain-shift benchmarks and a 75-client scalability study put FEDOT and its block-diagonal variant FEDOT(+B) ahead of prompt- and adapter-based federated baselines, with FEDOT(+B) reaching 86.10% average comprehensive accuracy.

What carries the argument

The central object is the client-specific orthogonal transformation $w_l^{(i)} \in \mathbb{R}^{d\times d}$, parameterized through the Cayley transform $w_l^{(i)} = (I+R^{(i)})(I-R^{(i)})^{-1}$ with $R^{(i)} = \tfrac12(X^{(i)} - (X^{(i)})^\top)$ skew-symmetric. By construction every eigenvalue of $w_l^{(i)}$ has magnitude 1, so the condition number $\kappa(w_l^{(i)})$—the ratio of largest to smallest singular value—is exactly 1; Theorem 1 uses that value to claim a tight $4\tau$ bound on cross-client gradient differences. The same object also plays a capacity-control role: a full orthogonal matrix has $d(d-1)/2$ degrees of freedom, and the block-diagonal variant reduces this further while keeping $\kappa=1$, which the paper connects to preserving the pretrained manifold structure and avoiding overfitting to local data.

What would settle it

Train the general-linear variant FedLT with its condition number forced to 1 at every step, for example by spectral normalization or by parameterizing the map as a scalar times an orthogonal matrix. If its gradient-conflict cosine similarity and generalization accuracy match FEDOT, then the orthogonality-specific mechanism is doing the work; if they differ, Theorem 1 is not the operative explanation.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that client-specific orthogonal transformations are the right local adaptation module for black-box foundation models in federated learning. A client computes $h' = w_l^{(i)} h$ with $w_l^{(i)}$ orthogonal, normalizes, and feeds the result to the shared classifier; the orthogonal matrix is produced at every step from a skew-symmetric parameter matrix via the Cayley transform, so the constraint is maintained throughout SGD. Theorem 1 states that the $\ell^2$ norm of the difference between two clients' gradients on the global classifier is bounded by $2\tau\big(\kappa(w_l^{(i)})+\kappa(w_l^{(j)})\big)$, which becomes $4\tau$ when both local maps are orthogonal, and the paper presents this as the theoretical reason orthogonality minimizes gradient conflict. Empirically, the paper reports that FEDOT and the block-diagonal FEDOT(+B) outperform PEFT- and prompt-based federated baselines on FEMNIST, PACS, Office-Home, VLCS, and TerraIncognita, that the orthogonal version maintains the highest mean pseudo-gradient cosine similarity during training, and that the block-diagonal variant achieves the best average comprehensive accuracy of 86.10%.

Load-bearing premise

The load-bearing premise is that a number called the condition number of each client's local linear map—how unevenly the map stretches distances—controls how much clients' update directions disagree during training. In the proof (Appendix C) the key comparison is tautological, so the stated $4\tau$ bound holds for every linear map and does not itself single out orthogonal transformations.

Editorial extensions

If this is right

  • If the gradient-conflict bound holds as claimed, federated training over proprietary encoders can proceed with ordinary SGD on the shared classifier, without zeroth-order gradient estimation or backpropagation through the foundation model.
  • Orthogonality of local maps makes the claimed bound independent of how different clients' transformations become, so severe non-IID data should not destabilize aggregation as much as unconstrained linear or nonlinear adapters.
  • Block-diagonal orthogonal maps preserve the $\kappa=1$ bound while cutting degrees of freedom, giving a tunable knob: fewer blocks for simpler tasks, more blocks for complex ones, without sacrificing the stated stability guarantee.
  • Only the $K\times d$ global classifier is communicated, which the paper estimates at about 5K parameters for FEMNIST; if correct, this makes the scheme attractive for cross-device settings where communication is the bottleneck.
  • Random initialization of the classifier is competitive with CLIP text-encoder initialization, so the framework is not tied to vision-language alignment and should extend to vision-only foundation models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Limitations section notes that sharing a global classifier leaves a white-box element; a natural follow-up is to hide the classifier too, for example by updating it server-side or with zeroth-order optimization of the head only.
  • Because the block-diagonal construction already separates degrees of freedom from the $\kappa=1$ bound, one could parameterize local maps as a scalar times an orthogonal matrix and treat the condition number as a tunable knob; comparing that against FEDOT would reveal whether the empirical gains come from isometry or from capacity control.
  • The external-orthogonalization recipe is modality-agnostic: applying client-specific orthogonal maps to token embeddings before a shared head would give a black-box, dual-privacy analogue of orthogonal fine-tuning for language models, with communication cost still limited to the head.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes FEDOT, a federated learning framework for black-box foundation models. Clients share a global classifier and keep client-specific orthogonal transformations applied to frozen encoder embeddings, initialized via the CLIP text encoder and optimized with the Cayley transform. The authors claim a theoretical result (Theorem 1) that orthogonal transformations minimize the upper bound on cross-client gradient differences, and they support the method with experiments on five domain-shift datasets, scalability tests up to 75 clients, and ablations comparing orthogonal, general linear, and nonlinear local adapters.

Significance. The empirical study is a strength: the paper reports multi-seed results on five benchmarks, detailed appendix tables, and a scalability analysis to 75 clients, and the best variant FEDOT(+B) achieves the highest average comprehensive accuracy (86.10%) against the considered baselines. The framework is simple and respects the black-box constraint. However, the theoretical contribution that the paper highlights as its main novelty is not established by the presented proof. If the theoretical overstatement is corrected or removed, the empirical contribution is a useful addition to personalized federated learning with frozen foundation models.

major comments (3)
  1. [Appendix C.1] The key inequality in the proof of Theorem 1, ||w_l I(x)||/||w_l I(x)|| ≤ κ(w_l), is a tautology: the left-hand side is identically 1 for every nonzero feature and every invertible w_l, so the condition number does not enter the bound. Repeating the argument with the correct value gives ||∇_{w_g}^{(i)} ℓ^{(i)} − ∇_{w_g}^{(j)} ℓ^{(j)}|| ≤ 4τ for any linear transformation, orthogonal or not. Consequently, Theorem 1 does not establish that κ=1 is optimal, and the claims in the abstract, Section 4.1, and the contribution list that orthogonality is theoretically shown to minimize gradient conflict are unsupported.
  2. [Section 5.2, Tables 2–3] The experiments compare FEDOT (orthogonal, d(d−1)/2 degrees of freedom) with FedLT (general linear, d^2 degrees of freedom) and attribute FedLT's lower gradient cosine similarity and lower accuracy to its larger condition number κ. This comparison conflates orthogonality with parameter capacity; an unconstrained general linear transformation has roughly twice the degrees of freedom, and the paper does not include a control that matches the number of degrees of freedom across the two architectures. The causal claim that orthogonality per se improves gradient alignment is therefore not established by the empirical design.
  3. [Section 5.2, Table 2] The reported pseudo-gradient cosine similarity is presented as validation of Theorem 1, but the theorem bounds only the norm of the difference of expected gradients; it has no direct implication for pairwise cosine similarity, since two gradients of small norm can still point in opposite directions. The measured similarity differences are an independent empirical observation, not a consequence of the theorem.
minor comments (4)
  1. [Table 1] Several cells contain missing value separators, such as the FedAdapter row for Office-Home where numbers are concatenated (e.g., '96.35±0.2381.33±0.2685.96±0.51'), which makes the table hard to read.
  2. [Throughout] The notation for the proposed method is inconsistent: the paper introduces 'FEDOT' but many tables and appendix sections use 'FedOT' (e.g., Tables 4–23 and Figures 3–5).
  3. [Appendix D] The sentence 'We provide a detailed analysis of runtime and computational overhead across representative FL baselines on the FEMNIST dataset in Tab. 3' refers to the wrong table; the runtime table appears in Appendix E, so the numbering should be corrected.
  4. [Table 1 and Section 5.2] FEDOT(+B) and FEDOT report identical results for PACS, Office-Home, and VLCS, so the reader cannot tell whether the block-diagonal variant is only used where it differs; the authors should clarify whether the hyperparameter search for the block size yields the full orthogonal matrix in those cases.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 1's optimality of orthogonal transformations is not derived: the proof's key normalized-feature quotient is identically 1, so the κ-dependent bound and the 'orthogonality is optimal' claim reduce to a definitional identity; the empirical benchmarks remain independent.

  1. self definitional [Appendix C.1, Eq. (1) / Section 4.1 Theorem 1]
    "By definition, the norm of the transformed normalized vector is bounded by the condition number κ of the transformation matrix: ∥w(i)l I(x)∥/∥w(i)l I(x)∥ ≤ κ(w(i)l), which leads to the final bound:"

    The left-hand side of this displayed inequality is identically 1 for every nonzero feature I(x) and every invertible w_l: it is the norm of a vector divided by itself. Hence '≤ κ(w_l)' is a vacuous inequality, and the subsequent bound 2τ[κ(w_l^(i)) + κ(w_l^(j))] is obtained by substituting κ for a quantity that is always 1. Consequently the theorem's conclusion that orthogonal transformations with κ=1 give the smallest bound 4τ is not derived from any property of orthogonal maps; the same 4τ bound applies to every linear transformation. The claimed optimality of orthogonality reduces to the definition κ(Q)=1 for orthogonal Q, i.e., the result is equivalent to its input by construction.

full rationale

The only derivation supporting the paper's central theoretical claim — that orthogonality minimizes gradient conflicts — is the proof of Theorem 1 in Appendix C.1. After the triangle inequality, the term to be bounded is ∥w_l I(x)∥/∥w_l I(x)∥, which equals 1 for every nonzero feature and every invertible w_l. The paper replaces this identity with '≤ κ(w_l)', producing a κ-dependent bound. Since κ≥1 for all matrices and κ=1 for orthogonal matrices by definition, the claimed conclusion that orthogonal transformations achieve the smallest possible upper bound 4τ is not a consequence of the geometry of orthogonality; the same 4τ upper bound holds for every linear transformation. Thus the theoretical 'prediction' of orthogonality's optimality reduces to the definition of the condition number rather than to a derived property. No fitted parameters are involved, and the self-citations in the paper (e.g., FedCLIP, OFT) are not load-bearing for this step. The experimental sections are substantial and independent: Table 2 measures pseudo-gradient cosine similarity, Table 3 measures final condition numbers and accuracies, and the benchmark comparisons do not rely on the validity of Theorem 1. The circularity is therefore partial and localized to the theoretical justification, giving a score of 6.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the assumption that external linear or orthogonal transformations preserve FM semantics while enabling adaptation, on the black-box access model, and on the interpretation of gradient conflicts via condition numbers. One per-dataset free parameter (block size r) is tuned and materially affects results. No new physical or model entities are introduced.

free parameters (3)
  • block size (number of blocks) r for FEDOT(+B) = 256 (FEMNIST), 1 (PACS, OfficeHome, VLCS), 4 (TerraIncognita)
    Chosen per dataset as optimal; directly controls expressiveness of local adaptation and materially affects personalization (Fig. 2).
  • softmax temperature τ = Not specified
    Appears in the theoretical bound (4τ) and in the classifier; a fixed hyperparameter, not fitted.
  • learning rate = 1e-3 (FEMNIST), 5e-5 (others)
    Reported as tuned; standard hyperparameter needed for reproduction.
assumptions (5)
  • domain assumption The foundation model encoder I(·) is frozen and clients only access output embeddings.
    Sec. 3.1 defines the black-box FL setting; the whole method is built around it.
  • domain assumption A linear transformation of the normalized embedding followed by a shared linear classifier can express both personalized and generalizable classifiers.
    Sec. 3.2 and 4.3 assume this architecture is sufficient; no nonlinear head is used in FedOT.
  • domain assumption Gradient conflict between clients can be measured by the ℓ2 norm of the difference of their global-classifier gradients, and depends on the condition number of local transforms.
    This is the premise of Theorem 1 and the interpretation of Table 2, but the proof's dependence on κ is vacuous.
  • standard math The inequality ∥w_l I(x)∥/∥w_l I(x)∥ ≤ κ(w_l) used in the proof is a valid bound, tautologically 1 ≤ κ.
    Appendix C.1; as written it merely says 1 is at most the condition number, which is always true.
  • domain assumption Clients' data are non-IID in the sense of domain shifts (style, writer, location).
    The chosen benchmarks define heterogeneity; the method's benefits are claimed under this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generalized and Personalized Federated Learning with Black-Box Foundation Models via Orthogonal Transformations." pith.science (2026). https://pith.science/paper/MXYTD5P3

@misc{pith2026250519888,
  author       = {Pith},
  title        = {Pith review of: Generalized and Personalized Federated Learning with Black-Box Foundation Models via Orthogonal Transformations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MXYTD5P3}},
  note         = {Machine review of arXiv:2505.19888}
}
read the original abstract

Federated Learning (FL) facilitates decentralized model training while preserving data privacy. However, achieving both robust generalization and effective personalization simultaneously in heterogeneous (non-IID) environments remains a formidable challenge. Furthermore, the widespread adoption of proprietary Foundation Models (FMs) introduces a critical requirement for dual privacy: (a) protecting sensitive client data and (b) securing the server's valuable intellectual property. This mandates strictly black-box access to the FM. To address these multifaceted challenges, we introduce FedOT, a novel FL framework optimized for black-box FMs. FedOT employs a shared global task-dependent classifier while facilitating local adaptation through client-specific orthogonal transformations applied externally to the FM embeddings. This architecture inherently guarantees that the FM's internal parameters remain inaccessible and unmodified. By enforcing orthogonality, FedOT effectively mitigates gradient conflicts across diverse clients, which is theoretically bounded, preserves the semantic integrity of the FM representations, and achieves robust performance under significant data heterogeneity. The synergy of global and local parameters optimally balances generalization and personalization, markedly outperforming baseline FL methods across diverse benchmarks. Extensive empirical analysis, including rigorous multi-seed validation and scalability assessments, substantiates the robustness, efficiency, and superior performance of FedOT.

Figures

Figures reproduced from arXiv: 2505.19888 by the authors.

Figure 1
Figure 1. Overview of our proposed FL framework, FEDOT. We leverage a pre-trained vision encoder in a strictly black-box FL environment. The image encoder, deployed via API or as a compiled binary, operates with client-specific orthogonal transformations (local parameters) applied externally to the feature embeddings for personalization, while utilizing a globally shared classifier for generalization. This design ensures dual… view at source ↗
Figure 2
Figure 2. Comparison of DOF and personalization accuracy on FEMNIST and OfficeHome. We vary the number of blocks R = 2 N . The left axis depicts the change in personalization accuracy relative to N = 0 (R = 1). The optimal DOF balances adaptability and structural preservation, varying with dataset complexity [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Comparison of generalization and personalization performance for local-only, global-only, and FEDOT. FEDOT consistently outperforms both ablated methods, highlighting the synergistic effect of global and local components. OfficeHome) achieve optimal performance with higher DOF, allowing the model to capture richer, domain-specific pat￾terns. This underscores the necessity of tuning the DOF according to the intrinsic… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Scalability analysis on FEMNIST. Average general￾ization performance on 5 unseen test clients as the number of participating clients increases from 1 to 75. FEDOT consistently improves over the CLIP Zero-Shot baseline and generally outper￾forms PromptFL, demonstrating …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 14 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 1

    arXiv 2023

  2. [2]

    Recognition in terra incognita

    Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. InECCV, pages 456–473, 2018. 2

    work page 2018

  3. [3]

    Hudson, Ehsan Adeli, Russ Alt- man, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Alt- man, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Ni- ladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste...

    arXiv 2021

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. Language models are few-shot learners. InNeurIPS, pages 1877–1901,

    work page 1901

  5. [5]

    Leaf: A benchmark for federated settings.arXiv preprint arXiv:1812.01097, 2018

    Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Kone ˇcný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings.arXiv preprint arXiv:1812.01097, 2018. 2

    arXiv 2018

  6. [6]

    Sur quelques propriétés des déterminants gauches

    Arthur Cayley. Sur quelques propriétés des déterminants gauches. 1846. 2

    work page

  7. [7]

    Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias

    Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. InICCV, pages 1657–1664, 2013. 2

    work page 2013

  8. [8]

    A survey on hetero- geneous federated learning.arXiv preprint arXiv:2210.04505,

    Dashan Gao, Xin Yao, and Qiang Yang. A survey on hetero- geneous federated learning.arXiv preprint arXiv:2210.04505,

    arXiv

Show all 19 references
  1. [9]

    Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wen- chao Xu. Promptfl: Let federated participants cooperatively learn prompts instead of models–federated learning in age of foundation model.IEEE Transactions on Mobile Computing, 23(5):5179–5194, 2023. 1

    2023

  2. [10]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InProc. Int. Conf. Mach. Learn. (ICML), pages 4904–4916, 2021. 1

    2021

  3. [11]

    Deeper, broader and artier domain generalization

    Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. InICCV, pages 5542–5550, 2017. 2

    2017

  4. [12]

    Fedclip: Fast generalization and personalization for clip in federated learning.arXiv preprint arXiv:2302.13485, 2023

    Wang Lu, Xixu Hu, Jindong Wang, and Xing Xie. Fedclip: Fast generalization and personalization for clip in federated learning.arXiv preprint arXiv:2302.13485, 2023. 1

    2023 arXiv

  5. [13]

    Communication- efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. InProc. Int. Conf. Artif. Intell. Stat. (AISTATS), pages 1273– 1282, 2017. 1

    2017

  6. [14]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca Antiga. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, pages 8026–8037, 2019. 2

    2019

  7. [15]

    Federated learn- ing with partial model personalization

    Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learn- ing with partial model personalization. InProc. Int. Conf. Mach. Learn. (ICML), pages 17716–17758, 2022. 1

    2022

  8. [16]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsuper- vised multitask learners. OpenAI blog, 2019. 1

    2019

  9. [17]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InProc. Int. Conf. Mach. Learn. (ICML), pages 8748–8763,

  10. [18]

    Federated reconstruc- tion: Partially local federated learning

    Karan Singhal, Hakim Sidahmed, Zachary Garrett, Shanshan Wu, John Rush, and Sushant Prakash. Federated reconstruc- tion: Partially local federated learning. InNeurIPS, pages 11220–11232, 2021. 1

    2021

  11. [19]

    Deep hashing network for unsupervised domain adaptation

    Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. InCVPR, pages 5018–5027,

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.