Pith. sign in

REVIEW 4 major objections 5 minor 29 references

FedQuad: Federated Stochastic Quadruplet Learning to Mitigate Data Heterogeneity

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FedQuad claims a reformulated quadruplet loss applied locally on each client prevents the global federated model's embeddings from collapsing when client data is non-IID, and reports consistent accuracy gains over existing federated metric-

desk verdict FedQuad is a plausible local-loss tweak that shows real gains on strongly non-IID settings, but the paper's own tables undercut the 'consistently outperforms' claim and the evaluation lacks error bars. read the letter →

arxiv 2509.04107 v1 pith:QCKFNUOF submitted 2025-09-04 cs.LG cs.CV

Ozgu Goksu , Nicolas Pugeault This is my paper
classification cs.LGcs.CV
keywords FederatedLearningDataHeterogeneityRepresentationalCollapseQuadrupletLossMetricContrastiveNon-IIDCIFAR-10/CIFAR-100
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FedQuad tries to fix representational collapse in federated learning when client data is non-IID by changing the local training loss. Instead of only aligning local and global models or adding regularisation, it builds a quadruplet for each anchor: one same-class positive and two negatives drawn from two different classes. The loss pulls the anchor toward the positive and pushes it away from both negatives, shrinking intra-class spread and widening inter-class separation in each client's embedding space. With plain FedAvg aggregation, the paper reports that this loss beats FedAvg, MOON, supervised contrastive, triplet, and classic quadruplet baselines across several Dirichlet partitions and client counts. The authors argue that this structure preserves both intra-class diversity and inter-class discriminability, preventing the global model's embeddings from collapsing after aggregation.

What carries the argument

The central object is the stochastic quadruplet loss ℓquad* (Eq. 5). For an anchor (xa), a positive (xp) from the same class, and two negatives (xn1, xn2) from two different classes that also differ from the anchor's class, the loss is [d(a,p)^2 - d(a,n1)^2 + m1]+ + [d(a,p)^2 - d(a,n2)^2 + m2]+, applied to non-normalised encoder embeddings. The two distinct negatives are the mechanism that increases inter-class variance; the anchor-positive term shrinks intra-class variance. Sampling is class-aware per client, building a class-to-index map and choosing two negative classes without replacement, then combining the loss with cross-entropy as ℓ = ℓce + βℓquad*.

What would settle it

Run FedQuad with a Dirichlet concentration below the tested range (for example α=0.05) or with 500–1000 clients on CIFAR-100, count how many clients have fewer than three classes, and compare test accuracy against FedAvg and a triplet variant; if accuracy collapses or the quadruplet loss cannot be formed on those clients while FedAvg still trains, the claim that FedQuad mitigates heterogeneity does not extend to that regime.

Watch

Extended reading notes

Core claim

The central claim is that reformulating the quadruplet loss so that it drops the negative-negative distance term and instead pushes the anchor away from two negatives sampled from distinct classes, then adding this term to cross-entropy on each client, yields a global model that keeps classes separated after FedAvg aggregation. The paper argues that this explicitly optimises lower intra-class variance and higher inter-class variance in client representations, and the reported accuracy tables show consistent gains over metric-learning baselines under i.i.d. and non-IID partitions. The method requires no extra global model, no prototypes, and no data exchange between clients.

Load-bearing premise

The loss is only defined when a client has samples from at least three distinct classes: the anchor's class, a positive from the same class, and two negative classes that differ from each other and from the anchor; the paper does not define a fallback for clients holding only one or two classes.

Editorial extensions

If this is right

  • With FedAvg aggregation left unchanged, FedQuad raises global accuracy over FedAvg and metric-learning baselines; for example, on CIFAR-10 with 10 clients and α=0.3, FedQuad reaches 80.13% versus FedAvg's 74.05%.
  • Under many clients, FedQuad avoids the sharp degradation seen in model-contrastive baselines; on CIFAR-100 with 200 clients, FedQuad reports 26.76% versus MOON's 1.08%.
  • The quadruplet loss is computed entirely from local data, so the privacy characteristics of FedAvg are preserved without any raw data exchange.
  • The method is designed to improve rare-class robustness because it samples informative negatives across classes even when the client's class distribution is imbalanced.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism behind FedQuad's gains is the explicit two-negative push, a natural extension is adaptive hard-negative sampling within each client's available classes, which could push further when clients hold many classes.
  • The requirement of at least three classes per client suggests FedQuad's advantage may invert in the extreme non-IID regime where clients hold one or two classes; a class-prototype or global-negative fallback could test that boundary.
  • The two-margin design opens a tuning dimension: m1 and m2 could be set per class-pair semantic distance, connecting the loss to class-hierarchy-aware objectives.
  • The ablation shows that removing cross-entropy sharply reduces accuracy, indicating the quadruplet term acts largely as a regulariser on top of the classifier; further ablating β could reveal whether the benefit comes mainly from the positive pull or the negative pushes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FedQuad, a federated learning method that augments the standard cross-entropy loss with a stochastic quadruplet loss on embeddings. For each anchor, the loss pulls a same-class positive close and pushes two negatives sampled from two different classes away, with two separate margins. The local objective is combined with FedAvg-style aggregation and evaluated on CIFAR-10 and CIFAR-100 under Dirichlet non-IID partitions with 10, 50, and 200 clients, comparing against FedAvg, SupConFL, TripletFL, QuadrupletFL, and MOON. The paper claims that FedQuad consistently outperforms all baselines across heterogeneity levels and mitigates representational collapse.

Significance. The idea of using a quadruplet-style, class-separated negative sampling loss as a local client objective in federated learning is interesting, and the paper includes a useful ablation and public code. If the empirical claim were supported, FedQuad would be a simple, aggregation-agnostic plug-in loss for improving feature discrimination under non-IID data. However, the paper's own tables contradict the headline superiority claim, the method is undefined for clients with fewer than three classes, and all reported results are single-run with no error bars. These issues are load-bearing rather than cosmetic, so the current evidence is not sufficient to support the central claim.

major comments (4)
  1. [Section V, Tables II and III] The central claim, 'FedQuad consistently outperforms all baseline methods across different levels of data heterogeneity,' is contradicted by the paper's own tables. Table II (CIFAR-10, α=0.5, 200 clients) reports FedQuad 58.89% versus FedAvg 60.20%, and Table III (CIFAR-100, i.i.d., 10 clients) reports FedQuad 51.27% versus FedAvg 51.33%. These are not marginal settings; they are central to the evaluation. The claim must be substantially revised, or the experiments must be redone with conditions that actually support it.
  2. [Section III-A, Eq. (5)] The quadruplet sampling requires each client to contain at least three distinct classes: the anchor class, the positive class, and two different negative classes. Under extreme non-IID partitions with many clients, a client holding one or two classes cannot form a valid quadruplet and the loss ℓquad* is undefined. No fallback is defined. This is especially relevant to the 200-client experiments and to the limitation discussion in Section VI, where the paper acknowledges many-client regimes weaken the loss but does not specify what happens when the precondition fails.
  3. [All experimental tables (Section IV-V)] Every accuracy table appears to report a single run with no standard deviation, confidence interval, or seed information. Given that Table II shows a 1.31-point deficit for FedQuad against FedAvg and Table III shows a 0.06-point difference, the claimed superiority cannot be statistically distinguished from noise. For a paper whose main contribution is an empirical superiority claim, multiple seeds and variance reporting are essential.
  4. [Algorithm 1, Section III] The text states that FedQuad 'remains applicable when only a small subset of clients participates in each federated learning round,' but Algorithm 1 loops over all N clients ('for i = 1 to N') and aggregates all of them. There is no client-sampling fraction or partial-participation mechanism. Please clarify the actual protocol and align the text with the algorithm.
minor comments (5)
  1. [References] Reference [6] is cited as FedProx, but the listed title is 'Federated minimax optimisation: Improved convergence analyses and algorithms' by Sharma et al. The original FedProx work (Li et al., 2020) should be cited instead, or the reference corrected.
  2. [Section V, Figure 3] The t-SNE visualizations are qualitative and taken at round 5. Adding a quantitative metric of inter-class/intra-class variance or class separability would strengthen the representational-collapse argument.
  3. [Section IV-A, Table V] The main experiments use 20 communication rounds, while the ablation in Table V is run for 5 communication rounds. This is understandable, but the text should explicitly state the round count for the ablation in the main setup paragraph, since it currently appears only in the table caption.
  4. [Throughout] There are several typographical and grammatical issues, e.g., 'Methodogy' in Section III, 'minimaxising' in the abstract/introduction, and the sentence 'Table I and III, our version of QuadrupletFL and FedQuad, consistently outperforms...' in Section III. A careful proofread is needed.
  5. [Section II-A] The paper positions itself against heterogeneity baselines but does not compare with FedProx or SCAFFOLD, both discussed in the background. Adding at least one of these standard methods would strengthen the evaluation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; FedQuad's loss is an explicit optimization objective and the performance claim is empirical, with no self-citation or fitted-input-as-prediction.

full rationale

The paper does not present a derivation chain that reduces to its inputs. Section III defines a composite local loss, Eq. (3)-(5): the cross-entropy term plus a quadruplet hinge loss that explicitly pulls the anchor toward a positive and pushes it away from two negatives sampled from two distinct classes. The claimed effects of smaller intra-class variance and larger inter-class variance are directly encoded in the loss definition rather than derived from a fitted quantity. Hyperparameters beta, m1, and m2 are selected in the Table V ablation and then fixed for the main experiments; this is ordinary hyperparameter selection, not a statistically forced prediction of a closely related quantity. There are no self-citations and no uniqueness theorem imported from the authors' prior work. The absence of a fallback when a client has fewer than three classes is a genuine limitation but not circularity. The statement that FedQuad 'consistently outperforms all baseline methods' is contradicted by the paper's own Tables II and III (e.g., CIFAR-10 with 200 clients: FedQuad 58.89 vs FedAvg 60.20; i.i.d. CIFAR-100: 51.27 vs 51.33), but that is an internal-consistency/correctness issue, not circularity. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on three hyperparameters tuned on one small ablation, on the unproven mechanism that local intra/inter-class variance control survives aggregation, on the requirement that clients hold at least three classes, and on the representativeness of the chosen benchmarks. No new entities or forces are introduced.

free parameters (3)
  • Beta (quadruplet loss weight) = 0.5
    Chosen as 'best' from ablation Table V on CIFAR-10 i.i.d. with 5 rounds, then reused in all main experiments.
  • Margin m1 = 1.0
    Tuned via Table V ablation; controls the separation radius for the first negative sample.
  • Margin m2 = 0.5
    Tuned via Table V ablation; controls the separation radius for the second negative sample.
assumptions (4)
  • domain assumption Minimizing intra-class variance and maximizing inter-class variance in each client's embedding space reduces representational collapse after global averaging.
    This is the paper's motivating mechanism, asserted in the Introduction and Discussion, but not proven or derived.
  • domain assumption Clients have local data spanning at least three classes so that valid quadruplets exist.
    Section III-A sampling requires two distinct negative classes; no fallback is specified for clients with one or two classes.
  • standard math Standard FedAvg weighted averaging of parameters is an appropriate aggregation rule.
    Eq. (1) and Algorithm 1 inherit FedAvg from [5] without modification.
  • domain assumption CIFAR-10 and CIFAR-100 with Dirichlet partitions are representative of federated data heterogeneity.
    Experimental setup in Section IV uses these datasets and splits to stand in for real-world heterogeneity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FedQuad: Federated Stochastic Quadruplet Learning to Mitigate Data Heterogeneity." pith.science (2026). https://pith.science/paper/QCKFNUOF

@misc{pith2026250904107,
  author       = {Pith},
  title        = {Pith review of: FedQuad: Federated Stochastic Quadruplet Learning to Mitigate Data Heterogeneity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QCKFNUOF}},
  note         = {Machine review of arXiv:2509.04107}
}
read the original abstract

Federated Learning (FL) provides decentralised model training, which effectively tackles problems such as distributed data and privacy preservation. However, the generalisation of global models frequently faces challenges from data heterogeneity among clients. This challenge becomes even more pronounced when datasets are limited in size and class imbalance. To address data heterogeneity, we propose a novel method, \textit{FedQuad}, that explicitly optimises smaller intra-class variance and larger inter-class variance across clients, thereby decreasing the negative impact of model aggregation on the global model over client representations. Our approach minimises the distance between similar pairs while maximising the distance between negative pairs, effectively disentangling client data in the shared feature space. We evaluate our method on the CIFAR-10 and CIFAR-100 datasets under various data distributions and with many clients, demonstrating superior performance compared to existing approaches. Furthermore, we provide a detailed analysis of metric learning-based strategies within both supervised and federated learning paradigms, highlighting their efficacy in addressing representational learning challenges in federated settings.

Figures

Figures reproduced from arXiv: 2509.04107 by the authors.

Figure 1
Figure 1. The representational collapse problem in FL. While [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the FedQuad local training framework. Each client minimises loss composed of the cross-entropy loss [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. t-SNE visualisation of learned representations at an [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Client-wise data distribution under non-i.i.d. partition [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 24 canonical work pages

  1. [1]

    Laion- 5b: An open large-scale dataset for training next generation image-text models,

    C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, *et al.*, “Laion- 5b: An open large-scale dataset for training next generation image-text models,” *Advances in Neural Information Processing Systems*, vol. 35, pp. 25278–25294, 2022

    work page 2022

  2. [2]

    ImageNet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2009, pp. 248–255

    work page 2009

  3. [3]

    Enhancing generalisation in federated learning with heterogeneous data: A comparative literature review,

    A. Mora, A. Bujari, and P. Bellavista, “Enhancing generalisation in federated learning with heterogeneous data: A comparative literature review,” Future Generation Computer Systems , Elsevier, 2024

    work page 2024

  4. [4]

    Federated learning on non-IID data: A survey,

    H. Zhu, J. Xu, S. Liu, and Y . Jin, “Federated learning on non-IID data: A survey,” Neurocomputing, vol. 465, pp. 371–390, 2021

    work page 2021

  5. [5]

    Communication-efficient learning of deep networks from decentralised data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralised data,” in Artificial Intelligence and Statistics , PMLR, 2017, pp. 1273– 1282

    work page 2017

  6. [6]

    Federated minimax optimisation: Improved convergence analyses and algorithms,

    P. Sharma, R. Panda, G. Joshi, and P. Varshney, “Federated minimax optimisation: Improved convergence analyses and algorithms,” in Proc. Int. Conf. Mach. Learn. , PMLR, 2022, pp. 19683–19730

    work page 2022

  7. [7]

    Scaffold: Stochastic controlled averaging for on-device federated learning,

    S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for on-device federated learning,” arXiv preprint arXiv:1910.06378 , vol. 2, no. 6, 2019

    arXiv 1910

  8. [8]

    Model-contrastive federated learning,

    Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 10713– 10722

    work page 2021

Show all 29 references
  1. [9]

    Relaxed contrastive learning for federated learning,

    S. Seo, J. Kim, G. Kim, and B. Han, “Relaxed contrastive learning for federated learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2024, pp. 12279–12288

    2024

  2. [10]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, “Learning multiple layers of features from tiny images,” Master’s thesis, University of Toronto, 2009

    2009

  3. [12]

    Supervised contrastive learning,

    P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y . Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” in Advances in Neural Information Processing Systems , vol. 33, 2020, pp. 18661–18673

    2020

  4. [13]

    Deep metric learning using triplet network,

    E. Hoffer and N. Ailon, “Deep metric learning using triplet network,” in Similarity-Based Pattern Recognition: Proc. 3rd Int. Workshop, SIM- BAD, Copenhagen, Denmark, Oct. 2015, pp. 84–92. Springer, 2015

    2015

  5. [14]

    Beyond triplet loss: A deep quadruplet network for person re-identification,

    W. Chen, X. Chen, J. Zhang, and K. Huang, “Beyond triplet loss: A deep quadruplet network for person re-identification,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2017, pp. 403–412

    2017

  6. [15]

    Hard negative examples are hard, but useful,

    H. Xuan, A. Stylianou, X. Liu, and R. Pless, “Hard negative examples are hard, but useful,” in Proc. Eur. Conf. Comput. Vis. (ECCV) , 2020, pp. 126–142

    2020

  7. [16]

    Federated learning with metric loss,

    H. Park, H. Hosseini, and S. Yun, “Federated learning with metric loss,” in Proc. Workshop on Federated Learning for User Privacy and Data Confidentiality in ICML , 2021

    2021

  8. [17]

    Privacy-preserving and robust federated deep metric learning,

    Y . Tian, X. Ke, Z. Tao, S. Ding, F. Xu, Q. Li, H. Han, S. Zhong, and X. Fu, “Privacy-preserving and robust federated deep metric learning,” in Proc. IEEE/ACM 30th Int. Symp. on Quality of Service (IWQoS) , 2022, pp. 1–11

    2022

  9. [18]

    Defending against adversarial attacks in federated learning on metric learning model,

    Z. Gu, J. Shi, Y . Yang, and L. He, “Defending against adversarial attacks in federated learning on metric learning model,” in Proc. IEEE 22nd Int. Conf. on Trust, Security and Privacy in Computing and Communications (TrustCom), 2023, pp. 197–206

    2023

  10. [19]

    Privacy preserving palmprint recognition via federated metric learning,

    H. Shao, C. Liu, X. Li, and D. Zhong, “Privacy preserving palmprint recognition via federated metric learning,” IEEE Trans. Inf. Forensics Security, vol. 19, pp. 878–891, 2023

    2023

  11. [20]

    Feddc: Federated learning with non-iid data via local drift decoupling and correction,

    L. Gao, H. Fu, L. Li, Y . Chen, M. Xu, and C.-Z. Xu, “Feddc: Federated learning with non-iid data via local drift decoupling and correction,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2022, pp. 10112–10121

    2022

  12. [21]

    Federated learning based on dynamic regularisation,

    D. Acar, Y . Zhao, R. Navarro, M. Mattina, P. Whatmough, and V . Saligrama, “Federated learning based on dynamic regularisation,” arXiv preprint arXiv:2111.04263, 2021

    2021 arXiv

  13. [22]

    Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clients,

    X. Fang, M. Ye, and B. Du, “Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clients,” IEEE Trans. Pattern Anal. Mach. Intell., 2025

    2025

  14. [23]

    Fedproc: Prototypical contrastive federated learning on non-iid data,

    X. Mu, Y . Shen, K. Cheng, X. Geng, J. Fu, T. Zhang, and Z. Zhang, “Fedproc: Prototypical contrastive federated learning on non-iid data,” Future Generation Computer Systems , vol. 143, pp. 93–104, 2023

    2023

  15. [24]

    FedCRL: Personalized federated learning with contrastive shared representations for label heterogeneity in non-IID data,

    C. Huang, X. Chen, Y . Zhang, and H. Wang, “FedCRL: Personalized federated learning with contrastive shared representations for label heterogeneity in non-IID data,” arXiv preprint arXiv:2404.17916, 2024

    2024

  16. [25]

    Federated learning from pre-trained models: A contrastive learning approach,

    Y . Tan, G. Long, J. Ma, L. Liu, T. Zhou, and J. Jiang, “Federated learning from pre-trained models: A contrastive learning approach,” Adv. Neural Inf. Process. Syst. , vol. 35, pp. 19332–19344, 2022

    2022

  17. [26]

    FedTrip: A resource-efficient federated learning method with triplet regularization,

    X. Li, M. Liu, S. Sun, Y . Wang, H. Jiang, and X. Jiang, “FedTrip: A resource-efficient federated learning method with triplet regularization,” in Proc. IEEE Int. Parallel and Distributed Processing Symp. (IPDPS) , 2023, pp. 809–819

    2023

  18. [27]

    A mutual informa- tion perspective on federated contrastive learning,

    C. Louizos, M. Reisser, and D. Korzhenkov, “A mutual informa- tion perspective on federated contrastive learning,” arXiv preprint arXiv:2405.02081, 2024

    2024 arXiv

  19. [28]

    Federated contrastive learning for decentral- ized unlabeled medical images,

    N. Dong and I. V oiculescu, “Federated contrastive learning for decentral- ized unlabeled medical images,” in Proc. Int. Conf. on Medical Image Computing and Computer-Assisted Intervention (MICCAI) , 2021, pp. 378–387

    2021

  20. [29]

    Local learning matters: Rethinking data heterogeneity in federated learning,

    M. Mendieta, T. Yang, P. Wang, M. Lee, Z. Ding, and C. Chen, “Local learning matters: Rethinking data heterogeneity in federated learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2022, pp. 8397–8406

    2022

  21. [30]

    Tackling data heterogeneity in federated learning with class prototypes,

    Y . Dai, Z. Chen, J. Li, S. Heinecke, L. Sun, and R. Xu, “Tackling data heterogeneity in federated learning with class prototypes,” in Proc. AAAI Conf. Artif. Intell. , vol. 37, 2023, pp. 7314–7322

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.