Pith. sign in

REVIEW 3 major objections 6 minor 46 references

Mitigating Resolution-Drift in Federated Learning: Case of Keypoint Detection

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Resolution-drift, a newly named failure mode in federated learning, degrades keypoint-detection models when clients hold images at different resolutions, and the paper's RAF method—multi-resolution knowledge distillation—restores accuracy…

desk verdict RAF's empirical story on resolution-drift is solid and worth engaging; its convergence proof has a genuine algebraic flaw and should be corrected or removed before publication. read the letter →

arxiv 2507.23461 v1 pith:DLQPXHYF submitted 2025-07-31 cs.CV cs.AI

Taeheon Lim , Joohyung Lee , Kyungjae Lee , Jungchan Cho This is my paper
classification cs.CVcs.AI
keywords federatedlearningresolutiondrifthumanposeestimationkeypointdetectionmulti-resolutionknowledgedistillationvisiontransformernon-IIDdatahigh-resolutionregression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper identifies a failure mode it calls resolution-drift in federated learning for non-classification vision tasks: when different clients hold images at different resolutions, the aggregated global keypoint-detection model overfits the resolutions it saw and loses accuracy on others, even though each client's data is privately held. The paper proposes RAF (resolution-adaptive federated learning), which adds a multi-resolution knowledge-distillation loss to each client's local training: the model's own higher-resolution heatmap output acts as a teacher for its lower-resolution output, with gradients stopped on the teacher. RAF is claimed to act as a resolution-aware regularizer that improves accuracy across both seen and unseen resolutions, and it is designed to plug into existing aggregation schemes such as FedAvg or FedProx without changing the server. The paper reports gains up to 27.6% at the lowest test resolution and presents a convergence analysis claiming the same O(1/T) rate as standard FedAvg. A sympathetic reader would care because resolution heterogeneity is common in real camera networks, and the proposed fix is local, modular, and aimed at a previously unnamed axis of non-IID data.

What carries the argument

The load-bearing object is the multi-resolution knowledge-distillation (MRKD) loss, Eq. (5): for each client and each downsampled level, the higher-resolution heatmap output (teacher) is detached from the gradient and compared, under a fixed linear upsampling operator, with the lower-resolution heatmap output (student), and the squared error is added to the task's heatmap MSE with weight α. This makes the local objective a regularized, resolution-aware quadratic form in the late-training regime, where the surrogate loss is claimed to agree with the original loss in value and gradient at each round's iterate. Two supporting mechanisms carry the practical claim: convolution-based global and local positional embeddings (from ResFormer) replace ViTPose's absolute positional embeddings so the same network can take inputs of any resolution, and the stop-gradient teacher makes distillation one-directional, from high to low resolution. The convergence argument runs through three bounds—L-smoothness, γ-strong convexity, and bounded gradient variance—which plug the RAF loss into a standard FedAvg convergence theorem to obtain O(1/T) for the optimality gap.

What would settle it

Re-run the RAF experiments while recording the norm of the backbone weight updates during the last rounds; if those updates are not close to zero relative to the final-layer updates, the feature-converged premise behind the O(1/T) theorem fails, and the local-equivalence of the surrogate loss at the current iterate would need to be checked at neighbouring iterates as well.

Watch

Extended reading notes

Core claim

The central claim is that resolution variability across federated clients is not a minor version of class-label heterogeneity but a distinct drift mechanism for high-resolution regression tasks. Because keypoint models must preserve spatial structure, a ViT trained at one resolution learns position-dependent features that do not transfer; averaging weights from clients trained at different resolutions produces a global model that is uncertain about scale and underperforms at every resolution, including the ones used in training. RAF counters this by having each client create downsampled copies of its own images and distill the high-resolution heatmap prediction into the lower-resolution heatmap predictions, minimizing their difference after a fixed upsampling operator. This forces low-resolution outputs to carry the same spatial information as high-resolution outputs, which the paper argues prevents overfitting to any single scale. The paper also replaces ViTPose's fixed absolute positional embeddings with convolution-based global and local positional embeddings so one backbone can process arbitrary resolutions, and it proves, under a feature-converged late-training assumption, that the RAF objective inherits FedAvg's O(1/T) convergence rate. The experiments on human pose estimation with MPII show RAF improving accuracy on seen and unseen resolutions relative to FedAvg and FedProx baselines.

Load-bearing premise

The proof's load-bearing premise is that in the late phase of training the ViT backbone has effectively stopped changing, so only the final layer's weights move; the paper does not verify this for its own ViTPose-trained models, and if the backbone is still changing, the claimed O(1/T) convergence guarantee is not established.

Editorial extensions

If this is right

  • If RAF's central claim is right, a single federated keypoint model can serve clients whose cameras produce very different resolutions, with accuracy on unseen resolutions no longer collapsing.
  • A low-resolution client can improve its own inference by interpolating its low-resolution image up to a mid-range resolution before feeding the model, because RAF-trained features behave well at those scales.
  • The method works as a local-training add-on, so existing FL aggregation logic (FedAvg or FedProx) stays untouched; the reported gains are not due to a new server algorithm.
  • The t-SNE evidence suggests the same resolution-robustness story extends to other spatial-detail tasks such as semantic segmentation, since high-resolution representation models cluster features by input resolution without the regularizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that MRKD is effectively a scale-equivariance regularizer; a natural comparison would be to explicit multi-scale augmentation or a resolution-conditional normalization, which might achieve similar robustness with less compute.
  • The convergence theorem is only as strong as the unverified feature-converged regime; an easy experiment is to track the norm of backbone weight updates in late training for the actual ViTPose model, which the paper does not report.
  • One testable extension is to apply the same MRKD recipe to depth estimation or super-resolution: if resolution-drift is a shared property of high-resolution regression, RAF should produce the same pattern of gains on unseen scales in those tasks.
  • The interpolation-at-inference result suggests a deployment recipe: rather than retraining for every camera resolution, operators can upsample low-resolution feeds to a canonical mid-high resolution before inference, and the paper's numbers give a concrete map of the accuracy/resolution trade-off.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript studies federated learning for human pose estimation when clients hold images of different resolutions. It defines 'resolution-drift' as the performance degradation observed when aggregating models trained on heterogeneous input resolutions, and proposes Resolution Adaptive Federated Learning (RAF). RAF augments standard FedAvg/FedProx local training with a multi-resolution knowledge-distillation loss: each client downsamples its own images to lower resolutions and trains its model so that low-resolution heatmap predictions match (detached) higher-resolution teacher predictions. The ViTPose backbone is modified with ResFormer-style convolution-based positional embeddings to accommodate variable input sizes. Experiments on MPII with simulated resolution-heterogeneous clients show consistent gains for RAF over the baselines at seen and unseen inference resolutions. Section IV.C provides a convergence analysis claiming an O(1/T) rate by invoking FedAvg's convergence theorem on a surrogate quadratic loss.

Significance. If the empirical results are reproducible, RAF is a practically valuable and modular contribution to FL for high-resolution regression: it requires no changes to server aggregation, works with both FedAvg and FedProx, and yields large gains at low and unseen resolutions in the reported MPII simulations. The resolution-drift axis is a useful reframing of non-IID data for non-classification tasks, and the paper's experiments on centralized versus federated settings help isolate the regularizing effect of distillation. The main weakness is the theoretical section, whose central equivalence is algebraically incorrect; for this reason the convergence guarantee, as stated, is not established. The empirical part, if separated from the flawed proof, is still a reasonable contribution.

major comments (3)
  1. [IV.C, Eqs. (8), (11), (13), (14)] The claimed local equivalence between L_k and \bar L_k is false. For one KD term, write A = U_i^{i-1}(\psi^{(i)})^T and B = (\psi^{(i-1)})^T. The original term (8) is ||B w_t - A w||^2, whose value at w_t is ||(B-A)w_t||^2 and whose gradient is 2 A^T(A-B)w_t. The surrogate term (11) is w^T A^T(B-A)w, whose value at w_t is w_t^T A^T(B-A)w_t and whose gradient is [A^T(B-A)+(B-A)^T A]w_t. These quantities are equal only under special symmetry conditions on A^T(B-A), which do not hold in general. Equations (13) and (14) are therefore not valid. Propositions 1-3 are proven for \bar L_k, and Theorem 1 does not apply to the loss actually minimized by RAF.
  2. [IV.C, Proposition 1, Eq. (18)] The gradient formula stated for the KD term is incorrect for both losses under consideration. The gradient of (8) is 2A^T(A w - B w_t), not 2A^T(A-B)w, because the teacher is frozen at w_t; the gradient of (11) contains the extra symmetrized term (B-A)^T A. As written, Eq. (18) does not derive the smoothness constant of either objective. The constant may be recoverable for the surrogate, but the proof needs to be rewritten.
  3. [IV.C, feature-converged assumption] The analysis is premised on a 'feature-converged' regime in which only the last affine layer changes. The paper does not validate this assumption for the ViTPose-S models used in the experiments (e.g., by measuring relative updates of backbone versus head weights or checking that the linearized loss tracks the true loss). Since this is the setup for the convergence result, the assumption must either be checked or the theorem explicitly presented as conditional on it.
minor comments (6)
  1. [V.A, Tables II-III and Figures 6, 8, 10] No uncertainty quantification is reported; please provide means and standard deviations over multiple seeds, given the small per-client dataset size (4,000 images).
  2. [Algorithm 1, line 15] The local update 'w_k ← w − η ∇_{w_k} L_k(w)' is ambiguous about whether w on the right-hand side is the global broadcast or the local iterate; please clarify the variables.
  3. [Table III] The entry 'Bilnear' should be 'Bilinear'; also, the claim that 57.2 is an 'inherent accuracy limit' should be justified or phrased as an empirical observation rather than a fundamental bound.
  4. [V.E and Figure 8c] For the lowest-resolution client, the paper states that no further downsampling is possible, yet it attributes gains to RAF for this client; please clarify how MRKD operates for the lowest-resolution client and how its benefit arises through the global model.
  5. [V.G, heading a] The heading 'Anlaysis' is a typo for 'Analysis'.
  6. [Abstract and Section VI] The generalizability claims to segmentation, depth estimation, and super-resolution are supported only by a t-SNE visualization on a Torchvision FCN; consider softening these claims or adding quantitative transfer experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RAF's central claims are benchmark-tested with stated hyperparameters and external citations, so the reported gains are not definitionally forced.

full rationale

The paper's claimed derivation chain is not circular. The central empirical claim—that RAF mitigates resolution-drift—is tested on an external benchmark (MPII, reference [44]) with stated hyperparameters (alpha=1, gamma=0.01, AdamW, batch sizes 32/64, initial learning rate 2.5e-4, Section V.A) and is evaluated at resolutions unseen during training (e.g., 320x240, 384x288, 512x384 in Table II). The MRKD loss (Eq. (5)) uses the model's own higher-resolution outputs as soft targets, i.e., self-distillation, but this is a regularizer whose effectiveness is measured on held-out data; it is not a quantity that equals the reported gains by construction. No author-overlapping citations are used: the references carrying the theoretical analysis ([40]–[43]) and the architecture component ([17]) are external, and no prior work by the present authors is invoked as load-bearing support. The convergence theorem (Section IV.C, Theorem 1) is an application of the standard FedAvg result [43] under stated smoothness, strong-convexity, and bounded-variance assumptions. I flag the proof's asserted local equivalence of Lk and \bar Lk (Eqs. (13)–(14)) as an omitted or, per the skeptic's algebra, incorrect proof step: the equality of function values and gradients is only asserted, not established, and the claimed O(1/T) rate depends on it. However, this is an internal proof-rigor or correctness concern, not circularity, because it does not reduce the theorem's conclusion to its assumptions or hide a fitted parameter in the statement. The paper is self-contained against an external benchmark, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on a few hand-set hyperparameters and on the transferability of ResFormer-style positional embeddings; the convergence claim additionally rests on an unverified lazy-training assumption. No new physical or architectural entities are postulated. Resolution-drift is a descriptive label for a measured phenomenon rather than an invented entity.

free parameters (3)
  • distillation weight alpha = 1.0
    Set by hand in Section V.A without an ablation. It controls the balance between task loss and MRKD regularization and enters the claimed convergence constants.
  • regularization coefficient gamma = 0.01
    Set by hand in Section V.A. It provides the strong-convexity modulus in the proof, so the claimed O(1/T) rate depends on a hand-chosen value.
  • number of resolution levels N_{k,res}
    The method requires each client to choose how many downsampled copies are created, but Section V never states the values used, so the empirical gains depend on an unreported design choice.
assumptions (4)
  • domain assumption The ViT backbone is feature-converged in late training, so optimization can be analyzed as a linear model with frozen features.
    Section IV.C invokes lazy-training and neural-collapse literature to justify this, but the paper does not verify that ViTPose in the reported FL runs operates in this regime.
  • domain assumption The FedAvg convergence conditions of Li et al. [43] apply to RAF once smoothness, strong convexity, and bounded variance are shown.
    Theorem 1 says the conditions of [43] are satisfied automatically, but client heterogeneity, local epoch counts, and the surrogate-objective equivalence are not checked in detail.
  • domain assumption ResFormer-style convolution-based positional embeddings transfer from classification to high-resolution keypoint regression.
    Section IV.B adopts GPE and LPE for ViTPose; the experiments test this transfer empirically, but there is no theoretical justification beyond ResFormer's classification results.
  • standard math Assumption 1: features, upsampling operators, and ground-truth heatmaps are bounded.
    These bounds are standard regularity assumptions used in Proposition 1 to Proposition 3, but their values are not measured in the experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Resolution-Drift in Federated Learning: Case of Keypoint Detection." pith.science (2026). https://pith.science/paper/DLQPXHYF

@misc{pith2026250723461,
  author       = {Pith},
  title        = {Pith review of: Mitigating Resolution-Drift in Federated Learning: Case of Keypoint Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DLQPXHYF}},
  note         = {Machine review of arXiv:2507.23461}
}
read the original abstract

The Federated Learning (FL) approach enables effective learning across distributed systems, while preserving user data privacy. To date, research has primarily focused on addressing statistical heterogeneity and communication efficiency, through which FL has achieved success in classification tasks. However, its application to non-classification tasks, such as human pose estimation, remains underexplored. This paper identifies and investigates a critical issue termed ``resolution-drift,'' where performance degrades significantly due to resolution variability across clients. Unlike class-level heterogeneity, resolution drift highlights the importance of resolution as another axis of not independent or identically distributed (non-IID) data. To address this issue, we present resolution-adaptive federated learning (RAF), a method that leverages heatmap-based knowledge distillation. Through multi-resolution knowledge distillation between higher-resolution outputs (teachers) and lower-resolution outputs (students), our approach enhances resolution robustness without overfitting. Extensive experiments and theoretical analysis demonstrate that RAF not only effectively mitigates resolution drift and achieves significant performance improvements, but also can be integrated seamlessly into existing FL frameworks. Furthermore, although this paper focuses on human pose estimation, our t-SNE analysis reveals distinct characteristics between classification and high-resolution representation tasks, supporting the generalizability of RAF to other tasks that rely on preserving spatial detail.

Figures

Figures reproduced from arXiv: 2507.23461 by the authors.

Figure 1
Figure 1. Multiple axes defining statistical heterogeneity: the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architectural difference between classification and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. For the Human Pose Estimation task in the centralized [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Overview of RAF when applied to clients with varying-resolution data. RAF fully exploits spatial information in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Detailed illustration of RAF local training on a high-resolution client. The client’s original dataset is downsampled to [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Inference accuracy on the human pose estimation task [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Heatmap inference comparison on low-resolution [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Inference accuracy on the HPE task across various input resolutions for different training regimes. “CL” denotes [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 10
Figure 10. Figure 10: RAF with one high-resolution (256 × 192) client and multiple low-resolution (128 × 96) clients. In this setup, each client has 1, 000 samples for training. significantly outperformed the centralized baseline (zero low￾resolution clients) at a low resolution. Recall fr…
Figure 9
Figure 9. Figure 9: Inference with an interpolated image. The top row [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 12
Figure 12. Figure 12: t-SNE visualizations of feature embeddings, colored [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 11
Figure 11. Figure 11: t-SNE visualization of feature embeddings (from [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 38 canonical work pages

  1. [1]

    Federated learning for iot devices with domain generalization,

    L. Zhang, X. Lei, Y . Shi, H. Huang, and C. Chen, “Federated learning for iot devices with domain generalization,” IEEE Internet of Things Journal, vol. 10, no. 11, pp. 9622–9633, 2023

    work page 2023

  2. [2]

    Communication-efficient federated learning for wireless edge intelligence in iot,

    J. Mills, J. Hu, and G. Min, “Communication-efficient federated learning for wireless edge intelligence in iot,” IEEE Internet of Things Journal , vol. 7, no. 7, pp. 5986–5994, 2020

    work page 2020

  3. [3]

    Edge intelligence: The confluence of edge computing and artificial intelligence,

    S. Deng, H. Zhao, W. Fang, J. Yin, S. Dustdar, and A. Y . Zomaya, “Edge intelligence: The confluence of edge computing and artificial intelligence,” IEEE Internet of Things Journal , vol. 7, no. 8, pp. 7457– 7469, 2020

    work page 2020

  4. [4]

    Wireless edge computing with latency and reliability guarantees,

    M. S. ElBamby, C. Perfecto, C.-F. Liu, J. Park, S. Samarakoon, X. Chen, and M. Bennis, “Wireless edge computing with latency and reliability guarantees,” Proceedings of the IEEE , vol. 107, no. 8, pp. 1717–1737, 2019

    work page 2019

  5. [5]

    Decentralized edge intelligence: A dynamic resource allocation framework for hierarchical federated learning,

    W. Y . B. Lim, J. S. Ng, Z. Xiong, J. Jin, Y . Zhang, D. Niyato, C. Leung, and C. Miao, “Decentralized edge intelligence: A dynamic resource allocation framework for hierarchical federated learning,” IEEE Transactions on Parallel and Distributed Systems , vol. 33, no. 3, pp. 536–550, 2022

    work page 2022

  6. [6]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Ag ¨uera y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , 2017

    work page 2017

  7. [7]

    Federated learning: Challenges, methods, and future directions,

    T. Li, A. K. Sahu, A. Talwalkar, and V . Smith, “Federated learning: Challenges, methods, and future directions,” IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50–60, 2020

    2020

  8. [8]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” in Proceedings of the Third Conference on Machine Learning and Systems , 2020

    work page 2020

Show all 46 references
  1. [9]

    Scaffold: Stochastic controlled averaging for federated learning,

    S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in Proceedings of the 37th International Conference on Machine Learning, 2020

    2020

  2. [10]

    Simple baselines for human pose estimation and tracking,

    B. Xiao, H. Wu, and Y . Wei, “Simple baselines for human pose estimation and tracking,” in 15th European Conference on Computer Vision, 2018

    2018

  3. [11]

    Vitpose: Simple vision transformer baselines for human pose estimation,

    Y . Xu, J. Zhang, Q. Zhang, and D. Tao, “Vitpose: Simple vision transformer baselines for human pose estimation,” in Advances in Neural Information Processing Systems 35 , 2022

    2022

  4. [12]

    Deep high-resolution repre- sentation learning for human pose estimation,

    K. Sun, B. Xiao, D. Liu, and J. Wang, “Deep high-resolution repre- sentation learning for human pose estimation,” in IEEE Conference on Computer Vision and Pattern Recognition , 2019

    2019

  5. [13]

    Redefining non-iid data in federated learning for computer vision tasks: Migrating from labels to embeddings for task-specific data distributions,

    K. Borazjani, P. Abdisarabshali, N. Khosravan, and S. Hosseinalipour, “Redefining non-iid data in federated learning for computer vision tasks: Migrating from labels to embeddings for task-specific data distributions,” 2025

    2025

  6. [14]

    Distilling the knowledge in a neural network,

    G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” CoRR, vol. abs/1503.02531, 2015

    2015 arXiv

  7. [15]

    Self-distillation amplifies regularization in hilbert space,

    H. Mobahi, M. Farajtabar, and P. Bartlett, “Self-distillation amplifies regularization in hilbert space,” in Advances in Neural Information Processing Systems 33 , vol. 33, 2020, pp. 3351–3361

    2020

  8. [16]

    Revisiting knowledge distillation via label smoothing regularization,

    L. Yuan, F. E. H. Tay, G. Li, T. Wang, and J. Feng, “Revisiting knowledge distillation via label smoothing regularization,” in IEEE Conference on Computer Vision and Pattern Recognition , 2020

    2020

  9. [17]

    Resformer: Scaling vits with multi-resolution training,

    R. Tian, Z. Wu, Q. Dai, H. Hu, Y . Qiao, and Y .-G. Jiang, “Resformer: Scaling vits with multi-resolution training,” in IEEE Conference on Computer Vision and Pattern Recognition , 2023

    2023

  10. [18]

    Tackling the ob- jective inconsistency problem in heterogeneous federated optimization,

    J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V . Poor, “Tackling the ob- jective inconsistency problem in heterogeneous federated optimization,” in Advances in Neural Information Processing Systems 33 , 2020

    2020

  11. [19]

    Model-contrastive federated learning,

    Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in IEEE Conference on Computer Vision and Pattern Recognition , 2021

    2021

  12. [20]

    Feddyn: A dynamic and efficient federated distillation approach on recommender system,

    C. Jin, X. Chen, Y . Gu, and Q. Li, “Feddyn: A dynamic and efficient federated distillation approach on recommender system,” in 28th IEEE International Conference on Parallel and Distributed Systems , 2022

    2022

  13. [21]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30 , 2017, pp. 5998–6008

    2017

  14. [22]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics , 2019

    2019

  15. [23]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” CoRR, vol. abs/2303.08774, 2023

    2023 arXiv

  16. [24]

    Llama: Open and efficient foundation language models,

    H. T. et al., “Llama: Open and efficient foundation language models,” CoRR, vol. abs/2302.13971, 2023

    2023 arXiv

  17. [25]

    Large concept models: Language modeling in a sentence representation space,

    L. team et al., “Large concept models: Language modeling in a sentence representation space,” CoRR, vol. abs/2412.08821, 2024

    2024 arXiv

  18. [26]

    An image is worth 16x16 words: Trans- formers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Trans- formers for image recognition at scale,” in 9th International Conference on Le...

    2021

  19. [27]

    Training data-efficient image transformers & distillation through attention,

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J ´egou, “Training data-efficient image transformers & distillation through attention,” in Proceedings of the 38th International Conference on Machine Learning , 2021

    2021

  20. [28]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in IEEE International Conference on Computer Vision , 2023

    2023

  21. [29]

    Dinov2: Learning robust visual features without supervi- sion,

    M. O. et al., “Dinov2: Learning robust visual features without supervi- sion,” Transactions on Machine Learning Research , vol. 2024, 2024

    2024

  22. [30]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems 33 , 2020

    2020

  23. [31]

    Audiolm: A language modeling approach to audio gen- eration,

    Z. B. et al., “Audiolm: A language modeling approach to audio gen- eration,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2523–2533, 2023

    2023

  24. [32]

    Ast: Audio spectrogram transformer,

    Y . Gong, Y .-A. Chung, and J. R. Glass, “Ast: Audio spectrogram transformer,” in Interspeech 2021, 2021, pp. 571–575

    2021

  25. [33]

    Learning transferable visual models from natural language supervision,

    A. R. et al., “Learning transferable visual models from natural language supervision,” in Proceedings of the 38th International Conference on Machine Learning, 2021

    2021

  26. [34]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. C. H. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International Conference on Machine Learning , 2023

    2023

  27. [35]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” in Advances in Neural Information Processing Systems 36 , 2023

    2023

  28. [36]

    The prospect of enhancing large-scale heterogeneous federated learning with foundation models,

    Y . Gao, Z. Hou, C. Yang, Z. Li, H. Yu, and X. Li, “The prospect of enhancing large-scale heterogeneous federated learning with foundation models,” in IEEE International Conference on Multimedia and Expo , 2024

    2024

  29. [37]

    Fedyolo: Augmenting federated learning with pretrained transformers,

    X. Z. et al., “Fedyolo: Augmenting federated learning with pretrained transformers,” CoRR, vol. abs/2307.04905, 2023

    2023 arXiv

  30. [38]

    Deep high-resolution representation learning for visual recognition,

    J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y . Zhao, D. Liu, Y . Mu, M. Tan, X. Wang, W. Liu, and B. Xiao, “Deep high-resolution representation learning for visual recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 43, no. 10, pp. 3349– 3364, 2021

    2021

  31. [39]

    Bootstrap your own latent – a new approach to self- supervised learning,

    J.-B. G. et al., “Bootstrap your own latent – a new approach to self- supervised learning,” in Advances in Neural Information Processing Systems 33, 2020

    2020

  32. [40]

    On lazy training in differentiable programming,

    L. Chizat, E. Oyallon, and F. Bach, “On lazy training in differentiable programming,” 2019

    2019

  33. [41]

    Gradient descent maximizes the margin of ho- mogeneous neural networks,

    K. Lyu and J. Li, “Gradient descent maximizes the margin of ho- mogeneous neural networks,” in International Conference on Learning Representations, 2019

    2019

  34. [42]

    Prevalence of neural collapse during the terminal phase of deep learning training,

    V . Papyan, X. Han, and D. L. Donoho, “Prevalence of neural collapse during the terminal phase of deep learning training,” Proceedings of the National Academy of Sciences , vol. 117, no. 40, pp. 24 652–24 663, 2020

    2020

  35. [43]

    On the convergence of fedavg on non-iid data,

    X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” in 8th International Conference on Learning Representations, 2020

    2020

  36. [44]

    2d human pose estimation: New benchmark and state of the art analysis,

    M. Andriluka, L. Pishchulin, P. V . Gehler, and B. Schiele, “2d human pose estimation: New benchmark and state of the art analysis,” in IEEE Conference on Computer Vision and Pattern Recognition , 2014

    2014

  37. [45]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conference on Computer Vision and Pattern Recognition, 2016

    2016

  38. [46]

    Fully convolutional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition, 2015

    2015

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.