Pith. sign in

REVIEW 5 major objections 5 minor 33 references

Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignment

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Task vectors can cross pretraining gaps when first rotated with a few labeled examples.

desk verdict The paper's per-task alignment matrices, learned on target labels, make the headline gains look like constrained few-shot fine-tuning rather than cross-model task-vector transfer. read the letter →

arxiv 2505.12021 v1 pith:YYP475QR submitted 2025-05-17 cs.CV

Kazuhiko Kawamoto , Atsuhiro Endo , Hiroshi Kera This is my paper · ORCID
classification cs.CV
keywords taskarithmeticvectorscross-modeltransferorthogonalalignmentfew-shotlearningVisionTransformerLoRAmodelmerging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Task arithmetic treats the weight change from fine-tuning as a vector that can be added to another model, but only if both start from the same pretrained weights. This paper asks whether a task vector can be reused in a model pretrained on different data, and answers yes if the vector is first rotated into the target model's coordinate system by per-layer orthogonal matrices learned from a small number of labeled images. On two Vision Transformers pretrained on different datasets and eight classification benchmarks, direct task vector transfer produces no gain, while the rotated vectors reach accuracy close to full-model few-shot fine-tuning. The payoff is that task-specific abilities stay modular and reusable instead of being fused into a jointly fine-tuned model.

What carries the argument

For every layer $l$, an orthogonal matrix $U_l$ is found by minimizing the target model's cross-entropy after applying the transformed task vector $U_l^{\top} \Delta W_l^{(S)} U_l$, plus a Frobenius penalty that keeps $U_l$ orthogonal. Each $U_l$ is initialized to the identity, so early updates do nothing to the task vector, which helps in the few-shot regime. Orthogonality preserves the Frobenius norm and the matrix rank of the update, so low-rank LoRA task vectors keep their structure, and under the relation $W^{(T)} \approx U^{\top} W^{(S)} U$ the transform exactly compensates a rotational mismatch between pretrained parameter spaces. Learned per task and per layer, the matrices are applied by summing all transformed task vectors onto the target weights.

What would settle it

Fit a rotation on one task from a source-target pair and apply the same rotation to an unseen task vector from the same source; if the unseen task transfers as well, the rotation is a model-level alignment, but if it fails, the method's gains come from task-specific fitting. A second check is to measure the residual $\|W^{(T)} - U^{\top} W^{(S)} U\|$ over layers after fitting $U$; a large residual contradicts the rotational-mismatch premise.

Watch

Extended reading notes

Core claim

The paper's central claim is that an orthogonal similarity transformation $U_l^{\top} \Delta W_l^{(S)} U_l$ maps a task vector computed on a source model into the parameter space of a target model pretrained on different data, and that the matrices $U_l$ can be learned from 100 labeled images per task. This preserves the norm and rank of each update and lets several task vectors be added to the target model at once. Across eight classification datasets and two Vision Transformers pretrained on different corpora, the rotated embedding-based task vectors average 71.9% accuracy and the LoRA-based vectors 69.9%, against 72.88% for full-model few-shot fine-tuning of the target and about 51.8% for direct task vector application. The authors conclude that task vectors can cross pretraining boundaries when aligned, keeping task-specific capabilities modular and reusable.

Load-bearing premise

The transfer rests on the assumption that a rotation learned from 100 labeled images genuinely carries a task vector's meaning across models, rather than merely re-solving the task in the target model.

Editorial extensions

If this is right

  • A source Vision Transformer fine-tuned once per task can act as a reusable library whose task vectors are applied to any same-architecture target model after a light alignment step.
  • Because the transformation preserves rank, low-rank LoRA-style task vectors can cross pretraining boundaries without losing their structure.
  • Selective fine-tuning of only embedding or LoRA parameters transfers better than full-parameter task vectors, so the compactness of the update matters as much as the alignment.
  • The small accuracy gap to full-model few-shot fine-tuning suggests that modularity can be bought at little cost.
  • The method requires only 100 labeled images per task, making the alignment step practical in low-supervision settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a rotation matrix fitted on one task from a source-target pair should also transfer a different task vector between the same pair; testing this would separate genuine model-level alignment from task-specific fitting.
  • The near-parity with few-shot fine-tuning leaves open the alternative reading that the cross-entropy used to learn the rotation is itself adapting the task to the target, in which case the method is a constrained fine-tuning rather than a true reuse of the source task vector.
  • Because orthogonal similarity acts on square matrices, the transfer applies only to same-shaped layers; rectangular layers would need a different alignment group, and the paper leaves that extension open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes a method for transferring task vectors between two Vision Transformers with different pre-training (YFCC100M source, LAION400M target). Task vectors are computed on the source model by fine-tuning only attention embedding layers or LoRA matrices. To apply them to the target model, the method learns per-layer orthogonal similarity transformations U_l by minimizing cross-entropy loss on 100 labeled images per target task plus a soft orthogonality penalty. Experiments on eight classification datasets report that the alignment raises average transfer accuracy from about 51.5% (direct transfer) to 71.9% for embedding-based vectors and 69.9% for LoRA-based vectors, approaching the 72.88% of a few-shot fine-tuned target model. The paper argues that the method preserves norm and rank of task vectors and maintains modularity and reusability.

Significance. If the central claim holds, the paper would meaningfully extend task arithmetic beyond the same-initialization assumption, which is a real restriction in existing task-vector literature. The method is simple, the code is released, and the evaluation covers eight datasets with ablations on alpha and lambda. The main risk is that the per-task learned orthogonal matrices U_{i,l} may act as a task-specific fitting mechanism rather than a true alignment of the two pre-trained parameter spaces. Because the paper never tests whether U learned on one task transfers to another task, the reported gains could be interpreted as constrained few-shot fine-tuning rather than evidence for cross-model task-vector transfer. This concern is load-bearing and needs to be resolved experimentally before the central claim is established.

major comments (5)
  1. [§3.2-3.3 and §4.2] There is an unresolved inconsistency between the motivating Equation (6), where U is a property of the two pre-trained models and independent of any task, and Section 4.2, which learns separate U_{i,l} for each task i using 100 labeled examples from that task. With task-specific U_{i,l}, the objective can improve accuracy by rotating the source update into a direction that fits the target labels, which is operationally constrained few-shot fine-tuning rather than reuse of the source task vector. The paper should either learn a shared U across tasks or, failing that, explicitly test whether a U learned on one task transfers to held-out tasks. Without such an experiment, the headline result (71.90% vs 51.48%) does not by itself support the claim of cross-model task-vector transfer.
  2. [Table 1] The caption of Table 1 states that FS-Merge [16] is included as a baseline, but no FS-Merge row appears in the table. Since Related Work explicitly discusses Foldable SuperNets as a method for merging transformers with different initializations, omitting this baseline prevents a direct comparison with existing cross-initialization merging techniques and makes the table caption misleading. Please add the missing baseline or correct the caption.
  3. [§4.1, §4.2, Table 1] The experiments report no error bars, standard deviations, or multiple seeds. The differences among Embed (w/ Ours) at 71.90%, LoRA (w/ Ours) at 69.90%, and Target model (w/ FT) at 72.88% may be within run-to-run noise, especially since each task uses only 100 images and stochastic optimization throughout. Additionally, the 'Target model (w/ FT)' baseline is underspecified: it is unclear whether it is a multi-task model trained jointly on all eight datasets with the same 100 images per task, and what architecture, epochs, and hyperparameters are used. Without this information, the claim that the method achieves 'performance comparable to few-shot fine-tuning' cannot be assessed quantitatively.
  4. [Appendix A, Table 2] The full-parameter setting is not well-defined with respect to the proposed transformation. The orthogonal similarity update U^T Delta W U in Equation (4) requires square parameter matrices, but full-model fine-tuning of a ViT-B/32 includes non-square matrices such as the MLP layers (3072 x 768) and the classification head. The appendix does not explain how these matrices are handled. Furthermore, the 41.57% average, far below the 51.81% target-only baseline, is explained only post hoc as 'noisy task vectors'; under the per-task fitting interpretation, this failure is expected if the source full fine-tuning overfits the 100 images and the learned U cannot compensate. The authors should specify the full-parameter procedure and address this alternative explanation.
  5. [Equations (4)-(5) and §B] The paper states that orthogonal similarity transformations exactly preserve Frobenius norm and rank, but the learned matrices U are only softly penalized by alpha * ||U^T U - I||^2, not constrained to be orthogonal. With alpha = 1.0, and especially with alpha = 0.3 in Table 3, the learned U can deviate substantially from orthogonality, so the norm- and rank-preservation claims hold only approximately. The authors should state this approximation explicitly and report the final orthogonality deviation (e.g., mean ||U^T U - I||_F) after training.
minor comments (5)
  1. [References] Reference [27] contains a typo: 'Patrick Helber and ohters' should be 'Patrick Helber and others'.
  2. [§4.2] The description 'Training uses 100 mini-batches, with images refreshed at each batch' is ambiguous. Please specify the mini-batch size, the number of optimization steps, and whether each of the 1,000 augmented images is used exactly once per epoch.
  3. [Figure 1] The sample images in Figure 1 are very small; higher-resolution crops would help the reader judge the difficulty and domain shift of the datasets.
  4. [Title and throughout] The term 'orthogonal alignment' is stronger than what is implemented, since orthogonality is only softly enforced. Consider using 'approximately orthogonal' or 'orthogonality-regularized' in the title or at least in key technical statements to avoid overclaiming.
  5. [§4.3] The text claims 'consistent and performance improvements across all datasets', but the results show that Embed (w/ Ours) on Cars is effectively unchanged (74.51 vs 74.58) and LoRA (w/ Ours) is worse than the target-only baseline on Cars, DTD, and SUN397. Please soften this statement to match the reported numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the orthogonal alignment matrices are fitted on few-shot training data and evaluated on held-out test sets, and Eq. (6) is a motivating assumption rather than a derived prediction.

full rationale

The paper does not present a first-principles derivation whose output is equivalent to its input. In Section 3.2, the orthogonal matrices U_l are learned by minimizing a cross-entropy loss on target-task labeled examples plus an orthogonality penalty; in Section 4.2, each U_{i,l} is trained on 100 sampled images per target dataset and then the adapted model is evaluated on the same task's test set. This is a standard supervised fitting-and-evaluation procedure, not a hidden reuse of the test labels or a parameter-free prediction. Equation (6), W^(T) ≈ U^T W^(S) U, is explicitly conditional ('If the source and target parameter matrices differ by a rotation...') and is not derived from data; it serves as motivation for choosing orthogonal similarity transformations. The claim that the method 'improves transfer accuracy' is an empirical, falsifiable result: the w/ Ours rows in Table 1 differ from the w/o Ours rows and from the target-only baseline, so the outcome is not forced by construction. The skeptic's concern that per-task U_{i,l} may be doing task-specific adaptation rather than true cross-model transfer is a validity or generalization concern, not circularity: the paper never claims the U matrices are parameter-free or that the transfer accuracy is a prediction made without target-task data. There are no load-bearing self-citations: the reference list contains no prior work by the present authors, and no uniqueness theorem is imported from the authors' own work. The norm-preservation and rank-preservation identities in Eqs. (4)-(5) are simple mathematical facts, not circularly defined quantities. The full-parameter failure in Appendix A is an empirical negative result and is interpreted as such. Overall, the derivation chain is self-contained and the evaluation protocol is held-out; the reported accuracy is a fitted outcome, not a prediction equivalent to its inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. The free parameters are the learned orthogonal matrices and two hand-chosen hyperparameters. The main unverified assumption is the rotation hypothesis linking the two parameter spaces.

free parameters (3)
  • Layer-wise orthogonal alignment matrices U_i,l = learned per task i and layer l
    Fitted to the target task's few-shot data via cross-entropy loss; these matrices encode the entire cross-model adaptation and are the core fitted quantity.
  • Regularization coefficient α = 1.0 (main experiments); ablated 0.3-2.0
    Chosen by hand; controls orthogonality penalty strength.
  • Task vector scaling λ = 1.0 (main experiments); ablated 0.3-2.0
    Scales the source task vector; tuned by hand.
assumptions (3)
  • ad hoc to paper The parameter spaces of the source and target models can be related by per-layer orthogonal similarity transformations: W^(T) ≈ U^T W^(S) U (Eq. 6).
    This assumption motivates the method but is never verified empirically; the learned U may simply reparameterize the few-shot fine-tuning.
  • domain assumption Fine-tuning only attention embedding or LoRA parameters yields transferable task vectors, while full-parameter task vectors do not.
    The paper uses this to justify the selective fine-tuning design; Appendix A shows full-parameter transfer is worse, but the reason is not fully explained.
  • standard math Standard deep learning optimization assumptions (differentiability, gradient descent convergence) hold.
    Required for training U_l and the source fine-tuning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignment." pith.science (2026). https://pith.science/paper/YYP475QR

@misc{pith2026250512021,
  author       = {Pith},
  title        = {Pith review of: Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYP475QR}},
  note         = {Machine review of arXiv:2505.12021}
}
read the original abstract

Task arithmetic enables efficient model editing by representing task-specific changes as vectors in parameter space. Task arithmetic typically assumes that the source and target models are initialized from the same pre-trained parameters. This assumption limits its applicability in cross-model transfer settings, where models are independently pre-trained on different datasets. To address this challenge, we propose a method based on few-shot orthogonal alignment, which aligns task vectors to the parameter space of a differently pre-trained target model. These transformations preserve key properties of task vectors, such as norm and rank, and are learned using only a small number of labeled examples. We evaluate the method using two Vision Transformers pre-trained on YFCC100M and LAION400M, and test on eight classification datasets. Experimental results show that our method improves transfer accuracy over direct task vector application and achieves performance comparable to few-shot fine-tuning, while maintaining the modularity and reusability of task vectors. Our code is available at https://github.com/kawakera-lab/CrossModelTransfer.

Figures

Figures reproduced from arXiv: 2505.12021 by the authors.

Figure 1
Figure 1. Sample images from the eight benchmark datasets used in our experiments. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 29 canonical work pages

  1. [16]

    Foldable supernets: Scalable merging of transformers with different initializations and tasks

    Edan Kinderman, Itay Hubara, Haggai Maron, and Daniel Soudry. Foldable supernets: Scalable merging of transformers with different initializations and tasks. arXiv preprint arXiv:2410.01483, 2024

    arXiv 2024

  2. [1]

    Do better imagenet models transfer better? In CVPR, pages 2661–2671, 2019

    Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In CVPR, pages 2661–2671, 2019

    work page 2019

  3. [2]

    Learning transferable visual models from natural language supervision

    Alec Radford et al. Learning transferable visual models from natural language supervision. In ICML, volume 139, pages 8748–8763, 2021

    work page 2021

  4. [3]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron et al. Emerging properties in self-supervised vision transformers. In ICCV, pages 9650–9660, 2021

    work page 2021

  5. [4]

    Segment anything

    Alexander Kirillov et al. Segment anything. In ICCV, pages 4015–4026, 2023

    work page 2023

  6. [5]

    Editing models with task arithmetic

    Gabriel Ilharco et al. Editing models with task arithmetic. In ICLR, 2023

    work page 2023

  7. [6]

    Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman et al. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, volume 162, pages 23965–23998, 2022

    work page 2022

  8. [7]

    Roy, and Michael Carbin

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, volume 119, pages 3259–3269, 2020

    work page 2020

Show all 33 references
  1. [8]

    Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture

    Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture. In CVPR, pages 16399–16409, 2022

    2022

  2. [9]

    Git re-basin: Merging models modulo permutation symmetries

    Sam Ainsworth, Ilya Eckstein, Paul Barham, Andy Brock, and Karen Simonyan. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2023

    2023

  3. [10]

    Zipit! merging models from different tasks without training

    George Stoica et al. Zipit! merging models from different tasks without training. In ICLR, 2024

    2024

  4. [11]

    Task arithmetic in the tangent space: Improved editing of pre-trained models

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In NeurIPS, volume 36, pages 66727–66754, 2023

    2023

  5. [12]

    Efficient model editing with task-localized sparse fine-tuning

    Leonardo Iurada, Marco Ciccone, and Tatiana Tommasi. Efficient model editing with task-localized sparse fine-tuning. In ICLR, 2025

    2025

  6. [13]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang et al. Adamerging: Adaptive model merging for multi-task learning. In ICLR, 2024

    2024

  7. [14]

    Localize-and-stitch: Efficient model merging via sparse task arithmetic

    Yifei He et al. Localize-and-stitch: Efficient model merging via sparse task arithmetic. Transactions on Machine Learning Research, 2025

    2025

  8. [15]

    Merging models with fisher-weighted averaging

    Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In NeurIPS, pages 17703– 17716, 2022

    2022

  9. [17]

    Zhang et al

    Frederic Z. Zhang et al. Knowledge composition using task vectors with learned anisotropic scaling. In NeurIPS, pages 67319–67354, 2024

    2024

  10. [18]

    Vision-language models create cross-modal task representations

    Grace Luo, Trevor Darrell, and Amir Bar. Vision-language models create cross-modal task representations. In ICML, 2025

    2025

  11. [19]

    Task arithmetic in trust region: A training-free model merging approach to navigate knowledge conflicts

    Wenju Sun et al. Task arithmetic in trust region: A training-free model merging approach to navigate knowledge conflicts. In ICLR, 2025

    2025

  12. [20]

    When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers

    Hongkang Li et al. When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers. In ICLR, 2025

    2025

  13. [21]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 6 Kazuhiko Kawamoto et al

    2021

  14. [22]

    Lora: Low-rank adaptation of large language models

    Edward J Hu et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022

    2022

  15. [23]

    Yfcc100m: The new data in multimedia research

    Bart Thomee et al. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64–73, 2016

    2016

  16. [24]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann et al. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021

    2021 arXiv

  17. [25]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In CVPR, pages 554–561, 2013

    2013

  18. [26]

    Describing textures in the wild

    Mircea Cimpoi et al. Describing textures in the wild. In CVPR, pages 3606–3613, 2014

    2014

  19. [27]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber and ohters. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217– 2226, 2019

    2019

  20. [28]

    The german traffic sign recognition benchmark: A multi-class classification competition

    Johannes Stallkamp et al. The german traffic sign recognition benchmark: A multi-class classification competition. In IJCNN, pages 1453–1460. IEEE, 2011

    2011

  21. [29]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  22. [30]

    Remote sensing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, 2017

    2017

  23. [31]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao et al. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, pages 3485–3492. IEEE, 2010

    2010

  24. [32]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer et al. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011

    2011

  25. [33]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. A Appendix: Orthogonal Transformation Using Full parameters We investigate whether task vectors constructed from all parameters of the pre-trained model (i.e., full fine-tuning) can benefit...

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.