{"id":"c8be9411-89d3-456b-9759-a6631cbe42c0","arxiv_id":"2505.12021","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An orthogonal similarity alignment learned from 100 images per task transfers task vectors from a YFCC100M-trained ViT to a LAION400M-trained ViT, reaching 71.9% average accuracy versus 72.9% for few-shot fine-tuning.","lead":"Task vectors are parameter-space edits that make a model behave as if trained on a new task. This paper shows how to transfer such vectors between two independently pre-trained vision models by learning a few-shot orthogonal rotation of each update, reaching accuracy close to standard few-shot fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-task U_{i,l} contradict the shared-rotation story of Eq. (6); without a shared-U or cross-task transfer test, the reported gains may be constrained fine-tuning rather than task-vector transfer.","rationale":"The reader's weakest_assumption is the same one that I find most load-bearing: Eq. (6) assumes a task-independent orthogonal similarity between source and target parameter spaces, and the paper never verifies it. My pass adds a specific, textual observation: Section 3.2 defines a shared {U_l}, while Section 4.2 learns U_{i,l} per task. This is not a minor detail; it changes what the experiment demonstrates. If each U_i is fit to the target task's labels, the good accuracies in Table 1 can be explained by the U's adapting the source update to the target task, with the source task vector acting as an initialization. The method then is a constrained few-shot fine-tuner, not a transfer mechanism, and the 'modularity and reusability' claim is weakened because each task requires storing and fitting a large orthogonal matrix. I do not claim the paper is wrong; the empirical improvement is real and code is released. The issue is that the central explanatory mechanism is untested and the implementation is inconsistent with the stated theory. The proposed shared-U experiment would settle the question. Because the reader already assigned CONDITIONAL, my read does not move the verdict; if the shared-U test fails, the verdict should be REJECT or the claim should be reframed.","tokens_in":7474,"tokens_out":10496,"duration_ms":109328,"concrete_test":"Learn a single shared U_l per layer (initialized to I) using the union of all eight tasks' 100-image few-shot sets with the same loss and α, then apply that same U to all eight task vectors and report the per-task and average accuracies. If the average stays near 71.90%, the rotation is a model-space property and the per-task matrices were unnecessary. If it falls back toward the 51.81% target-only/direct-transfer level, the per-task U_{i,l} were fitting task-specific target information. A complementary diagnostic from the same run: record the chordal distance between U_{i,l} and U_{j,l} from the original experiments; large distances across tasks contradict a single shared model-space rotation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.2 the method is defined with a single shared set of orthogonal matrices {U_l}; Eq. (6) states W^(T) ≈ U^T W^(S) U, so U is a property of the two pretrained models and should not depend on the task. Section 4.2 silently switches to per-task matrices U_{i,l} and trains each one on 100 labeled target examples from task i. Nothing in the paper reconciles this switch, and no experiment uses a shared U. Under the task-specific formulation, the cross-entropy objective can improve by rotating the source task vector into a direction that fits the target labels, which is operationally constrained few-shot fine-tuning rather than reuse of the task vector. The capacity is large: for d=768 each U_l has ~294k degrees of freedom, so per task the 48 attention matrices give ~14M parameters. The paper never tests Eq. (6) against the orthogonal Procrustes solution, and never checks whether a U learned on one task transfers to another task. Appendix A's full-parameter failure (41.57%, below the 51.81% target-only baseline) is consistent with U providing task-specific adaptation. If this reading is correct, the headline 71.90% vs 51.48% gain is not evidence for cross-model task-vector transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a method for transferring task vectors between two Vision Transformers with different pre-training (YFCC100M source, LAION400M target). Task vectors are computed on the source model by fine-tuning only attention embedding layers or LoRA matrices. To apply them to the target model, the method learns per-layer orthogonal similarity transformations U_l by minimizing cross-entropy loss on 100 labeled images per target task plus a soft orthogonality penalty. Experiments on eight classification datasets report that the alignment raises average transfer accuracy from about 51.5% (direct transfer) to 71.9% for embedding-based vectors and 69.9% for LoRA-based vectors, approaching the 72.88% of a few-shot fine-tuned target model. The paper argues that the method preserves norm and rank of task vectors and maintains modularity and reusability.","tokens_in":7822,"tokens_out":4251,"duration_ms":44250,"significance":"If the central claim holds, the paper would meaningfully extend task arithmetic beyond the same-initialization assumption, which is a real restriction in existing task-vector literature. The method is simple, the code is released, and the evaluation covers eight datasets with ablations on alpha and lambda. The main risk is that the per-task learned orthogonal matrices U_{i,l} may act as a task-specific fitting mechanism rather than a true alignment of the two pre-trained parameter spaces. Because the paper never tests whether U learned on one task transfers to another task, the reported gains could be interpreted as constrained few-shot fine-tuning rather than evidence for cross-model task-vector transfer. This concern is load-bearing and needs to be resolved experimentally before the central claim is established.","major_comments":[{"comment":"There is an unresolved inconsistency between the motivating Equation (6), where U is a property of the two pre-trained models and independent of any task, and Section 4.2, which learns separate U_{i,l} for each task i using 100 labeled examples from that task. With task-specific U_{i,l}, the objective can improve accuracy by rotating the source update into a direction that fits the target labels, which is operationally constrained few-shot fine-tuning rather than reuse of the source task vector. The paper should either learn a shared U across tasks or, failing that, explicitly test whether a U learned on one task transfers to held-out tasks. Without such an experiment, the headline result (71.90% vs 51.48%) does not by itself support the claim of cross-model task-vector transfer.","section":"§3.2-3.3 and §4.2"},{"comment":"The caption of Table 1 states that FS-Merge [16] is included as a baseline, but no FS-Merge row appears in the table. Since Related Work explicitly discusses Foldable SuperNets as a method for merging transformers with different initializations, omitting this baseline prevents a direct comparison with existing cross-initialization merging techniques and makes the table caption misleading. Please add the missing baseline or correct the caption.","section":"Table 1"},{"comment":"The experiments report no error bars, standard deviations, or multiple seeds. The differences among Embed (w/ Ours) at 71.90%, LoRA (w/ Ours) at 69.90%, and Target model (w/ FT) at 72.88% may be within run-to-run noise, especially since each task uses only 100 images and stochastic optimization throughout. Additionally, the 'Target model (w/ FT)' baseline is underspecified: it is unclear whether it is a multi-task model trained jointly on all eight datasets with the same 100 images per task, and what architecture, epochs, and hyperparameters are used. Without this information, the claim that the method achieves 'performance comparable to few-shot fine-tuning' cannot be assessed quantitatively.","section":"§4.1, §4.2, Table 1"},{"comment":"The full-parameter setting is not well-defined with respect to the proposed transformation. The orthogonal similarity update U^T Delta W U in Equation (4) requires square parameter matrices, but full-model fine-tuning of a ViT-B/32 includes non-square matrices such as the MLP layers (3072 x 768) and the classification head. The appendix does not explain how these matrices are handled. Furthermore, the 41.57% average, far below the 51.81% target-only baseline, is explained only post hoc as 'noisy task vectors'; under the per-task fitting interpretation, this failure is expected if the source full fine-tuning overfits the 100 images and the learned U cannot compensate. The authors should specify the full-parameter procedure and address this alternative explanation.","section":"Appendix A, Table 2"},{"comment":"The paper states that orthogonal similarity transformations exactly preserve Frobenius norm and rank, but the learned matrices U are only softly penalized by alpha * ||U^T U - I||^2, not constrained to be orthogonal. With alpha = 1.0, and especially with alpha = 0.3 in Table 3, the learned U can deviate substantially from orthogonality, so the norm- and rank-preservation claims hold only approximately. The authors should state this approximation explicitly and report the final orthogonality deviation (e.g., mean ||U^T U - I||_F) after training.","section":"Equations (4)-(5) and §B"}],"minor_comments":[{"comment":"Reference [27] contains a typo: 'Patrick Helber and ohters' should be 'Patrick Helber and others'.","section":"References"},{"comment":"The description 'Training uses 100 mini-batches, with images refreshed at each batch' is ambiguous. Please specify the mini-batch size, the number of optimization steps, and whether each of the 1,000 augmented images is used exactly once per epoch.","section":"§4.2"},{"comment":"The sample images in Figure 1 are very small; higher-resolution crops would help the reader judge the difficulty and domain shift of the datasets.","section":"Figure 1"},{"comment":"The term 'orthogonal alignment' is stronger than what is implemented, since orthogonality is only softly enforced. Consider using 'approximately orthogonal' or 'orthogonality-regularized' in the title or at least in key technical statements to avoid overclaiming.","section":"Title and throughout"},{"comment":"The text claims 'consistent and performance improvements across all datasets', but the results show that Embed (w/ Ours) on Cars is effectively unchanged (74.51 vs 74.58) and LoRA (w/ Ours) is worse than the target-only baseline on Cars, DTD, and SUN397. Please soften this statement to match the reported numbers.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The central issue is whether the method truly transfers task vectors or simply performs constrained few-shot fine-tuning. This is testable: if a U learned on one task transfers to other tasks, the alignment story is supported; if not, the paper's contribution is a few-shot adaptation method, which is a different claim. I would ask the authors to add a cross-task transfer experiment and a direct comparison with few-shot fine-tuning of the target model's embedding/LoRA layers using the same 100 images. The code release is a positive sign, but the underspecified 'Target model (w/ FT)' baseline and the missing FS-Merge row in Table 1 need correction. The manuscript is within the journal's scope, but the current evidence does not yet establish the headline phenomenon."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on the cross-model task-vector paper. The idea is clean and the results are big, but the evaluation undercuts the central claim.\n\nWhat is genuinely new: per-layer orthogonal similarity transforms as a way to move task vectors between differently pre-trained ViTs. Git Re-Basin uses permutations, ZipIt! uses feature stitching; rotating a task vector with an orthogonal matrix is not in the cited prior work. The benchmark is solid in scope: two real pre-trainings, eight tasks, and the raw gains are large—51.5 to 71.9 average with embedding vectors. The method is simple enough that it deserves a test.\n\nWhat worries me is the protocol. Section 3.2 describes a shared set of orthogonal matrices U_l, a property of the two pre-trained models that should not depend on the task. Section 4.2 silently switches to a separate U_{i,l} per task, each learned on 100 labeled examples from that task. That switch changes the meaning. With per-task U, the optimizer can rotate the source task vector until it fits the target labels. The capacity is high: for d=768, each U_l has about 2.9e5 free parameters, so 48 attention matrices give around 14M parameters per task. The reported 71.9% is close to the 72.9% of full few-shot fine-tuning on the same data, which is consistent with the rotation supplying task-specific adaptation rather than recovering a shared model-pair rotation.\n\nAppendix A strengthens that reading. Aligning full-parameter task vectors drops accuracy to 41.6%, below the target-only baseline of 51.8%. If U were recovering a true rotation between the models, full-parameter alignment should do at least as well as attention-only alignment. Instead it looks like per-task U fits noise.\n\nThe paper also has standard reporting gaps: no error bars, the FS-Merge baseline named in Table 1 never appears, and the 'Target model (w/ FT)' baseline is underspecified. Those are fixable. The load-bearing issue is the shared-U versus per-task-U discrepancy, and no experiment tests whether a U learned on one task transfers to another or whether a shared U learned from the model weights works at all.\n\nSo the paper is not a waste of time. The orthogonal alignment idea is worth a careful look, and a referee could push the authors to run two decisive controls: learn one shared U from W^S and W^T (via Procrustes or direct optimization) and test transfer; learn U on one task and apply it to another. If either works, the paper becomes strong. As it stands, the evidence supports 'constrained few-shot fine-tuning' as much as it supports 'task-vector transfer.' I'd send it to peer review, but I'd require those experiments before acceptance.","headline":"The paper's per-task alignment matrices, learned on target labels, make the headline gains look like constrained few-shot fine-tuning rather than cross-model task-vector transfer.","tokens_in":8256,"tokens_out":6303,"would_cite":false,"duration_ms":59477,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Task vectors can cross pretraining gaps when first rotated with a few labeled examples.","keywords":["task arithmetic","task vectors","cross-model transfer","orthogonal alignment","few-shot learning","Vision Transformer","LoRA","model merging"],"falsifier":"Fit a rotation on one task from a source-target pair and apply the same rotation to an unseen task vector from the same source; if the unseen task transfers as well, the rotation is a model-level alignment, but if it fails, the method's gains come from task-specific fitting. A second check is to measure the residual $\\|W^{(T)} - U^{\\top} W^{(S)} U\\|$ over layers after fitting $U$; a large residual contradicts the rotational-mismatch premise.","tokens_in":7312,"feed_emoji":"🔄","tokens_out":8217,"duration_ms":77203,"temperature":0.7,"pith_summary":"Task arithmetic treats the weight change from fine-tuning as a vector that can be added to another model, but only if both start from the same pretrained weights. This paper asks whether a task vector can be reused in a model pretrained on different data, and answers yes if the vector is first rotated into the target model's coordinate system by per-layer orthogonal matrices learned from a small number of labeled images. On two Vision Transformers pretrained on different datasets and eight classification benchmarks, direct task vector transfer produces no gain, while the rotated vectors reach accuracy close to full-model few-shot fine-tuning. The payoff is that task-specific abilities stay modular and reusable instead of being fused into a jointly fine-tuned model.","feed_headline":"Task vectors cross pretraining gaps via few-shot rotations","feed_subtitle":"With 100 labeled images per task, transformed vectors nearly match full few-shot fine-tuning on eight datasets.","key_machinery":"For every layer $l$, an orthogonal matrix $U_l$ is found by minimizing the target model's cross-entropy after applying the transformed task vector $U_l^{\\top} \\Delta W_l^{(S)} U_l$, plus a Frobenius penalty that keeps $U_l$ orthogonal. Each $U_l$ is initialized to the identity, so early updates do nothing to the task vector, which helps in the few-shot regime. Orthogonality preserves the Frobenius norm and the matrix rank of the update, so low-rank LoRA task vectors keep their structure, and under the relation $W^{(T)} \\approx U^{\\top} W^{(S)} U$ the transform exactly compensates a rotational mismatch between pretrained parameter spaces. Learned per task and per layer, the matrices are applied by summing all transformed task vectors onto the target weights.","core_discovery":"The paper's central claim is that an orthogonal similarity transformation $U_l^{\\top} \\Delta W_l^{(S)} U_l$ maps a task vector computed on a source model into the parameter space of a target model pretrained on different data, and that the matrices $U_l$ can be learned from 100 labeled images per task. This preserves the norm and rank of each update and lets several task vectors be added to the target model at once. Across eight classification datasets and two Vision Transformers pretrained on different corpora, the rotated embedding-based task vectors average 71.9% accuracy and the LoRA-based vectors 69.9%, against 72.88% for full-model few-shot fine-tuning of the target and about 51.8% for direct task vector application. The authors conclude that task vectors can cross pretraining boundaries when aligned, keeping task-specific capabilities modular and reusable.","pith_inferences":["Beyond the paper, a rotation matrix fitted on one task from a source-target pair should also transfer a different task vector between the same pair; testing this would separate genuine model-level alignment from task-specific fitting.","The near-parity with few-shot fine-tuning leaves open the alternative reading that the cross-entropy used to learn the rotation is itself adapting the task to the target, in which case the method is a constrained fine-tuning rather than a true reuse of the source task vector.","Because orthogonal similarity acts on square matrices, the transfer applies only to same-shaped layers; rectangular layers would need a different alignment group, and the paper leaves that extension open."],"forward_implications":["A source Vision Transformer fine-tuned once per task can act as a reusable library whose task vectors are applied to any same-architecture target model after a light alignment step.","Because the transformation preserves rank, low-rank LoRA-style task vectors can cross pretraining boundaries without losing their structure.","Selective fine-tuning of only embedding or LoRA parameters transfers better than full-parameter task vectors, so the compactness of the update matters as much as the alignment.","The small accuracy gap to full-model few-shot fine-tuning suggests that modularity can be bought at little cost.","The method requires only 100 labeled images per task, making the alignment step practical in low-supervision settings."],"supporting_citations":[{"why":"Defines task vectors and the same-initialization assumption that this paper removes.","marker":"[5]"},{"why":"Shows models can be aligned modulo permutation symmetries, framing the cross-model alignment problem.","marker":"[9]"},{"why":"Serves as the cross-initialization merging baseline in the benchmark.","marker":"[16]"},{"why":"Supplies the Vision Transformer architecture and the attention embedding layers used to construct task vectors.","marker":"[21]"},{"why":"Supplies LoRA low-rank updates used as alternative task vectors.","marker":"[22]"},{"why":"Provides the source pretraining corpus for the YFCC-pretrained model.","marker":"[23]"},{"why":"Provides the target pretraining corpus for the LAION-pretrained model.","marker":"[24]"}],"fun_headline_variants":["Task vectors jump pretraining gaps with few-shot rotations","Few-shot rotation aligns task vectors across models","Cross-model task vectors: 100 images per task beats direct transfer","Orthogonal alignment lets task vectors cross pretraining domains","Task vectors adapt to new pretraining with just 100 examples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The transfer rests on the assumption that a rotation learned from 100 labeled images genuinely carries a task vector's meaning across models, rather than merely re-solving the task in the target model.","fun_headline_variants_meta":{"raw":{"variants":["Task vectors jump pretraining gaps with few-shot rotations","Few-shot rotation aligns task vectors across models","Cross-model task vectors: 100 images per task beats direct transfer","Orthogonal alignment lets task vectors cross pretraining domains","Task vectors adapt to new pretraining with just 100 examples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1393,"prompt_tokens":894,"completion_tokens":499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":420}},"tokens_in":510,"tokens_out":499,"duration_ms":4277,"temperature":1.0,"reasoning_tokens":420,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:42:21.947820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit a rotation on one task from a source-target pair and apply the same rotation to an unseen task vector from the same source; if the unseen task transfers as well, the rotation is a model-level alignment, but if it fails, the method's gains come from task-specific fitting. A second check is to measure the residual $\\|W^{(T)} - U^{\\top} W^{(S)} U\\|$ over layers after fitting $U$; a large residual contradicts the rotational-mismatch premise.","supporting_citations":[{"cited_title":"Editing models with task arithmetic","cited_arxiv_id":null,"evidence_quote":"Defines task vectors and the same-initialization assumption that this paper removes."},{"cited_title":"Git re-basin: Merging models modulo permutation symmetries","cited_arxiv_id":null,"evidence_quote":"Shows models can be aligned modulo permutation symmetries, framing the cross-model alignment problem."},{"cited_title":"An image is worth 16x16 words: Transformers for image recognition at scale","cited_arxiv_id":null,"evidence_quote":"Supplies the Vision Transformer architecture and the attention embedding layers used to construct task vectors."},{"cited_title":"Lora: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"Supplies LoRA low-rank updates used as alternative task vectors."},{"cited_title":"Yfcc100m: The new data in multimedia research","cited_arxiv_id":null,"evidence_quote":"Provides the source pretraining corpus for the YFCC-pretrained model."}],"review_version":1}