{"id":"e5425fe9-c12f-43a3-a376-e2e5ba1ac3ad","arxiv_id":"2502.00264","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Transformer attention layers admit continuous rotation symmetries, and aligning a source model's attention weights by the optimal rotation before weight averaging improves model fusion accuracy.","lead":"The paper shows that rotating the query and key (and value and output) matrices in transformer attention layers by the same orthogonal matrix leaves the model's behavior unchanged, and it uses this continuous symmetry to align two models before merging them. A smart generalist might read it because it offers a simple plug-in step that improves existing weight-averaging model fusion methods for transformers, with closed-form math.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ViT fusion evidence is not decisive: baselines near chance, gains within noise, and full matching sometimes worse than attention-only; the substantial-improvement claim needs stronger evaluation.","rationale":"The mathematical contribution of the paper is sound: the rotation symmetry in Equation (5) is a genuine functional equivalence for self-attention layers, and Algorithm 1 correctly solves the Frobenius-distance matching problem for the stated pairwise objective. I do not see a flaw in the algebra or in the closed-form derivation. The load-bearing weakness is therefore not the theory but the empirical demonstration that this matching 'substantially improves model fusion.' The reader's conditional verdict already identifies small gains, missing error bars, and the validation-selection step in the ViT protocol. I go one step further: in the ViT experiments, several baselines operate at or below chance accuracy on CIFAR-10, which makes the absolute improvements hard to interpret, and the ablation shows the full algorithm can be worse than attention-only matching on Fisher merging. These issues directly affect the paper's central practical claim. My proposed check—re-running the ViT experiments in a regime where all methods produce meaningful accuracy, with multiple seeds—would determine whether the reported improvements are real or an artifact of the evaluation protocol. If the evidence does not survive, the verdict should remain conditional or be revised toward rejection of the strong empirical claims, while the theoretical result would still stand. I therefore recommend keeping the reader's CONDITIONAL verdict unchanged, pending this check.","tokens_in":23857,"tokens_out":23563,"duration_ms":227625,"concrete_test":"Rerun the ViT merging comparison on CIFAR-10 with models that produce non-trivial accuracy, e.g., fine-tune each ViT on CIFAR-10 or train a linear probe on frozen features for each merged model, and report mean plus/minus standard deviation over at least three independent seeds for every fusion method with and without matching. Check whether all baselines clearly exceed random chance, whether matched-versus-unmatched differences exceed the run-to-run noise, and whether the full matching algorithm outperforms its attention-only ablation. If the improvements vanish or fall within noise, the central empirical claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that rotation-symmetry matching substantially improves model fusion, but the strongest empirical support (Table 2, Figure 3) is not robust. The ViT protocol uses two pretrained models evaluated on CIFAR-10 without fine-tuning (Table 3 lists no ViT fine-tuning time), and the absolute numbers are troubling: Simple averaging without matching gives 7.60%, below the 10% random baseline; after matching it reaches 10.19%, still chance-level. Fisher and Regmean are 17.96% and 14.24% before matching, and the best OT baselines improve by only 0.02 to 0.10 accuracy points after matching (e.g., OT-ACTS 61.15 -> 61.25). No error bars or multiple seeds are reported, so these small differences could be run-to-run noise. Furthermore, the ablation in Figure 3 shows that for Fisher, attention-only matching (w/o FFN) achieves 20.21% versus 18.58% for the full method, meaning the complete plug-and-play module is not consistently better than its attention-only component. Given the near-chance performance of several baselines and the ambiguous ViT evaluation protocol, the headline empirical claim of substantial fusion improvement is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a continuous rotational symmetry for transformer self-attention layers, showing that applying independent orthogonal transformations to each head's query/key and value/output weight matrices (Equation 5) preserves the layer function exactly. It then proposes a plug-and-play parameter matching algorithm for model fusion: feedforward blocks are matched by permutation (Equation 7), attention blocks by a Kabsch-style orthogonal Procrustes solution (Algorithm 1), optionally followed by a rescaling step (Equations 11-12). The authors evaluate the method by fusing fine-tuned RoBERTa and DeBERTa models on NLP benchmarks and pretrained ViTs on CIFAR-10, reporting accuracy and loss-barrier results. The central algebraic derivation is straightforward and correct, and the matching algorithm is computationally efficient, but the empirical support for the strong claim of 'substantially improves model fusion' is thin and partly inconsistent with the reported tables and ablations.","tokens_in":24029,"tokens_out":4976,"duration_ms":57312,"significance":"If the results are confirmed, the paper makes a useful contribution: it identifies an exact, continuous symmetry of attention layers that goes beyond discrete permutation symmetries, and it contributes a closed-form, efficient matching procedure that can be attached to existing fusion methods. The vector-space derivation in Section 3.2 is clean and the complexity analysis is order-correct, and the authors are careful to include code, an appendix proof, and a limitations section. The strength of the paper is theoretical and algorithmic; the claimed empirical gains, however, are currently not established at the level needed to support the headline conclusion, and several load-bearing claims need to be either strengthened or carefully scoped.","major_comments":[{"comment":"The empirical evidence for the headline 'substantially improves model fusion' is not robust. In the ViT experiments (Table 2), several baselines are at or near chance: Simple averaging gives 7.60% without matching and 10.19% with matching, where random guessing on CIFAR-10 is 10%; the best OT baselines improve by only 0.02-0.10 accuracy points after matching (e.g., OT-ACTS 61.15 to 61.25). No error bars, variance estimates, or multiple seeds are reported, so differences of this size may be run-to-run noise. The ablation in Figure 3 compounds this concern: for Fisher, attention-only matching (w/o FFN) achieves 20.21% while the complete method achieves 18.58%, meaning the full plug-and-play module is not consistently better than its attention-only component. The statement that 'Our parameter matching algorithm consistently improves the performance of different model fusion methods' is also contradicted by Table 1 entries with zero or negative change, e.g., DeBERTa Simple on Emotion stays at 2.99 and Fisher on NER-CoNLL for RoBERTa decreases from 23.06 to 23.05.","section":"§5.2, Tables 1-2, Figure 3"},{"comment":"The closed-form solution R1 = U V^T is optimal over the orthogonal group, but not necessarily over the proper rotation group. If det(U V^T) = -1, the returned matrix is a reflection, not a rotation, and therefore lies outside the set R over which Equation (9) is minimized. Since Equation (5) and Algorithm 1 explicitly call R a rotation matrix, this distinction matters. The standard fix is the Umeyama correction (U diag(1,...,1,-1) V^T when the determinant is negative), which is cited in the paper but not implemented or discussed. If the authors intend R to mean the full orthogonal group, they should say so explicitly; otherwise Algorithm 1 should include the determinant-correction step to actually return a rotation.","section":"§4.2, Theorem 4.1, Algorithm 1"},{"comment":"The 'theoretically optimal parameter matching' claim is broader than what is proven. The optimality result in Theorem 4.1 is for the binary, rotation-only, Frobenius-norm objective; the rescaling step is applied sequentially after rotation, and the authors themselves note in Appendix D that the sequential approach 'does not guarantee global optimality for the joint optimization over (R, alpha)'. Likewise, the multi-model extension is acknowledged to lack the global optimality of the pairwise case. The paper should state these scoping conditions prominently in the main text and in the abstract-adjacent claims, so that 'optimal' is understood as optimal for the specific weight-space objective under the stated constraints, not as a guarantee about fusion utility.","section":"§4.2, Equations (11)-(12); Appendix D"},{"comment":"The matching objective minimizes Frobenius distance in weight space, which is only a proxy for the fusion objective. The paper motivates this by citing the convexity/closeness heuristic of Wortsman et al., and the loss-barrier experiment in Section 5.4 provides some supporting evidence, but no experiment establishes a monotone relationship between the specific rotation-aligned weight distance and the final fusion metric. Since the choice of the exact objective is what makes the closed-form solution possible, this limitation should be discussed explicitly; as written, the text moves from 'minimize parameter distance' to 'improves fusion' without acknowledging that the two are not guaranteed to be monotonically related across heterogeneous layers, heads, and merging methods.","section":"§4.2, Equation (8)"}],"minor_comments":[{"comment":"The theorem statement and Algorithm 1 say the solution is obtained by 'eigendecomposition', but the proof uses a singular value decomposition; the wording should be corrected to SVD everywhere.","section":"§4.2, Theorem 4.1 and Algorithm 1"},{"comment":"Equation (13) in the appendix duplicates the theorem statement as Equation (9) in the main text, and Equation (14) duplicates Equation (10); the numbering should be adjusted to avoid confusion.","section":"Appendix A"},{"comment":"The label 'OT-ACTS-EMB' in Figure 3 appears to be a typo for 'OT-ACTS-EMD' used in Table 2.","section":"Figure 3"},{"comment":"The phrase 'matching head layers yields greater improvements' is ambiguous: Figure 6(a) shows that early layers (low layer indices) matter most, but 'head layers' could be read as attention heads. Please use 'early layers' or 'initial layers' for clarity.","section":"§5.6"},{"comment":"The paper does not report the number of random seeds or any variance measure for any of the fusion results in Tables 1-2; such information is needed to assess the significance of the small differences reported.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The theoretical part of this paper is sound and potentially useful, and the matching algorithm is clean and cheap to run. My concern is that the empirical section, as currently presented, cannot support the strong 'substantially improves model fusion' conclusion: the ViT results are close to chance for several baselines, the differences are small, and the ablation is partly unfavorable to the full method. I would support acceptance after a revision that either strengthens the experiments (more seeds, error bars, clearer protocols) or substantially narrows the empirical claims, and after the orthogonal-group/rotation distinction in Theorem 4.1 is resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [colleague],\n\nThe rotation symmetry observation is correct and cleanly proven, and the matching algorithm is a sensible, practical application of it. But the empirical claims outrun the evidence, and the ViT results in particular do not survive close reading.\n\nWhat's new: the identity that per-head orthogonal rotations of Q/K (jointly) and V/O (jointly) preserve the attention layer's function, including biases, is a simple linear-algebra fact. The paper uses it to derive a closed-form parameter matching via an orthogonal Procrustes step, with a correct proof and O(d^3) complexity. That's a genuinely useful primitive for transformer model fusion, and the code is public. The authors are also honest in Appendix D about the multi-model optimality gap and the sequential rotation-then-rescaling approximation.\n\nWhere the paper is soft: the empirical support is thin relative to the \"substantially improves\" claim. Table 2 shows Simple averaging at 7.60% before matching—below the 10% random baseline—and 10.19% after, still chance level. Several gains are within 0.02–0.10 accuracy points; no error bars or multiple seeds are reported. The ViT protocol selects the merged model with the highest validation performance before reporting test accuracy, which can bias the result. The ablation in Figure 3 shows the full module is not consistently better than attention-only matching (Fisher: 18.58 vs 20.21; Simple: 10.19 vs 10.45). The NLP tables are mixed too: some DeBERTa improvements are real, but RoBERTa Simple shows no change on Emotion and NER. So \"consistent improvement\" is overstated.\n\nThat said, the core symmetry result and the algorithm are worth having. The connection between weight-space distance and fusion quality is a heuristic, and the paper doesn't overclaim the theory—it just overclaims the experiments.\n\nBottom line: this paper deserves a serious referee and a major revision on the experimental side. Researchers working on model merging will want to cite it for the rotation primitive. I'd bring it to a reading group, mostly to discuss the gap between a clean theoretical identity and the messiness of fusion evaluation.","headline":"Correct symmetry identity with a clean closed-form matching algorithm, but the empirical case for fusion gains is overstated, especially in the ViT results.","tokens_in":24626,"tokens_out":4848,"would_cite":true,"duration_ms":44708,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that rotation symmetries of transformer attention layers are continuous, and that aligning models by rotation improves fusion.","keywords":["rotation symmetry","parameter space symmetry","model fusion","self-attention","Orthogonal Procrustes","weight matching","transformer alignment","model merging"],"falsifier":"Run the proposed rotation on one of the two attention layers and compare its output to the original on a batch of inputs; any difference beyond numerical precision would refute the claimed invariance. Separately, construct or find a pair of models for which the distance-minimizing rotation gives a higher loss barrier than a deliberately worse rotation, which would break the presumed link between weight distance and fusion quality.","tokens_in":1752,"feed_emoji":"🔄","tokens_out":7006,"duration_ms":122103,"temperature":0.7,"pith_summary":"The paper introduces rotation symmetry for transformer self-attention layers: for each attention head, applying an orthogonal rotation to the query and key matrices, and the same kind of rotation to the value and output matrices, leaves the layer's computed function unchanged for every input. This makes the equivalence class of an attention layer continuous, beyond the discrete permutation symmetries used for MLPs. The paper then turns this symmetry into a model-fusion tool: a closed-form, Kabsch-style algorithm rotates one model's attention parameters to lie as close as possible to another model's parameters before the two are averaged or merged. Across language and vision transformers, this alignment consistently improves fusion accuracy, reduces the distance between merged models, and lowers the loss barrier along the interpolation path.","feed_headline":"Rotating attention weights makes transformers fuse better","feed_subtitle":"A closed-form rotation aligns functionally identical attention layers, improving merged models across NLP and vision tasks.","key_machinery":"The carrying object is the rotation-symmetry equivalence class of a self-attention layer, together with the reduction of parameter matching to an Orthogonal Procrustes problem: the optimal rotation satisfies $R = UV^⊤$, where $U Σ V^⊤$ is the singular value decomposition of a cross-covariance matrix built from the query, key, and bias matrices, and separately from the value, output, and bias matrices. Algorithm 1 applies this per head, treating one model as the anchor and rotating the other, and a rescaling step, a scalar $a$ multiplying the query/key blocks and dividing the value/output blocks, is layered on afterward.","core_discovery":"On its own terms, the paper's central claim is that a self-attention layer is invariant under per-head orthogonal transformations: with independent rotation matrices $R_{qk}^h$ and $R_{vo}^h$, the replacements $W_Q^h → (R_{qk}^h)^⊤ W_Q^h$, $b_Q^h → b_Q^h R_{qk}^h$, $W_K^h → (R_{qk}^h)^⊤ W_K^h$, $b_K^h → b_K^h R_{qk}^h$, $W_V^h → (R_{vo}^h)^⊤ W_V^h$, $b_V^h → b_V^h R_{vo}^h$, $W_O^h → W_O^h R_{vo}^h$ preserve the attention output for any input. The paper further claims that, given two attention layers, the rotation that minimizes the Frobenius (sum-of-squared-entries) distance between them is obtained in closed form from a singular value decomposition, and that matching one layer to the other by this rotation reduces the distance between end models, improves the fused model, and flattens the loss barrier between them.","pith_inferences":["An extension the paper does not test: the same rotation alignment should improve federated averaging or any weight-space interpolation method, because the mechanism claimed is the reduced weight distance rather than anything specific to the fusion baselines used.","Because heads rotate independently, the equivalence class grows with head count and hidden dimension; a testable consequence is that rotation alignment matters more for larger, wider transformers, which would explain the paper's observation that the larger model benefits more from matching.","The sequential rotation-then-rescaling is admitted to be a practical approximation; a joint optimization over $(R, a)$ is the natural next experiment and would isolate whether the remaining fusion gap comes from that approximation.","The early-layers-matter result is demonstrated on two GLUE tasks; whether it transfers to other benchmarks or model families is open."],"forward_implications":["Any transformer attention layer has infinitely many functionally identical parameter settings parameterized by continuous rotations, not just finitely many permutations.","The proposed matching raises the accuracy of simple, Fisher, RegMean, and optimal-transport fusion methods across RoBERTa, DeBERTa, and ViT models in the reported experiments.","Matched models are closer in parameter space and exhibit a lower loss barrier along linear interpolation than unmatched or permutation-matched models.","Matching only a few early attention layers already yields most of the fusion gain, so the method can be applied cheaply.","The matching overhead is small relative to fine-tuning or merging, about 1.6 to 3.5 seconds per model pair in the reported settings."],"supporting_citations":[{"why":"Defines the transformer self-attention layer whose query-key-value-output structure is the object of the rotation invariance derivation.","marker":"Vaswani et al., 2017"},{"why":"Establishes weight matching via permutation symmetry and provides the Git Re-Basin baseline that the paper's continuous rotation matching extends and compares against.","marker":"Ainsworth et al., 2023"},{"why":"Supplies the theoretical and empirical motivation that closer end models fuse better, which is the reason parameter matching improves fusion.","marker":"Wortsman et al., 2022"},{"why":"Provides the Orthogonal Procrustes solution used in the proof of the closed-form optimal rotation.","marker":"Schönemann, 1966"},{"why":"Names the rotation-matching procedure that Algorithm 1 adapts to attention parameters.","marker":"Kabsch, 1976"},{"why":"Provides the optimal-transport transformer fusion baselines and ViT experimental setup used for comparison.","marker":"Imfeld et al., 2024"},{"why":"Provides the RegMean merging baseline and the NLP datasets and codebase used for the in-domain and out-of-domain experiments.","marker":"Jin et al., 2023"},{"why":"Provides the Fisher-weighted averaging baseline whose fusion performance is compared with and without matching.","marker":"Matena & Raffel, 2022"},{"why":"Connects permutation invariance to linear mode connectivity, the loss-barrier motivation the paper builds on.","marker":"Entezari et al., 2022"},{"why":"Introduces optimal-transport neuron alignment for model fusion, an earlier matching approach generalized here.","marker":"Singh & Jaggi, 2020"}],"fun_headline_variants":["Rotation symmetry improves transformer fusion","SVD rotations align attention layers for fusion","Closed-form rotation enhances model merging","Continuous symmetry boosts transformer fusion"],"cache_read_input_tokens":26752,"weakest_assumption_plain":"The load-bearing premise is that minimizing the ordinary Euclidean distance between the matched attention weights is the right objective for fusion, so that a rotation making the weights numerically closer reliably makes the merged model better; the paper verifies this empirically but does not prove a monotone relationship.","fun_headline_variants_meta":{"raw":{"variants":["Rotation symmetry improves transformer fusion","SVD rotations align attention layers for fusion","Closed-form rotation enhances model merging","Continuous symmetry boosts transformer fusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000139,"raw_usage":{"total_tokens":1180,"prompt_tokens":993,"completion_tokens":187,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":140}},"tokens_in":609,"tokens_out":187,"duration_ms":2528,"temperature":1.0,"reasoning_tokens":140,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:35:40.219901+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed rotation on one of the two attention layers and compare its output to the original on a batch of inputs; any difference beyond numerical precision would refute the claimed invariance. Separately, construct or find a pair of models for which the distance-minimizing rotation gives a higher loss barrier than a deliberately worse rotation, which would break the presumed link between weight distance and fusion quality.","supporting_citations":[{"cited_title":"Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A","cited_arxiv_id":null,"evidence_quote":"Supplies the theoretical and empirical motivation that closer end models fuse better, which is the reason parameter matching improves fusion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the optimal-transport transformer fusion baselines and ViT experimental setup used for comparison."},{"cited_title":"Dataless knowledge fusion by merging weights of language models","cited_arxiv_id":null,"evidence_quote":"Provides the RegMean merging baseline and the NLP datasets and codebase used for the in-domain and out-of-domain experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Fisher-weighted averaging baseline whose fusion performance is compared with and without matching."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces optimal-transport neuron alignment for model fusion, an earlier matching approach generalized here."}],"review_version":1}