Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

HOFT: Householder Orthogonal Fine-tuning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that a single orthogonal matrix cannot express every adapted weight matrix with the same hyperspherical energy, and that two—one on each side—are required.

desk verdict Genuinely new two-sided orthogonal PEFT, but the paper's central theoretical claim about hyperspherical energy preservation is wrong as stated; the method itself is plausible and worth refereeing. read the letter →

arxiv 2505.16531 v2 pith:M33BKTXF submitted 2025-05-22 cs.LG

Alejandro Moreno Arcas , Albert Sanchis , Jorge Civera , Alfons Juan This is my paper
classification cs.LG
keywords orthogonalfine-tuningHouseholderreflectionCWYtransformparameter-efficienthypersphericalenergylow-rankadaptationLoRADo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that orthogonal fine-tuning—adapting a pretrained weight matrix by multiplying it with a learned orthogonal matrix—cannot reach every equally good adapted matrix with a single multiplication on one side. To cover all matrices that preserve the model's hyperspherical energy, the sum of inverse distances between column vectors, two orthogonal matrices are needed: one acting from the left and one from the right. The paper introduces HOFT, which builds both matrices cheaply from Householder reflections using the CWY transform and a fast inverse approximation, and SHOFT, which adds a learned scaling vector between the two orthogonal factors. Across commonsense reasoning, machine translation, subject-driven generation, and mathematical reasoning, HOFT and SHOFT match or beat LoRA, DoRA, OFT, and BOFT, often with fewer trainable parameters.

What carries the argument

The central object is the CWY transform, a fast way to multiply a chain of Householder reflections into a single orthogonal matrix: for a matrix U with linearly independent columns, the accumulated product equals I − U S⁻¹ Uᵀ, where S is the upper triangular part of UᵀU with its diagonal halved. HOFT uses this on both U and V to form Q_U and Q_V, and approximates S⁻¹ by the first two terms of its Neumann series, D⁻¹ − D⁻¹AD⁻¹, avoiding costly matrix inverses and yielding time complexity O(mn + (m+n)(r² + 2r)). The paired-vector initialization, (u₁,u₁,...,u_k,u_k), makes HOFT randomly initializable at the identity, a property the paper notes is absent in OFT and BOFT.

What would settle it

Take a random matrix M with distinct singular values and a nontrivial orthogonal factor Q_V that mixes columns; compute the column-wise hyperspherical energy of Eq. (7) before and after right multiplication. If the energies differ materially, the stated invariance fails, and the energy-preservation framing is not the mechanism that guarantees reachability; the reachability claim itself can be checked independently by SVD reconstruction with Q_U = ÛUᵀ and Q_V = V V̂ᵀ.

Watch

Extended reading notes

Core claim

The central claim, argued in Section 3.1, is that a single orthogonal matrix Q applied as cM = QM cannot express every adapted matrix cM whose singular values match M, because choosing Q = ÛUᵀ fixes the left singular vectors but leaves the right singular vectors V̂ unconstrained. To reach any such cM one must use two orthogonal matrices, cM = Q_U M Q_V, setting Q_U = ÛUᵀ and Q_V = V V̂ᵀ. HOFT realizes this two-sided construction by storing Householder vectors in the columns of U and V and accumulating them into orthogonal matrices via the CWY transform; the paper also gives a Procrustes-style bound showing one-sided adaptation can incur error up to 2√m‖M‖_F. SHOFT inserts a learned scaling vector between the two orthogonal factors, so it no longer preserves hyperspherical energy exactly and can adjust singular-value magnitudes.

Load-bearing premise

The load-bearing premise is that preserving hyperspherical energy—the sum of inverse distances between column vectors of the weight matrix—is the same as preserving singular values, and that multiplying by an orthogonal matrix on the right leaves that energy unchanged.

Editorial extensions

If this is right

  • One-sided orthogonal fine-tuning methods such as OFT, COFT, and BOFT cannot reach every adapted matrix with unchanged singular values, so their expressivity is strictly limited compared with HOFT.
  • HOFT and SHOFT can be randomly initialized at the identity by pairing identical Householder vectors, which OFT and BOFT cannot do.
  • The two-sided construction is cheap enough for practical use: HOFT is about 72.5% faster than DoRA on commonsense reasoning and about 72.1% faster than OFT on subject-driven generation, with peak memory between LoRA and DoRA.
  • SHOFT's additional scaling vector is the only extra component, and SHOFT reaches the best MATH accuracy among all compared methods, which the paper attributes to the scaling transformation adjusting singular-value magnitudes.
  • Quantized HOFT and SHOFT maintain the benefits of their non-quantized counterparts, outperforming QLoRA and QDoRA on most quantized mathematical reasoning benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reachability argument is sound for matrices with identical singular values, but the paper's further identification of 'same hyperspherical energy' with 'same singular values' is not implied by the SVD construction: right multiplication by Q_V mixes columns, so the column-wise energy of Eq. (7) is not generally invariant. A sympathetic reading is to treat the energy-preservation framing as moti
  • The one-sided Procrustes bound suggests a practical diagnostic: if validation error with one-sided orthogonal adaptation exceeds that of two-sided adaptation at matched parameter counts, the expressivity gap is likely the cause; the paper does not run this controlled comparison.
  • Weight decay independence of the CWY product, shown in Appendix B, implies that in SHOFT the scaling vector is the only place weight decay can act, which could allow separate control of geometry and magnitude during training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Householder Orthogonal Fine-tuning (HOFT), a parameter-efficient fine-tuning method that multiplies a frozen weight matrix M on the left and right by learned orthogonal matrices QU and QV obtained by accumulating Householder reflections via the CWY transform with an approximate inverse, and a scaled variant SHOFT that adds a per-row magnitude vector. The authors claim that two orthogonal matrices are necessary for full expressivity in orthogonal fine-tuning, by arguing that any matrix with the same 'hyperspherical energy' as M can be written as QU M QV. They evaluate HOFT and SHOFT on commonsense reasoning, machine translation, subject-driven generation, mathematical reasoning, and quantized settings, reporting competitive or better results than LoRA, DoRA, OFT, BOFT, and HRA.

Significance. If the theoretical framing were correct and the empirical comparisons sound, the contribution would be useful: it is an efficient two-sided orthogonal PEFT method with O(r(m+n)) parameters, parallelizable Householder accumulation, and a scaled variant, evaluated across a broad set of tasks and model families. The SVD reachability statement underlying the two-sided construction is mathematically true and, when properly formulated, gives a clear motivation for using both left and right orthogonal factors. The breadth of the experiments (four task areas, several model families, and quantized models) is a strength, as is the explicit analysis of the inverse approximation error. However, the paper does not ship code, reports no error bars, and its central 'hyperspherical energy preservation' claim is contradicted by the paper's own definition and measurements, so the current version requires major revision.

major comments (3)
  1. [Section 3.1, Section 3.3, Eq. (7)] The central theoretical claim that the two-sided product QU M QV preserves hyperspherical energy is incorrect under the paper's own definition. Eq. (7) defines HE(W) as a sum over inverse pairwise distances between columns of W. Left multiplication by an orthogonal matrix QU is an isometry of the column space and preserves these distances, but right multiplication by QV mixes columns, so ||(M QV)_i - (M QV)_j|| is not generally equal to ||M_i - M_j||. The SVD argument in Section 3.1 establishes a different, true statement: any matrix with the same singular values as M can be written as QU M QV. Equal singular values do not imply equal HE under Eq. (7). Indeed, Figure 5 in Appendix B.1 shows |HE(M) - HE(QU M QV)| growing rapidly with rank, which is direct evidence against the preservation claim. Please reformulate the theory in terms of singular-value preservation (or define a different, right-multiplication-invariant notion of energy), and revise the abstract, Introduction, Section 3.3, and the proof of Eq. (1) in Appendix C accordingly.
  2. [Table 3 and Table 8] The machine translation comparison for LLaMA3.1-8B French is confounded by a learning-rate mismatch: OFT and BOFT use LR = 3e-3 (Table 8) while HOFT uses 3e-4 and LoRA/DoRA use 1e-4. The collapsed BLEU scores for OFT (21.5) and BOFT (22.1) versus LoRA (46.8) are therefore likely an artifact of an ill-tuned learning rate rather than evidence about the methods. Please retune OFT and BOFT (or use a common, properly tuned schedule) and re-run the comparison, or restrict the claim that HOFT/SHOFT 'produce superior results' over orthogonal baselines.
  3. [Section 4, Tables 2, 5, 6] All numbers are reported from a single run with no error bars, confidence intervals, or significance tests, and many HOFT/SHOFT advantages are under one accuracy point (e.g., Table 2 LLaMA3.1-8B average: HOFT 87.4 vs LoRA 87.0; Table 5 GSM8K: HOFT 56.6 vs HRA 56.3). The claim that HOFT and SHOFT 'consistently match or outperform' the baselines is not strongly supported without an estimate of run-to-run variability. Please report multiple seeds or significance tests, at least for the main tables, and moderate the wording of the conclusions.
minor comments (4)
  1. [Appendix C, Eq. (1)] The proof computes ||bQV - I||_F^2 = 2m - 2Tr(bQV) and obtains 4m, but bQV is an n×n orthogonal matrix, so the correct expression is 2n - 2Tr(bQV) and the final bound should be 2√n ||M||_F, not 2√m ||M||_F. This is a typo only if m = n, but it should be fixed for the general rectangular case.
  2. [Theorem 1] The statement of Theorem 1 says the result holds for an arbitrary vector τ with τ_i ≠ 0, but the two-step computation of S (upper triangular part of U^T U, diagonal halved) implicitly assumes τ_i = u_i^T u_i / 2. For other τ, the expression is false; please state the required relation between τ and U.
  3. [Appendix B.1, Figure 1] In the sentence defining the error in Figure 1, the normalization is written as √n but the matrices are m×m; it should be √m (or the notation should be clarified).
  4. [Appendix E.2] The sentence 'We will include the complete experimental protocol, hyperparameter settings, and detailed results in the revised manuscript' is out of place in a submitted manuscript and should be removed or replaced by the actual details.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HOFT's central two-sided reachability claim is self-contained SVD linear algebra, and its empirical evaluations use held-out benchmarks with external baselines.

full rationale

The paper's central theoretical claim, in Sections 3.1 and 3.3, is that pre- and post-multiplying a matrix M by orthogonal matrices QU and QV can express any adapted matrix cM with the same singular values as M. This is established by a direct SVD construction: if M = U Sigma V^T and cM = bU Sigma bV^T, then setting QU = bU U^T and QV = V bV^T gives QU M QV = cM. This is a self-contained linear-algebra existence result, not a quantity fitted to data and then relabeled as a prediction. No fitted parameters are disguised as predictions, and no load-bearing self-citation appears: the cited orthogonal fine-tuning works (OFT, BOFT, HRA) are external prior work, not the present authors' own results. The paper does make an incorrect identification when it equates 'preserving hyperspherical energy' with bSigma = Sigma in Section 3.1: under the paper's own definition in Eq. 7, hyperspherical energy is a function of column distances, and right multiplication by QV need not preserve those distances. However, this is a mathematical correctness issue about the energy-preservation framing, not a circularity in the derivation chain; the reachability statement about singular-value-preserving matrices remains an independent, verifiable theorem. Empirical claims are tested on standard held-out benchmarks (common sense reasoning, machine translation, subject-driven generation, mathematical reasoning) against external baselines, so the results are not forced by construction. The paper's derivation is therefore self-contained with respect to its inputs, and the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. Its load-bearing assumptions are the inherited orthogonality-generalization hypothesis, the questionable identification of singular-value preservation with hyperspherical energy preservation, and the accuracy of the truncated Neumann inverse. The rank r is the main user-chosen quantity affecting performance.

free parameters (1)
  • Householder rank r = 16, 8, 2, and 4 across tasks
    Rank is chosen per experiment in Appendix A; Appendix E.1 shows accuracy varies substantially with r, so empirical claims depend on tuned ranks.
assumptions (4)
  • standard math Every real matrix M has an SVD M = U Σ V^T
    Used in Section 3.1 to define the orthogonal fine-tuning paradigm and the two-matrix construction.
  • domain assumption Orthogonal adaptation preserves pre-trained knowledge and improves generalization via hyperspherical energy
    Inherited from OFT and BOFT literature [36, 29, 26, 28]; motivates the method but is not independently established here.
  • ad hoc to paper Same singular values imply same hyperspherical energy, so energy preservation reduces to keeping Σ unchanged
    Section 3.1 equates hyperspherical energy conservation with \(\Sigma\) being unchanged; under the paper's own Eq. 7, column-wise hyperspherical energy also depends on singular vectors, so the identification is unjustified.
  • domain assumption The two-term Neumann approximation produces matrices close enough to orthogonal for training
    Section 3.2 and Figure 1 show the approximation error grows with r, and no training-time guarantee is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HOFT: Householder Orthogonal Fine-tuning." pith.science (2026). https://pith.science/paper/M33BKTXF

@misc{pith2026250516531,
  author       = {Pith},
  title        = {Pith review of: HOFT: Householder Orthogonal Fine-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M33BKTXF}},
  note         = {Machine review of arXiv:2505.16531}
}
read the original abstract

Adaptation of foundation models using low-rank methods is a widespread approach. Another way to adapt these models is to employ orthogonal fine-tuning methods, which are less time and memory efficient despite their good generalization properties. In this work, we propose Householder Orthogonal Fine-tuning (HOFT), a novel orthogonal fine-tuning method that aims to alleviate time and space complexity. Moreover, some theoretical properties of the orthogonal fine-tuning paradigm are explored. From this exploration, Scaled Householder Orthogonal Fine-tuning (SHOFT) is proposed. Both HOFT and SHOFT are evaluated in downstream tasks, namely commonsense reasoning, machine translation, subject-driven generation and mathematical reasoning. Compared with state-of-the-art adaptation methods, HOFT and SHOFT show comparable or better results.

Figures

Figures reproduced from arXiv: 2505.16531 by the authors.

Figure 1
Figure 1. Inverse approximation error 3 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Diagram of our proposed HOFT method Similar to HRA’s rank r [53], HOFT also employs r householder vectors. For both inverse ap￾proximations, the computational complexity is O [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Examples of training images of 3D icons and lego sets [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative results on lego sets and 3d icons datasets [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Hyperspherical energy difference [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 7
Figure 7. Figure 7: Rank exploration in Qwen2.5-7B The average accuracies of the PEFT methods across different ranks are shown in Figures 6 and 7. In [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Comparison of different prompts in 3D icons dataset [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Comparison of different prompts in lego sets dataset [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Deep Delta Learning

    cs.LG 2026-01 unverdicted novelty 7.0 of 10

    Replacing additive residual connections with a gated rank-1 delta update that interpolates identity, projection, and reflection slightly improves language modeling and downstream averages in reported 124M/353M runs.

Reference graph

Works this paper leans on

57 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    arXiv 2024

  2. [2]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning

    Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Associ- ation for Computational Linguistics and the 11th International Joint Conference on Nat- ural Langua...

  3. [3]

    Rotrnn: Modelling long sequences with rotations.arXiv preprint arXiv:2407.07239, 2024

    Kai Biegun, Rares Dolga, Jake Cunningham, and David Barber. Rotrnn: Modelling long sequences with rotations.arXiv preprint arXiv:2407.07239, 2024

    arXiv 2024

  4. [4]

    Piqa: Reasoning about phys- ical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  5. [5]

    An adaptive orthogonal convolution scheme for efficient and flexible cnn architectures.arXiv preprint arXiv:2501.07930, 2025

    Thibaut Boissin, Franck Mamalet, Thomas Fel, Agustin Martin Picard, Thomas Massena, and Mathieu Serrurier. An adaptive orthogonal convolution scheme for efficient and flexible cnn architectures.arXiv preprint arXiv:2501.07930, 2025

    arXiv 2025

  6. [6]

    Oran Brigham.The Fast Fourier Transform and Its Applications

    E. Oran Brigham.The Fast Fourier Transform and Its Applications. Prentice Hall, Englewood Cliffs, NJ, 2 edition, 1974. ISBN 978-0133075052

    work page 1974

  7. [7]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021

    2021

  8. [8]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT, pages 2924–2936, 2019

    work page 2019

Show all 57 references
  1. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  2. [10]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  3. [11]

    No language left behind: Scaling human-centered machine translation.arXiv preprint arXiv:2207.04672, 2022

    Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation.arXiv preprint arXiv:2207.04672, 2022. 10

    2022 arXiv

  4. [12]

    Qlora: Efficient finetuning of quantized llms.NeurIPS, 2024

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms.NeurIPS, 2024

    2024

  5. [13]

    O-vit: Orthogonal vision transformer

    Yanhong Fei, Yingjie Liu, Xian Wei, and Mingsong Chen. O-vit: Orthogonal vision transformer. arXiv preprint arXiv:2201.12133, 2022

    2022 arXiv

  6. [14]

    Golub and Charles F

    Gene H. Golub and Charles F. Van Loan.Matrix Computations. Johns Hopkins University Press, Baltimore, MD, 4th edition, 2013. ISBN 978-1421407944

    2013

  7. [15]

    Gower and Garmt B

    John C. Gower and Garmt B. Dijksterhuis.Procrustes Problems, volume 30 ofOxford Statistical Science Series. Oxford University Press, Oxford, UK, 2004. ISBN 9780198510581. doi: 10.1093/acprof:oso/9780198510581.001.0001

    2004

  8. [16]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  9. [17]

    To- wards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366, 2021

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. To- wards a unified view of parameter-efficient transfer learning.arXiv preprint arXiv:2110.04366, 2021

    2021 arXiv

  10. [18]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InInternational conference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  11. [19]

    Lora: Low-rank adaptation of large language models.ICLR, 1 (2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1 (2):3, 2022

    2022

  12. [20]

    Fedpara: Low-rank hadamard product for communication-efficient federated learning

    Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficient federated learning. InInternational Conference on Learning Representations, 2022

    2022

  13. [21]

    Quintana-Ortí, Robert van de Geijn, and Field G

    Thierry Joffrain, Tze Meng Low, Enrique S. Quintana-Ortí, Robert van de Geijn, and Field G. Van Zee. Accumulating householder transformations, revisited.ACM Trans. Math. Softw., 32(2):169–179, June 2006. ISSN 0098-3500. doi: 10.1145/1141885.1141886. URL https: //doi.org/10.114...

    2006

  14. [22]

    Kshitij Khare and Christopher V . Stewart. Random Orthogonal Matrices and the Cayley Transform.Bernoulli, 26(2):807–825, 2019

    2019

  15. [23]

    Kopiczko, Tijmen Blankevoort, and Yuki M

    Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adaptation.ICLR, 2024

    2024

  16. [24]

    Vb-lora: Extreme parameter efficient fine-tuning with vector banks.NeurIPS, 2024

    Yang Li, Shaobo Han, and Shihao Ji. Vb-lora: Extreme parameter efficient fine-tuning with vector banks.NeurIPS, 2024

    2024

  17. [25]

    Cwy parametrization: a solution for parallelized optimization of orthogonal and stiefel matrices

    Valerii Likhosherstov, Jared Davis, Krzysztof Choromanski, and Adrian Weller. Cwy parametrization: a solution for parallelized optimization of orthogonal and stiefel matrices. In International Conference on Artificial Intelligence and Statistics, pages 55–63. PMLR, 2021

    2021

  18. [26]

    Regularizing neural networks via minimizing hyperspherical energy.Conference on Computer Vision and Pattern Recognition, 2020

    Rongmei Lin, Weiyang Liu, Zhen Liu, Chen Feng, Zhiding Yu, James M Rehg, Li Xiong, and Le Song. Regularizing neural networks via minimizing hyperspherical energy.Conference on Computer Vision and Pattern Recognition, 2020

    2020

  19. [27]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024

    2024

  20. [28]

    Learning towards minimum hyperspherical energy.NeurIPS, 2018

    Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy.NeurIPS, 2018

    2018

  21. [29]

    Parameter-efficient orthogonal finetuning via butterfly factorization.International Conference on Learning Representations, 2024

    Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization.International Conference on Learning Representations, 2024. 11

    2024

  22. [30]

    Pissa: Principal singular values and singular vectors adaptation of large language models.Advances in Neural Information Processing Systems, 37:121038–121072, 2024

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models.Advances in Neural Information Processing Systems, 37:121038–121072, 2024

    2024

  23. [31]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, 2018

    2018

  24. [32]

    Orca-math: Unlocking the potential of slms in grade school math.arXiv preprint arXiv:2402.14830, 2024

    Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math.arXiv preprint arXiv:2402.14830, 2024

    2024 arXiv

  25. [33]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA, 2002. Association for Computational L...

    2002

  26. [34]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  27. [35]

    A call for clarity in reporting BLEU scores

    Matt Post. A call for clarity in reporting BLEU scores. InProceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-6319. URL https:// aclantholog...

    2018 doi

  28. [36]

    Controlling text-to-image diffusion by orthogonal finetuning

    Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320–79362, 2023

    2023

  29. [37]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  30. [38]

    COMET: A neural framework for MT evaluation

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online, November 2020. Association for Computational Lingu...

    2020 doi

  31. [39]

    Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Philipp Koehn, Loïc Barrault, Ond ˇrej Bojar, Fethi Bouga...

    2022

  32. [40]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  33. [41]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500...

    2023

  34. [42]

    Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    2021

  35. [43]

    Social iqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural L...

    2019

  36. [44]

    Elrt: Efficient low-rank training for compact convolutional neural networks.arXiv preprint arXiv:2401.10341, 2024

    Yang Sui, Miao Yin, Yu Gong, Jinqi Xiao, Huy Phan, and Bo Yuan. Elrt: Efficient low-rank training for compact convolutional neural networks.arXiv preprint arXiv:2401.10341, 2024

    2024 arXiv

  37. [45]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  38. [46]

    Dylora: Parameter- efficient tuning of pre-trained models using dynamic search-free low-rank adaptation

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter- efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computa- tional L...

    2023

  39. [47]

    Covost 2 and massively multilingual speech-to-text translation.arXiv preprint arXiv:2007.10310, 2020

    Changhan Wang, Anne Wu, and Juan Pino. Covost 2 and massively multilingual speech-to-text translation.arXiv preprint arXiv:2007.10310, 2020

    2007 arXiv

  40. [48]

    Towards better orthogonality regularization with disentangled norm in training deep cnns.arXiv preprint arXiv:2306.09939, 2023

    Changhao Wu, Shenan Zhang, Fangsong Long, Ziliang Yin, and Tuo Leng. Towards better orthogonality regularization with disentangled norm in training deep cnns.arXiv preprint arXiv:2306.09939, 2023

    2023 arXiv

  41. [49]

    Qa-lora: Quantization-aware low-rank adaptation of large language models

    Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. InICLR, 2024

    2024

  42. [50]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  43. [51]

    Navigating text-to-image customization: From lycoris fine-tuning to model evaluation

    Shih-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  44. [52]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  45. [53]

    Bridging the gap between low-rank and orthogonal adaptation via householder reflection adaptation.NeurIPS, 2024

    Shen Yuan, Haotian Liu, and Hongteng Xu. Bridging the gap between low-rank and orthogonal adaptation via householder reflection adaptation.NeurIPS, 2024

    2024

  46. [54]

    Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    2019

  47. [55]

    Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In11th Interna- tional Conference on Learning Representations, ICLR 2023, 2023

    2023

  48. [56]

    Efros, Eli Shechtman, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. InProceedings of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018), pages 586–595, Salt La...

    2018

  49. [2021]

    doi: 10.18653/v1/2021.acl-long.568

    Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.568. URL https://aclanthology.org/2021.acl-long.568/

    2021 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.