Pith. sign in

REVIEW 5 cited by

LoTR: Low Tensor Rank Weight Adaptation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.01376 v2 pith:OOAUB6RG submitted 2024-02-02 cs.CL cs.AIcs.LG

LoTR: Low Tensor Rank Weight Adaptation

Daniel Bershatsky , Daria Cherniuk , Talgat Daulbaev , Aleksandr Mikhalev , Ivan Oseledets This is my paper
classification cs.CL cs.AIcs.LG
keywords tensorfine-tuningllmslotrlow-rankadaptationallowsgradient
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fine-tuning LLMs are based on matrix factorization of gradient update. We introduce LoTR, a novel approach for parameter-efficient fine-tuning of LLMs which represents a gradient update to parameters in a form of tensor decomposition. Low-rank adapter for each layer is constructed as a product of three matrices, and tensor structure arises from sharing left and right multipliers of this product among layers. Simultaneous compression of a sequence of layers with low-rank tensor representation allows LoTR to archive even better parameter efficiency then LoRA especially for deep models. Moreover, the core tensor does not depend on original weight dimension and can be made arbitrary small, which allows for extremely cheap and fast downstream fine-tuning.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Chebyshev Manifold Adaptation

    cs.LG 2026-07 conditional novelty 6.0

    ChebyMA uses Chebyshev polynomial surfaces to parameterize weight updates, claiming a better parameter-accuracy trade-off than LoRA, TLoRA, and StelLA on CIFAR and text classification.

  2. KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices

    cs.CL 2026-01 conditional novelty 6.0

    KromHC uses Kronecker products of small doubly stochastic matrices to make Hyper-Connection residual matrices exactly balanced with O(n^2C) parameters, and matches or beats prior variants on small LLM pretraining runs.

  3. TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models

    cs.LG 2025-09 unverdicted novelty 6.0

    TeRA parametrizes high-rank LLM weight updates via a random Tucker-like tensor network with shared frozen factors and layer-specific scaling vectors, matching high-rank adapter performance at vector-level parameter counts.

  4. EinSort: Sorting is All We Need for Tensorizing LLM

    cs.LG 2026-06 unverdicted novelty 5.0

    Sorting tensor indices enables an adaptive tensorization method that discovers low-rank structure in LLM weights and KV caches, yielding better reconstruction quality than baselines.

  5. Low-Rank Adaptation Redux for Large Models

    cs.LG 2026-04 unverdicted novelty 3.0

    An overview revisits LoRA variants by categorizing advances in architectural design, efficient optimization, and applications while linking them to classical signal processing tools for principled fine-tuning.