Pith. sign in

REVIEW 9 cited by

LoLCATs: On Low-Rank Linearizing of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.10254 v3 pith:6MJZWBG3 submitted 2024-10-14 cs.LG cs.AIcs.CLstat.ML

Michael Zhang , Simran Arora , Rahul Chalamala , Alan Wu , Benjamin Spector , Aaryan Singhal , Krithik Ramesh , Christopher Ré This is my paper · ORCID
classification cs.LGcs.AIcs.CLstat.ML
keywords llmslinearizinglolcatsqualityattentionslinearsignificantlytraining
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recent works show we can linearize large language models (LLMs) -- swapping the quadratic attentions of popular Transformer-based LLMs with subquadratic analogs, such as linear attention -- avoiding the expensive pretraining costs. However, linearizing LLMs often significantly degrades model quality, still requires training over billions of tokens, and remains limited to smaller 1.3B to 7B LLMs. We thus propose Low-rank Linear Conversion via Attention Transfer (LoLCATs), a simple two-step method that improves LLM linearizing quality with orders of magnitudes less memory and compute. We base these steps on two findings. First, we can replace an LLM's softmax attentions with closely-approximating linear attentions, simply by training the linear attentions to match their softmax counterparts with an output MSE loss ("attention transfer"). Then, this enables adjusting for approximation errors and recovering LLM quality simply with low-rank adaptation (LoRA). LoLCATs significantly improves linearizing quality, training efficiency, and scalability. We significantly reduce the linearizing quality gap and produce state-of-the-art subquadratic LLMs from Llama 3 8B and Mistral 7B v0.1, leading to 20+ points of improvement on 5-shot MMLU. Furthermore, LoLCATs does so with only 0.2% of past methods' model parameters and 0.4% of their training tokens. Finally, we apply LoLCATs to create the first linearized 70B and 405B LLMs (50x larger than prior work). When compared with prior approaches under the same compute budgets, LoLCATs significantly improves linearizing quality, closing the gap between linearized and original Llama 3.1 70B and 405B LLMs by 77.8% and 78.1% on 5-shot MMLU.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning Sparse Representations of Multimodal Content for Enhanced Cold Item Recommendation

    cs.IR 2026-07 conditional novelty 6.0 of 10

    Sparse content embeddings with a pre-sparsification alpha-entmax activation outperform dense embeddings for cold-start item recommendation at lower storage cost, especially for users with multiple interests.

  2. The Key to Going Linear: Analysis-Driven Transformer Linearization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Delta-rule linear attention faithfully approximates softmax attention through key-dependent rank-1 projections, enabling efficient post-hoc linearization of LLMs up to 32B parameters.

  3. ELiTeFormer: An Efficient Transformer for FPGAs

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Hybrid linear attention plus ternary projections, co-designed with a multiplier-free PE, deliver 10× weight and 12.8× KV-cache compression with competitive MMLU and FPGA speedups over LLaMA 3 on A100.

  4. Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models

    cs.CL 2026-02 conditional novelty 6.0 of 10

    NAtS-L learns per-chunk routing between Gated DeltaNet and softmax attention, improving long-context retrieval and length extrapolation over fixed hybrids.

  5. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  6. On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention

    cs.LG 2025-06 conditional novelty 5.0 of 10

    On-the-fly distillation of Transformer layers to dual-state linear attention produces about 2.3x faster simulated LLM serving than Llama2-7B with roughly comparable benchmark accuracy.

  7. GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

    cs.AI 2026-06 reject novelty 4.0 of 10

    GLIDE allocates softmax attention only to early/middle transformer layers and linearizes later layers, cutting KV-cache I/O by up to ~62× while retaining 92–96% of baseline accuracy.

  8. A Survey on Latent Reasoning

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.

  9. TPTT: Transforming Pretrained Transformers into Titans

    cs.CL 2025-06 reject novelty 4.0 of 10

    TPTT claims to convert pretrained transformers into memory-augmented linear-attention models with LoRA fine-tuning, but the reported MMLU results are marred by implausibly low baseline scores.

Pith tools