Pith. sign in

REVIEW 41 cited by

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.12307 v3 pith:VDB27S7M submitted 2023-09-21 cs.CL cs.AIcs.LG

Yukang Chen , Shengju Qian , Haotian Tang , Xin Lai , Zhijian Liu , Song Han , Jiaya Jia This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords contextlonglorafine-tuningattentionmodelstrainingextendsextension
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present LongLoRA, an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs), with limited computation cost. Typically, training LLMs with long context sizes is computationally expensive, requiring extensive training hours and GPU resources. For example, training on the context length of 8192 needs 16x computational costs in self-attention layers as that of 2048. In this paper, we speed up the context extension of LLMs in two aspects. On the one hand, although dense global attention is needed during inference, fine-tuning the model can be effectively and efficiently done by sparse local attention. The proposed shifted sparse attention effectively enables context extension, leading to non-trivial computation saving with similar performance to fine-tuning with vanilla attention. Particularly, it can be implemented with only two lines of code in training, while being optional in inference. On the other hand, we revisit the parameter-efficient fine-tuning regime for context expansion. Notably, we find that LoRA for context extension works well under the premise of trainable embedding and normalization. LongLoRA combines this improved LoRA with S^2-Attn. LongLoRA demonstrates strong empirical results on various tasks on Llama2 models from 7B/13B to 70B. LongLoRA extends Llama2 7B from 4k context to 100k, or Llama2 70B to 32k on a single 8x A100 machine. LongLoRA extends models' context while retaining their original architectures, and is compatible with most existing techniques, like Flash-Attention2. In addition, we further conduct supervised fine-tuning with LongLoRA and our long instruction-following LongAlpaca dataset.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 41 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LiveMem: Maintaining Memory State Continuity in Long-Running LLM Inference

    cs.CL 2026-08 reject novelty 6.0 of 10

    LiveMem augments a pretrained LLM with a GDN2 recurrent state that is trained under context turnover, and claims the state remains useful after the supporting tokens leave attention.

  2. RH-RAG: Trustworthy Long-Form Generation for Privacy-Constrained Settings

    cs.CL 2026-08 reject novelty 6.0 of 10

    A multi-agent RAG framework that adds planning, bounded memory, and NLI-based revision to local 7-8B models, reported to improve faithfulness and coherence in long-form generation.

  3. Long-Context Fine-Tuning with Limited VRAM

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Hierarchical token routing with truncated backprop and RAM/NVMe KV storage extends QLoRA fine-tuning context from 2K to 16K on a 16 GB GPU, with matched quality at the 2K boundary.

  4. DOSA: A Tree-Guided, Self-Regressive Framework for Long Document Structure Analysis

    cs.AI 2026-07 conditional novelty 6.0 of 10

    DOSA's tree-guided self-regressive chunking improves multi-page document hierarchy parsing by up to 4 F1 and 19 TEDS over prior methods on DocHieNet.

  5. BiSCo-LLM: Lookup-Free Binary Spherical Coding for Extreme Low-Bit Large Language Model Compression

    cs.LG 2026-07 conditional novelty 6.0 of 10

    BiSCo-LLM achieves near-FP16 accuracy on Qwen3-8B at ~2 bits/weight using codebook-free binary spherical codes with residual coding and category-wise recovery distillation.

  6. Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?

    cs.CL 2025-09 conditional novelty 6.0 of 10

    Mask-GCG uses learnable masks to prune a minority of low-impact tokens from GCG attack suffixes, slightly improving speed while showing most tokens are necessary.

  7. Structured Attention Matters to Multimodal LLMs in Document Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Structured LaTeX encoding of OCR text, combined with document images, improves DocQA accuracy across four MLLMs and four benchmarks without any training.

  8. SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 7B writing model trained on plan-write-refine thinking data with multi-stage preference optimization matches or beats several larger models on long-form generation benchmarks.

  9. Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Chipmunk speeds up diffusion transformer inference by recomputing, at each step, only the top percent of attention and MLP activation columns that change most between steps, caching the rest in column-sparse GPU kernels.

  10. STORM: Benchmarking Visual Rating of MLLMs with a Comprehensive Ordinal Regression Dataset

    cs.CV 2025-06 conditional novelty 6.0 of 10

    STORM is a new multi-domain ordinal-regression benchmark with coarse-to-fine Chain-of-Thought prompts that improves MLLM zero-shot visual rating, though the 'universal' claim is bounded by its five curated domains.

  11. Training Long-Context LLMs Efficiently via Chunk-wise Optimization

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Chunk-wise backpropagation with a sparse, rescaled variant fine-tunes an 8B model on 16K tokens using a single RTX 3090 GPU.

  12. PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference

    cs.AR 2025-02 conditional novelty 6.0 of 10

    CENT, a simulated GPU-free CXL-connected processing-in-memory system, reports 2.3x higher LLM inference throughput and 5.2x better tokens-per-dollar than a 4x A100 GPU server.

  13. SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Sharing one MLP layer's weights across several layers plus low-rank adapters recovers most of a pretrained LLM's quality with a fraction of the storage and faster phone inference.

  14. A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)

    cs.CL 2025-02 conditional novelty 6.0 of 10

    GALI extends LLM context windows without training by reusing pretrained position intervals and interpolating attention logits with distance-scaled Gaussian noise.

  15. NExtLong: Toward Effective Long-Context Training without Long Documents

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Interleaving hard negative distractors between chunks of short documents improves long-context language model performance on HELMET and RULER.

  16. Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding

    cs.CL 2025-01 conditional novelty 6.0 of 10

    TAPE makes positional embeddings content-aware and equivariant, improving Transformer performance on arithmetic and long-context tasks and extending representational power to NC1-complete algorithms.

  17. V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    V2PE assigns visual tokens smaller and variable positional increments than text tokens, which allows a 2B vision-language model to effectively process multimodal sequences up to 1M tokens.

  18. Quantized Delta Weight Is Safety Keeper

    cs.CR 2024-11 conditional novelty 6.0 of 10

    One-bit quantization of fine-tuning delta weights suppresses alignment-breaking and backdoor attacks on LLMs, at the price of measurable utility loss.

  19. When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

    cs.CL 2024-11 conditional novelty 6.0 of 10

    BFloat16 rounding breaks RoPE's positional shift invariance, and AnchorAttention, which fixes the first token as a shared anchor, improves long-context training accuracy and speed.

  20. LLaSA: Large Language and Structured Data Assistant

    cs.CL 2024-11 conditional novelty 6.0 of 10

    A single hypergraph encoder and G-Former, pretrained on 25 million tables, can be appended to different LLMs to improve their performance on table, knowledge graph, and database structured knowledge grounding tasks.

  21. Docopilot: Improving Multimodal Models for Document-Level Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.

  22. EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices

    cs.DC 2025-07 conditional novelty 5.0 of 10

    EdgeLoRA combines automatic adapter routing, LRU caching with a memory pool, and grouped LoRA batching to serve thousands of LoRA adapters on edge devices with up to 4x higher throughput than llama.cpp.

  23. FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FLoE uses Fisher information to pick the transformer layers that matter and a Bayesian optimizer to set LoRA rank, cutting trainable parameters while keeping or improving accuracy.

  24. UniTEX: Universal High Fidelity Generative Texturing for 3D Shapes

    cs.CV 2025-05 conditional novelty 5.0 of 10

    UniTEX generates textures for 3D shapes by predicting continuous volumetric texture functions, bypassing UV maps.

  25. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.

  26. SELF: Self-Extend the Context Length With Logistic Growth Function

    cs.CL 2025-05 conditional novelty 5.0 of 10

    SELF replaces Self-Extend's fixed token grouping with a logistic-growth grouping schedule, giving mixed but sometimes large gains on long-context benchmarks.

  27. LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions

    cs.CL 2025-05 conditional novelty 5.0 of 10

    By prompting an aligned LLM with a document and the special token that precedes a user query, LongMagpie synthesizes long-context instruction data that outperforms prior datasets when used to fine-tune Llama-3-8B.

  28. RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?

    cs.LG 2025-01 reject novelty 5.0 of 10

    Long-CoT fine-tuning with QwQ-generated traces improves math and geometry benchmarks, and RedStar-DPO surpasses QwQ on AIME24, but several reported numbers conflict with the paper's own tables.

  29. LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuning

    cs.CL 2025-01 conditional novelty 5.0 of 10

    LeMo reduces long-context fine-tuning memory by eliminating low-informativeness tokens, predicting sparsity patterns, and optimizing kernels, while keeping perplexity close to LoRA.

  30. TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

    cs.CL 2025-01 conditional novelty 5.0 of 10

    TriAdaptLoRA, a triangular low-rank adaptation with importance-driven rank growth, achieves small average gains over LoRA, AdaLoRA, and IncreLoRA on GLUE and SQuAD 2.0, with mixed per-task results.

  31. Adjoint sharding for very long context training of state space models

    cs.LG 2025-01 reject novelty 5.0 of 10

    The paper derives an adjoint-based gradient sharding algorithm for SSMs and claims up to 3X memory reduction, but provides no experimental evidence for the central empirical claims.

  32. Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA

    cs.CL 2024-12 conditional novelty 5.0 of 10

    In multi-hop QA, language models answer worse when the required evidence documents are separated by distractor documents, not just when the evidence sits in the middle of the context.

  33. Compute-Accuracy Pareto Frontiers for Open-Source Reasoning Large Language Models

    cs.CL 2025-12 conditional novelty 4.0 of 10

    Across five reasoning benchmarks, sparse MoE models dominate the accuracy-vs-FLOPs Pareto frontier, inference-compute gains saturate, and wrong answers systematically consume more compute than correct ones.

  34. Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Dynamic chunking plus question-aware chunk selection improves long-context QA, but the headline numbers are partly inflated by choosing hyperparameters on the test benchmarks.

  35. NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

    cs.LG 2025-05 conditional novelty 4.0 of 10

    NQKV is a block-wise 4-bit KV cache quantizer that uses a normal-distribution lookup table to let OPT models run with larger batches or longer contexts at negligible measured accuracy cost.

  36. Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices

    cs.DC 2024-12 conditional novelty 4.0 of 10

    Assigning federated fine-tuning devices different numbers of LoRA layers near the output, with ranks increasing toward the output, reaches target accuracy 1.5-2.8x faster and with up to 42.3% less communication than e...

  37. Boosting Long-Context Management via Query-Guided Activation Refilling

    cs.CL 2024-12 conditional novelty 4.0 of 10

    ACRE uses a bi-layer KV cache with query-guided refilling to answer long-context questions beyond a model's native window, reporting gains over RAG and compression baselines.

  38. Next-Generation Phishing: How LLM Agents Empower Cyber Attackers

    cs.CR 2024-11 conditional novelty 4.0 of 10

    LLM-rephrased phishing emails evade current email detectors more often than original ones, and training on LLM-generated variants partly restores detection.

  39. Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models

    cs.CL 2025-09 reject novelty 2.0 of 10

    A gated-adapter method with a sparsity regularizer is reported to match or slightly beat full fine-tuning on MNLI and BoolQ using 1.4% of parameters, but lacks the experimental detail needed to verify the claim.

  40. LLMs are Also Effective Embedding Models: An In-depth Overview

    cs.CL 2024-12 conditional novelty 2.0 of 10

    A structured survey of using decoder-only LLMs as text embedding models, covering prompting, fine-tuning, data construction, benchmarks, and open problems.

  41. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

    cs.LG 2024-12 conditional

    A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.

Pith tools