REVIEW 6 cited by
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models (LLMs) face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. Meanwhile, extending the context window in LLMs through post-pretraining is highly resource-intensive. To address this, we introduce LongRecipe, an efficient training strategy for extending the context window of LLMs, including impactful token analysis, position index transformation, and training optimization strategies. It simulates long-sequence inputs while maintaining training efficiency and significantly improves the model's understanding of long-range dependencies. Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size, and reduces computational training resource over 85% compared to full sequence training. Furthermore, LongRecipe also preserves the original LLM's capabilities in general tasks. Ultimately, we can extend the effective context window of open-source LLMs from 8k to 128k, achieving performance close to GPT-4 with just one day of dedicated training using a single GPU with 80G memory. Our code is released at https://github.com/zhiyuanhubj/LongRecipe.
Forward citations
Cited by 6 Pith papers
-
Training Long-Context LLMs Efficiently via Chunk-wise Optimization
Chunk-wise backpropagation with a sparse, rescaled variant fine-tunes an 8B model on 16K tokens using a single RTX 3090 GPU.
-
LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation
LongReD reduces short-text performance loss after long-context extension by training the extended model to match the original model's hidden states on short texts and using skipped position indices to bridge short and...
-
Classic4Children: Adapting Chinese Literary Classics for Children with Large Language Model
InstructChild fine-tunes Qwen2-7B with personality and narrative structure instructions plus a readability reward, and reports better child-friendly adaptations of the Four Great Classical Novels than GPT-4o and other...
-
V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding
V2PE assigns visual tokens smaller and variable positional increments than text tokens, which allows a 2B vision-language model to effectively process multimodal sequences up to 1M tokens.
-
UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models
UORA is a LoRA/VeRA-style PEFT method that selectively reinitializes low-magnitude rows and columns of frozen random matrices, reaching LoRA-comparable performance with far fewer trainable parameters.
-
ConTextual: Improving Clinical Text Summarization in LLMs with Context-preserving Token Filtering and Knowledge Graphs
ConTextual filters clinical notes to attention-important tokens, augments them with a patient-specific knowledge graph, and generates summaries that outperform several baselines on MIMIC-BHC and SOAP summarization benchmarks.
Discussion (0). Continue with ORCID to comment.