Pith. sign in

REVIEW 10 cited by

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.10400 v3 pith:EFVKG5HS submitted 2023-09-19 cs.CL cs.LG

Dawei Zhu , Nan Yang , Liang Wang , Yifan Song , Wenhao Wu , Furu Wei , Sujian Li This is my paper · ORCID
classification cs.CLcs.LG
keywords contextlengthwindowposetrainingllmsefficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large Language Models (LLMs) are trained with a pre-defined context length, restricting their use in scenarios requiring long inputs. Previous efforts for adapting LLMs to a longer length usually requires fine-tuning with this target length (Full-length fine-tuning), suffering intensive training cost. To decouple train length from target length for efficient context window extension, we propose Positional Skip-wisE (PoSE) training that smartly simulates long inputs using a fixed context window. This is achieved by first dividing the original context window into several chunks, then designing distinct skipping bias terms to manipulate the position indices of each chunk. These bias terms and the lengths of each chunk are altered for every training example, allowing the model to adapt to all positions within target length. Experimental results show that PoSE greatly reduces memory and time overhead compared with Full-length fine-tuning, with minimal impact on performance. Leveraging this advantage, we have successfully extended the LLaMA model to 128k tokens using a 2k training context window. Furthermore, we empirically confirm that PoSE is compatible with all RoPE-based LLMs and position interpolation strategies. Notably, our method can potentially support infinite length, limited only by memory usage in inference. With ongoing progress for efficient inference, we believe PoSE can further scale the context window beyond 128k.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?

    cs.CL 2025-09 conditional novelty 6.0 of 10

    Mask-GCG uses learnable masks to prune a minority of low-impact tokens from GCG attack suffixes, slightly improving speed while showing most tokens are necessary.

  2. SeqPE: Transformer with Sequential Position Encoding

    cs.LG 2025-06 reject novelty 6.0 of 10

    SeqPE encodes each position as a symbolic digit sequence through a small Transformer, and with contrastive plus distillation losses it reports improved extrapolation in language, QA, and image classification.

  3. Native-Resolution Image Synthesis

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A single diffusion transformer trained on native-resolution ImageNet achieves state-of-the-art FID at 256 and 512, and extrapolates to 1024 and 1536 with moderate degradation.

  4. The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Fine-tuned LLMs separate roles via task-type and begin-of-text shortcuts; PFT, which inserts a gap into position IDs during fine-tuning, reduces those shortcuts and improves closed-domain attack robustness.

  5. A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)

    cs.CL 2025-02 conditional novelty 6.0 of 10

    GALI extends LLM context windows without training by reusing pretrained position intervals and interpolating attention logits with distance-scaled Gaussian noise.

  6. NExtLong: Toward Effective Long-Context Training without Long Documents

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Interleaving hard negative distractors between chunks of short documents improves long-context language model performance on HELMET and RULER.

  7. V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    V2PE assigns visual tokens smaller and variable positional increments than text tokens, which allows a 2B vision-language model to effectively process multimodal sequences up to 1M tokens.

  8. SELF: Self-Extend the Context Length With Logistic Growth Function

    cs.CL 2025-05 conditional novelty 5.0 of 10

    SELF replaces Self-Extend's fixed token grouping with a logistic-growth grouping schedule, giving mixed but sometimes large gains on long-context benchmarks.

  9. Generative Retrieval for Book search

    cs.IR 2025-01 conditional novelty 5.0 of 10

    GBS applies generative retrieval to book search by augmenting training data with hierarchical book identifiers and pseudo-queries, and encoding books with outline-based bi-level positions and retentive attention, repo...

  10. LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuning

    cs.CL 2025-01 conditional novelty 5.0 of 10

    LeMo reduces long-context fine-tuning memory by eliminating low-informativeness tokens, predicting sparsity patterns, and optimizing kernels, while keeping perplexity close to LoRA.

Pith tools