Selectively rescaling RoPE position indices for the top 48 key dimensions, each to a separately detected effective length, lets Llama-3-8B-8K reach 128K context and lifts Llama-3.1-70B's RULER score by about 20 points.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation
Selectively rescaling RoPE position indices for the top 48 key dimensions, each to a separately detected effective length, lets Llama-3-8B-8K reach 128K context and lifts Llama-3.1-70B's RULER score by about 20 points.