{"id":"562c044f-97e6-4a7f-8016-182ff34109c8","arxiv_id":"2504.18857","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Selectively rescaling RoPE position indices for the top 48 key dimensions, each to a separately detected effective length, lets Llama-3-8B-8K reach 128K context and lifts Llama-3.1-70B's RULER score by about 20 points.","lead":"A new training-free method, DPE, lets 8K-trained language models like Llama-3-8B handle 128K-token inputs by rescaling positional information only for the model's most important frequency dimensions. If it holds up, long-context capability could be added to existing models without expensive retraining, with gains even inside the original context window.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-model transfer of per-group effective lengths is unverified: E was detected only on Llama-3-8B, yet RoPE bases differ across models, so identical dimension indices map to different frequencies and headline gains may not transfer.","rationale":"The reader's weakest_assumption correctly identifies the transferability of effective lengths. I agree with the conditional verdict, and I have sharpened the concern: because RoPE bases are not identical across the evaluated models, the dimension-index groups in DPE do not represent the same physical rotation frequencies. If E was measured only on Llama-3-8B and applied elsewhere, the Qwen2.5 and Llama3.1-70B improvements are not yet accounted for. This is an empirical reproducibility risk, not an internal contradiction; the RULER gains on Llama-3-8B and Mistral already provide partial support for the method, and the released code makes the check feasible. The condition should require per-model effective-length detection (or a demonstration that normalizing by θ makes E transferable) and a recomputation of the headline tables. No change to the reader's conditional assessment is needed.","tokens_in":19461,"tokens_out":9592,"duration_ms":86044,"concrete_test":"Using the released code, run the Appendix C effective-length detection protocol (8 groups, w=1k, detecting lengths 1k-128k, 100-sample NIAH) separately for Mistral-7B-Instruct-v0.2, Qwen2.5-7B-Instruct, Llama3.1-8B-Instruct, and Llama3.1-70B-Instruct. Obtain per-model E vectors, then recompute the RULER 128k rows with per-model E and top-48 key dimensions. If any row moves by more than ~5 points compared with Table 2, or if the per-model E vectors differ systematically with rope_theta, the cross-model transfer assumption fails. As a supplementary check, group dimensions by log θ rather than by index and test whether the transferred E becomes stable across models.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 fixes C=8, w=1k, and top-48 key dimensions 'for all the models', and Appendix D.1 reports effective lengths E only for Llama-3-8B-Instruct, selected by NIAH accuracy ranking on that model. The load-bearing premise is that per-dimension-group effective relative distances transfer across models. This is insecure because RoPE's physical frequencies are θ_j = base^{-2j/d}; Llama-3 uses rope_theta=500000, Mistral-v0.2/v0.3 use 10000, Qwen2.5 uses ~1e6, and Llama3.1 differs in scale. The same dimension index j therefore corresponds to a different rotation frequency in each model, so the same group index need not have the same effective length. DPE's headline cross-model numbers (Qwen2.5 RULER 29.10 -> 70.78; Llama3.1-70B 66.41 -> 86.39) reuse the Llama-3-8B E if no per-model detection was performed, and the paper neither reports per-model E nor shows that it is unnecessary. In addition, the Llama-3-8B NIAH score in Table 1 is measured on the same task used to select E, so that row is partly a selection result rather than a prediction; the independent evidence is the RULER and InfiniteBench results on Llama-3-8B and Mistral. A held-out re-detection per model is therefore required before accepting the cross-model generality claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dimension-Wise Positional Embeddings Manipulation (DPE), a training-free method for extrapolating the context length of RoPE-based LLMs. DPE divides head dimensions into groups, detects a maximum 'effective relative distance' per group via Needle-in-a-Haystack (NIAH) accuracy on Llama-3-8B-Instruct, identifies per-head 'key dimensions' using a 2-norm attention-contribution metric, and rescales only those dimensions' relative position matrices during inference. Experiments on NIAH, RULER, and InfiniteBench report consistent gains over existing training-free baselines, including large improvements on Llama-3-8B (8K→128K), Mistral-7B, Qwen-2.5-7B, and Llama-3.1-70B, with the latter surpassing GPT-4-1106-preview on RULER.","tokens_in":19818,"tokens_out":7827,"duration_ms":69999,"significance":"If the cross-model generality is confirmed, DPE is a notable contribution: it is simple, requires no training, adds negligible overhead with FlashAttention-2, and is backed by ablations showing that selective dimension-wise manipulation outperforms uniform position scaling. The paper includes a useful analysis of differing effective lengths across RoPE dimensions and demonstrates that per-head 2-norm attention contribution can identify dimensions whose rescaling restores long-context performance. The release of code further supports reproducibility. However, the strength of the claims depends on unverified assumptions about the transferability of the effective lengths across models with different RoPE bases.","major_comments":[{"comment":"The effective lengths E reported in Appendix D.1 are selected by ranking NIAH accuracy on Llama-3-8B-Instruct (Figure 1b), and the same benchmark's score for that same model is then reported as a DPE outcome in Table 1. The Llama-3-8B NIAH row is therefore partly a selection result rather than an independent prediction, which inflates the apparent performance of DPE on that row. Please either provide a held-out evaluation (e.g., detect E on a different model or using perplexity, then evaluate NIAH on Llama-3-8B) or explicitly separate selection results from predictions when discussing Table 1.","section":"Section 3.1 / Appendix D.1 / Table 1"},{"comment":"The manuscript does not state whether the per-group effective lengths E are re-detected for each model or reused from Llama-3-8B. Because RoPE bases differ across the evaluated models (e.g., Llama-3 uses base 500000, Mistral-v0.2 uses 10000, and Qwen-2.5 uses approximately 1e6), the same dimension index corresponds to different angular frequencies across models; thus the transferability of E is not self-evident. If E is reused without re-detection, a transfer experiment (e.g., comparing fixed E with per-model re-detected E on Mistral or Qwen-2.5) is required to justify the cross-model claims; if E is re-detected per model, the per-model E values must be reported and the detection procedure must be described for each model. This is load-bearing for the claim that DPE generalizes across model families.","section":"Section 3.3 / Appendix D.1"},{"comment":"The procedure for computing the 2-norm attention contribution used to select key dimensions is underspecified. The text defines D_h using 'all the queries and keys of the sequence with length L' but does not say whether the 2-norm scores are computed on the evaluation sequence at inference time, on a separate calibration set, or on a short sequence within the training length, nor how many tokens are used. Since the top-48 selection is fixed for all models, please specify the exact data, sequence length, and aggregation procedure used to determine D_h; this is necessary for reproducibility and for interpreting the overhead of the method.","section":"Section 3.2"},{"comment":"No error bars, confidence intervals, or significance tests are reported anywhere in the paper, even though several headline comparisons are based on small margins (e.g., Llama-3-8B NIAH DPE 92.50 vs. Self-Extend 89.50, and Llama-3.1-70B RULER DPE 86.39 vs. GPT-4 81.2). Given that Table 1 uses only 100 test cases per model, the reported differences may be within sampling noise for some rows. Please report variance across repeated runs or bootstrap intervals, or at least the number of test items per cell, and state whether differences are statistically reliable.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The baseline name 'Rerope' should be 'ReRoPE', and 'trucated length' should be 'truncated length'.","section":"Appendix D.2"},{"comment":"The abstract refers to 'GPT-4-128K' while Table 2 reports 'GPT-4-1106-preview'; please harmonize the terminology.","section":"Abstract and Section 4"},{"comment":"There is a typo: 'demons trate' should be 'demonstrate'.","section":"Section 6 (Related Work)"},{"comment":"The notation using floor brackets is ambiguous; please define the floor division explicitly near Equation (7).","section":"Equations (7) and (9)"},{"comment":"The sentence 'it achieves 56 point' should read '56 points' or '56.08'.","section":"Section 4"},{"comment":"The table header spells 'Claude2' while the text says 'Claud2'; please correct the spelling.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the cross-model transfer of effective lengths: the paper should clarify whether E is re-detected per model and, if not, provide a transfer experiment. The NIAH selection on Llama-3-8B also makes that row partially circular; the independent RULER and InfiniteBench results on other models are the strongest evidence. The comparison to GPT-4-1106-preview should be double-checked against the original RULER evaluation protocol to ensure apples-to-apples comparability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the thing to know: DPE is a genuinely new trick — per-dimension effective lengths, and scaling only the top-48 key dimensions per head — and the gains look real on several benchmarks. But the headline 8K-to-128K claim for Llama-3-8B is partly a selection artifact, and the cross-model transfer of the effective lengths is unverified. Under that caveat, it's a solid, improvable paper.\n\nWhat is new: ReRoPE, Self-Extend, and DCA scale the whole relative-position matrix; YaRN and NTK scale frequencies uniformly. DPE instead detects, for each RoPE dimension group, the maximum relative distance that works (the 'effective length'), then manipulates only the top-48 dimensions per head, selected by a 2-norm attention metric. The ablations make the mechanism plausible: rank-1 effective lengths matter, top-k beats all dimensions, and the 2-norm picks out the right dimensions. The implementation is clean and efficient with Flash Attention, and they release code. The RULER and InfiniteBench results on Mistral, Qwen, and Llama-3.1 are substantial, and those models and benchmarks were not used to choose the hyperparameters.\n\nNow the soft spots, in proportion. The main one is circularity: the effective lengths E were fit to NIAH accuracy on Llama-3-8B (Appendix D.1), and the paper's flagship extrapolation result (92.5) is NIAH accuracy on the same model (Table 1). That row is a fitting score, not a prediction. The RULER results on that model and the cross-model results give independent support, so the method isn't empty, but the specific '8K to 128K' headline should be read cautiously.\n\nThe bigger theoretical risk — and I agree with your stress-test note — is cross-model transfer. The E values are only reported for Llama-3-8B, yet they're applied to Mistral, Qwen, and Llama-3.1 with different rope_theta. The same dimension index does not mean the same frequency. Section 3.1 says 'For the same group of dimensions, we assume that they correspond to the same effective relative distance,' but that's an assumption within a model, and the authors never verify it across models. If effective lengths don't transfer, the method's generality collapses. Re-detecting E per model — or at least showing sensitivity — is required.\n\nMinor but real: no error bars anywhere, and one table actually contradicts the abstract. In Table 1, on Mistral-7B-Instruct-v0.2, DPE gets 84.25 on NIAH, below ReRoPE's 87.00. The abstract claims DPE 'significantly surpasses' all baselines; that's not true for every model. Also, the strong Llama-3.1-70B claim has no baselines other than the original RoPE — we don't know if YaRN or Self-Extend would also lift it.\n\nWho this is for: anyone working on training-free length extrapolation or efficient long-context serving. It deserves refereeing, but as a conditional: the authors need to re-detect E on each model (or prove transfer), report variance, and add baselines on 70B. As is, I'd cite it, but I wouldn't take the 8K-to-128K headline at face value.","headline":"Genuinely new per-dimension RoPE manipulation with large observed gains, but the headline result is partly fitted and the cross-model transfer of effective lengths is unverified.","tokens_in":20389,"tokens_out":4800,"would_cite":true,"duration_ms":40891,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selectively rescaling 48 RoPE dimensions pushes an 8k-trained Llama to 128k with no training.","keywords":["length extrapolation","RoPE","positional embeddings","training-free context extension","long-context benchmarks","dimension-wise scaling","attention contribution","context window"],"falsifier":"Run DPE on a model not in the paper (e.g., another RoPE model family at 8k training length) twice: once with the paper's Llama-3-8B effective-length table fixed, and once with effective lengths re-detected by the same NIAH sweep on that model. If the re-detected lengths differ substantially from the fixed table and the fixed-table version loses most of the gain, the transfer assumption is falsified; likewise, evaluating Llama-3-8B+DPE at 256k should show collapse beyond the largest detected effective length if the mechanism is truly length-bounded.","tokens_in":19211,"feed_emoji":"📏","tokens_out":7512,"duration_ms":65766,"temperature":0.7,"pith_summary":"The paper tries to establish that the failure of rotary-position models on long contexts is not uniform across hidden dimensions: different RoPE frequency groups have different \"effective relative distances,\" and only a handful of dimensions actually carry the burden of length extrapolation. Its proposal, DPE, rescales the relative-position matrix of just the top 48 dimensions per head to their detected effective lengths, leaving other dimensions untouched. This is claimed to push an 8k-trained Llama-3-8B to 128k contexts without any continued training, beating YaRN and Self-Extend, and to raise Llama-3.1-70B's RULER average by more than 18 points within its trained window. If right, long-context capability becomes a cheap inference-time parameter that any RoPE model could tune.","feed_headline":"Rescale 48 RoPE dimensions: 8k-trained Llama hits 128k, no training","feed_subtitle":"Rescaling selected position dimensions beats YaRN and Self-Extend, and lifts a 70B model's RULER score by 18 points.","key_machinery":"The load-bearing object is the per-dimension relative-position matrix $P_j$, built from RoPE's rotational embeddings $R(\\theta_j, n-m)$ with frequencies $\\theta_j = b^{-2j/d}$. DPE detects an effective length $e_i$ for each of $C=8$ dimension groups by sweeping the detection length $t$ in Eq. (7) on a needle-in-a-haystack task, then identifies key dimensions per head by ranking the 2-norm attention contribution $\\|q^{(j)}_m\\|\\|k^{(j)}_n\\|$, which bounds $|\\langle q^{(j)}_m, k^{(j)}_n\\rangle|$ by Cauchy-Schwarz. Only the top 48 dimensions have their position matrices rescaled by $s_i = \\lfloor L/e_i \\rfloor$ outside a local window $w=1\\text{k}$ (Eq. (9)), so all relative positions land inside each group's detected comfort zone. The mechanism's work is to show that selective rescaling, not uniform rescaling, is what determines extrapolation success.","core_discovery":"DPE's central discovery is that RoPE dimensions are not interchangeable for length generalization. By sweeping the maximum relative position of each of eight dimension groups from 1k to 128k on a needle-in-a-haystack task, the paper finds that low- and high-frequency groups tolerate very long positions while middle frequencies collapse past the training length. A 2-norm attention-contribution ranking then identifies, for each attention head, the 48 dimensions whose positions matter most; scaling only those dimensions' position matrices to their per-group effective lengths (e.g., 65536 for dimensions 0-7, 4096 for dimensions 32-47 on Llama-3-8B) restores retrieval at 128k. The paper reports that this selective manipulation outperforms scaling all dimensions, lifts Llama-3-8B from 0 to 56.08 on RULER 128k, and lifts Llama-3.1-70B from 66.41 to 86.39, above the reported GPT-4-1106-preview score of 81.2.","pith_inferences":["The reported effective-length table (dimensions 0-7 at 64k, 8-15 at 16k, etc.) might double as a diagnostic: it exposes which RoPE frequencies a model actually learned to trust, so one could compare training distributions across base models.","Re-detecting effective lengths on each target model, rather than reusing the Llama-3-8B table for all models, is an obvious testable upgrade; if rankings shift across scales or families, per-model detection should outperform the paper's fixed settings.","DPE's gains inside the training window suggest position-matrix conditioning, not just out-of-distribution avoidance; it may be improvable by stacking with frequency-based rescaling such as base changes.","The 256k frontier is unexplored: nothing in the mechanism stops effective lengths from being extrapolated further, but the paper stops at 128k."],"forward_implications":["An 8k-trained Llama-3-8B-Instruct can answer needle-in-a-haystack queries at 128k with 92.5% accuracy, and 56.08 average on RULER's 13 tasks, without continued training.","The same recipe lifts models already trained to 128k inside their training length: Llama-3.1-70B goes from 66.41 to 86.39 on RULER, beating the reported GPT-4-1106-preview score of 81.2.","Only the top 48 of 64 RoPE dimension pairs need rescaling; scaling all dimensions is slightly worse, so the method is a model of dimensional sparsity, not a blanket adjustment.","Because the change lives in the position matrix, it composes with FlashAttention-2 and adds negligible inference overhead at 128k.","Effective lengths differ across dimension groups, so the method turns context extension into a per-group hyperparameter search rather than a global interpolation or exclusion choice."],"supporting_citations":[{"why":"Supplies the grouping strategy and the Self-Extend baseline that DPE must outperform; the local-window idea in Eq. (7) inherits its neighboring-token constraint.","marker":"Jin et al., 2024"},{"why":"DCA is a direct baseline for whole-matrix relative-position rescaling; DPE's selective-dimension variant is contrasted against it.","marker":"An et al., 2024a"},{"why":"ReRoPE introduces truncating the relative-position matrix, the baseline against which DPE's per-dimension scaling is framed.","marker":"Su, 2023"},{"why":"Needle-in-a-Haystack is the task used both to detect effective relative distances and to evaluate the 128k extrapolation results.","marker":"gkamradt, 2023"},{"why":"Supplies the 2-norm attention-contribution metric used to rank and select the top-48 key dimensions.","marker":"Barbero et al., 2025"},{"why":"RULER is the long-context benchmark where the paper reports its headline extrapolation and in-training-length gains.","marker":"Hsieh et al., 2024"},{"why":"YaRN is the strongest frequency-scaling baseline DPE compares against on NIAH and RULER.","marker":"Peng et al., 2023"},{"why":"Dynamic NTK scaling is the other frequency-adjusting baseline in the comparison set.","marker":"emozilla, 2023"}],"fun_headline_variants":["Scaling just 48 RoPE dims extends Llama to 128k","Training-free extrapolation: 8k Llama reaches 128k via 48 dims","48 key dimensions unlock 128k context for Llama without training","RoPE trick: rescale 48 dims, Llama-8k hits 128k, beats YaRN","Lift 70B RULER by 18 pts: selective RoPE scaling, no training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each group of RoPE dimensions has one stable effective relative distance, measured once on Llama-3-8B-Instruct with a needle-in-a-haystack sweep, and that this distance transfers to other models and settings; if that transfer fails, DPE's fixed \"C=8, w=1k, top-48 for all models\" recipe loses its anchor.","fun_headline_variants_meta":{"raw":{"variants":["Scaling just 48 RoPE dims extends Llama to 128k","Training-free extrapolation: 8k Llama reaches 128k via 48 dims","48 key dimensions unlock 128k context for Llama without training","RoPE trick: rescale 48 dims, Llama-8k hits 128k, beats YaRN","Lift 70B RULER by 18 pts: selective RoPE scaling, no training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1639,"prompt_tokens":1065,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":456}},"tokens_in":681,"tokens_out":574,"duration_ms":5547,"temperature":1.0,"reasoning_tokens":456,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:07:17.004204+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DPE on a model not in the paper (e.g., another RoPE model family at 8k training length) twice: once with the paper's Llama-3-8B effective-length table fixed, and once with effective lengths re-detected by the same NIAH sweep on that model. If the re-detected lengths differ substantially from the fixed table and the fixed-table version loses most of the gain, the transfer assumption is falsified; likewise, evaluating Llama-3-8B+DPE at 256k should show collapse beyond the largest detected effective length if the mechanism is truly length-bounded.","supporting_citations":[],"review_version":1}