Pith. sign in

REVIEW 4 major objections 6 minor 22 references

Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Positional contrastive decoding recovers gold tokens that long-context models rank high but fail to select.

desk verdict Promising training-free decoding heuristic held back by a broken theoretical proof and benchmark-tuned hyperparameters. read the letter →

arxiv 2506.08371 v2 pith:HBVOSDK2 submitted 2025-06-10 cs.CL

Zikai Xiao , Ziyang Wang , Wen Ma , Yan Zhang , Wei Shen , Yan Wang , Luqi Gong , Zuozhu Liu This is my paper
classification cs.CL
keywords long-contextLLMsposteriorsalienceattenuationpositionalcontrastivedecodingRoPEattentiondecaykey-valueretrievalbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that long-context LLMs degrade partly because the salience of the correct next token in the decoding distribution attenuates as the context lengthens, even though that token usually stays near the top of the ranking. To counter this, it proposes Positional Contrastive Decoding (PCD), a training-free decoding step that subtracts logits produced by an over-rotated variant of the rotary position embedding (RoPE) from the standard logits and scales the difference, $\tilde{L} = (1+\beta)L - \beta L^*$. The paper argues that this contrast slows the decay of long-range attention scores, letting the model exploit the short-to-long training it already received. If correct, the method is a cheap inference-time fix for long-context QA and retrieval, with reported gains such as 8k KV retrieval on Llama-3-8B-262k rising from 72.0 to 79.0.

What carries the argument

The machinery is the contrastive logit step applied to perturbed rotary position embeddings. PCD constructs a "local-aware" attention by interpolating between the standard RoPE base $B$ and a lower base $B'$ through a transition function $T(x)=2-\exp(\alpha x)$, so low-frequency blocks get extra rotation while high-frequency blocks stay close to their original angles. It feeds those over-rotated query/key matrices through the same model to obtain perturbed logits $L^*$, then returns $\tilde{L}=(1+\beta)L-\beta L^*$ restricted to the top $\gamma$ tokens by probability. The load-bearing identity is the spectral representation of the attention score as a sum of cosines, $S(k)=\sum_j A_j \cos(k\theta_j+\phi_j)$, together with the claim that the contrastive combination saturates the long-distance decay to a slower exponent; this is what is supposed to turn the short-to-long training signal into better long-range attention.

What would settle it

Compute the contrastive attention score $S_{CD}(k)=(1+\lambda)S(k)-\lambda S'(k)$ numerically for $k$ from 1 to 16384 with $B=10^6$, $B'=10^4$, and $d=512$, and check whether its log-log decay slope is shallower than the standard score's by roughly $\left(\ln B'/\ln B\right)^{2/d}$; if the contrastive score decays at the same rate or faster, the theorem's mechanism is falsified. A second check: apply PCD to a long-context model whose positional encoding is RoPE but whose attention head dimension differs, and see whether the predicted exponent shift matches the measured accuracy gain.

Watch

Extended reading notes

Core claim

The central discovery is a decoding-space phenomenon the paper calls Posterior Salience Attenuation (PSA): as input length grows, the gold token's reciprocal rank in the model's output distribution falls on tasks held at constant difficulty, yet the gold token remains inside the top handful of candidates even with a 128k vocabulary. Because the correct answer is still highly ranked, a decoding operation that amplifies salience can recover it. PCD does this by building "local-aware" logits from a RoPE variant whose low-frequency rotations are over-rotated (base lowered from $B$ to $B'$ with a smooth transition $T(x)=2-\exp(\alpha x)$), then forming $\tilde{L}=(1+\beta)L-\beta L^*$ and applying it to the top $\gamma$ tokens. The paper's spectral analysis claims the contrastive score's asymptotic decay exponent is at most $\left(\ln B'/\ln B\right)^{2/d}$ times that of the standard score, so the long-range signal survives better under PCD; experiments on RULER, InfiniteBench, and LongBench report consistent gains.

Load-bearing premise

The mathematical argument that PCD keeps long-range attention alive rests on a lower-bound estimate in the spectral derivation that stays positive and grows with distance rather than decaying; if that estimate is invalid, the proposed mechanism is unsupported even if the method still helps empirically.

Editorial extensions

If this is right

  • PCD raises InfiniteBench KV retrieval accuracy on Llama-3-8B-262k from 72.0 to 79.0 at 8k and from 52.0 to 55.0 at 16k.
  • On RULER variable tracking, PCD improves F1 on the 262k model from 74.02 to 81.80 at 4k and from 71.21 to 77.92 at 8k.
  • The spectral analysis implies the advantage of PCD grows with distance, since the contrastive score decays with a strictly smaller exponent than the standard score.
  • PCD is training-free and decoding-side, so it can be added to existing RoPE-based long-context models without fine-tuning or prompt rearrangement.
  • PCD mitigates but does not remove the context limit: it cannot extend the attention window itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the mechanism is genuinely spectral, a similar contrast should work with other relative position encodings (e.g., ALiBi or Kerple) by perturbing their slope parameters and contrasting the two logits; the paper tests only RoPE.
  • Beyond the paper: the PSA observation suggests a cheap diagnostic for when decoding-side correction will help: on a fixed task, increase context length while watching the gold token's reciprocal rank; a steep drop with a retained high rank would flag PCD-style methods as candidates.
  • Beyond the paper: the ablation table shows that some choices of the top-$\gamma$ window yield accuracy below the no-PCD base (71.5 versus 72.0 in the reported setting), so transferring PCD to new tasks will likely require re-tuning $\gamma$ rather than treating the default as universal.
  • Beyond the paper: because PCD reweights existing logits, it should compose with training-based long-context methods rather than replace them; a natural next experiment is PCD on top of fine-tuned long-context models with different RoPE bases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies why long-context LLMs degrade despite large nominal context windows. It defines a salience score S(L), observes that the gold token's rank in the decoding distribution shrinks as context grows, and proposes Positional Contrastive Decoding (PCD), which contrasts standard logits L with logits L* obtained by over-rotating low-frequency RoPE components using a transition function and a smaller base B', then computes L~ = (1+β)L − βL* on a top-γ subset. A spectral analysis in Appendix A.1 is offered as a mechanism, and experiments on RULER, InfiniteBench, and LongBench are used to claim consistent gains, including 8k KV retrieval accuracy on Llama-3-8B-262k rising from 72.0 to 79.0.

Significance. If the empirical gains survive a properly held-out evaluation, PCD is an attractive training-free decoding method with a distinctive positional-encoding interpretation. The paper is commendably concrete about its limitations and reports an ablation over the tested hyperparameter ranges. However, the theoretical derivation that is meant to explain the gain is invalid as written, and the headline result is currently tied to hyperparameters selected on the same InfiniteBench benchmark that is then used for evaluation, with no error bars and no released code. Since both the mechanism and the empirical headline are load-bearing, the contribution is significant but only conditionally supported.

major comments (4)
  1. [Appendix A.1, Theorem A.5 and Eq. (20)] The lower bound |S_CD(k)| ≥ λ Σ_{j≤j0} A_j k δ_j |sin(kθ_j)| − O(k^2) grows with k instead of decaying, so it does not establish the claimed asymptotic improvement in Eq. (17). Moreover, the Taylor expansion in Eq. (19) requires kδ_j to be small, which is not guaranteed for low-frequency components when B'/B = 1e-4 and k is large. The paper's only theoretical justification for PCD therefore needs either a corrected proof or explicit removal of the theorem.
  2. [Section 3.3/Table 1 and Table 2] The hyperparameters α, β, B'/B, and γ are tuned on InfiniteBench, and the same InfiniteBench KV-retrieval accuracy is reported in Table 2 as a central result. Because Table 1 itself reports variances up to 3.1 accuracy points across three runs, the 7.0-point improvement at 8k is comparable to the tuning noise. Please evaluate with hyperparameters fixed on a held-out validation set, or report robustness over a grid, and provide standard errors for the main tables.
  3. [Tables 2 and 3] No error bars, no number of independent runs, and no statistical significance tests are reported for the main comparisons, and the code and evaluation scripts are not released. Without these, an independent check of the claimed state-of-the-art result is impossible, and the small LongBench average gain of 0.89 points in Table 3 cannot be distinguished from noise.
  4. [Section 3.4 and Figure 4] The simulations used to support the decay claim are run with settings that do not match the tuned configuration, for example α=0.4 and β=0.6 in the main simulation versus α=0.2 and β=2.5 in Table 1, and a different B' relation (Section 3.4 uses B=10^6 and B'=10^4, while Table 1 recommends B'/B=1e-4). Please clarify whether these curves are illustrative or quantitative evidence, and if the latter, reconcile the parameter settings.
minor comments (6)
  1. [References] The reference list contains a malformed entry that begins 'TUNING ON SYNTHETIC DATA. From artificial needles to real haystacks: ...' with no authors and no year; it should be completed or removed.
  2. [Table 3] The reported PCD average of 26.87 does not match the arithmetic mean (26.73) of the seven listed task scores; please check the aggregation.
  3. [Section 3.2 and Table 2] The text states that PCD 'consistently enhances' model performance, but Table 2 shows a decrease on Llama-3-8B-8k at 8k KV retrieval (92.4→91.0) and on the 1048k variant for RULER Variable Tracking at 4k and 8k; please qualify the claim.
  4. [Section 2.2, Table 1, Section 3.4] The notation for B' is inconsistent: Section 2.2 and Table 1 express the perturbation as B'/B with a recommended value of 1e-4, while Section 3.4 states B=10^6 and B'=10^4, a ratio of 1e-2, and Figure 4 uses B'=100000 with B=283461213. Please specify absolute values and ratios consistently.
  5. [Appendix A.5] The citation '(DATA; An et al., 2024)' uses an unresolved placeholder DATA; either add the reference or replace the citation.
  6. [Section 3.4] The word 'substrated' should be 'subtracted'.

Circularity Check

2 steps flagged · score 4.0 of 10

Hyperparameters tuned on InfiniteBench are then reported as success on InfiniteBench, and the decay simulation restates PCD's defining formula; RULER and LongBench results remain independent evidence.

  1. fitted input called prediction [Section 3.3 (Hyperparameter Ablation), Table 1; Table 2 (InfiniteBench: KV Retrieval)]
    "The hyperparameter ablation on InfiniteBench is evaluated in Table 1. Optimal performance is achieved with β=2.5 ... Moderate frequency perturbation (B/B′ = 10^4) performs best. ... Table 2: ... InfiniteBench: KV Retrieval (Accuracy %) ... PCD 92.0(↑2.8%) 79.0(↑7.0%) 55.0(↑3.0%)"

    Table 1 selects the PCD hyperparameters (α=0.2, β=2.5, B'/B=1e-4, γ=30) by optimizing accuracy on InfiniteBench, with run-to-run variance up to 3.1 points. Table 2 then reports the headline PCD gains on InfiniteBench KV Retrieval (e.g., 72.0→79.0 at 8k) on the same benchmark. The reported 'state-of-the-art' result is therefore not a prediction on a held-out test set; it is the outcome of fitting the method's knobs to that benchmark. Because no held-out hyperparameter selection or error bars are provided for Table 2, the 7-point gain is partially forced by construction rather than independent evidence.

  2. self definitional [Section 3.4 (Simulating Long-Term Decay) and Appendix A.1, Eqs. (9), (12), (18)]
    "The results in Fig. 3 (b) demonstrate that the over-rotated variant exhibits sharper decay for local modeling, while PCD mitigates long-range decay, enhancing global awareness. To further validate the effectiveness of PCD in slowing down attention score decay, we conducted additional simulations under varying head dimensions, frequency parameters B′, and contrastive coefficients β."

    The simulation is generated from the same definitions used to construct PCD: S(k) is the spectral attention score of Eq. (9), S'(k) is the over-rotated score of Eq. (12), and the contrastive score is S_CD(k) = (1+λ)S(k) − λS'(k) in Eq. (18). Plotting these formulas and observing slower decay for S_CD is a direct consequence of the algebraic construction, not an empirical test of whether PCD alleviates attention degradation. The simulation therefore cannot falsify or independently confirm the claim; it restates the definition of the method.

full rationale

Two partial circularities are present, but the central claim is not wholly circular. First, PCD's hyperparameters are tuned on InfiniteBench (Table 1), and the same benchmark's KV Retrieval numbers are then reported as PCD's success in Table 2; this fitted-input-called-prediction pattern inflates the headline gain. Second, the 'long-term decay simulation' used to validate the mechanism is a numerical plot of the very formula that defines PCD, so it is a self-consistent illustration rather than independent support. Against these, the RULER and LongBench results in Tables 2 and 3 use the fixed tuned hyperparameters on different benchmarks and provide non-circular external evidence. No load-bearing self-citations were found; all cited prior work is external, and no uniqueness claim is imported from the authors' own prior papers. The questionable inequality in Theorem A.5 (a lower bound that grows with k rather than decays) is a mathematical-correctness problem, not a circularity problem, so it is not scored here.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces four hyperparameters fitted on a benchmark, and relies on a frequency-perturbation baseline and the high-rank observation as background assumptions. No new physical or conceptual entities are invented; the 'local-aware attention' is a constructed baseline rather than a separately evidenced mechanism.

free parameters (4)
  • beta (contrast intensity) = 2.5 (optimal in ablation; 0.6 and 1.5 in simulations)
    Controls how strongly the local-aware logits are subtracted in L~ = (1+beta)L - beta L*; tuned on InfiniteBench.
  • alpha (transition function coefficient) = 0.2 (optimal; 0.4 and 0.1 in simulations)
    Sets the smoothness of the transition T(x) = 2 - exp(alpha*x) between original and over-rotated frequencies; fitted on InfiniteBench.
  • B' / B (frequency base ratio) = 1e-4 (i.e., B' = 100 when B = 1e6 in Definition A.3; B' = 1e4 in experiments)
    Determines the amount of over-rotation in low-frequency RoPE components; tuned on InfiniteBench.
  • top-gamma = 30 (optimal; range tested 10-200)
    Number of top tokens to which the contrastive mechanism is applied; tuned on InfiniteBench.
assumptions (3)
  • standard math RoPE attention scores obey the spectral decomposition S(k) = sum_j A_j cos(k*theta_j + phi_j) and the long-term decay bound of Lemma A.4.
    Used in Appendix A.1 to derive decay rates; standard property of Rotary Position Embedding.
  • ad hoc to paper Perturbing low-frequency RoPE angles produces a 'local-aware attention' whose logits L* serve as a clean baseline that isolates the gains of short-to-long training.
    The paper assumes the frequency perturbed model is a faithful local baseline without validating that the difference (1+beta)L - beta*L* selectively amplifies long-range information rather than noise.
  • domain assumption Gold tokens remain in the top ranks even for incorrect predictions (Fig. 3a), so boosting salience will recover the gold token without harming others.
    Empirical observation on a limited set of retrieval examples; no statistical test or coverage analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding." pith.science (2026). https://pith.science/paper/HBVOSDK2

@misc{pith2026250608371,
  author       = {Pith},
  title        = {Pith review of: Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HBVOSDK2}},
  note         = {Machine review of arXiv:2506.08371}
}
read the original abstract

While Large Language Models (LLMs) support long contexts, they struggle with performance degradation within the context window. Current solutions incur prohibitive training costs, leaving statistical behaviors and cost-effective approaches underexplored. From the decoding perspective, we identify the Posterior Salience Attenuation (PSA) phenomenon, where the salience ratio correlates with long-text performance degradation. Notably, despite the attenuation, gold tokens still occupy high-ranking positions in the decoding space. Motivated by it, we propose the training-free Positional Contrastive Decoding (PCD) that contrasts the logits derived from long-aware attention with those from designed local-aware attention, enabling the model to focus on the gains introduced by large-scale short-to-long training. Through the analysis of long-term decay simulation, we demonstrate that PCD effectively alleviates attention score degradation. Experimental results show that PCD achieves state-of-the-art performance on long-context benchmarks.

Figures

Figures reproduced from arXiv: 2506.08371 by the authors.

Figure 2
Figure 2. PCD effectively alleviates the decrease in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. An illustration of PCD, contrasting logits from [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. (a) Distribution of gold label ranks across samples; (b) Single Layer Analysis: PCD mitigates salience [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Long-term decay simulation of attention scores under varying hyperparameter settings, including [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Dynamic visualization of logits as context length increases, illustrating how the logit value of the gold [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 6 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024. Make your llm fully utilize the context. arXiv preprint arXiv:2404.16811

    arXiv 2024

  4. [4]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://doi.org/10.18653/v1/2024.acl-long.172 L ong B ench: A bilingual, multitask benchmark for long context understanding . In Proceedings of the 62nd Annual Meeting of the Association for ...

    doi:10.18653/v1/2024.acl-long.172 2024

  5. [5]

    Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, et al. 2025. CoT-Kinetics : A theoretical modeling assessing lrm reasoning process. arXiv preprint arXiv:2505.13408

    arXiv 2025

  6. [6]

    Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. 2022. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386--8399

    work page 2022

  7. [7]

    Glass, and Pengcheng He

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. https://openreview.net/forum?id=Th6NyL07na Dola: Decoding by contrasting layers improves factuality in large language models . In The Twelfth International Conference on Learning Representations

    2024

  8. [8]

    From artificial needles to real haystacks: Im-proving retrieval capabilities in llms by fine-tuning on synthetic data

    TUNING ON SYNTHETIC DATA. From artificial needles to real haystacks: Im-proving retrieval capabilities in llms by fine-tuning on synthetic data

    work page

Show all 22 references
  1. [9]

    working memory

    Amanda Dsouza, Christopher Glaze, Changho Shin, and Frederic Sala. 2024. Evaluating language model context windows: A" working memory" test and inference-time correction. arXiv preprint arXiv:2407.03651

    2024 arXiv

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. 2024. Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models. arXiv preprint arXiv:2408.10151

    2024 arXiv

  4. [12]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024 a . https://openreview.net/forum?id=kIoBbc76Sy RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling

    2024

  5. [13]

    Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al. 2024 b . Found in the middle: Calibrating positional attention bias improves long context utilization. arXiv preprint arX...

    2024 arXiv

  6. [14]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

    2024

  7. [15]

    Taiming Lu, Muhan Gao, Kuai Yu, Adam Byerly, and Daniel Khashabi. 2024. Insights into llm long-context failures: When transformers know but don't tell. arXiv preprint arXiv:2406.14673

    2024 arXiv

  8. [16]

    Alexander Peysakhovich and Adam Lerer. 2023. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427

    2023 arXiv

  9. [17]

    Ofir Press, Noah Smith, and Mike Lewis. 2022. https://openreview.net/forum?id=R8sQPpGCv0 Train short, test long: Attention with linear biases enables input length extrapolation . In International Conference on Learning Representations

    2022

  10. [18]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  11. [19]

    Szymon Tworkowski, Konrad Staniszewski, Miko aj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Mi o \'s . 2024. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36

    2024

  12. [20]

    paraphrasing the original text

    Yijiong Yu. 2023. " paraphrasing the original text" makes high accuracy long-context qa. arXiv preprint arXiv:2312.11193

    2023 arXiv

  13. [21]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024 a . Infinite bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computat...

    2024

  14. [22]

    Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024 b . Found in the middle: How language models use long contexts better via plug-and-play positional encoding. arXiv preprint arXiv:2403.04797

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.