REVIEW 8 cited by
Attention Sorting Combats Recency Bias In Long Context Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Current language models often fail to incorporate long contexts efficiently during generation. We show that a major contributor to this issue are attention priors that are likely learned during pre-training: relevant information located earlier in context is attended to less on average. Yet even when models fail to use the information from a relevant document in their response, they still pay preferential attention to that document compared to an irrelevant document at the same position. We leverage this fact to introduce ``attention sorting'': perform one step of decoding, sort documents by the attention they receive (highest attention going last), repeat the process, generate the answer with the newly sorted context. We find that attention sorting improves performance of long context models. Our findings highlight some challenges in using off-the-shelf language models for retrieval augmented generation.
Forward citations
Cited by 8 Pith papers
-
Evidence Interfaces Shape How Retrieval-Augmented Readers Use Support
Top-k retrieval scores should be reported with complete-support coverage, because the same score can mix missing-support failures with reader-side interface effects.
-
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
Positional Contrastive Decoding, a training-free method that contrasts standard and over-rotated RoPE logits, improves long-context retrieval and QA by a few points.
-
SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence
SelfElicit uses deep-layer attention to automatically highlight relevant evidence sentences in the input context, yielding consistent QA accuracy gains across six instruction-tuned LLMs.
-
RoToR: Towards More Reliable Responses for Order-Invariant Inputs
RoToR makes a frozen LLM order-invariant by circularly rotating a single global sort of segment position IDs, and Selective Routing combines it with the original model for mixed lists.
-
LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
LISA replaces standard attention with linear attention plus indexer-selected sparse attention, claiming O(nM) inference cost and accuracy gains, but the complexity analysis ignores the indexer's full-context scoring.
-
Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding
DeepSeek-R1-distilled models show higher multi-document QA accuracy than their base counterparts and flatter position-bias curves, especially with 50-80 documents.
-
Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training
Attention heads in trained GPT-2 models develop temporal contiguity, recency, and primacy effects, and ablating induction heads removes the resulting serial-recall bias in outputs.
-
Analysis of LLM as a grammatical feature tagger for African American English
LLMs are worse than rule-based and transformer baselines at detecting Habitual Be and Multiple Negation in African American English, and their errors correlate with recency and formality biases.
Discussion (0). Continue with ORCID to comment.