Pith. sign in

REVIEW 10 cited by

Mitigate Position Bias in Large Language Models via Scaling a Single Dimension

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.02536 v3 pith:G7RWSXJR submitted 2024-06-04 cs.CL cs.LG

Yijiong Yu , Huiqiang Jiang , Xufang Luo , Qianhui Wu , Chin-Yew Lin , Dongsheng Li , Yuqing Yang , Yongfeng Huang
show 1 more author
Lili Qiu
This is my paper · ORCID
classification cs.CLcs.LG
keywords positionbiasmodelshiddenstatesattentiondimensionlanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large Language Models (LLMs) are increasingly applied in various real-world scenarios due to their excellent generalization capabilities and robust generative abilities. However, they exhibit position bias, also known as "lost in the middle", a phenomenon that is especially pronounced in long-context scenarios, which indicates the placement of the key information in different positions of a prompt can significantly affect accuracy. This paper first explores the micro-level manifestations of position bias, concluding that attention weights are a micro-level expression of position bias. It further identifies that, in addition to position embeddings, causal attention mask also contributes to position bias by creating position-specific hidden states. Based on these insights, we propose a method to mitigate position bias by scaling this positional hidden states. Experiments on the NaturalQuestions Multi-document QA, KV retrieval, LongBench and timeline reorder tasks, using various models including RoPE models, context windowextended models, and Alibi models, demonstrate the effectiveness and generalizability of our approach. Our method can improve performance by up to 15.2% by modifying just one dimension of hidden states. Our code is available at https://aka.ms/PositionalHidden.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MARS-RA: Rank Aggregation for Credit Assignment via Multimodal Comparisons in Embodied Multi-Agent Cooperation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Credit assignment via LMM pairwise comparisons plus Bradley–Terry rank aggregation and potential-based shaping improves cooperative MARL under sparse rewards and dynamic agent counts.

  2. Semantically Similar, Logically Distinct: Diagnosing the Semantic-Answerability Gap in Table RAG

    cs.AI 2026-07 conditional novelty 6.0 of 10

    On a new sibling-table benchmark, dense retrievers find the right table group but identify the uniquely answerable table at near-chance rates; cross-encoder reranking lifts top-1 from 18.2% to 57.4%.

  3. RoToR: Towards More Reliable Responses for Order-Invariant Inputs

    cs.CL 2025-02 conditional novelty 6.0 of 10

    RoToR makes a frozen LLM order-invariant by circularly rotating a single global sort of segment position IDs, and Selective Routing combines it with the original model for mixed lists.

  4. On the Emergence of Position Bias in Transformers

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Multi-layer causal attention provably drives every token's context toward the first token, while decay masks and RoPE introduce a competing distance bias that trades off against depth.

  5. The Chameleon Nature of LLMs: Quantifying Multi-Turn Stance Instability in Search-Enabled Language Models

    cs.CL 2025-10 conditional novelty 5.0 of 10

    On 1,180 web-enabled multi-turn conversations, GPT-4o-mini, Llama-4-Maverick, and Gemini-2.5-Flash all changed stance frequently (chameleon scores 0.39–0.51), with little effect from temperature.

  6. Position: Contextual Integrity is Inadequately Applied to Language Models

    cs.CY 2025-01 conditional novelty 5.0 of 10

    Many prior studies that apply Contextual Integrity to language models omit the theory's core tenets, which can make their privacy conclusions unreliable.

  7. On the Reasoning Capacity of AI Models and How to Quantify It

    cs.AI 2025-01 reject novelty 5.0 of 10

    Positional randomization on GPQA shows GPT-4o-mini's accuracy is inflated by position-dependent heuristics, but the paper's strategy-decomposition model is validated only by construction and contradicts its own accuracy data.

  8. Towards High Supervised Learning Utility Training Data Generation: Data Pruning and Column Reordering

    cs.LG 2025-07 reject novelty 4.0 of 10

    PRRO combines signal-based data pruning and column reordering to improve the supervised learning utility of synthetic tabular data, but its evaluation is undermined by data manipulation and an ill-defined correlation measure.

  9. SAMba-UNet: SAM2-Mamba UNet for Cardiac MRI in Medical Robotic Perception

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A combined SAM2, Mamba, and UNet architecture reports state-of-the-art Dice of 0.9103 on the ACDC cardiac MRI benchmark, with no code or error bars yet released.

  10. Parallel Key-Value Cache Fusion for Position Invariant RAG

    cs.AI 2025-01 conditional novelty 4.0 of 10

    KV-Fusion encodes each retrieved passage in parallel with identical local positions, fuses the key-value caches, and fine-tunes a decoder so answers do not depend on passage order.

Pith tools