Pith. sign in

REVIEW 10 cited by

Eliminating Position Bias of Language Models: A Mechanistic Approach

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.01100 v3 pith:T7UFRGIR submitted 2024-07-01 cs.CL cs.LG

Ziqi Wang , Hanlin Zhang , Xiner Li , Kuan-Hao Huang , Chi Han , Shuiwang Ji , Sham M. Kakade , Hao Peng
show 1 more author
Heng Ji
This is my paper · ORCID
classification cs.CLcs.LG
keywords biaspositionattentionmodelsperformancedocumentsreasoningapproach
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Position bias has proven to be a prevalent issue of modern language models (LMs), where the models prioritize content based on its position within the given context. This bias often leads to unexpected model failures and hurts performance, robustness, and reliability across various applications. Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings. Based on the analyses, we propose to eliminate position bias (e.g., different retrieved documents' orders in QA affect performance) with a training-free zero-shot approach. Our method changes the causal attention to bidirectional attention between documents and utilizes model attention values to decide the relative orders of documents instead of using the order provided in input prompts, therefore enabling Position-INvariant inferencE (PINE) at the document level. By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning. Notably, PINE is especially useful when adapting LMs for evaluating reasoning pairs: it consistently provides 8 to 10 percentage points performance gains, making Llama-3-70B-Instruct perform even better than GPT-4-0125-preview and GPT-4o-2024-08-06 on the RewardBench reasoning set.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

    cs.LG 2026-07 conditional novelty 6.5 of 10

    LLM-as-judge scoring biases concentrate in low-dimensional, type-specific activation subspaces that support bidirectional causal steering and cross-domain failure prediction.

  2. MARS-RA: Rank Aggregation for Credit Assignment via Multimodal Comparisons in Embodied Multi-Agent Cooperation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Credit assignment via LMM pairwise comparisons plus Bradley–Terry rank aggregation and potential-based shaping improves cooperative MARL under sparse rewards and dynamic agent counts.

  3. Adaptive Repetition for Mitigating Position Bias in LLM-Based Ranking

    cs.LG 2025-07 conditional novelty 6.0 of 10

    An adaptive early-stopping rule for repeated LLM judgments cuts position-bias mitigation cost by roughly 80 percent while keeping the consensus result.

  4. Reranking-based Generation for Unbiased Perspective Summarization

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LLM-based metrics (LLM-Coverage and ALIGN SCORE) are more reliable than ROUGE and BERTScore for political perspective summaries, and reranking plus DPO training improves both coverage and faithfulness.

  5. RoToR: Towards More Reliable Responses for Order-Invariant Inputs

    cs.CL 2025-02 conditional novelty 6.0 of 10

    RoToR makes a frozen LLM order-invariant by circularly rotating a single global sort of segment position IDs, and Selective Routing combines it with the original model for mixed lists.

  6. On the Emergence of Position Bias in Transformers

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Multi-layer causal attention provably drives every token's context toward the first token, while decay masks and RoPE introduce a competing distance bias that trades off against depth.

  7. LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

    cs.AI 2026-05 reject novelty 5.0 of 10

    LISA replaces standard attention with linear attention plus indexer-selected sparse attention, claiming O(nM) inference cost and accuracy gains, but the complexity analysis ignores the indexer's full-context scoring.

  8. On the Reasoning Capacity of AI Models and How to Quantify It

    cs.AI 2025-01 reject novelty 5.0 of 10

    Positional randomization on GPQA shows GPT-4o-mini's accuracy is inflated by position-dependent heuristics, but the paper's strategy-decomposition model is validated only by construction and contradicts its own accuracy data.

  9. Positional Bias in Binary Question Answering: How Uncertainty Shapes Model Preferences

    cs.CL 2025-06 reject novelty 4.0 of 10

    Positional bias in LLM binary QA is small for easy questions but grows sharply as uncertainty rises, especially in subjective persuasion tasks.

  10. EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta

    cs.CL 2024-12 reject novelty 4.0 of 10

    EQUATOR uses vector search to fetch a human reference answer and a binary LLM grader to score open-ended LLM responses, producing far lower scores than standard benchmarks.

Pith tools