Pith. sign in

REVIEW 3 cited by

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.16421 v2 pith:LIOUBTJP submitted 2024-01-29 cs.LG cs.AIcs.CLstat.ML

Zhenyu He , Guhao Feng , Shengjie Luo , Kai Yang , Liwei Wang , Jingjing Xu , Zhi Zhang , Hongxia Yang
show 1 more author
Di He
This is my paper
classification cs.LGcs.AIcs.CLstat.ML
keywords encodingpositionalbipeextrapolationbilevelcapabilitiesinformationinter-segment
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Explicitly disentangling semantic and positional streams in a Transformer encoder reveals that absolute positional representations collapse to a 2D document-structure manifold, attention heads specialize by role, and ...

  2. A Measure-Theoretic Analysis of Reasoning: Structural Generalization and Approximation Limits

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Applies optimal transport to bound OOD generalization error in Transformers via Lipschitz continuity and TC^0 circuit depth lower bounds for Dyck-k backtracking, supported by evaluations on 54 configurations.

  3. HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models

    cs.CL 2025-09 reject novelty 4.0 of 10

    HoPE replaces RoPE's sine/cosine rotations with hyperbolic functions plus an exponential damping term to enforce monotonic attention decay, but the claimed consistent superiority and the 'RoPE as special case' theorem...

Pith tools