Pith. sign in

REVIEW 17 cited by

Conditional Positional Encodings for Vision Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2102.10882 v3 pith:L4HBYMSG submitted 2021-02-22 cs.CV cs.AIcs.LG

Xiangxiang Chu , Zhi Tian , Bo Zhang , Xinlong Wang , Chunhua Shen This is my paper
classification cs.CVcs.AIcs.LG
keywords positionalconditionalcpvtencodingencodingsinputvisionposition
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable positional encodings, which are pre-defined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during training. Besides, CPE can keep the desired translation-invariance in the image classification task, resulting in improved performance. We implement CPE with a simple Position Encoding Generator (PEG) to get seamlessly incorporated into the current Transformer framework. Built on PEG, we present Conditional Position encoding Vision Transformer (CPVT). We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings and delivers outperforming results. Our code is available at https://github.com/Meituan-AutoML/CPVT .

Discussion (0). Sign in to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

    cs.CV 2021-03 accept novelty 8.0 of 10

    Swin Transformer reaches 87.3% ImageNet accuracy and sets new records on COCO detection and ADE20K segmentation by replacing global self-attention with shifted-window local attention inside a hierarchical pyramid.

  2. Masked-Token Prediction for Anomaly Detection at the Large Hadron Collider

    hep-ph 2026-04 unverdicted novelty 7.0 of 10

    The work demonstrates masked-token prediction with transformers for model-independent anomaly detection in LHC data, achieving strong results on top-rich BSM signatures like four-top production using VQ-VAE tokenization.

  3. Hierarchical Mesh Transformers with Topology-Guided Pretraining for Morphometric Analysis of Brain Structures

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    A hierarchical mesh transformer using topology-guided pretraining on simplicial complexes achieves state-of-the-art results on Alzheimer's classification, amyloid prediction, and focal cortical dysplasia detection fro...

  4. Massive Activations in Large Language Models

    cs.CL 2024-02 unverdicted novelty 7.0 of 10

    Massive activations are constant large values in LLMs that function as indispensable bias terms and concentrate attention probabilities on specific tokens.

  5. LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

    cs.CV 2023-03 conditional novelty 7.0 of 10

    LLaMA-Adapter turns frozen LLaMA 7B into a capable instruction follower using only 1.2M new parameters and zero-init attention, matching Alpaca while extending to image-conditioned reasoning on ScienceQA and COCO.

  6. ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts

    cs.CV 2026-07 accept novelty 6.0 of 10

    Multi-stage agglomerative distillation consolidates eight vision, vision-language, and slide-level pathology teachers into one backbone that ranks first on average across 96 downstream tasks.

  7. Device Passport: Enabling Spatio-Temporal Pretrained Models to Generalize Across Input Layouts

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Device Passport improves cross-layout transfer for biosignal models by learning expert mixture models from each channel's functional activity and metadata, outperforming baselines in transfer regimes.

  8. DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    DPPE decouples rotation and translation in camera positional encodings for multi-view transformers to resolve late-stage training stagnation and improve generalization in novel view synthesis.

  9. End-to-End Context Compression at Scale

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    LCLMs are scaled 0.6B-encoder 4B-decoder compressors pre-trained on over 350B tokens that improve the Pareto frontier for general-task performance, compression speed, and peak memory in long-context language model inference.

  10. Small Models, Strong Priors: Architectural Inductive Bias for Parameter-Efficient Neural PDE Solvers

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    WaveLiT combines wavelet tokenization, linear attention, and multiscale pyramids to produce parameter-efficient neural PDE solvers that match much larger models on TheWell benchmarks.

  11. ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference

    cs.CV 2025-07 conditional novelty 6.0 of 10

    ToFe freezes unimportant ViT tokens temporarily and reuses them later, achieving about 50% FLOPs reduction with less than 2% top-1 accuracy drop on ImageNet.

  12. LOOPE: Learnable Optimal Patch Order in Positional Embeddings for Vision Transformers

    cs.CV 2025-04 unverdicted novelty 6.0 of 10

    LOOPE learns a patch ordering for positional embeddings in ViTs and introduces the Three Cell Experiment benchmark that shows 30-35% gaps in positional retention versus the usual 4-6%.

  13. VistaRef: Boosting Visual Spatial Orientation Awareness for Pointing-to-Object Detection

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    VistaRef improves pointing-to-object detection accuracy by 14 points via local hand entity modeling, geometric ray modeling, and an orientation-consistent alignment loss.

  14. USEMA: a Scalable Efficient Mamba Like Attention for Medical Image Segmentation

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    USEMA is a hybrid UNet architecture merging CNNs with scalable Mamba-like attention (SEMA) that achieves better efficiency than transformers and superior segmentation accuracy than pure CNN or Mamba models across medi...

  15. SAC-MIL: Spatial-Aware Correlated Multiple Instance Learning for Histopathology Whole Slide Image Classification

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A spatial-aware MLP-Mixer with polar rotary position embedding and channel-shifting SAC blocks achieves state-of-the-art slide-level classification on CAMELYON-16, TCGA-LUNG, and TCGA-BRCA.

  16. CoPE: A Lightweight Complex Positional Encoding

    cs.LG 2025-08 reject novelty 4.0 of 10

    CoPE, a complex positional encoding with imaginary-position and first-layer phase-aware attention, shows small GLUE gains but its no-decay proof and speed advantage are not fully supported.

  17. Generalized SAM: Efficient Fine-Tuning of SAM for Variable Input Image Sizes

    cs.CV 2024-08 unverdicted novelty 4.0 of 10

    GSAM applies random cropping to enable variable input sizes for efficient SAM fine-tuning, claiming lower compute with comparable or higher accuracy on varied datasets.

Pith tools