Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

RegCache claims that precomputed middle-layer sink tokens act as universal registers, absorbing activation outliers in vision encoders so that low-bit quantization retains accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:26 UTC pith:YZAUWJXZ

load-bearing objection A practical, training-free plug-in that clearly improves low-bit activation quantization of vision encoders on standard benchmarks, but the mechanism story is only partially supported and the code is missing. the 3 major comments →

arxiv 2510.04547 v5 pith:YZAUWJXZ submitted 2025-10-06 cs.LG cs.CV

Activation Quantization of Vision Encoders Needs Prefixing Registers

Seunghyeon Kim , Taesun Yeom , Jinho Kim , Wonpyo Park , Kyuyeun Kim , Jaeho Lee This is my paper
classification cs.LG cs.CV
keywords activation quantizationvision transformersoutliersattention sinksregister tokenspost-training quantizationlow-bit inferenceRegCache
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the activation outliers which break low-bit quantization of vision transformers are not arbitrary per-image noise: they are 'sink' tokens that emerge in the middle layers and are nearly identical across images. On that basis it proposes RegCache, a training-free plug-in that precomputes a small universal register from reference images, inserts its key-value cache as a prefix at quantization-sensitive layers, and deletes the largest-norm tokens still present at test time. If the paper's claims hold, standard post-training quantization at 4 and 6 bits recovers a large fraction of full-precision accuracy across five vision encoders, in some cases moving from near-chance to usable accuracy. Because RegCache only adds and removes a few tokens, it layers on top of any existing quantizer at negligible computational cost.

Core claim

The paper's central discovery is that outlier tokens in vision encoders behave differently from those in language models and are nevertheless reusable. Layerwise sensitivity analysis shows that accuracy collapse under 8-bit quantization is localized to middle blocks where activation outliers first appear; in those blocks the largest-ℓ∞-norm token has mean pairwise cosine similarity 0.89 across images, while normal tokens score 0.26. The authors therefore hypothesize that middle-layer sink tokens from one image can serve as registers for another, and build RegCache on that hypothesis: curate top-norm tokens from a reference pool, average their key-value caches into a register, prefix it from

What carries the argument

The load-bearing mechanism is the register cache: a few key-value vectors built by averaging the cached keys and values of the 100 highest-ℓ∞-norm tokens sampled from a pool of reference images at the quantization-sensitive block and the blocks just before it. At inference, copies of this averaged cache are prefixed to the token stream of each relevant block, giving the model an external attention sink that concentrates extreme magnitudes; a companion step deletes the still-largest-norm tokens from the test image before they reach the sensitive block. The mechanism works because outlier tokens align on shared channels: an averaged external sink reproduces the same high-magnitude direction an

Load-bearing premise

The premise that carries the whole method is that a fixed register curated from reference images is universal—that the same few average tokens will absorb outliers for any test image and that the tokens deleted from the test image are always the semantically meaningless ones; if this universality breaks on inputs unlike the reference images, the method's accuracy gains (especially the largest ones) should disappear.

What would settle it

Take a vision encoder and a set of out-of-distribution images (e.g., medical scans, satellite imagery, or synthetic graphics) and measure, at the quantization-sensitive layer, (a) the mean cosine similarity between the test image's top-ℓ∞-norm tokens and the register tokens curated from natural images, and (b) quantized accuracy with and without RegCache. If the cosine similarity drops well below ~0.89 and the accuracy gain collapses while the unmodified outlier magnitude stays high, the universality claim is falsified; conversely, if the register still absorbs outliers and deletes only backgr

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At 4-bit and 6-bit precision, combining RegCache with existing quantization algorithms recovers large fractions of full-precision accuracy across five vision encoders; in several cases accuracy improves by tens of points over the same algorithm without RegCache.
  • The benefit extends beyond classification: zero-shot image-text retrieval accuracy improves consistently, and RegCache also helps when only weights are quantized.
  • Because the method is training-free and only adds or deletes a few tokens, the reported FLOPs overhead is at most about 0.2% and is often negative, with modest latency increases.
  • A prefix searched on one reference task transfers to other classification datasets without retuning, suggesting the register is a general feature of the encoder rather than a memorized artifact for a single benchmark.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next test, not run in the paper, is whether the same universal register survives domain shift: if a prefix curated on natural photographs is applied to encoders receiving medical, satellite, or synthetic images, the method's accuracy gain is the falsifiable prediction.
  • The paper's tokenization perspective implies a design principle: vision encoders pretrained with designated register tokens may be intrinsically easier to quantize, because sink tokens would emerge in early layers and can be handled like language-model sinks; this connects quantization research to architectural pretraining choices.
  • The authors note that the number of prefix copies and the deletion count are tuned per encoder; an extension would be to predict these hyperparameters from layerwise norm statistics, removing the main practical friction.
  • An untested but plausible corollary is that the same add-and-delete mechanism transfers to other non-autoregressive transformer families, such as diffusion generators, where activation ranges also bottleneck low-bit inference.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes RegCache, a training-free post-training activation-quantization plugin for vision transformers. The method has three steps: (1) curating register-candidate tokens as the highest-l_infinity-norm tokens at the quantization-sensitive layer on a reference image pool (50k ImageNet-1k training images); (2) caching the averaged KV values of these tokens as prefixes inserted from the sensitive layer onward (in some cases only at the sensitive layer); and (3) deleting the top-k~ highest-l_infinity-norm tokens at the sensitive layer at test time. The authors report consistent gains over PTQ4ViT, RepQ-ViT, NoisyQuant, and FIMA-Q across five vision encoders at 4/6/8-bit, plus retrieval results on MS-COCO and transfer to four other classification datasets. The central explanatory claim is that outlier tokens in middle layers of vision encoders are universal, semantically meaningless registers, so prefixed register caches absorb outliers and improve quantized accuracy.

Significance. If the central claim is upheld, RegCache would be a practically useful and inexpensive plug-in for low-bit vision-encoder quantization, with negligible overhead. The empirical breadth is a genuine strength: five different encoders, three base quantization methods, multiple bitwidths, retrieval and classification tasks, additional integrations with FIMA-Q, QuaRot, and AWQ, and cross-dataset transfer. The observation that middle-layer outlier tokens are similar across images is also interesting and worth further study. However, the paper's own component ablation (Table 6) does not support the register-centric interpretation, and the universality evidence is thin and tied to ImageNet-like data. The practical combined method appears to work, but the explanatory mechanism needs substantially more support before the claims in the abstract and Sec. 4 are established.

major comments (3)
  1. [Sec. 4.2/4.3 and Table 6] The central claim that precomputed universal register tokens are the active ingredient is not supported by the ablation. In Table 6, for SigLIP2 the prefix-caching component alone is worse than the baseline (23.82 vs 26.04) while token deletion alone gives most of the gain (69.06); for SigLIP, token deletion alone is far worse than baseline (42.41 vs 69.71) while prefix caching alone nearly matches the full method (74.37 vs 74.38). The combined method is effective in both cases, but the two components are not acting synergistically in the way the text claims: for each architecture, one component dominates and the other can be harmful. The paper needs a control that isolates the effect of the register content, e.g., random tokens, zero tokens, or a non-register average of normal tokens inserted as the same KV prefix, and an explicit comparison of the full method against deletion-only with
  2. [Sec. 4.1-4.3, Table 1, Table 4] The universality of the registers is the load-bearing assumption of the method, but the evidence is weak. Table 1 measures cosine similarity on only 64 ImageNet-1k validation images, and all hyperparameters (l_q, k, tau*, k~, insertion range) are selected on the ImageNet-1k training split, with the main headline being ImageNet-1k validation accuracy. The cross-dataset evaluation in Table 4 uses natural-image classification benchmarks that share ImageNet-like statistics and does not test far out-of-distribution inputs. Since the paper explicitly claims a 'universal middle-layer register for any input image at test phase' (Sec. 1), the authors should test the precomputed registers on a substantially different distribution (e.g., medical images, sketches, rendered scenes, or adversarially cropped data) and report whether the gains persist when the reference task is not ImageNet. As written,
  3. [Sec. 5.2, Table 6 and Limitations] The paper's own limitation statement says the number of prefix and deleted tokens 'must be selected heuristically for each vision encoder and base quantization algorithm.' Table 6 makes this more consequential: the sign of the contribution of each component flips across architectures, so the method is not a single robust mechanism but a combination whose components can each be harmful in isolation. The paper should report the sensitivity of the final result to tau* and k~ over a grid (at least for one representative architecture), and should present the full method's result against a simple per-token clipping or top-k-token-zeroing baseline. This is necessary to support the practical claim that the method is a reliable on-top module rather than a selected operating point.
minor comments (3)
  1. [Eq. (1) and Sec. 4.1] The notation for S = argtopk{... | z in Phi_{l_q}(x), for some x in I_ref} is unclear: it does not specify whether the top-k selection is over the union of all tokens from all reference images or per-image, and how ties/deduplication are handled. Please define the selection pool precisely.
  2. [Sec. 3.1 and Appendix] 'FC2 input' is used without definition. Define it (e.g., the input to the second fully-connected layer of the MLP block) at first use. Also, the caption of Table 12 says latency in 'm/s' but the values are reported as minutes and seconds; this should be fixed.
  3. [Sec. B, Lemma 1] Lemma 1 is a mathematical tautology about adding a large one-hot component to any two vectors; it does not by itself explain why actual outlier tokens have aligned channels. The paper should present direct evidence of channel overlap (e.g., the fraction of shared top-l_infinity channels across images), since that is the empirical premise on which the lemma's applicability rests.

Circularity Check

0 steps flagged

No meaningful circularity: RegCache's gains are empirical; the only mild issue is a self-cited LLM-prefixing seed and reference-task hyperparameter selection on ImageNet-1k train, neither of which forces the reported results.

full rationale

The paper's central claim is not derived by construction. RegCache curates register tokens by top-ℓ∞ norm at a quantization-sensitive layer, caches their KVs, and deletes the largest-norm tokens; whether this improves quantized accuracy is tested empirically across CLIP, OpenCLIP, SigLIP, SigLIP2, and DINOv2 on ImageNet-1k, MS-COCO, Cars, Flowers, Food-101, and CIFAR-100. No equation in the paper defines the reported accuracy in terms of the fitted quantities. The theoretical Lemma 1 in Sec. B explains why top-norm tokens are mutually similar once they share high-magnitude channels, but it is presented as an explanation of a measured observation (Tab. 1), not as a prediction of quantization performance. The main self-citation is [31], which supplies the LLM-side observation that prefixing attention sinks mitigates outliers; the present paper explicitly tests whether this transfers to vision encoders and shows non-trivial, architecture-dependent effects (e.g., Tab. 6: prefix caching alone can hurt SigLIP2), so the vision claim is not forced by the citation. Hyperparameters (sensitive layer, τ*, k̃) are selected on the ImageNet-1k training split, and the headline ImageNet-1k number is on the validation split, so this is standard model selection rather than fitting the reported quantity. The paper's own Limitations section (Sec. 6) acknowledges that these hyperparameters must be tuned per encoder and quantizer; that is a generality limitation, not a circularity. The reduction in max token norm (Tab. 5) is partly a mechanical consequence of deleting high-norm tokens, but the central accuracy gains are not derived from that statistic and are cross-validated on held-out datasets.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

The method is empirical; it contributes two empirical claims (middle-layer outliers; universal sink tokens) plus two search knobs (τ*, k̃). The 'theory' (Lemma 1) is a trivial asymptotic fact. The main free parameters are tuned on ImageNet-1k train, so the headline numbers are partially selected, not predicted.

free parameters (5)
  • k (number of register candidates) = 100
    Top-ℓ∞ tokens curated from 50k ImageNet-1k reference images (Sec. 4.1). Hand-chosen, not searched.
  • τ* (number of prefix copies) = searched in {1,...,15}
    Selected by highest reference-task accuracy on ImageNet-1k train (Sec. 4.2). Exact optimal values per model not reported.
  • k̃ (number of tokens deleted) = tuned per model/quantizer
    Number of top-ℓ∞ tokens removed at the sensitive layer, tuned on reference task (Sec. 4.3).
  • l_q (quantization-sensitive layer) = per model, e.g., block 5-8
    Layer selected by layerwise W8A8 sensitivity on reference task (Sec. 3.1, 4.1).
  • reference pool size / insertion range = 50,000 images; insert from l_q to final layer
    Pool size chosen; insertion range ('up to three preceding blocks') is heuristic (Sec. 4.1, 5.1).
axioms (4)
  • domain assumption The ImageNet classification reference task is representative of the deployment distribution; hyperparameters tuned on it transfer.
    Sec. 4.1-4.3; limitations admit heuristic search per model/quantizer.
  • domain assumption The highest-ℓ∞ tokens at the sensitive layer are universal registers (cosine similarity 0.89, Tab. 1) and semantically meaningless, so deleting them does not hurt downstream accuracy.
    Sec. 3.3, 4.3; Fig. 6.
  • domain assumption Layerwise W8A8 sensitivity identifies the block where outliers drive quantization loss, so prefix insertion must begin before this block.
    Sec. 3.1, Fig. 2; Sec. L.
  • standard math Lemma 1 (large additive coordinate dominates cosine similarity) is sufficient to explain cross-image universality.
    Sec. B, Lemma 1.
invented entities (1)
  • Precomputed register prefix tokens (averaged KV cache of reference-image outlier tokens) no independent evidence
    purpose: Act as attention sinks inserted into middle/final layers to absorb activation outliers during quantized inference
    These tokens are an algorithmic artifact curated by the method, not an independently measurable entity; their effect is observable only through the quantized model's output.

pith-pipeline@v1.3.0-alltime-deepseek · 19239 in / 16382 out tokens · 118286 ms · 2026-08-04T11:26:29.919662+00:00 · methodology

0 comments
read the original abstract

Large pretrained vision encoders are central to multimodal intelligence, powering applications from on-device vision processing to vision-language models. Since these applications often demand real-time processing of massive visual data, reducing the inference cost of vision encoders is critical. Quantization offers a practical path, but it remains challenging even at 8-bit precision due to so-called outliers. In this work, we propose $\textit{RegCache}$, a training-free algorithm that mitigates outliers in large-scale pretrained vision encoders and serves as a plug-in module that can be applied on top of other quantization methods. RegCache introduces outlier-prone yet semantically meaningless prefix tokens to the vision encoder, which prevent other tokens from having outliers. Notably, we observe that outliers in vision encoders behave differently from those in language models, motivating two technical innovations: middle-layer prefixing and token deletion. Experimental results show that our method consistently improves quantized model performance across various vision encoders, particularly in extremely low-bit regimes (e.g., 4-bit).

Figures

Figures reproduced from arXiv: 2510.04547 by Jaeho Lee, Jinho Kim, Kyuyeun Kim, SeungHyeon Kim, Taesun Yeom, Wonpyo Park.

Figure 1
Figure 1. Figure 1: (Left) Sink tokens in LLMs vs. vision encoders. In LLMs, well-known sink tokens exist in a closed-set vocabulary. In contrast, vision encoders take image inputs composed of diverse patches that are continuously mapped into an embedding space, making the dis￾covery of sink tokens more challenging. (Right) Activation magnitudes at the input of the 8th layer of CLIP-B/16, with and without RegCache (ImageNet-1… view at source ↗
Figure 2
Figure 2. Figure 2: (Top) Layerwise quantization sensitivity (%). We plot the zero-shot ImageNet-1k accuracy of various vision encoders when we quantize only one layer to W8A8. (Bottom) Maximum norm of the FC2 layer input tokens for each layer. We plot the largest ℓ∞-norm of all tokens in an image in a logarithmic scale, averaged over the ImageNet-1k validation set. The layer which is sensitive to quantization coincides with … view at source ↗
Figure 3
Figure 3. Figure 3: Emergence of outliers in foreground- and background-only images on SigLIP-B/16. Note that the curves for “original” and “BG-only” almost overlap. case of LLMs, where some tokens are clearly meaningless even at the first glance, e.g., ⟨BOS⟩, ⟨SEP⟩ (see [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the proposed method. We identify a universal register by analyzing the inputs of quantization-sensitive layers across blocks. During inference, the register is inserted into each block, and outlier tokens are removed from the most quantization-sensitive layer. Precisely, given some test image xtest, we select the tokens with the top-˜k ℓ∞ norm, i.e., D = argtop˜k  ∥z∥∞ [PITH_FULL_IMAGE:figure… view at source ↗
Figure 5
Figure 5. Figure 5: (Top) Layerwise quantization sensitivity (%). Zero-shot ImageNet-1k accuracy when we quantize only one layer to W8A8. (Bottom) Layerwise max token norms. The largest ℓ∞-norm of all tokens in an image on a logarithmic scale, averaged over the ImageNet￾1k validation set [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of register token searched from ImageNet-1k [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The frequency of top-1 max tokens in the input tensor of FC2 layers in different models. We evaluated DINOv2 on ImageNet￾1K and LLaMA3-8B on Wikitext-2 dataset [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Performance across prefix-insertion layers(%). The x-axis denotes the layer index where prefix insertion begins and the y-axis indicates zero-shot ImageNet-1k accuracy under W8A8 quantization. 7There is one exception: we do not use the Walsh–Hadamard matrix for the value and output projection matrices, as in the ViDiT-Q implementation. Also, we have empirically observed that na¨ıvely applying QuaRot on vis… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs

    cs.LG 2026-04 unverdicted novelty 7.0

    Sink-Token-aware Pruning (SToP) suppresses semantically uninformative sink tokens during visual token pruning in Video LLMs, boosting fine-grained performance even at 90% pruning rates across hallucination, reasoning,...

  2. Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs

    cs.LG 2026-04 conditional novelty 6.0

    Sink-Token-aware Pruning (SToP) uses a sink score to suppress attention-sink tokens during visual token pruning, improving fine-grained video understanding in Video LLMs at high pruning rates.

Reference graph

Works this paper leans on

45 extracted references · 7 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Quarot: Outlier- free 4-bit inference in rotated llms.Advances in Neural In- formation Processing Systems, 2024

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alis- tarh, Torsten Hoefler, and James Hensman. Quarot: Outlier- free 4-bit inference in rotated llms.Advances in Neural In- formation Processing Systems, 2024. 18

    2024

  2. [2]

    Understanding and overcoming the chal- lenges of efficient transformer quantization

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the chal- lenges of efficient transformer quantization. InEmpirical Methods in Natural Language Processing, 2021. 2

    2021

  3. [3]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. InEuropean conference on computer vision, 2014. 6

    2014

  4. [4]

    Prefixquant: Eliminating outliers by pre- fixed tokens for large language models quantization.arXiv preprint arXiv:2410.05265, 2024

    Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. Prefixquant: Eliminating outliers by pre- fixed tokens for large language models quantization.arXiv preprint arXiv:2410.05265, 2024. 1

    Pith/arXiv arXiv 2024

  5. [5]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023. 6

    2023

  6. [6]

    Low-bit quantization of neural networks for efficient infer- ence

    Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient infer- ence. InIEEE/CVF International Conference on Computer Vision Workshop, 2019. 1

    2019

  7. [7]

    Vision transformers need registers

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. InInterna- tional Conference on Learning Representations, 2024. 2, 3, 4, 16, 17

    2024

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InIEEE conference on computer vision and pat- tern recognition, 2009. 6

    2009

  9. [9]

    GPT3.int8(): 8-bit matrix multiplication for transformers at scale

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. InAdvances in neural information pro- cessing systems, 2022. 1, 2

    2022

  10. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representa- tions, 2021. 1

    2021

  11. [11]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhishek Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Aobo Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  12. [12]

    When attention sink emerges in language models: An empirical view

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. InIn- ternational Conference on Learning Representations, 2025. 17

    2025

  13. [13]

    Atten- tion score is not all you need for token importance indica- tor in kv cache reduction: Value also matters

    Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Atten- tion score is not all you need for token importance indica- tor in kv cache reduction: Value also matters. InEmpirical Methods in Natural Language Processing, 2024. 3

    2024

  14. [14]

    Vision transformers don’t need trained registers

    Nicholas Jiang, Amil Dravid, Alexei A Efros, and Yossi Gan- delsman. Vision transformers don’t need trained registers. In Advances in neural information processing systems, 2025. 3, 4, 16, 19

    2025

  15. [15]

    See what you are told: Visual attention sink in large multimodal models

    Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. InInternational Conference on Learn- ing Representations, 2025. 3

    2025

  16. [16]

    Openvla: An open- source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open- source vision-language-action model. InProceedings of The 8th Conference on ...

    2025

  17. [17]

    BERT busters: Outlier dimensions that disrupt transformers

    Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. BERT busters: Outlier dimensions that disrupt transformers. InFindings of the ACL, 2021. 2

    2021

  18. [18]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on com- puter vision workshops, 2013. 6

    2013

  19. [19]

    Repq- vit: Scale reparameterization for post-training quantization of vision transformers

    Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq- vit: Scale reparameterization for post-training quantization of vision transformers. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, 2023. 3, 6, 13

    2023

  20. [20]

    AWQ: Activation-aware weight quantization for on-device LLM compression and acceler- ation

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for on-device LLM compression and acceler- ation. InProceedings of Machine Learning and Systems,

  21. [21]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, 2014. 6

    2014

  22. [22]

    Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers

    Yijiang Liu, Huanrui Yang, Zhen Dong, Kurt Keutzer, Li Du, and Shanghang Zhang. Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 3, 6, 13

    2023

  23. [23]

    Post-training quantization for vision trans- former.Advances in Neural Information Processing Systems,

    Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision trans- former.Advances in Neural Information Processing Systems,

  24. [24]

    Spin- quant: LLM quantization with learned rotations

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge So- ran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spin- quant: LLM quantization with learned rotations. InInterna- tional Conference on Learning Representations, 2025. 18

    2025

  25. [25]

    Artifacts and attention sinks: Structured ap- proximations for efficient vision transformers.arXiv preprint arXiv:2507.16018, 2025

    Andrew Lu, Wentinn Liao, Liuhui Wang, Huzheng Yang, and Jianbo Shi. Artifacts and attention sinks: Structured ap- proximations for efficient vision transformers.arXiv preprint arXiv:2507.16018, 2025. 3

    Pith/arXiv arXiv 2025

  26. [26]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. InIn- dian conference on computer vision, graphics & image pro- cessing, 2008. 6

    2008

  27. [27]

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Je- gou, Julien Mairal, Patr...

    2024

  28. [28]

    Balamurugan Palanisamy, Vikas Hassija, Arpita Chatterjee, Arpita Mandal, Debanshi Chakraborty, Amit Pandey, G. S. S. Chalapathi, and Dhruv Kumar. Transformers for vision: A survey on innovative methods for computer vision.IEEE Access, 13:95496–95523, 2025. 1

    2025

  29. [29]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, 2021. 1, 6

    2021

  30. [30]

    Dinov3.arXiv preprint arXiv:2508.10104, 2025

    Oriane Sim ´eoni, Huy V V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Micha ¨el Ramamonjisoa, et al. Dinov3.arXiv preprint arXiv:2508.10104, 2025. 2

    Pith/arXiv arXiv 2025

  31. [31]

    Prefixing attention sinks can mitigate activation outliers for large language model quantization

    Seungwoo Son, Wonpyo Park, Woohyun Han, Kyuyeun Kim, and Jaeho Lee. Prefixing attention sinks can mitigate activation outliers for large language model quantization. In Empirical Methods in Natural Language Processing, 2024. 1, 2, 3, 5

    2024

  32. [32]

    Massive activations in large language models

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. InConference on Language Modeling, 2024. 1, 2, 3, 17

    2024

  33. [33]

    All bark and no bite: Rogue dimensions in transformer language models obscure representational quality

    William Timkey and Marten Van Schijndel. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. InEmpirical Methods in Natural Language Processing, 2021. 2

    2021

  34. [34]

    Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025. 6

    Pith/arXiv arXiv 2025

  35. [35]

    Fima-q: Post-training quantization for vi- sion transformers by fisher information matrix approxima- tion

    Zhuguanyu Wu, Shihe Wang, Jiayi Zhang, Jiaxin Chen, and Yunhong Wang. Fima-q: Post-training quantization for vi- sion transformers by fisher information matrix approxima- tion. InProceedings of the Computer Vision and Pattern Recognition Conference, 2025. 3, 8, 13

    2025

  36. [36]

    Aphq-vit: Post-training quan- tization with average perturbation hessian based reconstruc- tion for vision transformers

    Zhuguanyu Wu, Jiayi Zhang, Jiaxin Chen, Jinyang Guo, Di Huang, and Yunhong Wang. Aphq-vit: Post-training quan- tization with average perturbation hessian based reconstruc- tion for vision transformers. InProceedings of the Computer Vision and Pattern Recognition Conference, 2025. 3

    2025

  37. [37]

    SmoothQuant: Accurate and ef- ficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and ef- ficient post-training quantization for large language models. InInternational conference on Machine Learning, 2023. 1

    2023

  38. [38]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InInternational Conference on Learning Representations, 2024. 1, 3

    2024

  39. [39]

    Noise or signal: The role of image back- grounds in object recognition

    Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image back- grounds in object recognition. InInternational Conference on Learning Representations, 2021. 4

    2021

  40. [40]

    Mitigating quantization errors due to activation spikes in GLU-based LLMs.arXiv preprint 2405.14428, 2024

    Jaewoo Yang, Hayun Kim, and Younghoon Kim. Mitigating quantization errors due to activation spikes in GLU-based LLMs.arXiv preprint 2405.14428, 2024. 1

    Pith/arXiv arXiv 2024

  41. [41]

    Dopq- vit: Towards distribution-friendly and outlier-aware post- training quantization for vision transformers.arXiv preprint arXiv:2408.03291, 2024

    Lianwei Yang, Haisong Gong, and Qingyi Gu. Dopq- vit: Towards distribution-friendly and outlier-aware post- training quantization for vision transformers.arXiv preprint arXiv:2408.03291, 2024. 3

    Pith/arXiv arXiv 2024

  42. [42]

    Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization

    Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization. InEuropean conference on computer vision, 2022. 3, 6, 13

    2022

  43. [43]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, 2023. 6

    2023

  44. [44]

    w/ outliers

    Tianchen Zhao, Tongcheng Fang, Haofeng Huang, Rui Wan, Widyadewi Soedarmadji, Enshu Liu, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation. InIn- ternational Conference on Learning Representations, 2025. 18 Activation...

    arXiv 2025

  45. [45]

    7 As reported in Tab

    to apply Hadamard rotations to all linear layers in the transformer blocks. 7 As reported in Tab. 13, RegCache improves accuracy when used as a plug-in method, indicating that suppressing extreme outliers in the rotated domain can further enhance quantization performance. Table 13. Zero-shot image classification accuracy (%) on top of QuaRot [1] CLIP Open...