REVIEW 4 major objections 4 minor 1 cited by
ASCENT-ViT: Attention-based Scale-aware Concept Learning Framework for Enhanced Alignment in Vision Transformers
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Multiscale fusion boosts ViT concept explanations
desk verdict A useful new combination of multiscale CNN features and deformable attention for concept alignment, but the scale-awareness claim needs a matched-capacity control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Deformable Multi-Scale Fusion (DMSF) module, whose adaptive composition equation $z = \mathrm{norm}(z_q + \psi \cdot I \cdot \mathrm{MSDA}(z_q, p_q, c))$ is what actually injects scale awareness into the ViT. The Multi-Scale Deformable Attention (MSDA) operation uses the ViT's patch embeddings as queries and the CNN feature pyramid's multi-scale maps as keys and values, sampling reference points across scales so that each patch token can gather information from corresponding locations at different resolutions. The learnable vector $I$ (per token) and the scalar hyperparameter $\psi$ control the strength of the scale injection, and the layer norm keeps the fused vector in the same embedding space as the original patch tokens. This composite vector is then fed to the Concept-Representation Alignment Module (CRAM), which computes spatial and global attention matrices against human concept annotations to produce both the prediction and the explanation.
What would settle it
Replace the MSE module's multi-scale features with random noise of the same shapes, or retrain with a single fixed scale (e.g., only 1/16): if pixel-level concept TPR and accuracy on CUB200 or AWA2 do not drop materially, then the scale composition itself is not what drives the reported improvements.
Extended reading notes
Core claim
The paper's central claim is that the misalignment between ViT representations and human concept annotations is partly a scale problem, and that it can be repaired by composing multi-scale features into the patch stream. ASCENT-ViT does this in three stages: a Multi-scale Encoding module extracts features at 1/8, 1/16, and 1/32 resolution from a lightweight CNN; a Deformable Multi-Scale Fusion module treats the ViT's patch embeddings as queries for multi-scale deformable attention, producing $z = \mathrm{norm}(z_q + \psi \cdot I \cdot \mathrm{MSDA}(z_q, p_q, c))$; and a Concept-Representation Alignment Module aligns this composite representation with spatial and global concept annotations via two attention matrices. On CUB200, AWA2, KITS, Concept-MNIST, and Pascal aPY, the paper reports that ASCENT-ViT outperforms the attention-only CRAM baseline on both task accuracy and concept metrics, with the largest gains on backbones that lack built-in scale invariance.
Load-bearing premise
The entire gain rests on the assumption that a randomly initialized, lightweight CNN's three fixed-scale feature maps align semantically with the pretrained ViT's patch embeddings, so that the deformable fusion enriches the patch tokens instead of corrupting them.
Editorial extensions
If this is right
- ViT backbones can acquire CNN-like scale awareness in their explanations without architectural changes to the pretrained encoder.
- Concept explanations become more stable under rotations, crops, and zoom, because scale features anchor attention across transformations.
- The framework acts as a drop-in classification head, adding under roughly 3% parameters over the backbone.
- Gains over the attention-only baseline are largest for plain ViTs (positional bias only), while SWIN, which already has CNN-like inductive bias, shows roughly parity.
- Test-time interventions on learned concepts correct more mispredictions than attention-only or concept-bottleneck baselines.
Reading between the lines
- One extension the paper does not explore: the per-patch learnable vector $I$ could be visualized to reveal which patches accept scale information, turning the fusion into a diagnostic of where the ViT lacks scale awareness.
- Because the CNN encoder is randomly initialized and trained end-to-end, a natural test is to replace it with features drawn from the ViT's own intermediate layers; if the gains vanish, the benefit likely comes from the CNN's translation equivariance rather than from scale diversity per se.
- The fixed scale set (1/8, 1/16, 1/32) is likely dataset-dependent; systematic sweeps could show whether the method's benefit is robust to scale choices or tied to the natural object sizes in CUB200 and AWA2.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ASCENT-ViT, a plug-in concept-based explainability module for Vision Transformers that combines a Multi-scale Encoding (MSE) module (a random-initialized CNN producing features at three scales), a Deformable Multi-Scale Fusion (DMSF) module (deformable attention composing MSE features with ViT patch embeddings), and a Concept-Representation Alignment Module (CRAM) that aligns the fused representations with human-annotated concepts through spatial and global attention matrices. The authors evaluate ASCENT-ViT against a CRAM-only baseline (similar to Rigotti et al.) and CBM on CUB200, AWA2, KITS, Concept-MNIST, and Pascal aPY, across ViT-Base, ViT-Large, and SWIN backbones, reporting modest gains in classification accuracy and concept alignment metrics.
Significance. If the central attribution holds, the paper makes a useful contribution by introducing scale-aware inductive biases into concept-based explainability for ViTs, which is a novel direction relative to model-agnostic attention-only modules. The manuscript is also commendable for reporting results across multiple backbone architectures, providing detailed implementation descriptions, and making an anonymous code link available. The reported gains on CUB200 and KITS are consistent and plausible. However, the load-bearing claim that the multi-scale composition specifically causes the improvements is not yet adequately supported by the experimental design, and several reproducibility issues need to be resolved.
major comments (4)
- [§4.5 Table 2 and §4.3] The central attribution of the gains to 'scale awareness' is underdetermined. Table 2 adds MSE and DMSF sequentially, but no control experiment varies the content of the multi-scale features while holding capacity and hyperparameters fixed. Since the DMSF module contains 16 attention heads, 4 reference points, learnable offsets, a learnable I, and per-dataset tuned ψ, the improvements over 'Only CRAM' could plausibly come from added learnable capacity or better optimization rather than from the semantic scale pyramid. The authors should add at least one control: (i) untrained or randomly projected CNN features, (ii) noise features of the same shape, or (iii) a matched-capacity non-scale fusion module (e.g., a fully learnable projection of zq with the same parameter count). Without such a control, the paper's opening claim that multi-scale fusion improves accuracy and concept learning is not established.
- [§4.3 and Appendix A.1] The reported feature sizes in the MSE module are internally inconsistent. The text states scales 1/8, 1/16, and 1/32 correspond to c1=1024, c2=196, and c3=16, and that the concatenated size is 1029, but 1024+196+16=1236, while the stated scales for a 224×224 input would give flattened sizes 784, 196, and 49 (sum 1029). This discrepancy makes the exact configuration of the MSE module ambiguous and hinders reproduction. Please clarify the actual feature map dimensions (including any pooling or projection steps) and correct the summary size.
- [§4.1] The AWA2 concept annotations are generated with SAM v2 for 10 selected concepts, and the KITS concepts are derived from metadata or masks, but no validation of these machine-generated labels is reported. Because the Px TPR on AWA2 and KITS is a key part of the concept-alignment evidence, the model may be aligning to SAM artifacts rather than to human-understandable concepts. The authors should validate the generated labels on a held-out subset (e.g., IoU against manual annotations or a human study) or temper the claims for these datasets.
- [Table 1] Several of the reported differences between ASCENT-ViT and Only CRAM are within one standard deviation over three seeds, such as AWA2 ViT-Base accuracy (62.3±0.1 vs 62.3±0.2), AWA2 Px TPR (64.11±1.2 vs 64.02±0.5), Concept-MNIST digit classification (95.83±0.1 vs 95.71±0.1), and Pascal aPY accuracy (81.4±0.8 vs 81.0±0.8). The paper claims that ASCENT-ViT 'outperforms' on all datasets, but no significance tests or confidence intervals are provided, and Table 2 is a single run. Please report paired significance tests or effect sizes and adjust the strength of the claims accordingly, especially for datasets where the gains are within noise.
minor comments (4)
- [§4.2] The text says that for CUB200 the concept errors are evaluated with the Frobenius norm of the difference between ground-truth annotations and predicted concept attention matrices, citing the appendix, but the appendix does not include these CUB concept-error results; only Pascal concept errors appear in Table 6. Please either add the CUB results or remove the reference.
- [§4.4] The statement 'pixel-wise TPR improves by ∼4% on CUB200 and ∼5%' is not tied to a specific backbone; Table 1 shows the improvements on CUB200 range from 1.68 (ViT-Base) to 3.66 (ViT-Large), and only ViT-Large on AWA2 reaches near 5%. Please specify which backbone(s) the percentages refer to.
- [§4.3] The hyperparameter ψ is reported per dataset (1 for CUB, 2 for Concept-MNIST and Pascal aPY, 0.5 for AWA2 and KITS), but the paper does not state how these values were selected (e.g., via a validation split). Since ψ directly controls the strength of the external features, this detail is important for reproducibility and for assessing the per-dataset tuning concern.
- [Abstract / code link] The code link is an anonymous OpenScience link without a commit hash or version tag. For a methods paper that makes architectural contributions, a pinned code release (or an appendix with exact model configurations) is necessary to ensure reproducibility.
Circularity Check
No circularity: the concept-alignment quantities are trained and measured on separate data, and the reported gains are not defined into the inputs.
full rationale
The derivation chain is self-contained. MSE computes multi-scale features (Eqs. 1-2), DMSF composes them with ViT patch embeddings via MSDA (Eqs. 3-4), and CRAM forms attention between the composed representation z and concept annotations C (Eqs. 5-8). The training objective (Eq. 10) is classification loss plus a Frobenius alignment term ||ASpatial - H||F, while the reported concept metrics (Px TPR, pixel accuracy, misclassification error) are evaluated on held-out test annotations. Measuring how well the learned attention matrix predicts held-out human annotations is therefore a genuine predictive evaluation, not a re-statement of the training fit. The spatial concept metric does coincide with the quantity supervised in Eq. 10, but that is the intended supervised concept-learning task rather than a circular reduction. Self-citations, chiefly the C-MNIST dataset from Sinha et al. 2023, are not load-bearing: the central comparisons are against Rigotti et al.'s 'Only CRAM' and CBM baselines, and C-MNIST is only one of five datasets. The skeptic's concern that gains may be due to added parameters and per-dataset tuning is an experimental confound and underdetermination, not circularity. The reported feature-size inconsistency (c1=1024, c2=196, c3=16 summing to 1236 rather than the stated 1029) is a reproducibility defect, not a circular step. No load-bearing argument reduces to its own input or to an unverified self-citation chain.
Assumptions & free parameters
free parameters (4)
- psi (psi) =
1 for CUB, 2 for Concept-MNIST and Pascal aPY, 0.5 for AWA2 and KITS
- I initialization =
0.01
- S (number of scales) =
3 (1/8, 1/16, 1/32)
- lambda (explanation loss weight) =
1.0
assumptions (4)
- standard math Softmax attention and MSDA definitions from prior work are taken as given
- domain assumption Human-annotated concepts and localization maps are valid ground truth for training and evaluating concept alignment
- domain assumption SAM-generated annotations for AWA2 and KITS are reliable proxies for human concepts
- ad hoc to paper The residual composition z = norm(zq + psi * I * MSDA(...)) preserves patch semantics while adding scale awareness
Cite this review
Pith. "Pith review of ASCENT-ViT: Attention-based Scale-aware Concept Learning Framework for Enhanced Alignment in Vision Transformers." pith.science (2026). https://pith.science/paper/U54DINTT
@misc{pith2026250109221,
author = {Pith},
title = {Pith review of: ASCENT-ViT: Attention-based Scale-aware Concept Learning Framework for Enhanced Alignment in Vision Transformers},
year = {2026},
howpublished = {\url{https://pith.science/paper/U54DINTT}},
note = {Machine review of arXiv:2501.09221}
}
read the original abstract
As Vision Transformers (ViTs) are increasingly adopted in sensitive vision applications, there is a growing demand for improved interpretability. This has led to efforts to forward-align these models with carefully annotated abstract, human-understandable semantic entities - concepts. Concepts provide global rationales to the model predictions and can be quickly understood/intervened on by domain experts. Most current research focuses on designing model-agnostic, plug-and-play generic concept-based explainability modules that do not incorporate the inner workings of foundation models (e.g., inductive biases, scale invariance, etc.) during training. To alleviate this issue for ViTs, in this paper, we propose ASCENT-ViT, an attention-based, concept learning framework that effectively composes scale and position-aware representations from multiscale feature pyramids and ViT patch representations, respectively. Further, these representations are aligned with concept annotations through attention matrices - which incorporate spatial and global (semantic) concepts. ASCENT-ViT can be utilized as a classification head on top of standard ViT backbones for improved predictive performance and accurate and robust concept explanations as demonstrated on five datasets, including three widely used benchmarks (CUB, Pascal APY, Concept-MNIST) and 2 real-world datasets (AWA2, KITS).
Figures
Figures from the paper (8 more)
Forward citations
Cited by 1 Pith paper
-
Integrating attention into explanation frameworks for language and vision transformers
Two attention-based explanation methods are introduced: an attention-weighted Shapley decomposition for local attributions and attention-informed concept sensitivity for global explanations.
Reference graph
Works this paper leans on
-
[6]
Describing objects by their attributes
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE con- ference on computer vision and pattern recognition, pages 1778–1785. IEEE,
work page 2009
-
[7]
Ast: Audio spectrogram transformer
Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778,
-
[9]
Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L Griffiths. Are convolutional neural networks or trans- formers more like human vision? arXiv preprint arXiv:2105.07197,
-
[11]
Deformable detr: Deformable trans- formers for end-to-end object detection
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable trans- formers for end-to-end object detection. arXiv preprint arXiv:2010.04159,
arXiv 2010
-
[12]
We set the patch size to correspond to 16x16 pixels
A Appendix The Appendix is organized as follows: • Implementation Details • Effect of Model Type • Overhead of explainability modules • Visual comparison across ViT architectures • Robustness Analysis • Additional Results on Pascal dataset • Additional Results using ASCENT-ViT • Intervention on Test Datasets A.1 Implementation Details ViT Backbone. We set...
work page 2021
-
[1029]
The first convolu- tion block consists of three convolution blocks with the batch norm and ReLU activations followed by the Max pooling op- eration. The following convolution blocks consist of a single convolution layer of kernel size=3 and stride=2, as well as the batch norm and ReLU activations. DMSF Module. We utilize Multi-scale Deformable Atten- tion...
work page 2020
-
[2017]
The caltech-ucsd birds-200-2011 dataset
Catherine Wah, Steve Branson, Peter Welinder, Pietro Per- ona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset
work page 2011
-
[2018]
Image transformer for explainable au- tonomous driving system
Jiqian Dong, Sikai Chen, Shuya Zong, Tiantian Chen, and Samuel Labi. Image transformer for explainable au- tonomous driving system. In 2021 IEEE International In- telligent Transportation Systems Conference (ITSC), pages 2732–2737. IEEE,
work page 2021
Show all 13 references
-
[2019]
Vision transformer adapter for dense predictions
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534,
-
[2020]
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,
-
[2021]
On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258,
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Alt- man, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258,
-
[2023]
Sit: Self-supervised vision transformer
Sara Atito, Muhammad Awais, and Josef Kittler. Sit: Self-supervised vision transformer. arXiv preprint arXiv:2104.03602,
-
[2024]
A self- explaining neural architecture for generalizable concept learning
Sanchit Sinha, Guangzhi Xiong, and Aidong Zhang. A self- explaining neural architecture for generalizable concept learning. arXiv preprint arXiv:2405.00349,
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.