{"id":"53357e6c-0db4-42a9-91aa-1935792c1969","arxiv_id":"2501.09221","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ASCENT-ViT aligns ViT image representations with human-annotated concepts using multi-scale features and deformable attention, improving both accuracy and concept localization.","lead":"A new module called ASCENT-ViT helps vision AI models explain their decisions by tying predictions to human-labeled concepts like wing color or tumor region, and by using image features at multiple scales. It could make transformer-based image classifiers more trustworthy in high-stakes settings such as medical imaging.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gains may reflect added capacity or per-dataset tuning rather than the claimed scale-aware mechanism; no control experiment isolates the multi-scale feature content.","rationale":"The reader's weakest assumption concerns whether a randomly initialized CNN encoder and deformable attention can inject useful scale awareness into ViT patch embeddings. My concern sharpens that assumption into a falsifiable control problem: the paper never demonstrates that semantic content of the multi-scale features matters, because every comparison that adds MSE/DMSF also adds parameters and a per-dataset-tuned ψ. The strongest evidence for the central claim would be an ablation that holds capacity and tuning constant while varying only the scale-feature content. Absent that, the claim that multi-scale fusion is the operative ingredient remains plausible but unproven. I do not think this overturns the CONDITIONAL verdict, because the empirical trend is consistent and the proposed test could confirm the mechanism, but it does reinforce the need for the control experiments before full acceptance. The internal feature-size inconsistency further supports the need for code or exact implementation details.","tokens_in":16836,"tokens_out":4944,"duration_ms":53464,"concrete_test":"On CUB200 with ViT-Base, retrain ASCENT-ViT with the MSE feature pyramid replaced by (a) a fixed random CNN that is not updated during training, and (b) a noise feature pyramid with the same shapes, keeping DMSF/CRAM and all hyperparameters identical. Also train a matched-capacity control in which MSDA is replaced by ordinary self-attention over patch tokens only, with no multi-scale image features. If either random-feature variant matches the reported 85.71 accuracy or 83.93 Px TPR within one standard deviation, the scale-aware mechanism is not the source of the gain; if the matched-capacity non-scale control matches, the confound is added capacity. Report all variants over at least 10 seeds with the same protocol as Table 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that composing multi-scale CNN features with ViT patch embeddings via DMSF is what improves concept alignment and accuracy. The evidence does not rule out the simpler explanation that the gains come from added learnable parameters and per-dataset hyperparameter tuning. In Eq. 4, z = norm(zq + ψ·I·MSDA(zq, pq, c)); I is initialized to 0.01 and ψ is tuned per dataset (1 for CUB, 2 for C-MNIST/Pascal, 0.5 for AWA2/KITS). MSDA has 16 heads, 4 reference points, learnable offsets and attention weights, and the MSE CNN is randomly initialized. Such a module can learn arbitrary spatial pooling even if the 'scale pyramid' is semantically meaningless. The ablations in Table 2 add the CNN (w/ CRAM+MSE) and then DMSF, but they never vary the content of the scale features: no untrained-feature control, no noise-feature control, and no matched-capacity non-scale fusion baseline. Therefore the paper's attribution of improved accuracy/Px TPR to 'scale awareness' is underdetermined; added capacity or better optimization could explain the trend. There is also an internal inconsistency in the reported feature sizes: c1=1024, c2=196, and c3=16 sum to 1236, not the stated 1029, while the stated scales 1/8, 1/16, and 1/32 imply flattened sizes 784, 196, and 49 (sum 1029). This makes the exact MSE configuration unclear and impedes reproduction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ASCENT-ViT, a plug-in concept-based explainability module for Vision Transformers that combines a Multi-scale Encoding (MSE) module (a random-initialized CNN producing features at three scales), a Deformable Multi-Scale Fusion (DMSF) module (deformable attention composing MSE features with ViT patch embeddings), and a Concept-Representation Alignment Module (CRAM) that aligns the fused representations with human-annotated concepts through spatial and global attention matrices. The authors evaluate ASCENT-ViT against a CRAM-only baseline (similar to Rigotti et al.) and CBM on CUB200, AWA2, KITS, Concept-MNIST, and Pascal aPY, across ViT-Base, ViT-Large, and SWIN backbones, reporting modest gains in classification accuracy and concept alignment metrics.","tokens_in":17210,"tokens_out":5323,"duration_ms":50583,"significance":"If the central attribution holds, the paper makes a useful contribution by introducing scale-aware inductive biases into concept-based explainability for ViTs, which is a novel direction relative to model-agnostic attention-only modules. The manuscript is also commendable for reporting results across multiple backbone architectures, providing detailed implementation descriptions, and making an anonymous code link available. The reported gains on CUB200 and KITS are consistent and plausible. However, the load-bearing claim that the multi-scale composition specifically causes the improvements is not yet adequately supported by the experimental design, and several reproducibility issues need to be resolved.","major_comments":[{"comment":"The central attribution of the gains to 'scale awareness' is underdetermined. Table 2 adds MSE and DMSF sequentially, but no control experiment varies the content of the multi-scale features while holding capacity and hyperparameters fixed. Since the DMSF module contains 16 attention heads, 4 reference points, learnable offsets, a learnable I, and per-dataset tuned ψ, the improvements over 'Only CRAM' could plausibly come from added learnable capacity or better optimization rather than from the semantic scale pyramid. The authors should add at least one control: (i) untrained or randomly projected CNN features, (ii) noise features of the same shape, or (iii) a matched-capacity non-scale fusion module (e.g., a fully learnable projection of zq with the same parameter count). Without such a control, the paper's opening claim that multi-scale fusion improves accuracy and concept learning is not established.","section":"§4.5 Table 2 and §4.3"},{"comment":"The reported feature sizes in the MSE module are internally inconsistent. The text states scales 1/8, 1/16, and 1/32 correspond to c1=1024, c2=196, and c3=16, and that the concatenated size is 1029, but 1024+196+16=1236, while the stated scales for a 224×224 input would give flattened sizes 784, 196, and 49 (sum 1029). This discrepancy makes the exact configuration of the MSE module ambiguous and hinders reproduction. Please clarify the actual feature map dimensions (including any pooling or projection steps) and correct the summary size.","section":"§4.3 and Appendix A.1"},{"comment":"The AWA2 concept annotations are generated with SAM v2 for 10 selected concepts, and the KITS concepts are derived from metadata or masks, but no validation of these machine-generated labels is reported. Because the Px TPR on AWA2 and KITS is a key part of the concept-alignment evidence, the model may be aligning to SAM artifacts rather than to human-understandable concepts. The authors should validate the generated labels on a held-out subset (e.g., IoU against manual annotations or a human study) or temper the claims for these datasets.","section":"§4.1"},{"comment":"Several of the reported differences between ASCENT-ViT and Only CRAM are within one standard deviation over three seeds, such as AWA2 ViT-Base accuracy (62.3±0.1 vs 62.3±0.2), AWA2 Px TPR (64.11±1.2 vs 64.02±0.5), Concept-MNIST digit classification (95.83±0.1 vs 95.71±0.1), and Pascal aPY accuracy (81.4±0.8 vs 81.0±0.8). The paper claims that ASCENT-ViT 'outperforms' on all datasets, but no significance tests or confidence intervals are provided, and Table 2 is a single run. Please report paired significance tests or effect sizes and adjust the strength of the claims accordingly, especially for datasets where the gains are within noise.","section":"Table 1"}],"minor_comments":[{"comment":"The text says that for CUB200 the concept errors are evaluated with the Frobenius norm of the difference between ground-truth annotations and predicted concept attention matrices, citing the appendix, but the appendix does not include these CUB concept-error results; only Pascal concept errors appear in Table 6. Please either add the CUB results or remove the reference.","section":"§4.2"},{"comment":"The statement 'pixel-wise TPR improves by ∼4% on CUB200 and ∼5%' is not tied to a specific backbone; Table 1 shows the improvements on CUB200 range from 1.68 (ViT-Base) to 3.66 (ViT-Large), and only ViT-Large on AWA2 reaches near 5%. Please specify which backbone(s) the percentages refer to.","section":"§4.4"},{"comment":"The hyperparameter ψ is reported per dataset (1 for CUB, 2 for Concept-MNIST and Pascal aPY, 0.5 for AWA2 and KITS), but the paper does not state how these values were selected (e.g., via a validation split). Since ψ directly controls the strength of the external features, this detail is important for reproducibility and for assessing the per-dataset tuning concern.","section":"§4.3"},{"comment":"The code link is an anonymous OpenScience link without a commit hash or version tag. For a methods paper that makes architectural contributions, a pinned code release (or an appendix with exact model configurations) is necessary to ensure reproducibility.","section":"Abstract / code link"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is potentially interesting, but the attribution of improvements to the multi-scale mechanism needs stronger controls, and the AWA2/KITS labels need validation. I would encourage the editor to check whether the anonymous code link, if available, resolves the feature-size inconsistency and whether the per-dataset ψ tuning is done on a held-out validation set. The claims in the abstract and contributions are stronger than the table statistics support; the revision should align the claims with the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper assembles a new combination—CNN multiscale feature pyramid plus deformable attention fused into ViT patch embeddings, then aligned to concept annotations—and shows modest but consistent gains in accuracy and concept localization across five datasets and several ViT variants. That combination is genuinely new relative to CT, CBM, and BotCL, and the module overhead is small. The paper is worth a serious referee.\n\nWhat it does well: the architecture is coherent; Table 2 ablations progressively add MSE and DMSF and show gains; the result that SWIN-with-its-own-scale-bias gains almost nothing from ASCENT is a nice internal sanity check; evaluation spans CUB, AWA2, KITS, C-MNIST, Pascal aPY with 3 seeds and multiple backbones. The overhead table is honest: under 3% extra parameters.\n\nSoft spots, in order of severity. (1) The central attribution—that the gains come from scale-aware feature content—is underdetermined. MSDA and the randomly initialized CNN can learn arbitrary spatial pooling; added capacity could explain the trend. The ablations never vary the content of the scale features: no noise-feature control, no untrained-feature control, no matched-capacity non-scale fusion baseline. That is a genuine gap. (2) There's a concrete numbers bug in the MSE description: c1=1024, c2=196, c3=16 sum to 1236, not the stated 1029; the stated scales 1/8, 1/16, 1/32 for 224×224 would give 784, 196, 49. Either the scales or the sizes are wrong, and this needs correction for reproducibility. (3) psi is tuned per dataset (0.5 to 2), so the comparison to CRAM is not strictly parameter-free; the paper should report sensitivity across datasets and ideally a single psi. (4) AWA2 and KITS concept labels are SAM-generated without validation; code is anonymous with no commit hash. These are manageable but real.\n\nThe circularity concern is minor: Eq. 10 supervises the spatial attention matrix, and concept metrics are on test data, so that's fine.\n\nBottom line: the central claim probably holds in the narrow sense tested, but the evidence doesn't yet isolate the mechanism. A serious editor should send this to review, with the request for a matched-capacity control and correction of the feature-size inconsistency. If I were in the area, I'd read the revision.","headline":"A useful new combination of multiscale CNN features and deformable attention for concept alignment, but the scale-awareness claim needs a matched-capacity control.","tokens_in":17698,"tokens_out":2876,"would_cite":false,"duration_ms":28165,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multiscale fusion boosts ViT concept explanations","keywords":["concept-based explainability","vision transformers","multi-scale features","deformable attention","concept alignment","interpretability","feature pyramid","inductive bias"],"falsifier":"Replace the MSE module's multi-scale features with random noise of the same shapes, or retrain with a single fixed scale (e.g., only 1/16): if pixel-level concept TPR and accuracy on CUB200 or AWA2 do not drop materially, then the scale composition itself is not what drives the reported improvements.","tokens_in":16676,"feed_emoji":"🔍","tokens_out":5767,"duration_ms":49222,"temperature":0.7,"pith_summary":"ASCENT-ViT is a concept-based explainability framework for Vision Transformers. It claims that injecting CNN-style scale awareness into a ViT's patch embeddings via a multi-scale feature pyramid and deformable attention, then aligning the fused representation with human-annotated concepts through attention matrices, improves both prediction accuracy and the quality of concept explanations. The paper demonstrates consistent gains over a model-agnostic attention-only baseline across five datasets and seven ViT backbones. If correct, this shows that concept explainability modules should be designed around the inductive biases of the backbone rather than as generic plug-ins.","feed_headline":"Multiscale fusion boosts ViT concept explanations","feed_subtitle":"Injecting CNN-style scale features into patch embeddings improves concept alignment across five datasets and seven backbones.","key_machinery":"The load-bearing object is the Deformable Multi-Scale Fusion (DMSF) module, whose adaptive composition equation $z = \\mathrm{norm}(z_q + \\psi \\cdot I \\cdot \\mathrm{MSDA}(z_q, p_q, c))$ is what actually injects scale awareness into the ViT. The Multi-Scale Deformable Attention (MSDA) operation uses the ViT's patch embeddings as queries and the CNN feature pyramid's multi-scale maps as keys and values, sampling reference points across scales so that each patch token can gather information from corresponding locations at different resolutions. The learnable vector $I$ (per token) and the scalar hyperparameter $\\psi$ control the strength of the scale injection, and the layer norm keeps the fused vector in the same embedding space as the original patch tokens. This composite vector is then fed to the Concept-Representation Alignment Module (CRAM), which computes spatial and global attention matrices against human concept annotations to produce both the prediction and the explanation.","core_discovery":"The paper's central claim is that the misalignment between ViT representations and human concept annotations is partly a scale problem, and that it can be repaired by composing multi-scale features into the patch stream. ASCENT-ViT does this in three stages: a Multi-scale Encoding module extracts features at 1/8, 1/16, and 1/32 resolution from a lightweight CNN; a Deformable Multi-Scale Fusion module treats the ViT's patch embeddings as queries for multi-scale deformable attention, producing $z = \\mathrm{norm}(z_q + \\psi \\cdot I \\cdot \\mathrm{MSDA}(z_q, p_q, c))$; and a Concept-Representation Alignment Module aligns this composite representation with spatial and global concept annotations via two attention matrices. On CUB200, AWA2, KITS, Concept-MNIST, and Pascal aPY, the paper reports that ASCENT-ViT outperforms the attention-only CRAM baseline on both task accuracy and concept metrics, with the largest gains on backbones that lack built-in scale invariance.","pith_inferences":["One extension the paper does not explore: the per-patch learnable vector $I$ could be visualized to reveal which patches accept scale information, turning the fusion into a diagnostic of where the ViT lacks scale awareness.","Because the CNN encoder is randomly initialized and trained end-to-end, a natural test is to replace it with features drawn from the ViT's own intermediate layers; if the gains vanish, the benefit likely comes from the CNN's translation equivariance rather than from scale diversity per se.","The fixed scale set (1/8, 1/16, 1/32) is likely dataset-dependent; systematic sweeps could show whether the method's benefit is robust to scale choices or tied to the natural object sizes in CUB200 and AWA2."],"forward_implications":["ViT backbones can acquire CNN-like scale awareness in their explanations without architectural changes to the pretrained encoder.","Concept explanations become more stable under rotations, crops, and zoom, because scale features anchor attention across transformations.","The framework acts as a drop-in classification head, adding under roughly 3% parameters over the backbone.","Gains over the attention-only baseline are largest for plain ViTs (positional bias only), while SWIN, which already has CNN-like inductive bias, shows roughly parity.","Test-time interventions on learned concepts correct more mispredictions than attention-only or concept-bottleneck baselines."],"supporting_citations":[{"why":"Supplies the multi-scale deformable attention (MSDA) operation that the DMSF fusion module is built on.","marker":"[Zhu et al., 2020]"},{"why":"Defines the attention-based concept alignment baseline ('Only CRAM') that ASCENT-ViT improves upon.","marker":"[Rigotti et al., 2021]"},{"why":"Provides the concept bottleneck model (CBM) baseline and the concept-supervised training formulation.","marker":"[Koh et al., 2020]"},{"why":"Defines the Vision Transformer backbone whose patch embeddings are the query stream of the DMSF module.","marker":"[Dosovitskiy et al., 2020]"},{"why":"Motivates the feature pyramid structure used by the Multi-scale Encoding module.","marker":"[Lin et al., 2017]"}],"fun_headline_variants":["Multiscale fusion sharpens ViT concept alignment","Scale-aware concept learning boosts ViT explanations","ASCENT-ViT aligns concepts with multi-scale attention","CNN scale features improve ViT concept explanations","Multi-scale attention head aligns ViT with concepts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire gain rests on the assumption that a randomly initialized, lightweight CNN's three fixed-scale feature maps align semantically with the pretrained ViT's patch embeddings, so that the deformable fusion enriches the patch tokens instead of corrupting them.","fun_headline_variants_meta":{"raw":{"variants":["Multiscale fusion sharpens ViT concept alignment","Scale-aware concept learning boosts ViT explanations","ASCENT-ViT aligns concepts with multi-scale attention","CNN scale features improve ViT concept explanations","Multi-scale attention head aligns ViT with concepts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2585,"prompt_tokens":996,"completion_tokens":1589,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":1517}},"tokens_in":612,"tokens_out":1589,"duration_ms":12857,"temperature":1.0,"reasoning_tokens":1517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:09:08.561219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the MSE module's multi-scale features with random noise of the same shapes, or retrain with a single fixed scale (e.g., only 1/16): if pixel-level concept TPR and accuracy on CUB200 or AWA2 do not drop materially, then the scale composition itself is not what drives the reported improvements.","supporting_citations":[],"review_version":1}