{"id":"a5d8e5f3-74b8-4981-b4b6-a188d9ccbcc4","arxiv_id":"2502.08997","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"HierViT uses radiologist-defined attributes with prototype images and attention maps to classify lung nodules and skin lesions, with high accuracy on LIDC-IDRI and comparable accuracy on derm7pt.","lead":"HierViT is a vision transformer that classifies medical images by first scoring clinically defined visual attributes, then predicting the diagnosis from those scores, and it shows example images (prototypes) plus attention maps for each attribute. It aims to give radiologists explanations that resemble how a human expert reads a scan rather than a black-box output.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Target branch consumes full attribute token vectors, not attribute scores, so human-defined attributes may be epiphenomenal; no perturbation test shows target predictions depend on them, and the w-proto-inference variant is an oracle using ground-truth labels.","rationale":"Reading the paper in good faith, the method is a ViT encoder plus hierarchical attribute layers and a target branch; the central promise is that the final diagnosis is derived from human-defined attribute reasoning. That promise holds only if the attribute tokens form a genuine bottleneck and the target branch cannot exploit information unrelated to the attribute concepts. The architecture does not guarantee this: §2 feeds the whole stacked attribute token vectors into the target transformer. The attribute scores are produced by linear heads from those same vectors, so the training objective does not force the target to use the scalar scores. In fact, the 'w proto. inference' row in Table 1 shows that the target branch can classify from prototype vectors indexed by ground-truth attributes, but this is an oracle variant and thus cannot establish that the deployed model's predictions follow its own predicted attributes. The qualitative expert statement about attention maps is valuable but addresses localization, not whether the target logit depends on the attribute scores. A concept-bottleneck re-run is the decisive check. The reader's CONDITIONAL verdict already reflects these gaps; I would keep it unchanged. I only partially align with the reader's phrasing because the most fragile point here is the missing bottleneck/intervention rather than label reliability or completeness, although the two are related.","tokens_in":8153,"tokens_out":7143,"duration_ms":70749,"concrete_test":"Retrain HierViT with a true concept bottleneck: feed the target branch only the concatenated predicted attribute scores/logits (the outputs of the linear attribute heads in Eq. (1)) instead of the full attribute token vectors, keeping the encoder, losses, and hyperparameters otherwise identical. If target accuracy on LIDC-IDRI stays near 94.8% Within-1 and on derm7pt near 73.8%, the human-defined attributes are sufficient and the explanations are grounded. If accuracy drops materially (e.g., by more than 2–3 points), the published model relies on information in the attribute token vectors beyond the attribute scores, so the claimed attribute-based reasoning is not causally responsible for its predictions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim that HierViT is 'inherently interpretable' and that its target prediction is derived from human-defined attributes, the target must actually be a function of the attribute semantics. The architecture in §2 does not enforce this: after attribute layers, the target branch receives the full attribute token vectors (stacked, passed through a linear layer and a target transformer), not the scalar attribute scores produced by the linear heads in Eq. (1). Those token vectors are high-dimensional and can carry non-attribute information, so the target transformer could learn to predict malignancy or lesion class from shortcut features while the attribute heads provide plausible but post-hoc scores. The only experiment aimed at causal grounding, HierViT 'w proto. inference' in Table 1, replaces the attribute tokens with prototypes selected using the ground-truth attribute labels (an oracle), so it shows the target branch can classify when given true attributes, not that the deployed model follows its own predicted attributes. No intervention, perturbation, or bottleneck experiment is reported. If the target does not track the attribute scores, the provided explanation (ratings, prototypes, heatmaps) is not the model's reasoning and the central claim fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HierViT, a Vision Transformer with a hierarchical classification branch that predicts human-defined attribute scores (e.g., spiculation, sphericity for lung nodules; dermatoscopic features for skin lesions), uses those attribute token vectors to predict the final target, and offers interpretability through attribute scores, prototype images, and attention heatmaps. An optional decoder produces segmentation masks when available. The method is evaluated on LIDC-IDRI for lung nodule malignancy, where it reports a Within-1-Accuracy of 94.8% for the target, and on derm7pt for skin lesion classification, where it reports accuracy comparable to existing multi-task methods. The paper claims that HierViT is 'inherently interpretable' and that its reasoning is aligned with human reasoning because the target prediction is derived from human-defined attributes. The central scientific claims are that the hierarchical attribute branch improves accuracy over previous prototype-based methods and that the resulting explanations are causally tied to the model's decision process.","tokens_in":8556,"tokens_out":4574,"duration_ms":46816,"significance":"If the central interpretability claim were established, HierViT would be a useful contribution to explainable medical image classification: it combines ViT-based feature extraction with hierarchical attribute prediction, prototype learning, and attention visualization, and it reports results on two public benchmarks with confidence intervals. The architecture and losses are clearly specified, and the use of external benchmarks with radiologist-defined attributes is a strength. However, the paper does not currently provide evidence that the target prediction actually depends on the attribute semantics in the way the explanation claims. The only causal-looking experiment, the 'w proto. inference' variant in Table 1, is an oracle experiment that uses ground-truth attribute labels, and no perturbation or ablation test is reported. In addition, parts of the accuracy comparison mix incompatible metrics (Within-1-Accuracy versus binary accuracy) and the derm7pt differences are within statistical noise. These issues bear directly on the paper's two headline claims, so the manuscript needs substantial revision before the claims can be accepted.","major_comments":[{"comment":"The target branch consumes the stacked attribute token vectors, not the scalar attribute scores produced by Eq. (1). Nothing in the architecture or the losses forces the target prediction to depend on the human-defined attribute semantics; the token vectors are high-dimensional and can carry shortcut information unrelated to the attributes. The claim in Section 4 that 'these human-defined attributes ... feed into the target prediction branch' is therefore not established. Please add an intervention test, such as shuffling or masking the predicted attribute scores at inference and measuring the change in target accuracy, or training a variant whose target branch receives only the scalar attribute scores. This is necessary to support the central 'inherently interpretable' claim.","section":"Section 2 (Eqs. (1)–(3), Fig. 1)"},{"comment":"The 'w proto. inference' variant uses the ground-truth attribute value to select prototypes, as the text states ('The prototype's groundtruth attribute value is used for prediction, ignoring attribute heads'). This is an oracle experiment: it shows that the target branch can classify when given correct attribute information, but it does not show that the deployed model's target prediction follows its own predicted attributes. Please report a non-oracle variant that uses the model's predicted attribute scores or prototype selections, and explicitly discuss the oracle limitation in the text.","section":"Section 3.3 (Table 1, 'HierViT w proto. inference')"},{"comment":"The state-of-the-art comparison mixes incompatible metrics: TransUnet, Res-trans, and TransPND report binary classification accuracy (marked with an asterisk), while HierViT reports Within-1-Accuracy on the five-level malignancy rating. These numbers are not directly comparable, so the statement that HierViT 'outperforms SOTA methods' is not supported for those rows. Additionally, the LIDC-IDRI preprocessing (excluding nodules detected by fewer than three radiologists or smaller than 3 mm) and the cross-validation split may differ from the protocols used by the compared methods; please provide a same-protocol comparison or explicitly state the comparability limits.","section":"Section 3.3 (Table 1)"},{"comment":"On derm7pt, the reported differences between HierViT and FusionM4Net are within overlapping 95% binomial confidence intervals (e.g., target 73.8 [69.5, 78.1] versus 74.3 [70.0, 78.6]). The abstract's phrase 'superior and comparable prediction accuracy' and the text's statement that HierViT 'attain[s] the best accuracy in target prediction' overstate the evidence. Please report a paired significance test or a non-overlapping-interval criterion, or revise the claim to say that performance is statistically indistinguishable from the best baseline.","section":"Section 3.3 (Table 2)"},{"comment":"No ablation study is reported for the proposed components: the hierarchical attribute branch, the prototype loss, the decoder branch, the warm-up phase, and the choice of target-branch input. Since the paper's novelty is the combination of these components, ablations are needed to show which components contribute to target accuracy and to justify hyperparameters such as lambda_proto = 0.01 and push_step = 2. Without ablations, the performance and interpretability claims cannot be attributed to the proposed architecture.","section":"Section 3 (all experiments)"}],"minor_comments":[{"comment":"The code availability statement contains the placeholder 'https://github.com/XXX'; please provide the actual repository URL or remove the claim that the code is publicly available.","section":"Section 1"},{"comment":"The abbreviation 'CSE' for cross-entropy loss is nonstandard; the usual abbreviation is 'CE'. Please correct this throughout.","section":"Section 2 (Eq. (4))"},{"comment":"For LIDC-IDRI, please state whether the reported 95% confidence interval is computed over pooled predictions or over the five cross-validation folds, and clarify how many test samples contribute to the interval.","section":"Section 3.1"},{"comment":"The qualitative evaluation relies on a single expert's consultation; this should be described as an anecdotal check rather than as a systematic evaluation of explanation quality.","section":"Section 3.2"},{"comment":"Reference [27] (Pathak et al.) appears in the bibliography but is not cited in the text; please cite it where relevant or remove it.","section":"References"},{"comment":"The average column 'Ø' is the mean over attributes and target; please justify this equal-weight aggregation, since attributes and target have different numbers of classes and different clinical importance.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a borderline major revision rather than a reject: the architecture is clearly described and the LIDC Within-1-Accuracy improvement over Proto-Caps is a tangible result, but the central interpretability claim is currently unsupported because the target branch is not shown to depend on the attribute semantics. The 'w proto. inference' experiment is an oracle and should not be presented as evidence of causal reasoning. I would require the authors to add a perturbation/bottleneck experiment, a non-oracle prototype-inference variant, and a same-metric comparison on LIDC, and to soften the derm7pt and SOTA claims. The missing code link is also a reproducibility concern that should be fixed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious architecture paper with a clear method and a genuine LIDC accuracy gain, but its central interpretability claim is not yet supported because the target branch consumes token vectors, not attribute scores, and the only causal variant uses ground-truth attributes as an oracle.\n\nWhat's actually new: combining a ViT backbone with per-attribute transformer layers, prototype push (with real-image prototypes), and a target transformer that consumes the attribute token vectors. I don't know of another ViT paper that does exactly this. The method section is clear: losses, warm-up, push step, hyperparameters. That's good.\n\nThe LIDC result is solid: 94.8% Within-1-Accuracy with CI [94.5,95.1] vs Proto-Caps' 93.0 [92.7,93.3], so the gain is real, not noise. The derm7pt results are honestly presented as comparable, with overlapping CIs. The qualitative examples are useful, and the discussion includes the worrying finding from their user study that explanations can be persuasive even when wrong—that's a mature point.\n\nSoft spots. The big one: the target branch gets the full attribute token vectors (stacked, linear-projected, then a transformer), not the scalar attribute scores from Eq. (1). So nothing forces the target prediction to track the human-defined attributes. The attribute heads could be a side path while the target reads shortcuts from the token vectors. The 'w proto. inference' variant doesn't fix this: it replaces the tokens with prototype vectors selected using ground-truth attribute labels, so it's an oracle experiment. It shows the target branch can use true attributes, not that the deployed model uses its own predicted attributes. The authors describe this transparently but don't provide the missing test: perturb or mask the predicted attribute scores and show the target prediction changes accordingly.\n\nOther issues: the code link is still a placeholder, there are no ablations (attribute branch on/off, prototype loss weight, number of prototypes), and the LIDC table mixes Within-1-Accuracy with binary accuracy for the ViT baselines. Minor.\n\nBottom line: the accuracy story holds, the interpretability story is plausible but unproven. The paper deserves a proper referee—it's a genuine new architecture, clearly written, and the flaw is a missing experiment, not a fatal contradiction. If I reviewed it, I'd ask for a perturbation test at minimum. Bring it to reading group if you work on XAI for medical imaging.","headline":"A well-specified ViT+prototype hybrid with a real LIDC accuracy gain, but the interpretability claim needs a faithfulness test before it holds up.","tokens_in":8917,"tokens_out":2671,"would_cite":false,"duration_ms":26198,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HierViT is a Vision Transformer that classifies medical images by first scoring human-defined attributes, then showing prototype images and attention heatmaps, claiming both high accuracy and clinically aligned reasoning.","keywords":["Explainable AI","Hierarchical Prediction","Prototype Learning","Vision Transformer","Medical image classification","LIDC-IDRI","derm7pt","Attention heatmaps"],"falsifier":"A concrete intervention experiment would settle the causal claim: take a correctly classified LIDC-IDRI nodule, replace the attribute token for a high-stakes feature such as spiculation with the prototype vector of the opposite rating while leaving all image tokens fixed, and check whether the malignancy prediction shifts toward the clinically expected direction. If large attribute interventions leave target predictions essentially unchanged across many samples, the claim that the model reasons through human-defined attributes is falsified. A complementary check is to retrain without attribute supervision entirely and see whether target accuracy stays high; if it does, the attribute hierarchy is not carrying the diagnostic signal.","tokens_in":7987,"feed_emoji":"🩺","tokens_out":5786,"duration_ms":58442,"temperature":0.7,"pith_summary":"The paper introduces HierViT, a Vision Transformer that produces its final medical diagnosis only after scoring a set of human-defined visual attributes, showing a prototype image for each score, and highlighting the image region behind each attribute. On LIDC-IDRI, it reports 94.8% Within-1-Accuracy for lung-nodule malignancy, above the listed comparison methods, and on derm7pt it reaches a target accuracy comparable to the strongest baselines. The aim is to show that interpretability can be built into the architecture itself, so clinicians can check the model's reasoning in the same feature language they use, rather than relying on post-hoc attention maps. If this works, high-accuracy vision transformers become auditable tools for high-risk medical decisions.","feed_headline":"A ViT that explains itself hits 94.8% on lung-nodule malignancy","feed_subtitle":"These diagnoses route through doctor-defined attributes, prototype images, and attention maps—accuracy holds or improves.","key_machinery":"The load-bearing mechanism is the hierarchical attribute-token classifier: after the ViT encoder produces a token vector, a separate transformer layer per attribute scores that attribute, and these attribute token vectors are stacked and passed through a target transformer that makes the final prediction. Each attribute layer is paired with a prototype layer holding 16 learnable vectors per attribute class, trained with a Euclidean-distance loss; a push operation then replaces each prototype with the closest real training sample, so prototypes are actual images rather than synthetic abstractions. The optional ViT-based decoder adds a segmentation mask, and the training loss is the sum of target, attribute, and segmentation losses, with a warm-up phase before prototype learning begins. What makes the design 'inherently interpretable' is that the target head consumes only attribute-level representations, not raw image tokens, so the final decision is formally a function of human-defined feature scores.","core_discovery":"On its own terms, the paper's central discovery is that a Vision Transformer can be restructured so that the final classification is computed from attribute-level token vectors, each produced by a dedicated transformer layer and tied to learnable prototype vectors. The attribute branch scores clinically defined features, the prototype layer associates each score with real training images via a push operation, and the target branch consumes the stacked attribute tokens to make the final call. The paper reports that this design reaches 94.8% Within-1-Accuracy on LIDC-IDRI malignancy and 73.8% target accuracy on derm7pt, and that a prototype-inference variant, where the target is predicted from the closest prototypes instead of the attribute heads, retains 94.4% on LIDC-IDRI. This is presented as evidence that interpretability need not be added after the fact: the model's intermediate decisions can be expressed in the same features radiologists and dermatologists use.","pith_inferences":["A direct intervention experiment—swapping the attribute token for 'spiculation' to the opposite prototype or score while keeping the image tokens fixed—would test whether the target prediction actually moves in the clinically expected direction; the paper does not report such a test, but the architecture makes it straightforward.","If attribute labels are noisy or subjective, the target branch might learn to exploit correlations among attributes rather than their clinical meaning; one way to check is to corrupt or hold out attribute labels during training and measure how much target accuracy and explanation fidelity degrade.","The prototype push operation stores real training images, so the explanations inherit any dataset bias; on underrepresented nodule or lesion subtypes, the nearest prototypes may be misleading even if the attribute score is correct.","The reported expert consultation on LIDC-IDRI is qualitative; a formal evaluation with multiple radiologists, measuring how often they change their diagnosis after seeing the attributes and prototypes, would quantify whether the explanations truly align with human reasoning."],"forward_implications":["A clinician could validate a prediction by checking each attribute score, comparing the prototype image with the inference image, and inspecting the attribute-specific attention heatmap; mismatches would flag an unreliable call before it reaches the diagnosis.","The prototype-inference variant shows that even when the target head is forced to use prototype vectors rather than learned attribute heads, LIDC-IDRI accuracy only drops from 94.8% to 94.4%, suggesting that explanations and accuracy do not have to trade off.","Because the architecture is built on a standard ViT encoder, the same hierarchical attribute design could be transferred to other medical imaging tasks where discrete clinical attributes exist, such as breast imaging or chest radiograph reporting.","On derm7pt, treating lesion features as a hierarchy rather than as independent multi-label outputs yields accuracy comparable to the best listed baselines, indicating that structuring the task by clinical attributes does not cost prediction performance.","The attention heatmaps are not generic saliency maps but are attached to individual attributes, giving a potential new way to localize pathological structures: the spiculation heatmap focuses on nodule edges, while the sphericity and lobulation heatmaps focus on the interior."],"supporting_citations":[{"why":"Supplies the ViT encoder backbone and the pre-trained ImageNet-1K weights used by every branch of HierViT.","marker":"[1]"},{"why":"Provides the prototype-learning formulation that the prototype layers adapt for attribute-specific visual examples.","marker":"[7]"},{"why":"Prior hierarchical prototype method (Proto-Caps) that motivates the architecture, the warm-up phase, and the 'w. proto. inference' variant, and serves as a comparison baseline.","marker":"[8]"},{"why":"LIDC-IDRI dataset source, providing the lung nodule CT images and segmentations used in all LIDC experiments.","marker":"[15]"},{"why":"Describes the radiologist annotation protocol that defines the eight appearance attributes and the malignancy ratings used as targets and supervision.","marker":"[16]"},{"why":"Introduces the derm7pt dataset, its seven-point checklist features, and the test split used for comparability.","marker":"[18]"},{"why":"X-Caps baseline that encodes visual attributes in capsules for explainable diagnosis; HierViT compares against its attribute and target accuracies.","marker":"[22]"},{"why":"User study showing that attribute-and-prototype explanations affect radiologist confidence, supporting the paper's trust and usability claims.","marker":"[10]"}],"fun_headline_variants":["Self-explaining ViT hits 94.8% on lung nodules","ViT that thinks like doctors: 94.8% on lung scans","Interpretable ViT: 94.8% accuracy with human-like reasoning","HierViT: interpretable ViT with 94.8% lung nodule accuracy","Prototype-guided ViT explains its lung nodule calls at 94.8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the radiologist- and dermatologist-defined attribute labels are complete and reliable enough that scoring these attributes first is the right causal path to the diagnosis; if those labels are noisy or miss the true decision boundary, the target branch may learn to bypass the attribute semantics and the promised human-aligned reasoning does not hold.","fun_headline_variants_meta":{"raw":{"variants":["Self-explaining ViT hits 94.8% on lung nodules","ViT that thinks like doctors: 94.8% on lung scans","Interpretable ViT: 94.8% accuracy with human-like reasoning","HierViT: interpretable ViT with 94.8% lung nodule accuracy","Prototype-guided ViT explains its lung nodule calls at 94.8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000954,"raw_usage":{"total_tokens":4058,"prompt_tokens":926,"completion_tokens":3132,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":3027}},"tokens_in":542,"tokens_out":3132,"duration_ms":21536,"temperature":1.0,"reasoning_tokens":3027,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:57:35.311845+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete intervention experiment would settle the causal claim: take a correctly classified LIDC-IDRI nodule, replace the attribute token for a high-stakes feature such as spiculation with the prototype vector of the opposite rating while leaving all image tokens fixed, and check whether the malignancy prediction shifts toward the clinically expected direction. If large attribute interventions leave target predictions essentially unchanged across many samples, the claim that the model reasons through human-defined attributes is falsified. A complementary check is to retrain without attribute supervision entirely and see whether target accuracy stays high; if it does, the attribute hierarchy is not carrying the diagnostic signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ViT encoder backbone and the pre-trained ImageNet-1K weights used by every branch of HierViT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prototype-learning formulation that the prototype layers adapt for attribute-specific visual examples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior hierarchical prototype method (Proto-Caps) that motivates the architecture, the warm-up phase, and the 'w. proto. inference' variant, and serves as a comparison baseline."},{"cited_title":"InTCIA (2015) https://doi.org/10","cited_arxiv_id":null,"evidence_quote":"LIDC-IDRI dataset source, providing the lung nodule CT images and segmentations used in all LIDC experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"X-Caps baseline that encodes visual attributes in capsules for explainable diagnosis; HierViT compares against its attribute and target accuracies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"User study showing that attribute-and-prototype explanations affect radiologist confidence, supporting the paper's trust and usability claims."}],"review_version":1}