REVIEW 4 major objections 7 minor 23 references
Developing Explainable Machine Learning Model using Augmented Concept Activation Vector
T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Measuring a concept's true pull on a model's decision
desk verdict Plausible TCAV-style idea with a useful clinical direction, but the isolation claim is unsupported: missing control augmentations, no error bars, and an unused entropy metric make the quantitative claims premature. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the augmented activation vector: for a correctly classified input $d_i$ lacking concept $C$, the network's layer-$l$ activation $\mathbf{V}_l = f(d_i)$ is compared with the activation of the same input after overlaying the concept, $\mathbf{V}_l^a = f(d_i \oplus C)$, and the mean absolute deviation $\Delta V = \frac{1}{n}\sum_i |\mathbf{V}_l^a - \mathbf{V}_l|$ is taken as the concept's influence. The method also uses cosine-similarity angles between the shifted vector and average healthy-versus-disease reference vectors, and an entropy term $H = -\sum_i p_i \log p_i$ to account for how pattern frequency in imbalanced data affects the model's decisions. These measurements are made on a single trained network, which is what distinguishes ACAV from TCAV's two-dataset procedure.
What would settle it
Augment healthy fundus images with a randomly textured patch matched for size, location, and low-level statistics but carrying no medical concept, then measure the activation deviation; if the deviation is comparable to that of the fatty-dot or cotton-wool pattern, the measured shift is not concept-specific and the isolation claim fails.
Extended reading notes
Core claim
The paper proposes Augmented Concept Activation Vector (ACAV), a post-hoc explanation method that quantifies how much a high-level concept contributes to a model's classification by overlaying the concept pattern onto input images that lack it, then comparing the activation vector of the layer before the output before and after augmentation. The shift is summarized by the average absolute deviation $\Delta V = \frac{1}{n}\sum_i |\mathbf{V}_l^a - \mathbf{V}_l|$, by the fraction of augmented samples that flip to the opposite class, and by an entropy-based imbalance metric $H = -\sum_{i=1}^C p_i\log p_i$ that weighs concept prevalence. In experiments on fundus images augmented with cotton-wool, fatty-dot, and bleeding patterns, and on brain MRI images augmented with small, medium, and large tumors, the measured cosine-similarity angles between the augmented activation vectors and the disease-class reference vectors shrink as more or larger patterns are added, which the authors interpret as evidence that the concept's impact has been isolated and quantitatively measured.
Load-bearing premise
Adding a concept pattern to an image is assumed to change only that high-level concept, leaving brightness, edges, texture, and local contrast otherwise untouched, so any activation shift can be attributed uniquely to the concept.
Editorial extensions
If this is right
- A single trained model can be interrogated for the quantitative contribution of each visual concept, removing the need for a second concept-trained network.
- Rare but clinically valued patterns can be evaluated even when they appear infrequently in the dataset, because the concept is artificially added rather than looked for in existing samples.
- The results quantify not only whether a pattern matters but by how much, in degrees of activation-vector shift and in class-flip rates, which could be directly reported in a model audit.
- The method is architecture-agnostic and could be applied to any neural network and any concept with a visual representation, including shape, size, intensity, and location variations.
- Measured shifts that align with clinical expectations suggest ACAV could surface unexpected dependencies, such as a model leaning on a pattern practitioners believe is secondary.
Reading between the lines
- The isolation claim rests on the untested premise that overlaying a concept changes only the high-level concept; a control augmentation that disturbs low-level statistics without adding a concept would reveal whether the measured shift is genuinely concept-specific.
- If ACAV is reliable, it could serve as a debugging tool for medical imaging models, flagging when a classifier over-weights a pattern that is correlated with the label but not causally related.
- The entropy-based imbalance metric could be extended into a weighting scheme for explanations, so that 'rare but decisive' features are presented with a prevalence-adjusted confidence.
- The method could be tested on a synthetic dataset with known ground-truth concept contributions to check whether the measured angles and class-flip rates recover the true ranking of concept influence.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Augmented Concept Activation Vector (ACAV), a post-hoc explainability method that measures the impact of high-level visual concepts on a trained neural network's decisions. The method augments input images with concept patterns (e.g., cotton-wool spots, fatty dots, bleeding, tumor patches) and measures shifts in the network's internal activation vectors using Equation (3), with additional cosine-angle metrics in Tables 3-4. Experiments on RFMiD fundus images and a brain MRI dataset claim to isolate the impact of radiomic patterns and tumor size on classification. The central claims are that ACAV can 'isolate the impact of a given high-level concept and accurately measure it quantitatively' and can handle imbalanced data where rare patterns matter.
Significance. If validated, ACAV would be a lightweight, single-model alternative to TCAV-like methods for concept attribution in medical imaging, avoiding the need for a separately trained concept model and working directly on raw images rather than extracted features. The paper also targets a practically important problem: quantifying the influence of rare radiomic patterns in imbalanced datasets. These are genuine strengths of the proposal. However, as presented, the experimental evidence does not support the isolation and quantitative-accuracy claims: there is no control augmentation condition, no error bars or significance tests, and the reported metrics are not consistently tied to the equations that define them. The core idea is plausible, but the current manuscript is not a validated method.
major comments (4)
- [Section 3, Eq. (3), Tables 1-4] The central claim that ACAV 'can isolate the impact of a given high-level concept and accurately measure it quantitatively' (Abstract) is not supported by the experiments because no control augmentation is ever run. Augmenting an image with a cotton-wool or fatty-dot patch changes low-level statistics (edges, intensity, contrast, location) as well as the semantic concept; Tables 1-4 therefore measure the combined effect of all these changes. Without a random-patch condition, a phase-scrambled or texture-matched control, or an augmentation with a known irrelevant concept, the observed activation deviations cannot be attributed uniquely to the concept. This is a missing control experiment, not a disagreement with consensus.
- [Section 4, Tables 1-4] The quantitative claims are made without any estimate of variability or statistical significance. Each experiment is based on 50 images, yet the tables report only point values (e.g., average absolute deviations of 0.03-0.10 in Table 1 and angle differences of 3-60 degrees in Table 3) with no confidence intervals, standard deviations, significance tests, or multiple-seed runs. Given the small magnitudes of some deviations, these numbers could easily be within sampling noise; the conclusion that the model is 'sensitive' or 'insensitive' to a pattern is therefore not established. I request error bars, hypothesis tests, and comparison against the control condition from the previous comment.
- [Section 3, Eq. (3) vs Section 4, Tables 1-4] The metrics reported in the results do not match the definitions in Section 3. Equation (3) defines the average absolute deviation of activation vectors, and the text defines a first metric as the ratio of data points assigned to a different class after augmentation; neither quantity appears in the tables. Instead, Tables 1-2 report 'average norm vector' and 'average absolute deviation' (with no formula), and Tables 3-4 report cosine angles in degrees. The entropy metric of Eq. (4) is never used in the experiments. Because the reported numbers cannot be traced to the defined metrics, the paper does not provide an operational definition of the claimed quantitative measurement.
- [Section 4, Figure 5 and softmax threshold] The model description is internally contradictory. The text states that 'the last layer includes five neurons having soft-max activation function' and later states that 'the last layer includes a single neuron with softmax activation function,' with a 0.6/0.4 confidence threshold. These are incompatible descriptions of the output layer, and the threshold choice affects which images are counted as correctly classified and hence which activation vectors enter the averages in Tables 1-4. The architecture must be specified precisely for the quantitative results to be reproducible.
minor comments (7)
- [Section 3] There are multiple typos and malformed notation: 'prposed', 'date points', and the set definition 'AC : {aC|aC = a L C, a∈ TN , C isaconceptpattern}' are garbled; the augmentation operator 'di L C' is never defined.
- [Eq. (3)] Equation (3) writes f(di) for both the original and the augmented input; the augmented input should be f(dia) or equivalent, as written it suggests the same input is fed twice.
- [Tables 1-2] The column labels 'Average Norm Vector' are ambiguous: they do not state whether these are Euclidean norms, cosine similarities, or something else, and the unit of 'Average Absolute Deviation' in Table 1 is not defined.
- [Section 4] The train/validation/test split, class balance, and preprocessing (e.g., image resolution for fundus images) are not described for either dataset, so the experiments cannot be reproduced.
- [Section 4, final paragraph] The statement that the results were 'validated by a domain expert' gives no protocol; please describe what was shown to the expert and what the confirmation consisted of.
- [References] Several references contain formatting artifacts (e.g., '[PdSMBR VCdS19]', '[A VL+14]', '[LR VL+12]'); please clean the citation keys.
- [Abstract and Introduction] The second aim, 'determine the prevalence of frequent patterns' in imbalanced datasets, is not addressed by any reported prevalence measure or by use of Eq. (4).
Circularity Check
No significant circularity: the measured activation deviations are empirical outputs, not constructed from the claim itself.
full rationale
The paper's derivation chain is: train a CNN on healthy versus diabetic fundus images; define high-level concept patterns; augment healthy images with those patterns; measure the resulting activation-vector deviation (Eq. 3) and cosine-similarity angles against healthy and diabetic reference vectors (Tables 1-4). The reported quantities are direct empirical measurements from the trained model, not quantities defined in terms of the paper's own conclusions. The claim that the method 'can isolate the impact of a given high-level concept' does rest on the untested assumption that augmentation changes only the concept and not low-level pixel statistics; however, that is a missing control experiment and a validity threat, not a circularity. No equation is shown to be equivalent to another by construction, no fitted parameter is renamed as a prediction, and no load-bearing result is imported from a self-citation. The only self-citation, [HNB+23], is used for a generic statement about feature-selection dimensionality reduction and is not load-bearing. Accordingly, the central derivation is self-contained as a measurement procedure, and no specific circular step can be exhibited from the text.
Assumptions & free parameters
free parameters (4)
- CNN model weights =
not provided
- softmax confidence thresholds =
0.6 (healthy) / 0.4 (unhealthy)
- number of augmented patterns per image =
single vs multiple (unspecified counts)
- tumor size scaling categories =
large, average, small with no quantitative sizes
assumptions (5)
- domain assumption The classification process considers the existence and prevalence of specific concept patterns in the input.
- domain assumption Augmenting a concept pattern into a data point preserves the context and avoids external factors, so any activation change is attributable to the concept.
- ad hoc to paper The deviation of activation vectors (Eq. 3) and the cosine angles to class reference vectors are valid quantitative measures of concept impact.
- domain assumption The RFMiD classes 2 and 3 can be excluded without biasing the two-class healthy/diabetic setup.
- domain assumption A domain expert's qualitative confirmation is sufficient validation that the measured impacts reflect clinical reality.
Cite this review
Pith. "Pith review of Developing Explainable Machine Learning Model using Augmented Concept Activation Vector." pith.science (2026). https://pith.science/paper/XS3APSEI
@misc{pith2026241219208,
author = {Pith},
title = {Pith review of: Developing Explainable Machine Learning Model using Augmented Concept Activation Vector},
year = {2026},
howpublished = {\url{https://pith.science/paper/XS3APSEI}},
note = {Machine review of arXiv:2412.19208}
}
read the original abstract
Machine learning models use high dimensional feature spaces to map their inputs to the corresponding class labels. However, these features often do not have a one-to-one correspondence with physical concepts understandable by humans, which hinders the ability to provide a meaningful explanation for the decisions made by these models. We propose a method for measuring the correlation between high-level concepts and the decisions made by a machine learning model. Our method can isolate the impact of a given high-level concept and accurately measure it quantitatively. Additionally, this study aims to determine the prevalence of frequent patterns in machine learning models, which often occur in imbalanced datasets. We have successfully applied the proposed method to fundus images and managed to quantitatively measure the impact of radiomic patterns on the model decisions.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Structural compression of convolutional neural networks
Reza Abbasi-Asl and Bin Yu. Structural compression of convolutional neural networks. arXiv preprint arXiv:1705.07356 , 2017
arXiv 2017
-
[2]
Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach
Hugo JWL Aerts, Emmanuel Rios Velazquez, Ralph TH Leijenaar, Chintan Parmar, Patrick Grossmann, Sara Carvalho, Johan Bussink, Ren \'e Monshouwer, Benjamin Haibe-Kains, Derek Rietveld, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nature communications , 5(1):4006, 2014
work page 2014
-
[3]
Adaptive feature selection using an autoencoder and classifier: Applied to a radiomics case
Reza Hassanpour, Niels Netten, Tony Busker, Mortaza Shoae Bargh, and Sunil Choenni. Adaptive feature selection using an autoencoder and classifier: Applied to a radiomics case. In Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing , pages 1256--1259, 2023
work page 2023
-
[4]
Radiomics: the process and the challenges
Virendra Kumar, Yuhua Gu, Satrajit Basu, Anders Berglund, Steven A Eschrich, Matthew B Schabath, Kenneth Forster, Hugo JWL Aerts, Andre Dekker, David Fenstermacher, et al. Radiomics: the process and the challenges. Magnetic resonance imaging , 30(9):1234--1248, 2012
work page 2012
-
[5]
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning , pages 2668--2677. PMLR, 2018
work page 2018
-
[6]
Radiomics: extracting more information from medical images using advanced feature analysis
Philippe Lambin, Emmanuel Rios-Velazquez, Ralph Leijenaar, Sara Carvalho, Ruud GPM Van Stiphout, Patrick Granton, Catharina ML Zegers, Robert Gillies, Ronald Boellard, Andr \'e Dekker, et al. Radiomics: extracting more information from medical images using advanced feature analysis. European journal of cancer , 48(4):441--446, 2012
work page 2012
-
[7]
An explainable machine learning framework for multiple medical datasets classification
Mostarina Mitu, SM Mahedy Hasan, Anwar Hossain Efat, Md Fakrul Taraque, Nahrin Jannat, and Mahjabin Oishe. An explainable machine learning framework for multiple medical datasets classification. In 2023 International Conference on Next-Generation Computing, IoT and Machine Learning (NCIM) , pages 1--6. IEEE, 2023
work page 2023
-
[8]
Interpretable and explainable machine learning: a methods-centric overview with concrete examples
Ri c ards Marcinkevi c s and Julia E Vogt. Interpretable and explainable machine learning: a methods-centric overview with concrete examples. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 13(3):e1493, 2023
work page 2023
Show all 23 references
-
[9]
Quantitative analysis of lesion morphology and texture features for diagnostic prediction in breast mri
Ke Nie, Jeon-Hor Chen, J Yu Hon, Yong Chu, Orhan Nalcioglu, and Min-Ying Su. Quantitative analysis of lesion morphology and texture features for diagnostic prediction in breast mri. Academic radiology , 15(12):1513--1525, 2008
2008
-
[10]
Explanation of machine learning models using shapley additive explanation and application for real data in hospital
Yasunobu Nohara, Koutarou Matsumoto, Hidehisa Soejima, and Naoki Nakashima. Explanation of machine learning models using shapley additive explanation and application for real data in hospital. Computer Methods and Programs in Biomedicine , 214:106584, 2022
2022
-
[11]
Attention u-net: Learning where to look for the pancreas
Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 , 2018
2018 arXiv
-
[12]
Local interpretable model-agnostic explanations for classification of lymph node metastases
Iam Palatnik de Sousa, Marley Maria Bernardes Rebuzzi Vellasco, and Eduardo Costa da Silva. Local interpretable model-agnostic explanations for classification of lymph node metastases. Sensors , 19(13):2969, 2019
2019
-
[13]
Radiomic machine-learning classifiers for prognostic biomarkers of head and neck cancer
Chintan Parmar, Patrick Grossmann, Derek Rietveld, Michelle M Rietbergen, Philippe Lambin, and Hugo JWL Aerts. Radiomic machine-learning classifiers for prognostic biomarkers of head and neck cancer. Frontiers in oncology , 5:272, 2015
2015
-
[14]
Book review: Max kuhn and kjell johnson
Dimitris Rizopoulos. Book review: Max kuhn and kjell johnson. applied predictive modeling. new york, springer. Biometrics , 74(1):383--383, 2018
2018
-
[15]
why should i trust you?
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135--1144, 2016
2016
-
[16]
Grad-cam: Visual explanations from deep networks via gradient-based localization
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618--626, 2017
2017
-
[17]
Striving for simplicity: The all convolutional net
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 , 2014
2014 arXiv
-
[18]
Deep inside convolutional networks: Visualising image classification models and saliency maps
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 , 2013
2013 arXiv
-
[19]
Detecting statistical interactions from neural network weights
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. arXiv preprint arXiv:1705.04977 , 2017
2017 arXiv
-
[20]
Sunet: A lesion regularized model for simultaneous diabetic retinopathy and diabetic macular edema grading
Zhi Tu, Shenghua Gao, Kang Zhou, Xianing Chen, Huazhu Fu, Zaiwang Gu, Jun Cheng, Zehao Yu, and Jiang Liu. Sunet: A lesion regularized model for simultaneous diabetic retinopathy and diabetic macular edema grading. In 2020 IEEE 17th International Symposium on Biomedical Imaging...
2020
-
[21]
Visualizing and understanding convolutional networks
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13 , pages 818--833. Springer, 2014
2014
-
[22]
Ibex: an open infrastructure software platform to facilitate collaborative work in radiomics
Lifei Zhang, David V Fried, Xenia J Fave, Luke A Hunter, Jinzhong Yang, and Laurence E Court. Ibex: an open infrastructure software platform to facilitate collaborative work in radiomics. Medical physics , 42(3):1341--1353, 2015
2015
-
[23]
Learning deep features for discriminative localization
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2921--2929, 2016
2016
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.