Pith. sign in

REVIEW 4 major objections 7 minor 23 references

Developing Explainable Machine Learning Model using Augmented Concept Activation Vector

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read Measuring a concept's true pull on a model's decision

desk verdict Plausible TCAV-style idea with a useful clinical direction, but the isolation claim is unsupported: missing control augmentations, no error bars, and an unused entropy metric make the quantitative claims premature. read the letter →

arxiv 2412.19208 v1 pith:XS3APSEI submitted 2024-12-26 cs.LG

classification cs.LG
keywords conceptactivationvectorexplainableAIradiomicsfundusimagesmodelinterpretabilityimbalanceddatamedicalimagingpost-hocexplanation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a trained classifier's dependence on a human-understandable visual concept can be measured directly by adding that concept to an otherwise normal input and observing how the network's internal activation vector shifts. If true, it would give clinicians and model developers a way to ask, of a single already-trained model, how much a specific radiomic pattern or lesion shape actually drives a decision, without training a second model or collecting separate concept images. The authors apply the method to retinal fundus images augmented with diabetic-eye patterns and to brain MRI images augmented with differently sized tumors, reporting that activation vectors move toward the disease class as patterns are added. They conclude that ACAV isolates the impact of high-level patterns independently of the surrounding data.

What carries the argument

The central object is the augmented activation vector: for a correctly classified input $d_i$ lacking concept $C$, the network's layer-$l$ activation $\mathbf{V}_l = f(d_i)$ is compared with the activation of the same input after overlaying the concept, $\mathbf{V}_l^a = f(d_i \oplus C)$, and the mean absolute deviation $\Delta V = \frac{1}{n}\sum_i |\mathbf{V}_l^a - \mathbf{V}_l|$ is taken as the concept's influence. The method also uses cosine-similarity angles between the shifted vector and average healthy-versus-disease reference vectors, and an entropy term $H = -\sum_i p_i \log p_i$ to account for how pattern frequency in imbalanced data affects the model's decisions. These measurements are made on a single trained network, which is what distinguishes ACAV from TCAV's two-dataset procedure.

What would settle it

Augment healthy fundus images with a randomly textured patch matched for size, location, and low-level statistics but carrying no medical concept, then measure the activation deviation; if the deviation is comparable to that of the fatty-dot or cotton-wool pattern, the measured shift is not concept-specific and the isolation claim fails.

Watch

Extended reading notes

Core claim

The paper proposes Augmented Concept Activation Vector (ACAV), a post-hoc explanation method that quantifies how much a high-level concept contributes to a model's classification by overlaying the concept pattern onto input images that lack it, then comparing the activation vector of the layer before the output before and after augmentation. The shift is summarized by the average absolute deviation $\Delta V = \frac{1}{n}\sum_i |\mathbf{V}_l^a - \mathbf{V}_l|$, by the fraction of augmented samples that flip to the opposite class, and by an entropy-based imbalance metric $H = -\sum_{i=1}^C p_i\log p_i$ that weighs concept prevalence. In experiments on fundus images augmented with cotton-wool, fatty-dot, and bleeding patterns, and on brain MRI images augmented with small, medium, and large tumors, the measured cosine-similarity angles between the augmented activation vectors and the disease-class reference vectors shrink as more or larger patterns are added, which the authors interpret as evidence that the concept's impact has been isolated and quantitatively measured.

Load-bearing premise

Adding a concept pattern to an image is assumed to change only that high-level concept, leaving brightness, edges, texture, and local contrast otherwise untouched, so any activation shift can be attributed uniquely to the concept.

Editorial extensions

If this is right

  • A single trained model can be interrogated for the quantitative contribution of each visual concept, removing the need for a second concept-trained network.
  • Rare but clinically valued patterns can be evaluated even when they appear infrequently in the dataset, because the concept is artificially added rather than looked for in existing samples.
  • The results quantify not only whether a pattern matters but by how much, in degrees of activation-vector shift and in class-flip rates, which could be directly reported in a model audit.
  • The method is architecture-agnostic and could be applied to any neural network and any concept with a visual representation, including shape, size, intensity, and location variations.
  • Measured shifts that align with clinical expectations suggest ACAV could surface unexpected dependencies, such as a model leaning on a pattern practitioners believe is secondary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The isolation claim rests on the untested premise that overlaying a concept changes only the high-level concept; a control augmentation that disturbs low-level statistics without adding a concept would reveal whether the measured shift is genuinely concept-specific.
  • If ACAV is reliable, it could serve as a debugging tool for medical imaging models, flagging when a classifier over-weights a pattern that is correlated with the label but not causally related.
  • The entropy-based imbalance metric could be extended into a weighting scheme for explanations, so that 'rare but decisive' features are presented with a prevalence-adjusted confidence.
  • The method could be tested on a synthetic dataset with known ground-truth concept contributions to check whether the measured angles and class-flip rates recover the true ranking of concept influence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Augmented Concept Activation Vector (ACAV), a post-hoc explainability method that measures the impact of high-level visual concepts on a trained neural network's decisions. The method augments input images with concept patterns (e.g., cotton-wool spots, fatty dots, bleeding, tumor patches) and measures shifts in the network's internal activation vectors using Equation (3), with additional cosine-angle metrics in Tables 3-4. Experiments on RFMiD fundus images and a brain MRI dataset claim to isolate the impact of radiomic patterns and tumor size on classification. The central claims are that ACAV can 'isolate the impact of a given high-level concept and accurately measure it quantitatively' and can handle imbalanced data where rare patterns matter.

Significance. If validated, ACAV would be a lightweight, single-model alternative to TCAV-like methods for concept attribution in medical imaging, avoiding the need for a separately trained concept model and working directly on raw images rather than extracted features. The paper also targets a practically important problem: quantifying the influence of rare radiomic patterns in imbalanced datasets. These are genuine strengths of the proposal. However, as presented, the experimental evidence does not support the isolation and quantitative-accuracy claims: there is no control augmentation condition, no error bars or significance tests, and the reported metrics are not consistently tied to the equations that define them. The core idea is plausible, but the current manuscript is not a validated method.

major comments (4)
  1. [Section 3, Eq. (3), Tables 1-4] The central claim that ACAV 'can isolate the impact of a given high-level concept and accurately measure it quantitatively' (Abstract) is not supported by the experiments because no control augmentation is ever run. Augmenting an image with a cotton-wool or fatty-dot patch changes low-level statistics (edges, intensity, contrast, location) as well as the semantic concept; Tables 1-4 therefore measure the combined effect of all these changes. Without a random-patch condition, a phase-scrambled or texture-matched control, or an augmentation with a known irrelevant concept, the observed activation deviations cannot be attributed uniquely to the concept. This is a missing control experiment, not a disagreement with consensus.
  2. [Section 4, Tables 1-4] The quantitative claims are made without any estimate of variability or statistical significance. Each experiment is based on 50 images, yet the tables report only point values (e.g., average absolute deviations of 0.03-0.10 in Table 1 and angle differences of 3-60 degrees in Table 3) with no confidence intervals, standard deviations, significance tests, or multiple-seed runs. Given the small magnitudes of some deviations, these numbers could easily be within sampling noise; the conclusion that the model is 'sensitive' or 'insensitive' to a pattern is therefore not established. I request error bars, hypothesis tests, and comparison against the control condition from the previous comment.
  3. [Section 3, Eq. (3) vs Section 4, Tables 1-4] The metrics reported in the results do not match the definitions in Section 3. Equation (3) defines the average absolute deviation of activation vectors, and the text defines a first metric as the ratio of data points assigned to a different class after augmentation; neither quantity appears in the tables. Instead, Tables 1-2 report 'average norm vector' and 'average absolute deviation' (with no formula), and Tables 3-4 report cosine angles in degrees. The entropy metric of Eq. (4) is never used in the experiments. Because the reported numbers cannot be traced to the defined metrics, the paper does not provide an operational definition of the claimed quantitative measurement.
  4. [Section 4, Figure 5 and softmax threshold] The model description is internally contradictory. The text states that 'the last layer includes five neurons having soft-max activation function' and later states that 'the last layer includes a single neuron with softmax activation function,' with a 0.6/0.4 confidence threshold. These are incompatible descriptions of the output layer, and the threshold choice affects which images are counted as correctly classified and hence which activation vectors enter the averages in Tables 1-4. The architecture must be specified precisely for the quantitative results to be reproducible.
minor comments (7)
  1. [Section 3] There are multiple typos and malformed notation: 'prposed', 'date points', and the set definition 'AC : {aC|aC = a L C, a∈ TN , C isaconceptpattern}' are garbled; the augmentation operator 'di L C' is never defined.
  2. [Eq. (3)] Equation (3) writes f(di) for both the original and the augmented input; the augmented input should be f(dia) or equivalent, as written it suggests the same input is fed twice.
  3. [Tables 1-2] The column labels 'Average Norm Vector' are ambiguous: they do not state whether these are Euclidean norms, cosine similarities, or something else, and the unit of 'Average Absolute Deviation' in Table 1 is not defined.
  4. [Section 4] The train/validation/test split, class balance, and preprocessing (e.g., image resolution for fundus images) are not described for either dataset, so the experiments cannot be reproduced.
  5. [Section 4, final paragraph] The statement that the results were 'validated by a domain expert' gives no protocol; please describe what was shown to the expert and what the confirmation consisted of.
  6. [References] Several references contain formatting artifacts (e.g., '[PdSMBR VCdS19]', '[A VL+14]', '[LR VL+12]'); please clean the citation keys.
  7. [Abstract and Introduction] The second aim, 'determine the prevalence of frequent patterns' in imbalanced datasets, is not addressed by any reported prevalence measure or by use of Eq. (4).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the measured activation deviations are empirical outputs, not constructed from the claim itself.

full rationale

The paper's derivation chain is: train a CNN on healthy versus diabetic fundus images; define high-level concept patterns; augment healthy images with those patterns; measure the resulting activation-vector deviation (Eq. 3) and cosine-similarity angles against healthy and diabetic reference vectors (Tables 1-4). The reported quantities are direct empirical measurements from the trained model, not quantities defined in terms of the paper's own conclusions. The claim that the method 'can isolate the impact of a given high-level concept' does rest on the untested assumption that augmentation changes only the concept and not low-level pixel statistics; however, that is a missing control experiment and a validity threat, not a circularity. No equation is shown to be equivalent to another by construction, no fitted parameter is renamed as a prediction, and no load-bearing result is imported from a self-citation. The only self-citation, [HNB+23], is used for a generic statement about feature-selection dimensionality reduction and is not load-bearing. Accordingly, the central derivation is self-contained as a measurement procedure, and no specific circular step can be exhibited from the text.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests chiefly on the trained model and on the untested assumption that augmentation isolates the concept. No new physical entities are postulated. The free parameters are the unstated training setup, hand-chosen confidence thresholds, and undefined augmentation scales.

free parameters (4)
  • CNN model weights = not provided
    The activation deviations and angles are computed from one unstated training run of a CNN on the two-class RFMiD split; no weights, seeds, or training details are given, so all reported numbers depend on an unobservable fitted model.
  • softmax confidence thresholds = 0.6 (healthy) / 0.4 (unhealthy)
    Hand-chosen thresholds used to filter low-confidence test samples before computing activation vectors; changing them would change which images enter the analysis.
  • number of augmented patterns per image = single vs multiple (unspecified counts)
    The comparison between single and multiple pattern augmentation is central to Tables 1 and 2, but the paper does not state how many patterns are added in the multiple case.
  • tumor size scaling categories = large, average, small with no quantitative sizes
    For the brain MRI experiment, segmented tumors are scaled into three qualitative categories; the boundaries are not specified, so the angle measurements depend on an unstated scale choice.
assumptions (5)
  • domain assumption The classification process considers the existence and prevalence of specific concept patterns in the input.
    Stated in Section 3 before Eq. 1; if the trained model does not encode these concepts, the activation deviations do not measure concept impact.
  • domain assumption Augmenting a concept pattern into a data point preserves the context and avoids external factors, so any activation change is attributable to the concept.
    Stated in Section 3 as 'The main advantage of augmenting the concept pattern to a data point is preserving the context of the sample and avoiding external factors.' This is the key isolation assumption and is not tested against a control augmentation.
  • ad hoc to paper The deviation of activation vectors (Eq. 3) and the cosine angles to class reference vectors are valid quantitative measures of concept impact.
    The paper defines these metrics but provides no calibration, benchmark, or noise model showing that a given deviation corresponds to a particular amount of concept influence.
  • domain assumption The RFMiD classes 2 and 3 can be excluded without biasing the two-class healthy/diabetic setup.
    Section 4 excludes the two intermediate diabetic severity classes and later uses some class 2/3 images as augmentation sources, which may introduce distribution shift not accounted for.
  • domain assumption A domain expert's qualitative confirmation is sufficient validation that the measured impacts reflect clinical reality.
    The final paragraph of Section 4 states results were validated by an ophthalmologist; this is a subjective check, not a quantitative ground truth, yet it is the only validation offered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Developing Explainable Machine Learning Model using Augmented Concept Activation Vector." pith.science (2026). https://pith.science/paper/XS3APSEI

@misc{pith2026241219208,
  author       = {Pith},
  title        = {Pith review of: Developing Explainable Machine Learning Model using Augmented Concept Activation Vector},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XS3APSEI}},
  note         = {Machine review of arXiv:2412.19208}
}
read the original abstract

Machine learning models use high dimensional feature spaces to map their inputs to the corresponding class labels. However, these features often do not have a one-to-one correspondence with physical concepts understandable by humans, which hinders the ability to provide a meaningful explanation for the decisions made by these models. We propose a method for measuring the correlation between high-level concepts and the decisions made by a machine learning model. Our method can isolate the impact of a given high-level concept and accurately measure it quantitatively. Additionally, this study aims to determine the prevalence of frequent patterns in machine learning models, which often occur in imbalanced datasets. We have successfully applied the proposed method to fundus images and managed to quantitatively measure the impact of radiomic patterns on the model decisions.

Figures

Figures reproduced from arXiv: 2412.19208 by the authors.

Figure 1
Figure 1. Schematic diagram of the proposed method. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Healthy images augmented with cotton-wool radiomic pattern. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Healthy images augmented with cotton-wool and fatty dots radiomic patterns. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Images with bleeding pattern augmented with fatty dots radiomic pattern. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Schematic diagram of the classifying model. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Samples images used in the second experiment. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Sample images from the Brain MRI dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Sample augmented images with different pattern scales from the Brain MRI dataset. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 15 canonical work pages

  1. [1]

    Structural compression of convolutional neural networks

    Reza Abbasi-Asl and Bin Yu. Structural compression of convolutional neural networks. arXiv preprint arXiv:1705.07356 , 2017

  2. [2]

    Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach

    Hugo JWL Aerts, Emmanuel Rios Velazquez, Ralph TH Leijenaar, Chintan Parmar, Patrick Grossmann, Sara Carvalho, Johan Bussink, Ren \'e Monshouwer, Benjamin Haibe-Kains, Derek Rietveld, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nature communications , 5(1):4006, 2014

  3. [3]

    Adaptive feature selection using an autoencoder and classifier: Applied to a radiomics case

    Reza Hassanpour, Niels Netten, Tony Busker, Mortaza Shoae Bargh, and Sunil Choenni. Adaptive feature selection using an autoencoder and classifier: Applied to a radiomics case. In Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing , pages 1256--1259, 2023

  4. [4]

    Radiomics: the process and the challenges

    Virendra Kumar, Yuhua Gu, Satrajit Basu, Anders Berglund, Steven A Eschrich, Matthew B Schabath, Kenneth Forster, Hugo JWL Aerts, Andre Dekker, David Fenstermacher, et al. Radiomics: the process and the challenges. Magnetic resonance imaging , 30(9):1234--1248, 2012

  5. [5]

    Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning , pages 2668--2677. PMLR, 2018

  6. [6]

    Radiomics: extracting more information from medical images using advanced feature analysis

    Philippe Lambin, Emmanuel Rios-Velazquez, Ralph Leijenaar, Sara Carvalho, Ruud GPM Van Stiphout, Patrick Granton, Catharina ML Zegers, Robert Gillies, Ronald Boellard, Andr \'e Dekker, et al. Radiomics: extracting more information from medical images using advanced feature analysis. European journal of cancer , 48(4):441--446, 2012

  7. [7]

    An explainable machine learning framework for multiple medical datasets classification

    Mostarina Mitu, SM Mahedy Hasan, Anwar Hossain Efat, Md Fakrul Taraque, Nahrin Jannat, and Mahjabin Oishe. An explainable machine learning framework for multiple medical datasets classification. In 2023 International Conference on Next-Generation Computing, IoT and Machine Learning (NCIM) , pages 1--6. IEEE, 2023

  8. [8]

    Interpretable and explainable machine learning: a methods-centric overview with concrete examples

    Ri c ards Marcinkevi c s and Julia E Vogt. Interpretable and explainable machine learning: a methods-centric overview with concrete examples. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 13(3):e1493, 2023

Show all 23 references
  1. [9]

    Quantitative analysis of lesion morphology and texture features for diagnostic prediction in breast mri

    Ke Nie, Jeon-Hor Chen, J Yu Hon, Yong Chu, Orhan Nalcioglu, and Min-Ying Su. Quantitative analysis of lesion morphology and texture features for diagnostic prediction in breast mri. Academic radiology , 15(12):1513--1525, 2008

  2. [10]

    Explanation of machine learning models using shapley additive explanation and application for real data in hospital

    Yasunobu Nohara, Koutarou Matsumoto, Hidehisa Soejima, and Naoki Nakashima. Explanation of machine learning models using shapley additive explanation and application for real data in hospital. Computer Methods and Programs in Biomedicine , 214:106584, 2022

  3. [11]

    Attention u-net: Learning where to look for the pancreas

    Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 , 2018

  4. [12]

    Local interpretable model-agnostic explanations for classification of lymph node metastases

    Iam Palatnik de Sousa, Marley Maria Bernardes Rebuzzi Vellasco, and Eduardo Costa da Silva. Local interpretable model-agnostic explanations for classification of lymph node metastases. Sensors , 19(13):2969, 2019

  5. [13]

    Radiomic machine-learning classifiers for prognostic biomarkers of head and neck cancer

    Chintan Parmar, Patrick Grossmann, Derek Rietveld, Michelle M Rietbergen, Philippe Lambin, and Hugo JWL Aerts. Radiomic machine-learning classifiers for prognostic biomarkers of head and neck cancer. Frontiers in oncology , 5:272, 2015

  6. [14]

    Book review: Max kuhn and kjell johnson

    Dimitris Rizopoulos. Book review: Max kuhn and kjell johnson. applied predictive modeling. new york, springer. Biometrics , 74(1):383--383, 2018

  7. [15]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135--1144, 2016

  8. [16]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618--626, 2017

  9. [17]

    Striving for simplicity: The all convolutional net

    Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 , 2014

  10. [18]

    Deep inside convolutional networks: Visualising image classification models and saliency maps

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 , 2013

  11. [19]

    Detecting statistical interactions from neural network weights

    Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. arXiv preprint arXiv:1705.04977 , 2017

  12. [20]

    Sunet: A lesion regularized model for simultaneous diabetic retinopathy and diabetic macular edema grading

    Zhi Tu, Shenghua Gao, Kang Zhou, Xianing Chen, Huazhu Fu, Zaiwang Gu, Jun Cheng, Zehao Yu, and Jiang Liu. Sunet: A lesion regularized model for simultaneous diabetic retinopathy and diabetic macular edema grading. In 2020 IEEE 17th International Symposium on Biomedical Imaging...

  13. [21]

    Visualizing and understanding convolutional networks

    Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13 , pages 818--833. Springer, 2014

  14. [22]

    Ibex: an open infrastructure software platform to facilitate collaborative work in radiomics

    Lifei Zhang, David V Fried, Xenia J Fave, Luke A Hunter, Jinzhong Yang, and Laurence E Court. Ibex: an open infrastructure software platform to facilitate collaborative work in radiomics. Medical physics , 42(3):1341--1353, 2015

  15. [23]

    Learning deep features for discriminative localization

    Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2921--2929, 2016

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.