{"id":"b3d85e90-9200-491f-8590-e1c7b29e2ea6","arxiv_id":"1909.01498","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Applying network dissection, Grad-CAM, activation maximization, and test-time dropout to three 2D brain tumor segmentation networks suggests filter-level concepts and coarse-to-fine localization, with high uncertainty on misclassified regions.","lead":"This paper probes the internal workings of three brain tumor segmentation neural networks using existing interpretability tools. It finds evidence that individual filters learn recognizable concepts such as edema and tumor core, and that the networks narrow their attention from the whole brain to fine tumor details.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The disentanglement and top-down localization claims rest on threshold-tuned, visually selected feature maps; no full-filter statistics or null baseline are reported.","rationale":"The reader's weakest assumption already targets the arbitrariness of the dissection threshold and visual concept assignment; my stress-test agrees and sharpens it: the same concerns also apply to the Grad-CAM hierarchy claim, and the remedy is the same: report full-filter distributions and compare against null baselines. No formal verification, code release, or trained-model release is provided, and the qualitative results are the only evidence for the paper's headline claim. This does not make the paper fraudulent or internally inconsistent; it means the central claim is currently conditional on additional quantitative validation. Since the reader's verdict is already CONDITIONAL, my analysis does not move the verdict; it reinforces the stated condition. The paper's own caveats (§4.1 admits the feature visualizations' faithfulness is unclear, and §6 says more experiments are needed) support this reading.","tokens_in":11827,"tokens_out":5009,"duration_ms":54990,"concrete_test":"Run network dissection over all filters of all three models on the 48-volume test set and on two controls: untrained (random-weight) versions of the same architectures and feature maps with randomly permuted spatial structure. Report, per layer, the fraction of filters with IoU≥c for each BraTS label (stating c), and sweep the quantile in Eq. (1) over 0.001-0.1. The 0.01-quantile choice is only justified if the concept-detector fraction exceeds the best control fraction by a clear margin; otherwise the reported concepts are an artifact of threshold tuning and largest-blob post-processing. Separately, for §3.2, compute per-layer mean Grad-CAM IoU with brain and tumor masks over all test slices to test the claimed coarse-to-fine attention progression quantitatively.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that brain-tumor segmentation CNNs organize as human-understandable, filter-level disentangled concepts and localize top-down is supported only by selected panels in Figures 2-5. Network Dissection in §3.1 is not reported as a distribution: the text never states the IoU threshold c in Eq. (2), nor the number or fraction of filters that qualify as concept detectors, nor how filters were selected for display. The threshold T in Eq. (1) is explicitly chosen because the 0.01 quantile 'gives the best results' on the same data, and masks are post-processed to keep only the largest activated connected component inside the brain; both steps can manufacture high IoU for large structures (e.g. whole tumor) from any noisy feature map. Implicit concepts (brain, edema, grey matter) are assigned by visual inspection. For the hierarchy claim in §3.2, no quantitative attention metric is computed over the test set; the observation that attention moves from brain to tumor to subregions is derived from a few displayed Grad-CAM maps. Therefore the empirical assertion is not yet distinguished from thresholding and selection artifacts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies a pipeline of interpretability and uncertainty techniques to three 2D brain tumor segmentation networks (DenseUnet, ResUnet, SimUnet) trained on the BraTS 2018 dataset. The three analysis strands are: network dissection to identify filter-level concepts, Grad-CAM to trace layer-wise spatial attention, and test-time dropout to estimate epistemic uncertainty; activation maximization is also used to visualize filter inputs. The central empirical claims are that individual filters learn human-understandable disentangled concepts (e.g., whole tumor, edema, tumor core, and implicit concepts such as brain or white matter), that the models localize tumors in a top-down hierarchical manner (brain first, then tumor, then fine subregions), that skip/residual connections help early localization, and that misclassified regions are often associated with high uncertainty. The paper is framed as an introductory study and explicitly notes limitations, including the uncertain faithfulness of activation maximization and the difficulty of extending the analysis to 3D models.","tokens_in":12167,"tokens_out":3338,"duration_ms":34618,"significance":"If substantiated, the paper's findings would be valuable for the medical imaging interpretability community: filter-level disentanglement and hierarchical localization in segmentation networks would support the idea that such models can be inspected in human-understandable terms, potentially increasing clinician trust. The paper has several strengths: it applies established interpretability methods (network dissection, Grad-CAM, test-time dropout) to a medically relevant task, compares three architectures in a controlled setting, uses the public BraTS 2018 benchmark, and is candid about the exploratory nature of some results. However, the central claims rest on selective qualitative evidence and post hoc threshold choices rather than systematic quantitative evaluation. The current manuscript therefore does not yet establish the disentanglement and top-down localization assertions, but the gap is addressable with additional analysis rather than being fundamentally unworkable.","major_comments":[{"comment":"The concept-detector criterion is incompletely specified: the IoU threshold c in Eq. (2) is never stated, and the text reports no distribution of IoU values, no number or fraction of filters that pass the threshold, and no breakdown by layer or network. Without these statistics, the statement that 'individual filters learn explicit as well as implicit disentangled concepts' is supported only by selected panels in Figures 2-4. Please report the full statistics: the number of filters evaluated per layer, the distribution of IoU values, the value of c used, and the fraction of filters exceeding c. In addition, provide a null baseline (for example, thresholded random noise masks or filters with shuffled activations) to demonstrate that the observed IoU values cannot be obtained by the thresholding and post-processing procedure alone.","section":"3.1, Eq. (2)"},{"comment":"The threshold T in Eq. (1) is chosen as the 0.01 quantile because it 'gives the best results' on the same data used for evaluation, and this is a post hoc selection on the test distribution. The post-processing step, which keeps only the largest activated connected component inside the brain, can further inflate IoU for large structures such as the whole tumor, even from noisy activation maps. In addition, the implicit concepts (brain, edema, grey matter) are assigned by visual inspection with no stated protocol or inter-rater validation. Please specify a pre-specified or cross-validated threshold selection procedure, document the concept labeling rule, and report quantitative concept-detection results (mean and standard deviation of IoU, per-concept filter counts) over the entire held-out validation set rather than only selected examples.","section":"3.1, Eq. (1) and post-processing paragraph"},{"comment":"The top-down hierarchical localization claim is based on qualitative inspection of selected Grad-CAM maps in Figure 5, with no quantitative attention metric computed over the test set. The assertion that 'attention moves from the brain to the tumor and finally to finer subregions' needs to be substantiated with a layer-wise localization score, such as the overlap of the Grad-CAM map with the brain mask, the whole-tumor mask, and the tumor subregion masks, averaged over all validation volumes for each model. Please also state how the displayed layers and images were selected, and report whether the top-down pattern is consistently observed across the full dataset or only in the shown examples.","section":"3.2"}],"minor_comments":[{"comment":"The phrase 'In-Network Dissection' appears to be missing a space and an article; it should likely read 'In Network Dissection' or 'In the Network Dissection pipeline'.","section":"3.1 opening"},{"comment":"The Dice scores in Table 1 are reported as single values without variance; please include standard deviations or confidence intervals, since the 48-volume validation set is large enough to support such statistics.","section":"Table 1"},{"comment":"The regularization coefficients are given as 'approximate values' (λ∼1e-4, γ∼1e-2, ζ∼1e-5) and jitter/rotation parameters as ∼8 pixels and ∼10 degrees; please state the exact values used or provide a sensitivity table, as these choices directly affect the activation-maximization visualizations.","section":"4.2"},{"comment":"The caption contains a typo: 'possiblity' should be 'possibility'.","section":"Figure 1 caption"},{"comment":"The total variation term in Eq. (7) uses square brackets that appear to denote absolute values, but the absolute value operator is not shown; please clarify the notation.","section":"4.2.2, Eq. (7)"},{"comment":"The claim that misclassified regions are often associated with high uncertainty is supported only by selected examples in Figure 7; please add a quantitative evaluation, for instance comparing the distribution of uncertainty values in correctly versus incorrectly classified voxels across the validation set.","section":"5"}],"recommendation":"major_revision","confidential_remarks":"This is an exploratory interpretability study that applies established tools to brain tumor segmentation. The main weakness is that the central claims about disentangled concepts and top-down localization are supported only by selected qualitative examples and post hoc threshold choices. The paper is not beyond repair; a revision that adds systematic quantitative statistics, null baselines, and test-set-wide evaluation would make it a solid contribution. In its current form, the evidence is too anecdotal for a journal-level claim, but the scope of the paper is appropriate for a specialized medical imaging or interpretability venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on interpretability for medical segmentation, but read it as a pilot, not a demonstration. The authors train three standard 2D U-Net variants on BraTS 2018 and run network dissection, Grad-CAM, activation maximization, and Monte Carlo dropout. What's new is the application: those tools were built for classification on ImageNet, and it's useful to see them behave in a 3-class medical segmentation setting. They also make a thoughtful observation that skip/residual connections seem to let the downsampling path localize tumors earlier than a plain encoder-decoder. The uncertainty maps aligning with misclassifications is a nice qualitative result, and the authors are unusually candid about what they can't claim - they explicitly say the activation maximization images may not be faithful.\n\nThe soft spots are the ones the attached note flags, and they're real. The network dissection threshold is chosen because it 'gives the best results' on the same data, and then masks are post-processed to keep only the largest connected component inside the brain. That can inflate IoU even for noisy feature maps. The paper never reports how many filters qualified as concept detectors, what the IoU threshold c was, or any distribution over filters. Implicit concepts like edema and white matter are assigned by visual inspection. The top-down hierarchy claim rests on a few selected Grad-CAM panels, with no quantitative attention metric over the test set. And Table 1 reports Dice without variance, with no code or trained models released for checking.\n\nNone of this is fatal if you take the paper at its stated level: an introductory study. The authors say as much in the conclusion and future work. But the abstract and conclusion still put 'disentangled concepts' and 'hierarchical approach' as central findings, and those are under-supported. I'd want to see full-filter statistics, a null baseline with random filters, and independent test-set evaluation before believing the claims are robust.\n\nWho gets value: anyone setting up interpretability experiments on medical segmentation, or looking for a benchmark of what can go wrong with network dissection in this domain. It deserves peer review, but as a conditional accept with major revision, not as a definitive result. I'd cite it as an early demonstration, with a caveat.","headline":"A genuinely honest pilot study applying interpretability tools to brain tumor segmentation, but the central claims about disentangled concepts and top-down localization are only qualitatively supported.","tokens_in":12547,"tokens_out":2533,"would_cite":true,"duration_ms":23323,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Brain tumor segmentation networks learn human-understandable concepts at the filter level and localize tumors from whole brain to fine subregions.","keywords":["brain tumor segmentation","interpretability","network dissection","Grad-CAM","activation maximization","test-time dropout","uncertainty estimation","BraTS 2018"],"falsifier":"Retrain the same model architecture on the same BraTS data with several random seeds and rerun network dissection: if the same layer positions do not reproducibly yield the same concept assignments, or if the IoU scores at the 0.01 quantile are no higher than at other quantiles or than scores from randomly permuted filter-to-concept mappings, the filter-level disentanglement is an artifact of the analysis rather than a stable property of the network.","tokens_in":11634,"feed_emoji":"🧠","tokens_out":6910,"duration_ms":61864,"temperature":0.7,"pith_summary":"This paper tries to establish that brain tumor segmentation networks are not black boxes in the way often assumed: their internal filters specialize in human-understandable concepts, and their attention sweeps from whole structures to fine details. Such a result matters because it would let clinicians inspect and trust a segmentation network's reasoning, and would give researchers a concrete target for debugging and improving models. The authors demonstrate this on three standard 2D encoder-decoder segmentation networks trained on the BraTS 2018 dataset, using four complementary techniques: network dissection, gradient-weighted class activation maps, activation maximization, and test-time dropout uncertainty. If correct, the finding means that human-readable organization can emerge without explicit supervision, which is encouraging for medical adoption of deep learning segmentation.","feed_headline":"Brain tumor networks organize into human-readable filters","feed_subtitle":"Filters specialize in whole tumor, edema, and core; attention moves brain-first to fine detail.","key_machinery":"The machinery is network dissection, which turns a filter's activation map into a concept mask by thresholding at the $0.01$ quantile of its dataset-wide activation distribution, post-processing the mask, and scoring it with $\\mathrm{IoU}(M_{k,l}(x), gt) \\ge c$ against labeled tumor subregions. A detector is declared when the IoU exceeds a threshold $c$. Grad-CAM computes neuron importance weights via gradients of the global-average-pooled segmentation output and produces a ReLU-weighted attention map per layer. Activation maximization with jitter, total-variation, and style regularization constructs pre-images that maximally excite a filter. Test-time dropout approximates Bayesian inference by averaging $T$ stochastic forward passes and estimating the variance. Together these four tools connect individual filters to concepts, trace spatial attention over layers, visualize filter preferences, and quantify prediction confidence.","core_discovery":"The authors' claim, stated on their own terms, is that brain tumor segmentation networks do not encode tumor knowledge in an unstructured way: individual filters learn disentangled, human-understandable concepts, with separate filters detecting the whole tumor, edema, and tumor core, and even implicit concepts like white and grey matter that were never part of the training labels. Across three differently structured 2D segmentation networks trained on BraTS 2018, thresholded activation masks from individual filters matched these concepts, and Grad-CAM attention maps showed the networks first fixating on the brain, then the tumor region, then the fine subregions. The paper presents this as evidence that segmentation models take a top-down, hierarchical approach to localization, that skip and residual connections accelerate spatial localization in early layers, and that misclassified regions carry high uncertainty. It does not claim state-of-the-art segmentation accuracy; it claims that these human-understandable traces of inference are present and extractable in ordinary segmentation networks.","pith_inferences":["The authors' threshold-choice procedure could be turned into a hypothesis test: if filter-to-concept assignments are stable across random initializations and across thresholds away from the hand-picked 0.01 quantile, disentanglement is a property of the task; if not, it is an artifact of the analysis.","A natural next experiment is to feed the discovered implicit concept masks (brain, white matter) back into the network as auxiliary input channels and measure whether segmentation accuracy improves; that would turn the descriptive claim into a causal one.","Extending the same pipeline to 3D segmentation models is an open question the authors flag; one could compute concept consistency across adjacent slices to see whether disentanglement survives volumetric inference.","The correlation between uncertainty and misclassification suggests a concrete clinical workflow: model proposes, uncertainty map highlights, radiologist adjudicates; this workflow's net benefit could be measured against unassisted reading in a reader study."],"forward_implications":["Filter-level concept detectors mean a segmentation prediction can be audited by checking which filters fired, rather than treating the network as an opaque whole.","Skip and residual connections appear to carry spatial information toward early layers: DenseUnet and ResUnet localize the tumor within their first layers, while the connection-free SimUnet only localizes it near the output.","The layer-wise attention sequence (background to brain to tumor to fine subregions) implies segmentation networks follow the same global-to-local ordering documented in human visual perception.","Misclassified tumor regions coincide with high test-time-dropout uncertainty, so uncertainty maps can flag where a radiologist should double-check the model."],"supporting_citations":[{"why":"Supplies the network dissection procedure (activation thresholding plus IoU scoring) used to identify filter-level concepts.","marker":"Bau et al., 2017"},{"why":"Supplies Grad-CAM, which the paper adapts to show layer-wise spatial attention toward the tumor.","marker":"Selvaraju et al., 2017"},{"why":"Supplies test-time dropout as approximate Bayesian inference, the basis of the uncertainty maps.","marker":"Gal and Ghahramani, 2016"},{"why":"Defines the BraTS benchmark and its tumor subregion labels (whole tumor, core, enhancing tumor) used as ground truth.","marker":"Menze et al. (2014)"},{"why":"Provides the BraTS 2018 challenge dataset and its training/validation organization used to train and test the models.","marker":"Bakas et al. (2018)"},{"why":"Introduces activation maximization, the optimization used to visualize internal filter preferences.","marker":"Erhan et al., 2009"},{"why":"Supplies the style regularizer that constrains optimized images to resemble the input data distribution.","marker":"Li et al., 2017"},{"why":"Provides the global precedence effect used to interpret the observed coarse-to-fine attention as human-like.","marker":"Navon, 1977"}],"fun_headline_variants":["Tumor segmentation nets learn human-readable filters","Brain tumor AI separates tumor, edema, core in filters","Tumor nets use hierarchical attention: brain, tumor, detail","Segmentation networks reveal human-understandable tumor concepts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of disentangled concepts rests on the assumption that a per-filter activation threshold set at the 0.01 quantile, chosen because it gives the best concept scores, and the visual assignment of unlabeled concepts truly reflect what each filter detects; if the threshold choice and assignment are arbitrary, the apparent concepts could be artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Tumor segmentation nets learn human-readable filters","Brain tumor AI separates tumor, edema, core in filters","Tumor nets use hierarchical attention: brain, tumor, detail","Segmentation networks reveal human-understandable tumor concepts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2962,"prompt_tokens":997,"completion_tokens":1965,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1901}},"tokens_in":613,"tokens_out":1965,"duration_ms":14282,"temperature":1.0,"reasoning_tokens":1901,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:15:24.418590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same model architecture on the same BraTS data with several random seeds and rerun network dissection: if the same layer positions do not reproducibly yield the same concept assignments, or if the IoU scores at the 0.01 quantile are no higher than at other quantiles or than scores from randomly permuted filter-to-concept mappings, the filter-level disentanglement is an artifact of the analysis rather than a stable property of the network.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the network dissection procedure (activation thresholding plus IoU scoring) used to identify filter-level concepts."},{"cited_title":"R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D","cited_arxiv_id":null,"evidence_quote":"Supplies Grad-CAM, which the paper adapts to show layer-wise spatial attention toward the tumor."},{"cited_title":"and Ghahramani, Z","cited_arxiv_id":null,"evidence_quote":"Supplies test-time dropout as approximate Bayesian inference, the basis of the uncertainty maps."},{"cited_title":"H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., et al","cited_arxiv_id":null,"evidence_quote":"Defines the BraTS benchmark and its tumor subregion labels (whole tumor, core, enhancing tumor) used as ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces activation maximization, the optimization used to visualize internal filter preferences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the global precedence effect used to interpret the observed coarse-to-fine attention as human-like."}],"review_version":1}