{"id":"dbce2d53-e919-43fb-8a3d-39a41810c1e9","arxiv_id":"2504.15100","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A demonstration that global and local sensitivity analysis plus activation maximization can highlight influential features in small neural networks for tabular and image data.","lead":"This paper applies three standard sensitivity-analysis tools (Sobol indices, pixel-perturbation heatmaps, and activation maximization) to small neural networks trained on diabetes, CIFAR-10, and lung-ultrasound data. It reports that the tools highlight clinically plausible features, such as glucose and BMI, and that class-activation images can reveal which image regions drive predictions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LUS interpretability claim rests on a model with no reported held-out accuracy; without test evaluation, the AM features cannot be distinguished from overfitting or dataset artifacts.","rationale":"The reader's conditional verdict is reasonable. The strongest positive evidence in the paper is that code is released, the internal numbers are consistent, and the Section 2.3 feature-subset experiment gives a direct check that the high-Sobol features retain most accuracy. My concern targets the second half of the central claim, which is less supported. Section 4 reports 94% training accuracy only; no held-out metric appears anywhere in the text. The authors compare with Grad-CAM from Born et al., but that comparison does not substitute for evaluation of their own model. The self-identified boundary-artifact limitation makes the missing test accuracy more consequential, not less. The Sobol independence concern raised by the reader is real and would affect the ranking in Section 2.3, but the subset experiment partially mitigates it; the LUS claim has no similar quantitative backstop. Therefore I agree with the reader's conditional verdict: the paper should be accepted only if the LUS model's generalization and the link between AM saliency and clinical features are demonstrated. I do not see a reason to reject: the methods are standard, the demonstrations are reproducible in principle, and the qualitative patterns are at least plausible.","tokens_in":11263,"tokens_out":5736,"duration_ms":57189,"concrete_test":"Using the covid19_ultrasound data and the provided VGG-16 code, define a patient-level held-out split, retrain or fine-tune the classifier, and report test accuracy, per-class precision/recall, and confusion matrix. If held-out accuracy is not substantially above chance or is far below the 94% training accuracy, the AM visualizations in Figure 15 cannot be interpreted as evidence of clinically meaningful feature extraction; if it is high, additionally compute a quantitative overlap (e.g., Dice) between AM saliency regions and expert-marked B-lines/A-lines/consolidations on a small annotated subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing condition for the Section 4 claim that activation maximization 'effectively highlights the medical features' is that the VGG-16 model trained on lung ultrasound is actually a functioning classifier on unseen data. The paper reports only 'accuracy of 94% on the training dataset' and never reports test or validation accuracy, per-class performance, or the split used. Without this, the AM heatmaps in Figures 15-16 could reflect a model that memorized training artifacts such as probe positioning, gain settings, text overlays, or patient-level confounds, rather than true B-lines, A-lines, or consolidations. The manuscript itself acknowledges the risk: 'some boundaries might be mistakenly recognized as significant medical features and could lead to misclassification.' Since the central claim is that the method surfaces clinically plausible, class-relevant features, the link between model generalization and visualization is load-bearing: if held-out accuracy is near chance, the 'effectively highlights' conclusion collapses. The diabetes feature-ranking claim is comparatively better supported by the ablative experiment (4-feature model at about 80% vs remaining-features at about 69%), though that result also lacks confidence intervals and multi-seed variability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper demonstrates three sensitivity-analysis tools on neural-network models: Sobol global sensitivity analysis on a small fully connected network trained on tabular diabetes data; local input-pixel perturbation sensitivity heatmaps on VGG-16 and ResNet-18 trained on CIFAR-10; and activation maximization (AM) on the same convolutional models, applied finally to a lung-ultrasound (LUS) classification task. The central claims are that Sobol indices identify Glucose, BMI, and Age as the leading diabetes predictors and that reducing the input features from eight to the four most influential (Glucose, BMI, Age, Insulin) loses little accuracy; that local sensitivity maps show decreasing sensitivity with depth in VGG-16 and preserved sensitivity in ResNet-18; and that AM highlights clinically plausible features (B-lines, consolidations, A-lines) in LUS images, more focused than Grad-CAM. The authors provide code repositories for the experiments.","tokens_in":11524,"tokens_out":3759,"duration_ms":35952,"significance":"If the claims hold, the paper would provide a useful empirical illustration that global sensitivity analysis can guide feature pruning in small tabular MLPs and that activation maximization can localize diagnostically relevant ultrasound features. The strengths are the availability of code, the use of established libraries (SALib), the convergence check of Sobol indices with increasing sample size, and the comparisons with PCA and Grad-CAM. However, the load-bearing evidence is qualitative and single-run in the convolutional and medical-imaging parts, and the feature-reduction claim is not supported by statistical confidence measures. As it stands, the contribution is a set of demonstrations rather than a validated methodology.","major_comments":[{"comment":"The claim that the VGG-16 model 'successfully distinguishes' COVID-19, bacterial pneumonia, and normal lungs in Section 4, and that activation maximization 'effectively highlights the medical features', rests on a model for which only 94% training accuracy is reported. Without a held-out test accuracy, per-class metrics, the train/test split, and class balance, the AM heatmaps in Figures 15-16 cannot be separated from patterns learned on dataset artifacts such as probe positioning, gain settings, or text overlays. The manuscript itself acknowledges that 'some boundaries might be mistakenly recognized as significant medical features,' but this does not rescue the claim. Please report test/validation accuracy and per-class performance, and ideally quantify or blind-evaluate the AM and Grad-CAM localizations against clinician annotations.","section":"Section 4 and Conclusion"},{"comment":"The Sobol analysis treats the eight clinical inputs as independent random variables sampled uniformly over their observed ranges via SALib. Clinical variables such as Glucose and Insulin, or Age and BMI, are correlated, and Sobol indices are not uniquely interpretable under dependent inputs; the ranking of Glucose, BMI, Age, and Insulin, and consequently the feature-subset experiment, could change under the true joint distribution. Please state explicitly the sampling ranges and either sample from a dependence-aware distribution (e.g., an empirical copula or a Gaussian copula fitted to the data) or provide a sensitivity check showing that the ranking is robust to the independence assumption. This is load-bearing for the paper's main feature-identification claim.","section":"Section 2.2-2.3"},{"comment":"The feature-reduction comparison reports test accuracy of 82.5% for the full model, 'about 80%' for the four-feature model, and 'about 69%' for the remaining four features, without confidence intervals, number of seeds, or run-to-run variability. At the dataset size used, these differences are within typical training noise, so the statement that feature reduction occurs 'without significant loss of the accuracy' is not supported by the presented evidence. Please report means and standard deviations over multiple random seeds and a statistical comparison (e.g., a paired test or McNemar's test).","section":"Section 2.3"},{"comment":"The conclusion that sensitivity 'gradually decreases with depth' in VGG-16 and that ResNet-18 preserves sensitivity is based on visual inspection of heatmaps for a single class (airplanes) from a single model, and the plotted scales in Figures 5-10 differ by orders of magnitude (10^0 down to 10^-2). This trend may be partly an artifact of the chosen normalization or output scaling rather than a property of the models. Please provide quantitative summary statistics (e.g., mean or quantiles of the sensitivity per block or layer, aggregated over multiple classes) under a fixed normalization across layers, before drawing conclusions about depth-dependent sensitivity.","section":"Section 3.1 and 3.4"}],"minor_comments":[{"comment":"There are several typos and grammatical errors, including 'ust' for 'use', 'Iy' for 'It', 'classily' for 'classify', and inconsistent capitalization of 'Grad-Cam' versus 'Grad-CAM'; a thorough proofread is needed.","section":"Throughout"},{"comment":"The local-sensitivity formula in Section 3 is typeset poorly and is not numbered; the notation for the block operator B_q and the perturbation epsilon should be defined more carefully, including the norm used (the formula shows a squared L2 norm, but the surrounding text says 'sensitivity of the pixel').","section":"Section 3"},{"comment":"The activation maximization update in equation (4) uses two coefficients epsilon_1 and epsilon_2, while equation (5) only has epsilon_1; the relationship between these coefficients and the reported values (epsilon_1 = 0.1, epsilon_2 = 0.1) should be clarified.","section":"Section 3.3"},{"comment":"The PCA comparison is reported without methodological details, such as whether the features were standardized before PCA, how many components were retained, and how 'most important indicators' were derived from the loadings; please add these details.","section":"Section 2.4"},{"comment":"Some references appear mismatched: reference [13] is cited for the diabetes dataset but appears to describe the ADAP learning algorithm, and reference [14] is listed as a Kaggle dataset; please verify the citation-to-source correspondence.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is primarily an application and demonstration of known sensitivity-analysis tools, with limited methodological novelty. The main correctness risk is the missing held-out evaluation for the lung-ultrasound classifier, which is fixable within the manuscript's scope if the authors provide the missing experiments and statistical support. In its current form the novelty and rigor are below what I would expect for a mathematical journal, but it could be suitable for an applied venue after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is not a new method paper. It applies Sobol global sensitivity, local sensitivity heatmaps, and activation maximization to a tiny diabetes net, VGG-16/ResNet-18 on CIFAR-10, and a VGG-16 on lung ultrasound. That's fine; its value is as a careful worked demonstration with code.\n\nThe Sobol analysis on the 8-input diabetes network is the cleanest part. The convergence plot across sample sizes 128 to 65536 is a nice touch, and the feature-subset experiment (four most sensitive features ~80% vs four least ~69% test accuracy) directly supports the claim that the sensitivity ranking is usable for feature reduction. The PCA comparison is also honest and useful. The CIFAR-10 visualizations are standard, but the exposition is clear.\n\nThe load-bearing problem is the lung-ultrasound section. The paper reports only 94% training accuracy and never gives test or validation accuracy, nor the split used. The AM heatmaps in Figures 15-16 might be picking up genuine B-lines, A-lines, and consolidations, but they might equally be artifacts of a model that memorized probe position, gain settings, or text overlays. The authors themselves write that 'some boundaries might be mistakenly recognized as significant medical features,' so they see the risk, but they don't close it with a held-out evaluation. That is fixable but essential before claiming the method 'successfully distinguishes' the three classes.\n\nSecond: the diabetes ablative experiments appear to be a single run. 82.5% vs 80% vs 69% has no confidence intervals or seeds, so we don't know whether the 4-feature model is statistically equivalent to the full model. The Sobol ranking also assumes the inputs are independent and uniformly sampled over observed ranges; clinical variables like BMI, age, and blood pressure are correlated, so the indices are conditional on that assumption. The paper doesn't flag this, and it matters for the feature-ranking claim.\n\nThe CIFAR-10 sensitivity-depth observations are qualitative. They reproduce known results about early layers encoding edges and color and residual connections preserving gradient flow, so they are plausible but not quantified.\n\nOverall, the paper is honest, reproducible, and scoped correctly except for the LUS overreach. The math is standard and the claims are mostly aligned with what the experiments support. I would send it to a serious referee, but I would expect a substantial revision: report held-out accuracy on the LUS data, add uncertainty estimates or multi-seed runs for the diabetes experiment, and soften the 'successfully distinguishing' language.\n\nThe likely audience is practitioners in applied ML and medical imaging who want a worked example of sensitivity-based interpretation, not method developers. My recommendation: treat it as a revise-and-resubmit candidate and give it referee time.","headline":"A straightforward, reproducible demonstration of standard sensitivity/interpretability tools on small models; the diabetes feature-ranking part holds up reasonably, but the lung-ultrasound claim needs held-out accuracy before it earns its conclusion.","tokens_in":11990,"tokens_out":1985,"would_cite":false,"duration_ms":18425,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65C05","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper demonstrates that sensitivity-analysis methods can identify which inputs a neural network actually depends on, from glucose, BMI, and age in a diabetes model to B-lines, consolidations, and A-lines in lung ultrasound.","keywords":["convolutional networks","Sobol sensitivity analysis","activation maximization","feedforward networks","local sensitivity analysis","explainable AI","feature selection","lung ultrasound"],"falsifier":"Recompute the Sobol indices using samples drawn from the empirical joint distribution of the eight clinical variables (for example, resampling from the actual dataset or a fitted copula) instead of independent uniform ranges, then retrain the four-feature network. If Glucose, BMI, and Age stop dominating the ranking, or if the four-feature accuracy falls well below the reported 80%, the feature-selection conclusion is an artifact of the independence assumption rather than a stable property of the diabetes data.","tokens_in":11086,"feed_emoji":"🔍","tokens_out":8815,"duration_ms":71855,"temperature":0.7,"pith_summary":"This paper sets out to show that sensitivity analysis can make neural networks interpretable without specialized machinery: standard variance-based and gradient-based tools reveal which inputs actually drive a network's output. On a small fully connected network for an open diabetes dataset, Sobol global sensitivity indices rank glucose, BMI, and age as the leading features, and retraining on just four of the eight inputs keeps test accuracy near 80% versus 82.5% with all inputs. On convolutional networks, where global analysis is too expensive, the paper turns to local pixel-perturbation sensitivity and activation maximization, finding that sensitivity fades with depth in VGG-16, that residual connections in ResNet-18 preserve deep-layer sensitivity, and that activation maximization brings out clinically recognized signs in lung ultrasound. If correct, these results make a practical case: sensitivity analysis can support feature selection in small tabular models and sanity-check what a medical image classifier actually looks at.","feed_headline":"Sensitivity tools expose the inputs a neural net truly depends on","feed_subtitle":"Applied to diabetes and lung ultrasound, the tools flag glucose, BMI, age, and key ultrasound signs.","key_machinery":"The machinery is fourfold, each tied to a measurement. First, Sobol's variance decomposition splits the network output $f(x_1,\\dots,x_k)$ into $f_0 + \\sum_i f_i(x_i) + \\sum_{i<j} f_{ij}(x_i,x_j) + \\cdots$, with first-order index $S_i = V_i/V(y)$ and total index $S_{Ti} = 1 - V_{\\sim i}/V(y)$; this is what produces the Glucose/BMI/Age ranking. Second, local sensitivity computes, for each pixel and color channel, the change in the block output norm when that pixel is perturbed: $s_{ij}^C = \\|B_q(\\dots,x_{ij}+\\varepsilon,\\dots)\\|_2 - \\|B_q(\\dots,x_{ij},\\dots)\\|_2$, giving the heatmaps that show depth-dependent decay. Third, activation maximization solves $x^* = \\arg\\max_x a_{ij}(\\theta,x)$ by gradient ascent, regularized by total variation so the synthetic images stay close to natural ones; this yields the feature-visualization and lung-ultrasound images. Fourth, the residual block $\\mathrm{Res}(x)=x+F_{\\mathrm{act}}(x)$ is the structural reason ResNet-18 preserves sensitivity in deep layers, since the identity shortcut lets input perturbations survive propagation.","core_discovery":"The central discovery is that sensitivity analysis is not just a diagnostic curiosity but a usable tool for deciding what a network really uses. For the eight-input fully connected diabetes network, Sobol global sensitivity indices rank Glucose, BMI, and Age as the dominant first-order contributors; retraining the same architecture on the four most influential features (Glucose, BMI, Age, Insulin) keeps test accuracy at about 80% against 82.5% with all eight inputs, while the four least influential features alone reach only about 69%. In the convolutional setting, the paper finds that local perturbation sensitivity in VGG-16 declines through the five blocks (reaching scale $10^{-2}$), that ResNet-18's residual blocks keep later-layer sensitivity near $10^0$, and that activation maximization with total-variation regularization produces class images that capture distinctive contours (airplane wings, deer antlers, cat ears). Applied to lung ultrasound, the same activation-maximization method highlights B-lines for COVID-19, consolidations for bacterial pneumonia, and A-lines for healthy lungs, matching clinical signs and Grad-CAM heatmaps but with a more focused sensitive area.","pith_inferences":["A natural next test the paper does not run: recompute the Sobol indices with correlated sampling from the empirical joint distribution. If the Glucose/BMI/Age ranking survives, the method becomes a low-cost screening tool for other small clinical models; if not, the ranking is an artifact of the independence assumption.","The depth-sensitivity decay in VGG-16 suggests a quantitative architectural heuristic: plot the mean local-sensitivity magnitude per block and stop adding blocks once it drops below a threshold. This is implied by the paper's data but not stated as a design rule.","The paper's claim that activation maximization is more effective than Grad-CAM on ultrasound is qualitative; a localization benchmark (e.g., comparing highlighted regions against expert-annotated A/B-lines) would turn that comparison into a measurable claim.","Because the same activation-maximization procedure exposes boundary artifacts that mimic B-lines, it could double as a data-audit tool: run it before training a medical classifier to see which image features are spuriously learnable."],"forward_implications":["On tabular clinical data with few features, Sobol indices can serve as a feature-selection step: the paper's four-feature network retains roughly 80% test accuracy, only about 2.5 points below the full model.","VGG-16's sensitivity falls from scale $10^0$ in early blocks to $10^{-2}$ in the fifth block, implying that adding ever-deeper plain convolutional blocks has diminishing returns for capturing input-level features.","ResNet-18's residual blocks maintain deep-layer sensitivity around $10^0$, so the identity shortcut is a concrete mechanism that keeps later layers responsive to input perturbations.","Activation maximization on lung ultrasound highlights clinically named signs—B-lines, consolidations, A-lines—and matches Grad-CAM observations, but with narrower high-sensitivity regions, suggesting it can complement gradient-based attribution in medical imaging.","When a class image is activated by parameters of another class, the network inserts features of the target class (e.g., antlers into a deer image), showing how activation maximization can reveal class-confusing cues that adversarial attacks might exploit."],"supporting_citations":[{"why":"Supplies the clinical diabetes dataset used in the tabular experiments.","marker":"[14]"},{"why":"Defines the Sobol variance decomposition and sensitivity indices that produce the feature ranking.","marker":"[16]"},{"why":"Provides the software implementation used to compute the Sobol indices.","marker":"[18]"},{"why":"Supports the quasi-random sampling whose convergence with sample size is shown in Figure 4.","marker":"[19]"},{"why":"Defines the VGG-16 architecture used for the CIFAR-10 and lung-ultrasound experiments.","marker":"[23]"},{"why":"Introduces the activation-maximization formulation behind the synthetic feature images.","marker":"[28]"},{"why":"Introduces the residual-block construction whose sensitivity-preserving effect is tested with ResNet-18.","marker":"[29]"},{"why":"Supplies the lung-ultrasound data and the clinical signs used to interpret the activation-maximization outputs.","marker":"[30]"},{"why":"Defines Grad-CAM, the baseline the paper compares against activation maximization on lung ultrasound.","marker":"[32]"}],"fun_headline_variants":["Sensitivity analysis reveals the features neural nets actually rely on","Global sensitivity ranks diabetes inputs, slims network with minimal loss","Explaining neural nets: which inputs matter and which don't","Sensitivity methods show what CNNs see: from glucose to ultrasound B-lines","Trim neural nets by asking which inputs they truly use"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking of Glucose, BMI, and Age as the dominant diabetes features rests on treating the eight clinical inputs as independent random variables sampled uniformly over their observed ranges, and on assuming this particular small network's sensitivities mirror the true relationship between those features and diabetes.","fun_headline_variants_meta":{"raw":{"variants":["Sensitivity analysis reveals the features neural nets actually rely on","Global sensitivity ranks diabetes inputs, slims network with minimal loss","Explaining neural nets: which inputs matter and which don't","Sensitivity methods show what CNNs see: from glucose to ultrasound B-lines","Trim neural nets by asking which inputs they truly use"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2836,"prompt_tokens":957,"completion_tokens":1879,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1792}},"tokens_in":573,"tokens_out":1879,"duration_ms":10375,"temperature":1.0,"reasoning_tokens":1792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:32:22.911744+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the Sobol indices using samples drawn from the empirical joint distribution of the eight clinical variables (for example, resampling from the actual dataset or a fitted copula) instead of independent uniform ranges, then retrain the four-feature network. If Glucose, BMI, and Age stop dominating the ranking, or if the four-feature accuracy falls well below the reported 80%, the feature-selection conclusion is an artifact of the independence assumption rather than a stable property of the diabetes data.","supporting_citations":[{"cited_title":"Diabetes dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the clinical diabetes dataset used in the tabular experiments."},{"cited_title":"Global sensitivity indices for nonlin- ear mathematical models and their Monte Carlo estimates,","cited_arxiv_id":null,"evidence_quote":"Defines the Sobol variance decomposition and sensitivity indices that produce the feature ranking."},{"cited_title":"SALib: An open- source python library for sensitivity analysis,","cited_arxiv_id":null,"evidence_quote":"Provides the software implementation used to compute the Sobol indices."},{"cited_title":"On quasi-monte carlo integrations,","cited_arxiv_id":null,"evidence_quote":"Supports the quasi-random sampling whose convergence with sample size is shown in Figure 4."},{"cited_title":"Deep resid- ual learning for image recognition,","cited_arxiv_id":null,"evidence_quote":"Introduces the residual-block construction whose sensitivity-preserving effect is tested with ResNet-18."},{"cited_title":"Accelerating de- tection of lung pathologies with explainable ultra- sound image analysis,","cited_arxiv_id":null,"evidence_quote":"Supplies the lung-ultrasound data and the clinical signs used to interpret the activation-maximization outputs."},{"cited_title":"Grad-cam: Visual explanations from deep networks via gradient-based localization,","cited_arxiv_id":null,"evidence_quote":"Defines Grad-CAM, the baseline the paper compares against activation maximization on lung ultrasound."}],"review_version":1}