{"id":"4c73f9d0-1d67-427c-bf50-6147eff395dc","arxiv_id":"2412.02012","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"INSIGHT, a weakly-supervised aggregator with built-in heatmap generation, achieves strong classification and segmentation on CT and whole-slide pathology benchmarks using only image-level labels.","lead":"INSIGHT is a new method for analyzing large medical images using only image-level labels, producing built-in heatmaps of disease regions. It combines small and large convolutional modules and a special pooling step to improve both classification accuracy and lesion localization on CT scans and pathology slides.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never specifies how gradients flow through the non-differentiable Otsu threshold in Eq.","rationale":"I agree with the reader that the most load-bearing concern is the unspecified gradient flow through the non-differentiable Otsu threshold in Eq. (2). The central claim depends on the heatmap H being a trainable pathway from the classification loss to the detection and context modules. If the hard threshold is applied in the training forward pass as the pseudocode suggests, the indicator function blocks gradients for below-threshold entries, which could prevent learning on precisely the small, clinically important lesions the method claims to capture. The paper's strong empirical results imply that some mechanism — perhaps a straight-through estimator or thresholding only at inference — must be in place, but the text does not describe it. This is a reproducibility and correctness risk, not just a stylistic omission. Other issues, such as the Virchow2 classification table showing WiKG with higher AUC than INSIGHT and the ablation text claiming Dice improves when Table 6 actually shows a Dice decrease from Row 3 to Row 4 on CAMELYON16, are real but secondary; they affect the strength of the claims rather than the core architecture. The paper has genuine strengths: public code, evaluation on multiple benchmarks, ablations, and qualitative heatmap comparisons. The proposed concrete test — inspecting the released code and running controlled ablations around the threshold — would settle whether the described method is actually the evaluated method, so the conditional verdict should stand until that clarification is provided.","tokens_in":16978,"tokens_out":8118,"duration_ms":79063,"concrete_test":"Run the released code (project website) and inspect the training forward/backward for Eq. (2). Determine whether the Otsu threshold is applied during training and how the indicator influences the gradient. Then run three small-scale CAMELYON16 training experiments: (a) the published code as-is, (b) the same model with the hard threshold removed from the training forward pass (using H directly in SmoothMax), and (c) the model with a straight-through estimator for the indicator. Compare AUC and Dice on the validation set. If (a) trains successfully and matches the paper's reported results, document the gradient mechanism; if (a) does not converge or (b)/(c) differ substantially, the paper must be revised to state the actual training-time operation of Eq. (2).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that INSIGHT's internal heatmaps are learned end-to-end and drive both classification and localization — depends on Eq. (3) connecting the classification loss to the detection and context modules through the thresholded heatmap H′ = H·I(H>T) in Eq. (2). The indicator function has zero derivative almost everywhere, so for any heatmap entry below the Otsu threshold T, dL/dH_i = 0 through this path. The paper does not describe a straight-through estimator, a soft threshold, or a decision to apply Eq. (2) only at inference. The pseudocode in Appendix D places the thresholding before SmoothMax pooling with no training/inference distinction, implying it is part of the training forward pass. If that is the actual implementation, then only above-threshold regions receive a learning signal; small or low-confidence lesions that fall below T are never pushed upward, which can cause degenerate training or failure to localize subtle findings. Conversely, if the authors used a differentiable approximation or applied the threshold only during evaluation, the method as described differs from the one evaluated. This is not a mere implementation detail: the heatmap is the only pathway from the loss to the convolutional modules, so the architecture's claimed behavior rests on this unspecified gradient flow. The reported strong results suggest some mechanism exists, but the paper's silence on it makes the central claim unfalsifiable from the text alone and blocks reproduction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces INSIGHT, a weakly-supervised aggregator for whole-slide images and CT volumes. Starting from pre-trained spatial embeddings, INSIGHT uses a detection module with small convolutional kernels and a context module with larger kernels to produce an internal heatmap, which is then thresholded with Otsu's method and aggregated by SmoothMax pooling to yield classification predictions. Training uses binary cross-entropy plus a spectral-decoupling regularizer. The authors evaluate on MosMed, CAMELYON16, and BRACS, reporting classification AUC and weakly-labeled Dice scores, and compare built-in heatmaps against Grad-CAM. The central claim is that end-to-end learning with image-level labels produces both state-of-the-art classification and high-quality internal heatmaps that align with diagnostically relevant regions without post-hoc visualization.","tokens_in":17198,"tokens_out":5371,"duration_ms":53288,"significance":"If the method works as described, INSIGHT is a useful contribution: it replaces post-hoc saliency methods with a built-in heatmap pathway, demonstrates compatibility with several foundation models (UNI, Virchow2, DINOv2), and reports gains on small-lesion Dice on the held-out CAMELYON16 test set. The evaluation is largely sound in design: it uses external public benchmarks with held-out test labels, hyperparameters are selected on a validation set, and code is publicly available, so the main quantitative claims are not circular. However, the paper currently leaves a load-bearing training detail unspecified: how gradients flow through the non-differentiable Otsu threshold in Eq. (2). In addition, the ablation narrative in Sec. 5.5 is internally inconsistent with Table 6. These issues must be resolved before the central claim can be considered reproducible.","major_comments":[{"comment":"The paper never specifies how gradients flow through the hard threshold H′ = H · I(H > T). The indicator function has zero derivative almost everywhere, so if this operation is part of the training graph, no learning signal reaches the detection or context modules for any heatmap entry below the Otsu threshold; if the threshold is instead applied only at inference, then the method described in Sec. 3.2 is not the method that was trained and evaluated. The pseudocode in Appendix D places the thresholding before SmoothMax pooling with no training/inference distinction, implying it is in the training forward pass. Because H′ is the only path from the classification loss to the convolutional modules, this is a load-bearing point for the paper's central claim. Please state explicitly whether a straight-through estimator, a soft threshold, or inference-only thresholding is used, and provide evidence about training behavior when lesion activations fall below the threshold.","section":"Sec. 3.2, Eq. (2); Appendix D"},{"comment":"The text claims that adding regularization (Row 4) 'improves both AUC and Dice across datasets,' but Table 6 shows that on CAMELYON16 the Dice score decreases from 76.7 ± 15.7 (Row 3) to 74.6 ± 19.1 (Row 4) when the regularizer is added, while AUC increases from 0.969 to 0.990. The narrative also mentions that 'label smoothing' benefits BRACS, but label smoothing is not described in Sec. 3.3 or in the ablation table. Please correct the text to match the numbers and specify exactly which regularizer or combination of regularizers is included in the Rg column.","section":"Sec. 5.5, Table 6"},{"comment":"On MosMed, the reported segmentation Dice of 42.7 ± 15.3 for INSIGHT overlaps within one standard deviation with the 3D GAN baseline (41.2 ± 14.7) and with 3D U-Net* (40.5 ± 21.3), so the claim of 'state-of-the-art performance' on this dataset is not statistically substantiated. The abstract's broader claim of 'high weakly-labeled semantic segmentation performance' rests mainly on the CAMELYON16 results. Please provide significance testing across the five folds for the MosMed Dice comparison, or temper the claim accordingly.","section":"Sec. 5.1, Table 1; Sec. 5.2"}],"minor_comments":[{"comment":"Equation (3) defines SmoothMax pooling over entries H′_i, while the pseudocode in Appendix D pools over the stitched full heatmap H_full; please clarify whether pooling is applied per patch and then aggregated or directly on the full-resolution heatmap.","section":"Sec. 3.2 and Appendix D"},{"comment":"The citation to Maddison et al. (2016) refers to the Concrete distribution paper, which is not the standard reference for the SmoothMax (softmax-weighted average) pooling used here; consider citing a more direct source or defining the operator without that citation.","section":"Sec. 3.2"},{"comment":"For MosMed-B, please state explicitly how the Dice score is aggregated over the 50 volumes (per-volume mean, slice-wise mean, or pooled over all voxels) so that the reported mean and standard deviation are interpretable.","section":"Sec. 4.1"},{"comment":"The input description says CT volumes are resized to 518 × 518 × 32, but the encoder input is 224 × 224; please clarify how slices are sampled or cropped before feature extraction.","section":"Sec. 4.1 and Appendix A"},{"comment":"The sentence ending '...producing calibrated, built-in heatmaps as part of its predictions, as shown in Fig. 3. ensures alignment between the model’s outputs and diagnostic reasoning.' is a fragment and should be rewritten.","section":"Sec. 2.3"},{"comment":"The claim that SmoothMax pooling 'yields a substantial Dice increase of 34.6 on CAMELYON16' is consistent with Table 6 (from 42.1 to 76.7), but the text later says the increase is 34.6 without specifying the baseline row; please clarify that the comparison is between Row 2 and Row 3.","section":"Sec. 5.5"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the non-differentiable Otsu threshold is legitimate and should be the primary focus of the revision. If the authors confirm that a straight-through or soft-threshold mechanism is used, the issue may be quickly resolved, but the current text makes the central training mechanism unfalsifiable. The ablation inconsistency in Sec. 5.5 is also straightforward to fix but must be addressed. The MosMed segmentation claim is weaker than the CAMELYON16 claim and should be reframed or statistically supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: INSIGHT is a credible engineering contribution that combines known pieces (detection/context convs, SmoothMax pooling, Otsu) in a new arrangement and shows real gains in weakly-supervised segmentation on CAMELYON16, MosMed, and BRACS. The paper deserves a serious referee, but it has a load-bearing gap in the method description that needs to be fixed.\n\nWhat's actually new: the architecture details—1x1 conv detection, 3x3 conv context suppression, multiplicative gating, then SmoothMax pooling over Otsu-thresholded heatmaps—form a combination I haven't seen in the cited MIL literature. The empirical work is solid: three public benchmarks, held-out test sets, hyperparameters tuned on validation only, stratified analysis of lesion sizes with permutation tests, and runtime comparisons. The segmentation numbers are the strongest part: on CAMELYON16 with UNI, Dice 74.6 vs 67.7 for the best baseline; with Virchow2, 78.3 vs 66.8. Those are non-trivial margins. The qualitative heatmaps also look more coherent than Grad-CAM.\n\nNow the soft spots, in order.\n\nFirst, and most important: the paper never says how gradients flow through the Otsu threshold in Eq. (2). The indicator function has zero derivative almost everywhere, and the pseudocode in Appendix D applies the threshold before SmoothMax pooling with no training/inference distinction. If the implementation uses a straight-through estimator or a soft threshold, that needs to be stated. If it doesn't, then below-threshold entries get no learning signal and the training story in Section 3.2 is incomplete. This is not a minor detail; it's the path from the loss to the convolutional modules. The strong empirical results suggest some mechanism works, but the paper as written is not reproducible on this point.\n\nSecond, the ablation text in Sec. 5.5 says regularization \"improves both AUC and Dice across datasets,\" but Table 6 shows CAMELYON16 Dice dropping from 76.7 to 74.6 when the regularizer is added. That's a direct contradiction that should be fixed.\n\nThird, the MosMed comparison set is thin—only one classification baseline plus a supervised U-Net and a GAN for segmentation—so the \"state-of-the-art\" classification claim there is under-supported. Also, BRACS AUCs have no error bars.\n\nFourth, the abstract's \"state-of-the-art classification results\" is too broad: with Virchow2 on CAMELYON16, WiKG gets AUC 0.996 vs INSIGHT's 0.992.\n\nNone of these are fatal. They are addressable, and the core architecture is sound enough to warrant revision. This paper is for researchers working on weakly supervised medical image analysis, especially those using pre-trained patch/slice embeddings. It deserves a serious referee, and I would engage with a revised version.","headline":"A credible empirical aggregator with a real reproducibility gap: the Otsu threshold step in Eq. (2) has no specified gradient path, and the ablation text contradicts its own table.","tokens_in":17763,"tokens_out":2890,"would_cite":true,"duration_ms":26808,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"INSIGHT claims that a weakly supervised aggregator can use only image-level labels to produce both strong classification and built-in diagnostic heatmaps, reporting AUC 0.990 and Dice 74.6% on CAMELYON16.","keywords":["weakly supervised learning","multiple instance learning","whole-slide images","chest CT volumes","interpretable heatmaps","semantic segmentation","foundation model embeddings","SmoothMax pooling"],"falsifier":"Instrument the gradient of the classification loss with respect to the detection and context module outputs during early training on CAMELYON16. If every below-threshold heatmap position has exactly zero gradient in every batch, then the threshold in Eq. (2) blocks learning and some unstated gradient approximation is carrying the reported result; if nonzero gradients appear, an implementation detail equivalent to a soft threshold is doing the work.","tokens_in":16709,"feed_emoji":"🩺","tokens_out":13668,"duration_ms":109707,"temperature":0.7,"pith_summary":"INSIGHT is a weakly supervised aggregator built around the idea that a heatmap should be part of the prediction mechanism, not an afterthought. The paper claims that by preserving spatial resolution in pre-trained embeddings and fusing a fine-detail detection map with a context-suppression map, a model can classify disease and localize it in the same forward pass using only slide- or volume-level labels. On three benchmarks—COVID-19 chest CT, metastatic breast cancer detection in lymph-node whole-slide images, and multi-label breast carcinoma subtyping—INSIGHT reports higher AUC and Dice than the compared MIL aggregators, with the largest localization gains on small lesions. If this holds, explainable medical image analysis becomes much cheaper: clinicians get diagnostic heatmaps without pixel-level annotations or post-hoc visualization tools.","feed_headline":"Built-in heatmaps from weak labels beat MIL and Grad-CAM","feed_subtitle":"On CT and pathology slides, one forward pass classifies disease and localizes lesions, reaching 78.3% Dice.","key_machinery":"The carrying object is the internal heatmap $H$ that INSIGHT produces before pooling. The detection module uses $1\\times1$ convolutions to capture fine-grained local evidence, the context module uses $3\\times3$ convolutions to build a suppression signal, and the fusion rule $H = \\sigma((1-\\sigma(H_{\\mathrm{Con}}))\\odot H_{\\mathrm{Det}})$ lets context gate the detection map. Otsu's algorithm selects a threshold $T$ and applies $H' = H\\cdot \\mathbb{I}(H>T)$ to remove low-activation background, and SmoothMax pooling, $\\hat{y}=\\sum_i H'_i e^{\\alpha H'_i}/\\sum_i e^{\\alpha H'_i}$, aggregates the thresholded heatmap into a prediction while keeping spatial resolution intact until the final step. The training objective combines binary cross-entropy with spectral decoupling regularization, and the heatmap itself serves as the explanation.","core_discovery":"On its own terms, the central discovery is that a weakly supervised aggregator can be designed so that heatmap generation is an inductive bias rather than a post-hoc interpretation, and that this improves both tasks at once. INSIGHT forms a fused heatmap $H = \\sigma((1-\\sigma(H_{\\mathrm{Con}}))\\odot H_{\\mathrm{Det}})$, where a detection module built from small convolutions highlights fine details and a context module with larger convolutions suppresses false positives; Otsu thresholding then isolates high-saliency regions, and SmoothMax pooling turns the thresholded heatmap into the categorical prediction. With only image-level labels, the paper reports AUC 0.990 and Dice 74.6% on CAMELYON16 using UNI features (Dice 78.3% with Virchow2), AUC 0.962 on MosMed CT, and multi-label AUC gains of 3.3 points for ADH and FEA on BRACS. It also reports that these built-in heatmaps substantially outperform Grad-CAM in Dice on the same encoders, and that the small-lesion Dice advantage over MIL baselines is statistically significant.","pith_inferences":["Because INSIGHT only needs a spatially preserving pretrained encoder, the same detection-context fusion could be dropped onto newer foundation models as they appear; the paper tests two pathology encoders, but a natural extension is to measure whether the Dice gains persist with each new encoder generation.","The reported small-lesion gains suggest that Otsu thresholding plus SmoothMax pooling is acting as a spatial denoiser; a direct test would be to replace the hard threshold with a differentiable soft threshold and compare Dice, which would also remove the gradient-flow concern in Eq. (2).","The built-in heatmaps could be used as pseudo-labels to bootstrap fully supervised segmentation or to let clinicians refine, rather than create, annotations; the paper mentions this as future value but does not evaluate it.","The use of one global Otsu threshold per heatmap may be a limitation when a slide contains lesions with very different intensity profiles; testing per-region or per-scale thresholding would show whether a single threshold is load-bearing for the localization claim."],"forward_implications":["On CAMELYON16 with UNI features, INSIGHT reports AUC 0.990 and Dice 74.6%, 6.9 Dice points above the best compared MIL baseline; with Virchow2 features, Dice reaches 78.3%.","On MosMed COVID-19 CT, INSIGHT reports classification AUC 0.962 versus 0.914 for the best baseline, and volume-level heatmap Dice 42.7%, above both a fully supervised 3D U-Net (40.5%) and a weakly supervised 3D GAN (41.2%).","On the multi-label BRACS dataset, INSIGHT improves ADH and FEA AUC by 3.3 points over the best compared aggregator while matching or beating it on DCIS and Invasive, indicating that per-class heatmaps can separate coexisting lesion subtypes.","In the stratified lesion-size analysis, INSIGHT's Dice advantage over MIL baselines is largest for small and moderate lesions—for example, 42.8% versus 20.7% for CLAM-MB with UNI—with differences significant at $p<0.01$.","The ablation study attributes the localization gain to the two architectural choices: context suppression alone raises CAMELYON16 Dice by 16.6 points, and adding SmoothMax pooling raises it by another 34.6 points."],"supporting_citations":[{"why":"Provides UNI, the self-supervised pathology encoder whose spatially preserved feature maps INSIGHT uses for CAMELYON16 and BRACS.","marker":"Chen et al. 2024"},{"why":"Supplies the CAMELYON16 WSI benchmark with pixel-level tumor annotations used to compute Dice.","marker":"Litjens et al. 2018"},{"why":"Supplies the BRACS multi-label breast carcinoma dataset used for subtype classification.","marker":"Brancati et al. 2021"},{"why":"Supplies the MosMed chest CT dataset, including the voxel-labeled subset used for segmentation evaluation.","marker":"Morozov et al. 2020"},{"why":"Supplies the DINOv2 ViT used as the CT slice encoder for MosMed.","marker":"Oquab et al. 2024"},{"why":"Supplies Virchow2, the second pathology foundation model used to show that INSIGHT's gains are not tied to one encoder.","marker":"Zimmermann et al. 2024"},{"why":"Provides the ABMIL attention-based MIL baseline that INSIGHT compares against.","marker":"Ilse et al. 2018"},{"why":"Provides the CLAM toolbox used for WSI patch extraction and the CLAM-SB/MB baselines.","marker":"Lu et al. 2021"},{"why":"Provides the threshold-selection algorithm used to binarize the fused heatmap in Eq. (2).","marker":"Otsu, 1979"},{"why":"Underlies SmoothMax pooling, the operator that converts thresholded heatmaps into predictions.","marker":"Maddison et al. 2016"}],"fun_headline_variants":["Weak-label heatmaps outperform MIL and Grad-CAM in one pass","Heatmap inductive bias lifts weak-label classification and segmentation","Fused weak-label heatmaps beat post-hoc CAM on small lesions","One pass classifies and localizes via built-in heatmaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that end-to-end training works despite the non-differentiable Otsu threshold in $H' = H\\cdot \\mathbb{I}(H>T)$; if gradients cannot reach the detection and context modules for below-threshold regions, the described heatmap-learning mechanism would not actually be what is being optimized.","fun_headline_variants_meta":{"raw":{"variants":["Weak-label heatmaps outperform MIL and Grad-CAM in one pass","Heatmap inductive bias lifts weak-label classification and segmentation","Fused weak-label heatmaps beat post-hoc CAM on small lesions","One pass classifies and localizes via built-in heatmaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000805,"raw_usage":{"total_tokens":3531,"prompt_tokens":939,"completion_tokens":2592,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2530}},"tokens_in":555,"tokens_out":2592,"duration_ms":214969,"temperature":1.0,"reasoning_tokens":2530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:55:46.065693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument the gradient of the classification loss with respect to the detection and context module outputs during early training on CAMELYON16. If every below-threshold heatmap position has exactly zero gradient in every batch, then the threshold in Eq. (2) blocks learning and some unstated gradient approximation is carrying the reported result; if nonzero gradients appear, an implementation detail equivalent to a soft threshold is doing the work.","supporting_citations":[],"review_version":1}