{"id":"ad00cc5d-85c5-47f2-8a72-3f5fefcfde44","arxiv_id":"2504.14783","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Dropping the top-k most important instances in a bag regularizes MIL training and improves classification; the proposed MIL-Dropout applies this idea to existing MIL aggregators.","lead":"This paper studies dropout in multiple instance learning and finds that dropping the most important instances in a bag, instead of random ones, improves classification and generalizes better. It introduces MIL-Dropout, a plug-in that picks and removes these instances, reporting accuracy gains across five MIL benchmarks and two whole-slide image datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Confound: augmented models get an extra three-layer feature extractor (Appendix B.1.2); without a same-capacity no-dropout control, the central claim that MIL-Dropout itself causes the gains is unsupported.","rationale":"The paper's own text (Section 5.4 and Appendix B) states that all MIL-Dropout integrations use a 'unified specific architecture involving adding three fully connected layers as the shallow feature extractor f_theta before entering MIL aggregation', while baselines are taken from original configurations consuming raw features. This is an explicit design asymmetry, not an artifact of the reader's interpretation. Because the headline results (avg +9.72% accuracy on benchmarks, +2.3%/+2.05% accuracy on WSIs) are the only evidence for the central claim, the absence of an augmented no-dropout control leaves the causal role of the dropout mask unidentified. The investigation experiments (Appendix A) are better controlled among dropout strategies but still lack a no-dropout arm within the same three-layer architecture, so they cannot rescue the main comparison. The transformer failure (Appendix C.2, Table 4) and test-set-tuned K/G (Appendix C.4.1) are real but secondary; the confound is the single most load-bearing issue. I therefore agree with the reader's CONDITIONAL verdict and the weakest_assumption, and the concrete test above would settle it.","tokens_in":18673,"tokens_out":2492,"duration_ms":17949,"concrete_test":"Re-run the five-benchmark comparison (Table 1) and Camelyon16/TCGA-NSCLC (Table 2) with a fourth arm: same three-layer feature extractor f_theta (256/128/64 and 512/256/128/64 respectively), same optimizer/epochs, but with MIL-Dropout disabled (mask all ones, or remove masking while keeping normalization gamma=1). If this no-dropout augmented control matches or exceeds the reported +MIL-Dropout numbers, the central claim fails. A minimal version: insert the MLP into ABMIL on MUSK1/MUSK2 and Camelyon16, train 5 runs, compare mean AUC/accuracy against Table 1/Table 2 entries.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that MIL-Dropout, not its surrounding machinery, boosts existing MIL methods at negligible cost. But in every main comparison (Table 1, Table 2), each augmented model adds a trainable three-layer MLP (256/128/64 for benchmarks; 512/256, 256/128, 128/64 for WSIs) before the aggregator, while baselines consume the raw 166/230/1024/512-dim features directly (Appendix B.1.2, B.2.3). Therefore the +MIL-Dropout tables conflate two interventions: extra representation capacity/trainable parameters and the dropout mask. The paper's own ablation controls (Fig. 5) vary only K and G within the augmented architecture; they never include an 'extra MLP, no dropout' arm. The investigation experiments in Appendix A do use the same three-layer extractor for both DropNeuron and DropInstance, which supports the top-k observation against DropNeuron, but they do not compare against an augmented no-dropout baseline. Thus the claimed 2-6 point AUC gains in Table 2 and 9.7-point accuracy gains in Table 1 cannot be attributed to MIL-Dropout; an equally plausible explanation is that the added MLP simply gives the aggregator a richer learned embedding. This is the reader's weakest_assumption, and I agree it is the load-bearing soft spot. Additional concerns (hyperparameter selection on test data, failed transformer integration) are secondary; the confound alone is sufficient to move the verdict from accept to conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies dropout in multiple instance learning. Section 3 reports investigation experiments showing that, within an ABMIL architecture with a shared three-layer feature extractor, dropping the top-k most important instances yields lower gradient-direction error (GDE), flatter minima under epsilon-sharpness, and better noise robustness than DropNeuron, random, or bottom-k strategies. Based on this, Section 4 proposes MIL-Dropout, which computes instance importance by average pooling over embedding dimensions and drops the top-k instances plus G instances most similar to each top-k instance, with a normalization factor gamma. Sections 5-6 evaluate MIL-Dropout by adding a three-layer feature extractor and the dropout to ABMIL, ABMIL-Gated, DSMIL, TransMIL, and DTFD-MIL on five MIL benchmark datasets and two WSI datasets, reporting large gains (e.g., 9.72% average accuracy on benchmarks and 2-4% AUC on WSI). The appendix includes ablations over K and G, attention-map visualization, a failed attempt to integrate into transformer blocks, and extra survival experiments.","tokens_in":18952,"tokens_out":5317,"duration_ms":43684,"significance":"If the reported effect is causal, MIL-Dropout would be a cheap, architecture-agnostic regularizer for MIL, and the paper's investigation of gradient-direction error, sharpness, and noise robustness would be a useful empirical contribution. The investigation experiments in Section 3 and Appendix A are internally well designed: the same three-layer extractor is used for all dropout variants, and the GDE and sharpness measurements support the top-k observation against DropNeuron. Code is released, and the rebuttal appendix adds extra datasets and survival tasks. However, the main evaluation in Tables 1 and 2 conflates MIL-Dropout with an added trainable feature extractor, so the central quantitative claim is not currently established. The hyperparameter selection on the evaluation datasets further inflates the reported gains. With a proper control arm and a validated selection protocol, the paper's contribution could become solid; as written, the evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The +MIL-Dropout models add a three-layer MLP (256-128-64 on benchmarks; 512/256-256/128-128/64 on WSIs) before the aggregator, while baselines consume raw features (166/230/1024/512 dimensions) directly. No \"extra MLP without dropout\" control is included, so the gains in Table 1 (e.g., 9.72% average accuracy) and Table 2 (e.g., up to 4% AUC) cannot be attributed to the masking mechanism; they may be due to the extra representation capacity. The investigation experiments in Appendix A use the same extractor for both DropNeuron and DropInstance, which supports the internal comparison, but they also lack an augmented no-dropout arm. Please add control experiments that apply the same f_theta without MIL-Dropout, and ideally with plain neuron dropout, for every aggregator and dataset in Tables 1 and 2.","section":"Section 5.4, Appendix B.1.2, B.2.3, Tables 1-2"},{"comment":"The optimal values of K and G are identified from Figure 5 on CAMELYON16 and TCGA-NSCLC, and Appendix C.4.1 states that all experiments in Table 2 use these optimal values. The paper does not describe a validation-based selection procedure or a separate test set for hyperparameter selection. If Figure 5 is computed on the same test partitions used in Table 2, the reported gains are optimistically selected. Please either use validation-based selection, report results across a grid without test-set selection, or justify that the chosen K and G are not tuned to the test set.","section":"Section 6.3, Appendix C.4.1"},{"comment":"The paper repeatedly claims \"theoretical guarantees\" for top-k dropping, but Section 3.4 is a heuristic explanation and the sharpness/GDE analyses are empirical measurements with a standard Taylor expansion; no formal theorem or proof is stated. In addition, the claim that MIL-Dropout can be \"seamlessly integrated into existing MIL frameworks\" is contradicted by Appendix C.2 and C.3, which report that integration into transformer blocks fails to converge (Table 4 shows 52.5 AUC versus 86.5 for TransMIL). Please temper these claims and specify in the abstract and impact statement that the method is intended for shallow feature extractors before MIL aggregation.","section":"Section 3.4, Impact Statement, Conclusion"}],"minor_comments":[{"comment":"The dataset name \"CAMELOYON16\" is misspelled and should be \"CAMELYON16\".","section":"Section 5.2, Table 2"},{"comment":"The sentence \"MIL aggregators trained with features extracted using SimCLR outperforms that trained with features extracted by SimCLR\" appears to compare SimCLR with itself; the second instance should presumably be \"ImageNet.\"","section":"Section 6.2"},{"comment":"The complexity statement \"The complexity is substantially reduced to the fact that O(ND(l)) due to K is typically much smaller than N\" is garbled; the stated complexity includes O(K(N-K)D(l)), so with K constant it is O(ND(l)). Please rewrite for clarity.","section":"Section 4.0.1"},{"comment":"The sentence \"this paper also serves as a theoretical supplement to PDL\" is unclear because PDL is not described and no supplement is provided in this manuscript; please either remove the claim or clarify the relationship.","section":"Related Work"},{"comment":"The observation that restricting dropout to at most 10% of the average instance count yields consistent improvements is stated only in the rebuttal appendix; if this is a design principle, it should be incorporated into the main text.","section":"Appendix C.4.3"},{"comment":"It is unclear why the sigmoid is applied to the average-pooled values and how the importance scores are normalized across instances before top-k selection; please clarify.","section":"Section 4, Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"The core confound is fixable with a modest control experiment: rerunning Tables 1 and 2 with an 'extra MLP, no dropout' baseline would directly test whether MIL-Dropout itself causes the gains. The K/G selection protocol also needs clarification to avoid test-set tuning. The investigation experiments in Section 3 are a useful contribution and could be published with the appropriate controls. The authors appear responsive based on the additional experiments in the rebuttal appendix, so I would encourage a major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—quick read of the MIL-Dropout paper. The one thing to know: the top-k instance-drop result is real enough to take seriously, but the headline comparisons don't support the headline claim. In every main table, the +MIL-Dropout models get an extra three-layer feature extractor (256/128/64, or larger on WSI) that the baselines don't receive. So the 2–6 point AUC gains and ~10 point accuracy jumps could just be the extra learned embedding, not the dropout. Without a same-capacity no-dropout arm, the paper doesn't isolate the mechanism. That's a load-bearing flaw, not a quibble.\n\nWhat's genuinely good: the investigation in Section 3 and Appendix A. Within a fixed architecture (ABMIL with the same 3-layer MLP for all arms), dropping the top-k instances by attention clearly beats random or bottom-k dropping, reduces gradient direction error, and lands in a flatter minimum. That's counterintuitive and worth knowing. The GDE framing is a nice addition. The proposed MIL-Dropout itself—average-pooling importance plus similarity-based neighborhood dropping—is a reasonable way to operationalize the idea, and the complexity analysis is honest.\n\nSecondary soft spots: K and G are tuned on the same WSI datasets whose final results are reported (C.4.1 says so explicitly), so some selection bias is baked in. The 'theoretical guarantees' language overstates what is actually a heuristic explanation. And the appendix admits MIL-Dropout fails inside transformer blocks, which undercuts the 'seamless integration' claim. These are all real but minor next to the confound.\n\nWho this is for: anyone actively working on MIL/WSI aggregation. The top-k finding could inform future regularizers even if this specific method doesn't hold up. I'd send it to review, not desk-reject: the observation is novel and the investigation is mostly well executed. But I'd make the authors add a no-dropout control with matched capacity and hold out hyperparameters before believing the reported gains.","headline":"The top-k instance-drop observation is genuinely worth testing, but the main tables conflate MIL-Dropout with an added feature extractor, so the advertised gains are not established.","tokens_in":19526,"tokens_out":2843,"would_cite":false,"duration_ms":24847,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that removing the top-ranked instances from a bag during MIL training—the opposite of what intuition suggests—consistently lowers gradient error, flattens the loss landscape, and improves classification, and it packages…","keywords":["multiple instance learning","dropout","whole slide image classification","weak supervision","gradient direction error","flat minima","attention","regularization"],"falsifier":"Take the CAMELYON16 setup and train the same three-layer-feature-extractor ABMIL variant under three conditions: no dropout, random instance dropout of the same mask size, and top-k MIL-Dropout. If random-dropping or no-dropping matches MIL-Dropout in accuracy, F1, AUC, and epsilon-sharpness, then the top-k selection rule is not what carries the reported gains.","tokens_in":18422,"feed_emoji":"🔬","tokens_out":6432,"duration_ms":54081,"temperature":0.7,"pith_summary":"Multiple instance learning (MIL) labels a whole bag of instances from a single bag label, an approach widely used to classify gigapixel whole-slide images in pathology. Because MIL is usually trained in two stages with a fixed feature extractor, the bag features are noisy and the aggregator can overfit to a few strongly discriminative instances. This paper tries to establish that the standard deep-learning remedy, dropout, works in MIL when applied in a particular way: drop the top-k most important instances rather than random or least important ones. The paper observes that this unusual masking reduces the gradient direction error during training, leads to flatter minima, and improves accuracy and robustness across five MIL benchmarks and two WSI datasets, all with no additional learnable parameters and negligible computational overhead.","feed_headline":"Dropping the top-k instances improves weak-label classifiers","feed_subtitle":"A dropout that removes the bag's most-attended evidence forces the model to find other positive clues, and it generalizes better.","key_machinery":"MIL-Dropout is a non-parametric instance-selection mask inserted into the shallow feature extractor of an existing MIL pipeline. At layer $l$, each instance's importance is the sigmoid of its average-pooled embedding; the top-K instances are chosen, and then for each of them the G most cosine-similar instances among the rest are added to the drop set. The mask zeroes those rows of the feature map and rescales the survivors by $\\gamma = N/(N-K(1+G))$ to stabilize training. The load-bearing object is not the mask itself but the ranking: the mask removes the currently most attended evidence, forcing the aggregator to discover additional positive instances while the bag label stays unchanged. In experiments the mask is placed after each of three added fully-connected layers before the ABMIL, DSMIL, TransMIL, or DTFD-MIL aggregator.","core_discovery":"The central discovery is the counterintuitive empirical finding that, within a bag, removing the instances the model currently considers most important is not harmful but beneficial. A positive bag remains positive after masking, so the network must find alternative, less obvious positive evidence; this behaves like data augmentation and prevents the model from locking onto a single group of easy instances. The paper frames this in optimization terms: among the strategies tested (random, bottom-k, top-k, none), only top-k dropping consistently lowers the gradient direction error relative to the whole-dataset gradient and drives the model to flatter local minima, which is associated with better generalization under Gaussian, Poisson, and salt-and-pepper noise. The paper packages the finding as MIL-Dropout, which computes instance importance with a parameter-free average-pooling attention score, selects the top-K instances plus the G neighbors most similar to each, masks them, and rescales the remaining features.","pith_inferences":["If the masking effect is causal, MIL-Dropout is effectively a hard-example mining regularizer: it forces the bag-level model to explain the positive label with evidence outside the current maximum-attention cluster, which should help in tasks where positive evidence is spatially scattered or where easy positives are unrepresentative of the whole disease.","The drop-top-k rule could be combined with instance-level uncertainty or with a curriculum that anneals K from large to small, turning the method into a schedule rather than a fixed hyperparameter; the paper does not test this.","Because the augmented models in the main comparisons add a three-layer feature extractor that baselines do not have, the cleanest test of the dropout hypothesis would be an ablation that adds the same extractor with dropout disabled; the paper reports no such control.","The same importance-ranking idea could be ported to other weakly supervised problems such as video anomaly detection or 3D point-cloud classification, which the paper lists as future work rather than testing."],"forward_implications":["Plugging MIL-Dropout into the simple ABMIL and ABMIL-Gated aggregators lifts average accuracy by roughly 9.7 to 9.8 points across MUSK1, MUSK2, FOX, TIGER, and ELEPHANT, reaching state-of-the-art numbers on those benchmarks.","On CAMELYON16 and TCGA-NSCLC, MIL-Dropout adds on average about 2.0 to 2.5 points of accuracy, F1, and AUC across four MIL aggregators with both ImageNet and self-supervised (SimCLR) features.","The method improves robustness to test-time noise and sharpens attention-based lesion localization, so the model's high-attention regions agree better with annotated tumor regions.","Because MIL-Dropout adds no learnable parameters and only $\\mathcal{O}(N \\log N + K(N-K)D^{(l)})$ mask computation per layer, it can be applied to any existing attention-based, transformer-based, or distillation-based MIL aggregator at negligible cost.","The paper's extra experiments show MIL-Dropout also improves the concordance index on survival prediction for ABMIL, DSMIL, TransMIL, and DTFD-MIL on TCGA-LUAD and TCGA-BRCA."],"supporting_citations":[{"why":"Defines the ABMIL aggregator that serves as the main testbed and supplies the attention-based notion of instance importance used throughout the paper.","marker":"Ilse et al., 2018"},{"why":"Introduces DSMIL, provides WSI preprocessing and SimCLR features, and is cited for the observation that randomly dropping instances already helps MIL.","marker":"Li et al., 2021"},{"why":"Provides DTFD-MIL, the ResNet-50 ImageNet features, and the benchmark train/test splits used for CAMELYON16.","marker":"Zhang et al., 2022"},{"why":"Contributes TransMIL, the transformer-based MIL baseline that the paper augments with MIL-Dropout.","marker":"Shao et al., 2021"},{"why":"Supplies the gradient-direction-error diagnostic that the paper uses to explain why top-k dropping improves optimization.","marker":"Liu et al., 2023"},{"why":"Defines epsilon-sharpness, the flat-minimum measure the paper uses to connect dropout to generalization.","marker":"Dinh et al., 2017"},{"why":"Defines the dropout operation that MIL-Dropout adapts to the instance level.","marker":"Hinton et al., 2012"},{"why":"Establishes dropout as a regularizer and is the reference for not applying dropout at inference.","marker":"Srivastava et al., 2014"}],"fun_headline_variants":["Drop top-k bag instances for better MIL generalization","MIL-Dropout: mask the top evidence to boost weak-label models","Remove the strongest clues to improve MIL classification","Counterintuitive: top-k dropping strengthens MIL training","In MIL, dropping key instances yields robust features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the accuracy and robustness gains come from dropping the top-k instances themselves, not from the extra three-layer feature extractor that only the augmented models receive, and the paper includes no control run with that extractor but without the dropout mask.","fun_headline_variants_meta":{"raw":{"variants":["Drop top-k bag instances for better MIL generalization","MIL-Dropout: mask the top evidence to boost weak-label models","Remove the strongest clues to improve MIL classification","Counterintuitive: top-k dropping strengthens MIL training","In MIL, dropping key instances yields robust features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000813,"raw_usage":{"total_tokens":3572,"prompt_tokens":959,"completion_tokens":2613,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":2536}},"tokens_in":575,"tokens_out":2613,"duration_ms":17713,"temperature":1.0,"reasoning_tokens":2536,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:41:01.067218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the CAMELYON16 setup and train the same three-layer-feature-extractor ABMIL variant under three conditions: no dropout, random instance dropout of the same mask size, and top-k MIL-Dropout. If random-dropping or no-dropping matches MIL-Dropout in accuracy, F1, AUC, and epsilon-sharpness, then the top-k selection rule is not what carries the reported gains.","supporting_citations":[{"cited_title":"E., and Zheng, Y","cited_arxiv_id":null,"evidence_quote":"Provides DTFD-MIL, the ResNet-50 ImageNet features, and the benchmark train/test splits used for CAMELYON16."},{"cited_title":"Transmil: Transformer based correlated multiple instance learning for whole slide image classification","cited_arxiv_id":null,"evidence_quote":"Contributes TransMIL, the transformer-based MIL baseline that the paper augments with MIL-Dropout."},{"cited_title":"Dropout reduces underfitting","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-direction-error diagnostic that the paper uses to explain why top-k dropping improves optimization."}],"review_version":1}