{"id":"d2034cf4-b07e-42e9-98ee-3f754f14ce90","arxiv_id":"2411.14750","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SATOMIL estimates patient-level ulcerative colitis severity from multiple endoscopic images using selective aggregator tokens, outperforming existing MIL methods and simple image-level baselines on two datasets.","lead":"This paper introduces SATOMIL, a deep learning method that estimates ulcerative colitis severity for a patient from a set of colonoscopy images, using only the patient-level severity score already recorded in hospitals. The method uses multiple attention tokens, each trained to focus on images above a severity threshold, and reports higher accuracy than existing multiple-instance learning methods on two datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The private-dataset clinical labels are assumed to equal the maximum severity over the bag's still images (Sec. 3.1); if they encode video or clinician judgment instead, the real-clinical-setting results do not test the claimed max-instance MIL task.","rationale":"The paper is a serious attempt at a clinically motivated MIL formulation, and it provides useful evidence for the mechanism: a publicly available codebase, 5-fold cross-validation, ablations isolating the selective aggregator tokens, attention-weight visualizations showing that tokens attend to high-severity instances, and consistent point-estimate improvements over strong MIL and ordinal-classification baselines. The reader's conditional verdict is appropriate. The single most load-bearing concern is the label-specification assumption for the private dataset. The entire real-clinical-setting claim rests on the assertion that patient-level severity equals the maximum severity over the still images in the bag. LIMUC satisfies this by construction, but the private dataset does not demonstrate it. If the clinical labels were determined from video or from clinician judgment beyond the still images, the private-dataset results would not measure the proposed max-instance MIL task, and the method's advantage could reflect incidental correlations rather than correct selective aggregation. This is not an accusation of data mishandling; it is an unverified external-validity condition. The reader's weakest assumed assumption identifies exactly this, so I agree with the reader. A second concern, the absence of error bars and significance testing, is real but secondary: if the label-specification check fails, the comparison is moot regardless of error bars; if it passes, the point-estimate gaps would still need variance estimates before claiming state-of-the-art. The concrete test above would settle the primary concern, allowing the conditional verdict to be upgraded or the private-dataset conclusions to be re-scoped.","tokens_in":13763,"tokens_out":6456,"duration_ms":69411,"concrete_test":"On the private dataset, compare the clinical patient-level severity label with the maximum of the independently made image-level severity annotations for each bag, reporting exact-match accuracy and quadratic weighted kappa between the two. If kappa is below approximately 0.9 or exact match is below approximately 90%, the clinical labels contain information not present in the still-image bags, and the private-dataset evaluation does not test the claimed max-instance ordinal MIL task.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SATOMIL estimates patient-level UC severity from a bag of still images using only patient-level labels, where the patient label is defined as the maximum severity over the bag (Sec. 3.1). This is precisely what makes the ordinal max-MIL formulation and the selective-aggregation mechanism the right model. On LIMUC the definition holds by construction, because bag labels are manufactured as the max of image-level annotations. On the private dataset, however, the patient-level labels come from clinical records, and the paper does not verify that those clinical labels equal the maximum severity of the captured still images. If endoscopic video, the clinician's overall impression, or images outside the recorded set contributed to the clinical score, then the training target is not a function of the bag, and the model is trained and evaluated against a mis-specified label. That would not merely weaken the private-dataset accuracy numbers; it would invalidate the abstract's claim that the method works 'in real clinical settings' as a max-instance MIL method. This is load-bearing because it is the link between the algorithm's inductive bias (selectively attending to the most severe instance) and the clinical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses patient-level estimation of ulcerative colitis (UC) severity from a bag of endoscopic images, using only patient-level labels during training. The task is formulated as ordinal multiple-instance learning with a max-instance assumption: the patient-level severity is defined as the highest severity among the images in the bag. The proposed SATOMIL introduces K-1 selective aggregator tokens into a transformer; the k-th token aggregates instance features to support the binary classification task O_i^k = 1(Y_i > k), and the final severity is obtained by thresholding and summing the K-1 binary predictions. The method is evaluated on LIMUC, where bag labels are manufactured by taking the maximum of image-level annotations, and on a private clinical dataset with patient-level severity labels. Comparisons are reported against ten MIL baselines, five ordinal classification variants, two image-level supervised baselines, plus ablations and attention/feature visualizations. The paper reports consistent numerical improvements in accuracy, quadratic weighted kappa, and macro-F1, and claims that the patient-level model trained without image-level labels outperforms image-level supervised baselines.","tokens_in":14004,"tokens_out":7601,"duration_ms":81237,"significance":"If the empirical results are reliable, the paper makes a practical contribution: clinical records that already contain patient-level severity can be used directly to train a model for patient-level UC grading, without the expensive step of obtaining image-level annotations. The selective aggregator token design is a simple and interpretable mechanism targeted at the max-instance ordinal MIL setting, and the reported attention visualizations, feature-space plots, and confusion matrices provide qualitative evidence that the mechanism behaves as intended. The public code release and the evaluation on a real clinical dataset are further strengths. However, the strength of the empirical claims is currently limited. The private-dataset labels are assumed to equal the maximum severity of the supplied still images, but that assumption is not verified; the comparative claims rest on point estimates without error bars or significance tests; and the claim of outperforming image-level methods is supported by only two simple baselines and a single metric. These issues concern the central claims and should be addressed before the results can be fully relied upon.","major_comments":[{"comment":"The task definition states that patient-level severity is the highest severity among all individual images of the patient, but the private dataset is described only as containing patient-level severity labels, without specifying what those clinical labels encode. On LIMUC the max-instance assumption holds by construction because bag labels are manufactured as the maximum of image-level annotations; thus LIMUC cannot validate the assumption on real clinical labels. On the private dataset, if the recorded severity was based on video, on images outside the collected bag, or on the clinician's overall impression, the training target is not a function of the bag of still images supplied to the model, and the 'real clinical settings' claim in the abstract would not actually test the max-instance MIL task. The authors should state the clinical scoring protocol explicitly and quantify the agreement between the clinical patient-level label and the maximum of the image-level annotations that were 'additionally made for evaluation.' Reporting performance separately for concordant versus discordant bags, or at least discussing the magnitude of disagreement, would clarify whether the private-data evaluation matches the problem definition.","section":"Section 3.1 and Section 4 (Dataset)"},{"comment":"All comparisons are reported as point estimates from a single 5-fold cross-validation, with no standard deviations, confidence intervals, or paired significance tests. Several margins are small, for example LIMUC kappa 0.826 versus 0.813 for Transformer in Table 1, LIMUC accuracy 0.690 versus 0.669 in Table 2, and private accuracy 0.683 versus 0.670 in Table 3. Since the paper's central claim is that SATOMIL 'outperformed' the state-of-the-art MIL methods, the authors should provide repeated runs with different random seeds and a paired significance test, or at least per-fold results, so that the reader can distinguish a systematic improvement from noise. Without this, the comparative statements in the abstract and Section 4.1 are not fully supported.","section":"Tables 1-4"},{"comment":"The abstract states that the method 'outperformed the previous image-level methods,' but Table 4 compares only two simple image-level baselines (CL and K-rank), reports only kappa, and does not include the existing image-level UC severity estimators cited in the Related Work, such as Kadota et al., Polat et al., and Takenaka et al. Moreover, the differences for the two implemented baselines are small (LIMUC kappa 0.826 versus 0.818, private kappa 0.774 versus 0.760) and are presented without uncertainty. The comparative claim should either be restricted to the implemented baselines or supported by experiments against the existing image-level methods; in addition, the same evaluation metrics (accuracy, macro-F1, and kappa) should be reported for all methods in Table 4.","section":"Section 4.3 and Table 4"}],"minor_comments":[{"comment":"The description of the selective aggregation layer is garbled and appears with inconsistent notation, for example 'key j = Key(t0j), qi j = Query(e0ij)' and the displayed equation for tilde-a_i_k with missing parentheses in the softmax denominator. The manuscript also contains two near-identical copies of parts of Section 3.3; please remove the duplication and unify the notation.","section":"Section 3.3"},{"comment":"The attention visualizations are qualitative and report only the top 50% of attention weights. Please state how attention weights are averaged across heads and, if possible, report quantitative agreement between attention scores and instance-level severity labels, which are available for LIMUC.","section":"Figures 7 and 8"},{"comment":"The reference numbering is inconsistent (for example, PyTorch appears as [17] and Adam as [16] in the implementation details, while the numbered list assigns different numbers), and some reference entries are duplicated in the bibliography. The WACV submission header and 'CONFIDENTIAL REVIEW COPY' markers also remain in the text.","section":"References and front matter"},{"comment":"The private dataset is referred to as 'Anonymous Hospital' in the first version and as 'Kyoto Second Red Cross Hospital' in the second version of the text; use the approved anonymization consistently.","section":"Dataset description"},{"comment":"The indicator function 1{·} is used in Equation (3) but is defined only implicitly in the K-rank label extension; define it at first use in Section 3.3.","section":"Equation (3)"}],"recommendation":"major_revision","confidential_remarks":"The technical idea is sound and the paper fits the scope of the venue. The main risk is the private-dataset label semantics: without a protocol statement and a comparison between the clinical patient-level labels and the max of the additional image-level annotations, the real-clinical claim remains vulnerable. The paper also needs a statistical grounding for its comparative claims and a thorough editorial cleanup of the duplicated and garbled sections. I do not see a circularity or citation-pattern problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline on this one: the selective aggregator token idea is real and worth knowing. SATOMIL trains K-1 tokens, each with a query attending to instances whose severity exceeds a threshold, so the model gets a separate bag-level feature for every class boundary. That is genuinely different from K-rank MIL (which aggregates output scores) and from a single class token (which has to balance all boundaries at once). The ablation against Transformer K-rank isolates this contribution, and the attention/feature visualizations show the tokens behaving as intended. I buy the mechanism.\n\nThe experiments are solid in scope: ten MIL baselines on LIMUC and a private dataset, with consistent gains across accuracy, kappa, and macro-F1, plus code released. The result that patient-level labels alone beat two image-level supervised baselines is a nice practical point, though those baselines (plain CE and K-rank) are not the strongest image-level methods in the literature.\n\nNow the soft spots, in increasing order of concern.\n\nFirst, everything is reported as point estimates from 5-fold CV, no error bars or significance tests. Some of the headline margins are thin (e.g., kappa 0.826 vs 0.818 on LIMUC against K-rank image-level; 0.774 vs 0.760 on private). This is fixable but needs to be in the revision.\n\nSecond, the 'outperforms image-level methods' claim is supported only by two simple classifiers. It is a modest claim, but the wording in the abstract overreaches slightly.\n\nThird, the private dataset label assumption. The method assumes the patient-level severity is the max over the bag's still images. For LIMUC this is true by construction. For the private dataset, labels come from clinical records. The paper describes the clinical practice as recording the most severe area of the images, which would make the assumption true, but it does not verify that the recorded score is actually a function of the collected still images. If video or the clinician's overall impression contributed, the 'real clinical setting' experiment is not testing the max-instance task the model is biased for. This is the most important thing to ask the authors to clarify.\n\nOverall, the central idea is good, the evidence is mostly there, and the flaws are addressable. It deserves a serious referee, and I'd want to see the revision land before believing the strongest clinical claim.\n\nRecommendation: send to peer review; require error bars, stronger image-level baselines or softened claims, and explicit discussion of how the private-dataset labels relate to the bag contents.","headline":"A genuinely new aggregation mechanism for ordinal max-MIL, with solid but point-estimate evidence; the main open question is whether the clinical labels really are a function of the bag's still images.","tokens_in":14472,"tokens_out":2931,"would_cite":false,"duration_ms":28104,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer with $K-1$ selective aggregator tokens, each assigned to a severity threshold, estimates patient-level ulcerative colitis severity from bags of endoscopic images using only patient-level labels; it reports higher accuracy…","keywords":["ordinal multiple-instance learning","ulcerative colitis severity estimation","selective aggregator transformer","patient-level diagnosis","endoscopic image analysis","rank labels","medical image classification"],"falsifier":"Compare recorded patient-level severity scores against the maximum of independently re-annotated severity scores over every frame of the corresponding endoscopy videos; if a substantial fraction of recorded scores exceed the still-frame maximum, the max-still-image premise fails and the method would need video-based aggregation or differently constructed bags.","tokens_in":13613,"feed_emoji":"🩺","tokens_out":13603,"duration_ms":111738,"temperature":0.7,"pith_summary":"Ulcerative colitis severity is recorded in the clinic at the patient level: among the 20 to 40 endoscopic images taken from a patient, only the severity of the most severe area is kept, and individual images are not labeled. This paper tries to learn that patient-level estimate directly from such records, formulating the task as ordinal multiple-instance learning in which each patient is a bag of images and the bag label is the maximum instance severity. The proposed method, SATOMIL, places $K-1$ selective aggregator tokens in a transformer; token $k$ is trained to aggregate only images whose severity exceeds $k$, so each bag-level feature is specialized to one class boundary. The paper reports that this beats state-of-the-art MIL methods on two datasets and, trained only on patient-level labels, also beats image-level classifiers that were trained with per-image supervision. If correct, this would let large retrospective clinical databases train severity graders without any additional annotation.","feed_headline":"Patient-level UC severity from labels alone beats supervised baselines","feed_subtitle":"Selective aggregator tokens pick severe images, beating supervised image-level classifiers.","key_machinery":"The load-bearing mechanism is the selective aggregator token: a trainable query token inside a transformer that acts as a classifier-specific pooling head. For each threshold $k$, token $t_k$ computes scaled dot-product attention over all instance features in the bag, with a mask that prevents token-to-token attention, and its output $a_k$ feeds a binary classifier $g_k$ for the rank label $O_k = \\mathbb{1}\\{Y > k\\}$. Training with binary cross-entropy forces $t_k$ to attend mainly to instances above $k$, so the model builds $K-1$ bag-level feature spaces, one per adjacent-class boundary, instead of one shared space that mixes severe and non-severe instances.","core_discovery":"The central claim is that max-severity estimation in ordinal multiple-instance learning should allocate a separate aggregator token to each severity threshold instead of compressing a bag into one feature vector. SATOMIL maintains $K-1$ learned tokens; the $k$-th token's self-attention is trained to give high weight to instances with severity above $k$ and low weight to the rest, producing a bag-level feature for exactly the boundary between severity $k$ and $k+1$. The final severity is a sum of threshold decisions, $\\hat{Y}^i = 1+\\sum_{k=1}^{K-1}\\mathbb{1}\\{\\hat{O}^i_k > 0.5\\}$. In the paper's experiments this design raises accuracy, quadratic weighted kappa, and macro-F1 over ten MIL baselines and over five transformer variants augmented with ordinal classification losses on both the public LIMUC dataset and a private hospital dataset; the reported kappa advantage over image-level supervised classifiers is 0.826 versus 0.782 and 0.818 on LIMUC, and 0.774 versus 0.755 and 0.760 on the private dataset.","pith_inferences":["Editorial inference: if the recorded patient label is a max over the whole video rather than over the saved still frames, SATOMIL's training target is mis-specified; a natural extension is to collect a small set of video-derived labels and measure the drop in accuracy.","Editorial inference: for bags where the most severe instance is very rare, the high-$k$ tokens must attend to very few instances, so the method may benefit from attention temperature annealing or explicit hard-positive mining.","Editorial inference: because each threshold classifier returns a probability, the margin between the highest and second-highest threshold probabilities could serve as a confidence measure for deferring uncertain patient-level diagnoses to a clinician.","Editorial inference: the private dataset comes from a single hospital; testing across endoscopy hardware and centers would reveal whether the reported advantage over image-level supervision persists under domain shift."],"forward_implications":["Retrospective hospital records, where only the worst severity is noted, become sufficient training data for patient-level severity graders, removing the need for costly per-image annotation.","The same per-threshold token design should apply to any max-severity bag task, such as tumor grading in whole-slide pathology or retinopathy severity from multiple fundus photographs.","Each token answers an explicit threshold question, so the binary probabilities provide a natural ordinal calibration and can identify which images in a bag drive the final patient score.","The ablation comparing Transformer with Transformer K-rank implies that ordinal labels alone do not fix max-severity MIL; the per-threshold selective aggregation is the component responsible for the gain."],"supporting_citations":[{"why":"Supplies the public LIMUC dataset with image-level annotations, from which patient bags are constructed by taking the max severity.","marker":"[19]"},{"why":"The closest prior work, K-rank MIL for patient-level UC severity; serves as the main ordinal MIL baseline to beat.","marker":"[21]"},{"why":"Attention-based deep MIL motivates the feature-aggregation strategy and provides the Feature+Attention baseline.","marker":"[7]"},{"why":"Provides the output aggregation baselines whose poor loss propagation the paper uses to justify feature aggregation.","marker":"[27]"},{"why":"Provides the feature mean and max pooling baselines for MIL.","marker":"[8]"},{"why":"DSMIL, a whole-slide MIL method, is a baseline that the proposed method outperforms.","marker":"[12]"},{"why":"AdditiveTransMIL, a whole-slide MIL method, is a baseline in the comparison.","marker":"[9]"},{"why":"IBMIL, a whole-slide MIL method, is a baseline in the comparison.","marker":"[14]"},{"why":"Supplies the K-rank ordinal label decomposition used to turn one ordinal label into $K-1$ binary threshold labels.","marker":"[15]"}],"fun_headline_variants":["Per-threshold tokens sharpen UC severity estimation","Selective aggregator tokens lift MIL for colitis grading","New transformer scores UC severity from multiple images","Ordinal MIL with K-1 tokens beats image-level baselines","Threshold-wise attention improves ulcerative colitis grading"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recorded patient-level severity equals the maximum severity among the still images in the bag; if a clinician's score comes from video motion or from frames not among the collected stills, the training label is not the target the model actually learns.","fun_headline_variants_meta":{"raw":{"variants":["Per-threshold tokens sharpen UC severity estimation","Selective aggregator tokens lift MIL for colitis grading","New transformer scores UC severity from multiple images","Ordinal MIL with K-1 tokens beats image-level baselines","Threshold-wise attention improves ulcerative colitis grading"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1355,"prompt_tokens":977,"completion_tokens":378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":593,"tokens_out":378,"duration_ms":4822,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:55:27.786151+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare recorded patient-level severity scores against the maximum of independently re-annotated severity scores over every frame of the corresponding endoscopy videos; if a substantial fraction of recorded scores exceed the still-frame maximum, the max-still-image premise fails and the method would need video-based aggregation or differently constructed bags.","supporting_citations":[{"cited_title":"Interventional bag multi-instance learning on whole-slide pathological images","cited_arxiv_id":null,"evidence_quote":"IBMIL, a whole-slide MIL method, is a baseline in the comparison."},{"cited_title":"Class distance weighted cross-entropy loss for ulcerative colitis severity es- timation","cited_arxiv_id":null,"evidence_quote":"Supplies the public LIMUC dataset with image-level annotations, from which patient bags are constructed by taking the max severity."},{"cited_title":"Automatic estimation of ulcera- tive colitis severity from endoscopy videos using ordinal multi-instance learning","cited_arxiv_id":null,"evidence_quote":"The closest prior work, K-rank MIL for patient-level UC severity; serves as the main ordinal MIL baseline to beat."},{"cited_title":"Attention-based deep multiple instance learning","cited_arxiv_id":null,"evidence_quote":"Attention-based deep MIL motivates the feature-aggregation strategy and provides the Feature+Attention baseline."},{"cited_title":"Tomczak, and Max Welling","cited_arxiv_id":null,"evidence_quote":"Provides the feature mean and max pooling baselines for MIL."},{"cited_title":"Dual-stream multi- ple instance learning network for whole slide image clas- sification with self-supervised contrastive learning","cited_arxiv_id":null,"evidence_quote":"DSMIL, a whole-slide MIL method, is a baseline that the proposed method outperforms."},{"cited_title":"Additive mil: intrinsically interpretable multiple instance learning for pathology","cited_arxiv_id":null,"evidence_quote":"AdditiveTransMIL, a whole-slide MIL method, is a baseline in the comparison."},{"cited_title":"Ordinal regression with multiple output cnn for age estimation","cited_arxiv_id":null,"evidence_quote":"Supplies the K-rank ordinal label decomposition used to turn one ordinal label into $K-1$ binary threshold labels."}],"review_version":1}