REVIEW 3 major objections 5 minor 1 cited by
Ordinal Multiple-instance Learning for Ulcerative Colitis Severity Estimation with Selective Aggregated Transformer
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A transformer with $K-1$ selective aggregator tokens, each assigned to a severity threshold, estimates patient-level ulcerative colitis severity from bags of endoscopic images using only patient-level labels; it reports higher accuracy…
desk verdict A genuinely new aggregation mechanism for ordinal max-MIL, with solid but point-estimate evidence; the main open question is whether the clinical labels really are a function of the bag's still images. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the selective aggregator token: a trainable query token inside a transformer that acts as a classifier-specific pooling head. For each threshold $k$, token $t_k$ computes scaled dot-product attention over all instance features in the bag, with a mask that prevents token-to-token attention, and its output $a_k$ feeds a binary classifier $g_k$ for the rank label $O_k = \mathbb{1}\{Y > k\}$. Training with binary cross-entropy forces $t_k$ to attend mainly to instances above $k$, so the model builds $K-1$ bag-level feature spaces, one per adjacent-class boundary, instead of one shared space that mixes severe and non-severe instances.
What would settle it
Compare recorded patient-level severity scores against the maximum of independently re-annotated severity scores over every frame of the corresponding endoscopy videos; if a substantial fraction of recorded scores exceed the still-frame maximum, the max-still-image premise fails and the method would need video-based aggregation or differently constructed bags.
Extended reading notes
Core claim
The central claim is that max-severity estimation in ordinal multiple-instance learning should allocate a separate aggregator token to each severity threshold instead of compressing a bag into one feature vector. SATOMIL maintains $K-1$ learned tokens; the $k$-th token's self-attention is trained to give high weight to instances with severity above $k$ and low weight to the rest, producing a bag-level feature for exactly the boundary between severity $k$ and $k+1$. The final severity is a sum of threshold decisions, $\hat{Y}^i = 1+\sum_{k=1}^{K-1}\mathbb{1}\{\hat{O}^i_k > 0.5\}$. In the paper's experiments this design raises accuracy, quadratic weighted kappa, and macro-F1 over ten MIL baselines and over five transformer variants augmented with ordinal classification losses on both the public LIMUC dataset and a private hospital dataset; the reported kappa advantage over image-level supervised classifiers is 0.826 versus 0.782 and 0.818 on LIMUC, and 0.774 versus 0.755 and 0.760 on the private dataset.
Load-bearing premise
The recorded patient-level severity equals the maximum severity among the still images in the bag; if a clinician's score comes from video motion or from frames not among the collected stills, the training label is not the target the model actually learns.
Editorial extensions
If this is right
- Retrospective hospital records, where only the worst severity is noted, become sufficient training data for patient-level severity graders, removing the need for costly per-image annotation.
- The same per-threshold token design should apply to any max-severity bag task, such as tumor grading in whole-slide pathology or retinopathy severity from multiple fundus photographs.
- Each token answers an explicit threshold question, so the binary probabilities provide a natural ordinal calibration and can identify which images in a bag drive the final patient score.
- The ablation comparing Transformer with Transformer K-rank implies that ordinal labels alone do not fix max-severity MIL; the per-threshold selective aggregation is the component responsible for the gain.
Reading between the lines
- Editorial inference: if the recorded patient label is a max over the whole video rather than over the saved still frames, SATOMIL's training target is mis-specified; a natural extension is to collect a small set of video-derived labels and measure the drop in accuracy.
- Editorial inference: for bags where the most severe instance is very rare, the high-$k$ tokens must attend to very few instances, so the method may benefit from attention temperature annealing or explicit hard-positive mining.
- Editorial inference: because each threshold classifier returns a probability, the margin between the highest and second-highest threshold probabilities could serve as a confidence measure for deferring uncertain patient-level diagnoses to a clinician.
- Editorial inference: the private dataset comes from a single hospital; testing across endoscopy hardware and centers would reveal whether the reported advantage over image-level supervision persists under domain shift.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses patient-level estimation of ulcerative colitis (UC) severity from a bag of endoscopic images, using only patient-level labels during training. The task is formulated as ordinal multiple-instance learning with a max-instance assumption: the patient-level severity is defined as the highest severity among the images in the bag. The proposed SATOMIL introduces K-1 selective aggregator tokens into a transformer; the k-th token aggregates instance features to support the binary classification task O_i^k = 1(Y_i > k), and the final severity is obtained by thresholding and summing the K-1 binary predictions. The method is evaluated on LIMUC, where bag labels are manufactured by taking the maximum of image-level annotations, and on a private clinical dataset with patient-level severity labels. Comparisons are reported against ten MIL baselines, five ordinal classification variants, two image-level supervised baselines, plus ablations and attention/feature visualizations. The paper reports consistent numerical improvements in accuracy, quadratic weighted kappa, and macro-F1, and claims that the patient-level model trained without image-level labels outperforms image-level supervised baselines.
Significance. If the empirical results are reliable, the paper makes a practical contribution: clinical records that already contain patient-level severity can be used directly to train a model for patient-level UC grading, without the expensive step of obtaining image-level annotations. The selective aggregator token design is a simple and interpretable mechanism targeted at the max-instance ordinal MIL setting, and the reported attention visualizations, feature-space plots, and confusion matrices provide qualitative evidence that the mechanism behaves as intended. The public code release and the evaluation on a real clinical dataset are further strengths. However, the strength of the empirical claims is currently limited. The private-dataset labels are assumed to equal the maximum severity of the supplied still images, but that assumption is not verified; the comparative claims rest on point estimates without error bars or significance tests; and the claim of outperforming image-level methods is supported by only two simple baselines and a single metric. These issues concern the central claims and should be addressed before the results can be fully relied upon.
major comments (3)
- [Section 3.1 and Section 4 (Dataset)] The task definition states that patient-level severity is the highest severity among all individual images of the patient, but the private dataset is described only as containing patient-level severity labels, without specifying what those clinical labels encode. On LIMUC the max-instance assumption holds by construction because bag labels are manufactured as the maximum of image-level annotations; thus LIMUC cannot validate the assumption on real clinical labels. On the private dataset, if the recorded severity was based on video, on images outside the collected bag, or on the clinician's overall impression, the training target is not a function of the bag of still images supplied to the model, and the 'real clinical settings' claim in the abstract would not actually test the max-instance MIL task. The authors should state the clinical scoring protocol explicitly and quantify the agreement between the clinical patient-level label and the maximum of the image-level annotations that were 'additionally made for evaluation.' Reporting performance separately for concordant versus discordant bags, or at least discussing the magnitude of disagreement, would clarify whether the private-data evaluation matches the problem definition.
- [Tables 1-4] All comparisons are reported as point estimates from a single 5-fold cross-validation, with no standard deviations, confidence intervals, or paired significance tests. Several margins are small, for example LIMUC kappa 0.826 versus 0.813 for Transformer in Table 1, LIMUC accuracy 0.690 versus 0.669 in Table 2, and private accuracy 0.683 versus 0.670 in Table 3. Since the paper's central claim is that SATOMIL 'outperformed' the state-of-the-art MIL methods, the authors should provide repeated runs with different random seeds and a paired significance test, or at least per-fold results, so that the reader can distinguish a systematic improvement from noise. Without this, the comparative statements in the abstract and Section 4.1 are not fully supported.
- [Section 4.3 and Table 4] The abstract states that the method 'outperformed the previous image-level methods,' but Table 4 compares only two simple image-level baselines (CL and K-rank), reports only kappa, and does not include the existing image-level UC severity estimators cited in the Related Work, such as Kadota et al., Polat et al., and Takenaka et al. Moreover, the differences for the two implemented baselines are small (LIMUC kappa 0.826 versus 0.818, private kappa 0.774 versus 0.760) and are presented without uncertainty. The comparative claim should either be restricted to the implemented baselines or supported by experiments against the existing image-level methods; in addition, the same evaluation metrics (accuracy, macro-F1, and kappa) should be reported for all methods in Table 4.
minor comments (5)
- [Section 3.3] The description of the selective aggregation layer is garbled and appears with inconsistent notation, for example 'key j = Key(t0j), qi j = Query(e0ij)' and the displayed equation for tilde-a_i_k with missing parentheses in the softmax denominator. The manuscript also contains two near-identical copies of parts of Section 3.3; please remove the duplication and unify the notation.
- [Figures 7 and 8] The attention visualizations are qualitative and report only the top 50% of attention weights. Please state how attention weights are averaged across heads and, if possible, report quantitative agreement between attention scores and instance-level severity labels, which are available for LIMUC.
- [References and front matter] The reference numbering is inconsistent (for example, PyTorch appears as [17] and Adam as [16] in the implementation details, while the numbered list assigns different numbers), and some reference entries are duplicated in the bibliography. The WACV submission header and 'CONFIDENTIAL REVIEW COPY' markers also remain in the text.
- [Dataset description] The private dataset is referred to as 'Anonymous Hospital' in the first version and as 'Kyoto Second Red Cross Hospital' in the second version of the text; use the approved anonymization consistently.
- [Equation (3)] The indicator function 1{·} is used in Equation (3) but is defined only implicitly in the K-rank label extension; define it at first use in Section 3.3.
Circularity Check
No significant circularity; the derivation is self-contained and the clinical-label assumption is a validity risk, not a circular step.
full rationale
The derivation chain is self-contained. The patient-level target is defined in Sec. 3.1 as the highest severity among bag instances, and on LIMUC the bag label is manufactured from image-level maxima ("we assigned the label of the most severe part of the image within each bag to the bag itself"), which is exactly the task definition, not a circular input. SATOMIL is trained only from these bag labels: the K-rank binary targets O_i^k = 1{Y_i>k} are deterministic re-encodings of the provided bag label, and the final estimate Y_hat = 1 + sum 1{O_hat^k>0.5} inverts that encoding. No component of the model is fitted to image-level labels; the image-level annotations on the private dataset are explicitly reserved for evaluation ("the image-level annotations were additionally made for evaluation"). The private clinical labels are an external ground truth; whether they equal the max over captured still images is a label-validity assumption that could affect conclusions, but the paper does not define those labels in terms of its own outputs, so it is a correctness risk rather than circularity. The only references with author overlap (Kadota et al., Takezaki et al.) are cited as earlier image-level severity-estimation work and are not load-bearing for the ordinal MIL formulation or the selective-aggregation mechanism; the K-rank encoding itself is attributed to external sources (Cao et al., Niu et al.). All comparisons are against external MIL baselines and ordinal classifiers on fixed test folds. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. Verdict: no significant circularity.
Assumptions & free parameters
free parameters (3)
- binarization threshold =
0.5
- learning rate =
3e-6
- early stopping patience =
100 patients
assumptions (3)
- domain assumption Patient-level severity equals the maximum of instance severities within a bag.
- standard math K-rank binary decomposition O_i^k = 1{Y_i > k} is a valid encoding of ordinal labels.
- domain assumption The attention mechanism can learn to specialize each token to instances above threshold k through bag-level BCE loss alone.
Cite this review
Pith. "Pith review of Ordinal Multiple-instance Learning for Ulcerative Colitis Severity Estimation with Selective Aggregated Transformer." pith.science (2026). https://pith.science/paper/YHFATKUD
@misc{pith2026241114750,
author = {Pith},
title = {Pith review of: Ordinal Multiple-instance Learning for Ulcerative Colitis Severity Estimation with Selective Aggregated Transformer},
year = {2026},
howpublished = {\url{https://pith.science/paper/YHFATKUD}},
note = {Machine review of arXiv:2411.14750}
}
read the original abstract
Patient-level diagnosis of severity in ulcerative colitis (UC) is common in real clinical settings, where the most severe score in a patient is recorded. However, previous UC classification methods (i.e., image-level estimation) mainly assumed the input was a single image. Thus, these methods can not utilize severity labels recorded in real clinical settings. In this paper, we propose a patient-level severity estimation method by a transformer with selective aggregator tokens, where a severity label is estimated from multiple images taken from a patient, similar to a clinical setting. Our method can effectively aggregate features of severe parts from a set of images captured in each patient, and it facilitates improving the discriminative ability between adjacent severity classes. Experiments demonstrate the effectiveness of the proposed method on two datasets compared with the state-of-the-art MIL methods. Moreover, we evaluated our method in real clinical settings and confirmed that our method outperformed the previous image-level methods. The code is publicly available at https://github.com/Shiku-Kaito/Ordinal-Multiple-instance-Learning-for-Ulcerative-Colitis-Severity-Estimation.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 1 Pith paper
-
Medical-Knowledge Driven Multiple Instance Learning for Classifying Severe Abdominal Anomalies on Prenatal Ultrasound
A medical-knowledge-driven multiple instance learning framework classifies fetal abdominal anomalies at case level from whole ultrasound examination image pools, without standard plane localization.
Reference graph
Works this paper leans on
-
[1]
Qi Bi, Shuang Yu, Wei Ji, Cheng Bian, Lijun Gong, Han- ruo Liu, Kai Ma, and Yefeng Zheng. Local-global dual perception based deep multiple instance learning for reti- nal disease classification. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th Inter- national Conference, Strasbourg, France, September 27– October 1, 2021, Pro...
work page 2021
-
[2]
Rank consistent ordinal regression for neural networks with ap- plication to age estimation
Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with ap- plication to age estimation. Pattern Recognition Letters , 140:325–331, 2020
work page 2020
-
[3]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009
2009
-
[4]
Soft labels for ordinal regres- sion
Raul Diaz and Amit Marathe. Soft labels for ordinal regres- sion. In the IEEE/CVF conference on computer vision and pattern recognition, pages 4738–4747, 2019
work page 2019
-
[5]
Joint estimation of age, gender and ethnicity: Cca vs
Guodong Guo and Guowang Mu. Joint estimation of age, gender and ethnicity: Cca vs. pls. In 2013 10th IEEE inter- national conference and workshops on automatic face and gesture recognition (FG), pages 1–6. IEEE, 2013
work page 2013
-
[6]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In the IEEE conference on computer vision and pattern recognition , pages 770–778, 2016
work page 2016
-
[7]
Attention-based deep multiple instance learning
Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In Inter- national conference on machine learning, pages 2127–2136. PMLR, 2018
work page 2018
-
[8]
Maximilian Ilse, Jakub M. Tomczak, and Max Welling. Deep multiple instance learning for digital histopathology. In Handbook of Medical Image Computing and Computer As- sisted Intervention, pages 521–546. Academic Press, 2020
work page 2020
Show all 28 references
-
[9]
Additive mil: intrinsically interpretable multiple instance learning for pathology
Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Taylor-Weiner, Limin Yu, and Aaditya Prakash. Additive mil: intrinsically interpretable multiple instance learning for pathology. Advances in Neural Information Processing Sys- tems, 35:20689–20702, 2022
2022
-
[10]
Automatic estimation of ulcerative colitis severity by learning to rank with calibration.IEEE Access, 10:25688– 25695, 2022
Takeaki Kadota, Kentaro Abe, Ryoma Bise, Takuji Kawa- mura, Naokuni Sakiyama, Kiyohito Tanaka, and Seiichi Uchida. Automatic estimation of ulcerative colitis severity by learning to rank with calibration.IEEE Access, 10:25688– 25695, 2022
2022
-
[11]
Deep bayesian active-learning- to-rank for endoscopic image data
Takeaki Kadota, Hideaki Hayashi, Ryoma Bise, Kiyohito Tanaka, and Seiichi Uchida. Deep bayesian active-learning- to-rank for endoscopic image data. In Annual Conference on Medical Image Understanding and Analysis, pages 609–622. Springer, 2022
2022
-
[12]
Dual-stream multi- ple instance learning network for whole slide image clas- sification with self-supervised contrastive learning
Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multi- ple instance learning network for whole slide image clas- sification with self-supervised contrastive learning. In the IEEE/CVF conference on computer vision and pattern recognition, pages 14318–14328, 2021
2021
-
[13]
Learning probabilistic ordinal embeddings for uncertainty-aware regression
Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng, and Jie Zhou. Learning probabilistic ordinal embeddings for uncertainty-aware regression. In the IEEE/CVF conference on computer vision and pattern recognition , pages 13896– 13905, 2021
2021
-
[14]
Interventional bag multi-instance learning on whole-slide pathological images
Tiancheng Lin, Zhimiao Yu, Hongyu Hu, Yi Xu, and Chang- Wen Chen. Interventional bag multi-instance learning on whole-slide pathological images. In the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 19830–19839, 2023
2023
-
[15]
Ordinal regression with multiple output cnn for age estimation
Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output cnn for age estimation. In the IEEE conference on computer vision and pattern recognition, pages 4920–4928, 2016
2016
-
[16]
Adam: A method for stochastic optimization
Kingma Diederik P and Ba Jimmy. Adam: A method for stochastic optimization. In arXiv, 2014
2014
-
[17]
Pytorch: An im- perative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems,...
2019
-
[18]
P. O. Pinheiro and R. Collobert. From image-level to pixel- level labeling with convolutional networks. In Computer Vi- sion and Pattern Recognition, pages 1713–1721, 2015
2015
-
[19]
Class distance weighted cross-entropy loss for ulcerative colitis severity es- timation
Gorkem Polat, Ilkay Ergenc, Haluk Tarik Kani, Yesim Ozen Alahdab, Ozlen Atug, and Alptekin Temizel. Class distance weighted cross-entropy loss for ulcerative colitis severity es- timation. In Annual Conference on Medical Image Under- standing and Analysis, pages 157–171. Sprin...
2022
-
[20]
Multi instance neural net- works
Jan Ramon and Luc De Raedt. Multi instance neural net- works. In International Conference on Machine Learning workshop, pages 53–60, 2000
2000
-
[21]
Automatic estimation of ulcera- tive colitis severity from endoscopy videos using ordinal multi-instance learning
Evan Schwab, Gabriela Oana Cula, Kristopher Standish, Stephen SF Yip, Aleksandar Stojmirovic, Louis Ghanem, and Christel Chehoud. Automatic estimation of ulcera- tive colitis severity from endoscopy videos using ordinal multi-instance learning. Computer Methods in Biomechan- i...
2022
-
[22]
Transmil: Transformer based correlated multiple instance learning for whole slide image classification
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136–2147, 2021
2021
-
[23]
Performance of a deep learning model vs human reviewers in grading endoscopic disease sever- ity of patients with ulcerative colitis
Ryan W Stidham, Wenshuo Liu, Shrinivas Bishu, Michael D Rice, Peter DR Higgins, Ji Zhu, Brahmajee K Nallamothu, and Akbar K Waljee. Performance of a deep learning model vs human reviewers in grading endoscopic disease sever- ity of patients with ulcerative colitis. JAMA networ...
2019
-
[24]
Development and validation of a deep neu- ral network for accurate evaluation of endoscopic images from patients with ulcerative colitis
Kento Takenaka, Kazuo Ohtsuka, Toshimitsu Fujii, Mariko Negi, Kohei Suzuki, Hiromichi Shimizu, Shiori Oshima, Shintaro Akiyama, Maiko Motobayashi, Masakazu Naga- hori, et al. Development and validation of a deep neu- ral network for accurate evaluation of endoscopic images fro...
2020
-
[25]
Disease Severity Regression with Contin- uous Data Augmentation
Shumpei Takezaki, Kiyohito Tanaka, Seiichi Uchida, and Takeaki Kadota. Disease Severity Regression with Contin- uous Data Augmentation. In International Symposium on Biomedical Imaging, pages 1–5, 2023
2023
-
[26]
Controlling class layout for deep ordi- nal classification via constrained proxies learning
Cong Wang, Zhiwei Jiang, Yafeng Yin, Zifeng Cheng, Ship- ing Ge, and Qing Gu. Controlling class layout for deep ordi- nal classification via constrained proxies learning. Proceed- ings of the Association for the Advancement of Artificial In- telligence Conference on Artificial...
2023
-
[27]
Revisiting multiple instance neural networks
Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks. Pattern Recognition, 74:15–24, 2018
2018
-
[28]
Mil-vt: Multiple instance learning enhanced vision transformer for fundus image classification
Shuang Yu, Kai Ma, Qi Bi, Cheng Bian, Munan Ning, Nan- jun He, Yuexiang Li, Hanruo Liu, and Yefeng Zheng. Mil-vt: Multiple instance learning enhanced vision transformer for fundus image classification. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: ...
2021
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.