REVIEW 4 major objections 8 minor 9 references
Deep Learning for Retinal Degeneration Assessment: A Comprehensive Analysis of the MARIO Challenge
T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The MARIO benchmark shows deep-learning models classify AMD activity change between two OCT scans as reliably as ophthalmologists agree with each other, while three-month prediction of disease evolution remains at chance for all twelve…
desk verdict A well-run challenge paper with a genuinely useful new longitudinal AMD benchmark and a convincing negative result for three-month prediction, but the abstract's claim that AI performs as well as a physician is not backed by a measured physician baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the MARIO benchmark itself: a longitudinal dataset of 136 patients whose consecutive OCT volumes were registered and whose matched B-scan pairs carry a seven-class expert annotation, collapsed into three clinically meaningful classes — Reduced, Stable, Worsened — plus an uninterpretable category. Test-set labels were produced independently by two ophthalmologists, so their inter-annotator agreement (about 75% on the contested distinctions) defines the human baseline the algorithms are measured against; training labels came from one annotator. The scoring machinery combines F1, specificity, and rank correlation for both tasks, adds quadratic weighted kappa for the ordinal forecasting task, and weights Task 2 at 65% of the final rank. The Algerian dataset of five patients serves as a built-in domain-shift probe for population and device differences.
What would settle it
Have a third retinal specialist adjudicate the disputed B-scan pairs, then score the top Task 1 models against the adjudicated labels: if accuracy on the absent-versus-stable and eliminated-versus-reduced splits falls to chance, the parity claim is an artifact of matching noisy labels rather than clinical truth. For Task 2, ask two clinicians to forecast the three-month outcome from the same baseline images: if the clinicians' own inter-rater kappa is also near zero, the negative result reflects the information content of a single scan rather than a failure of AI.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that change detection and change prediction sit at very different maturity levels. On the Brest test set the top Task 1 system (MIPLAB) scored an F1 of 0.858, a rank correlation of 0.692, and a specificity of 0.922 against the ophthalmologists' labels — a level the organizers read as physician parity, given that the two expert annotators themselves disagreed on about a quarter of B-scan pairs on the distinctions between absent and stable activity and between eliminated and reduced activity. On Task 2 the same system's scores fell to a rank correlation of 0.187 and a quadratic weighted kappa of 0.192, and no other finalist did better; the paper reads this as evidence that three-month evolution under anti-VEGF therapy is not yet predictable from a single scan. The paper further reports that performance on the five-patient Algerian dataset, acquired with a different device and population, reorders the teams substantially, with the method most dependent on volumetric structure collapsing, while several teams that ranked mid-table on Brest improved their standing.
Load-bearing premise
The claim that AI performs as well as a physician on Task 1 rests on treating the two annotators' manual labels as the definition of physician-level judgment, even though those two experts disagreed on about 25% of B-scan pairs on the very distinctions — absent versus stable activity, and eliminated versus reduced activity — that the task asks models to make.
Editorial extensions
If this is right
- A machine-readable benchmark for longitudinal AMD activity monitoring now exists, so the ability of an algorithm to read disease change between visits can be scored objectively and compared across methods.
- Because the best Task 1 models already operate at the level of agreement between expert annotators, the practical ceiling on this label set is set by the ground truth, not by model capacity.
- Three-month forecasting from a single OCT scan is at or near chance for all twelve finalists, and since Task 2 carries 65% of the final score, the leaderboard is dominated by the unsolved prediction problem.
- Rankings on the Brest dataset did not reliably predict rankings on the Algerian dataset, confirming that population and device shift materially change model ordering.
- Models pre-trained on retinal OCT and small, well-designed architectures matched or beat much larger networks, indicating that pretraining and design choices matter more than parameter count.
Reading between the lines
- A testable extension the paper leaves implicit: deploy the Task 1 classifiers as a triage filter inside treat-and-extend protocols, flagging visits where activity changed so the clinician can review those scans first — this use case needs only the solved change classification, not the unsolved forecast.
- The Task 2 near-chance result may in part reflect a confound the dataset cannot resolve: the anti-VEGF injections actually administered inside the three-month window are not modeled, yet they directly shape the future state; a dataset recording injection dates would clarify whether the ceiling is an information limit or a missing-variable problem.
- The concentration of annotator disagreement on inactive versus stable and eliminated versus reduced suggests the collapse from seven to three classes may discard clinically meaningful gradations; an ordinal activity score instead of three classes could raise the ceiling for humans and algorithms alike.
- The documented ranking shifts between Brest and Tlemcen — with teams like DF41 and STEP dropping sharply while MIPLAB, MIC group 6, and FERLIV stayed near the top — indicate that single-dataset leaderboards can mis-rank generalizability; reporting rankings per dataset alongside a pooled rank would be a safer standard for future challenges.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports the design, execution, and post-hoc analysis of the MARIO challenge, a MICCAI 2024 competition for longitudinal monitoring of neovascular age-related macular degeneration (AMD) from OCT. Twelve finalist teams solved two tasks: Task 1 classifies the evolution of neovascular activity between pairs of consecutive OCT B-scans from 136 patients in Brest, France, and Task 2 predicts three-month disease evolution from a single time point. The ranking protocol (Section 6.1) fixes metric weights before the challenge, combining F1-score, rank correlation, and specificity for Task 1, and additionally quadratic weighted kappa for Task 2. The organizers evaluate submitted models on an auxiliary Algerian dataset of five patients to probe population and device shift. The central claims are that top algorithms reach physician-level performance on Task 1 (best Task 1 F1 of 0.858, Table 5) and that no team can yet predict Task 2 evolution (best quadratic weighted kappa of 0.192, Table 6). The paper additionally presents bootstrap rank stability analysis, McNemar pairwise tests, and an analysis of model parameters versus ranking.
Significance. The challenge addresses a real clinical gap that existing public datasets such as RETOUCH and OCTAGON do not cover: longitudinal detection of neovascular activity changes in treated AMD patients. The evaluation design is transparent: the ranking formulas (Eqs. 1-3) were defined in advance, the evaluation code is public (Section 3.2), all twelve finalist pipelines are described with code links (Tables 3-4), and the inclusion of an Algerian cohort is a step toward generalizability assessment. These features make the benchmark potentially useful to the OCT and clinical AI communities. However, the manuscript's headline claim that AI 'performs as well as a physician' on Task 1 is not backed by a quantified physician baseline anywhere in the text, and the statistical analyses in Section 6.7 treat B-scans within a patient as independent observations despite the external cohort containing only five patients. These issues are load-bearing for the abstract's two central claims and must be resolved before the findings can be accepted at face value.
major comments (4)
- [§4.1, §4.5, Tables 5-7] The abstract claims that AI 'performs as well as a physician in measuring AMD progression (Task 1)', but the manuscript never reports a physician-versus-physician agreement computed on the test set in any of the ranking metrics (F1, rank correlation, specificity). Section 4.1 states that inter-annotator agreement was 'established as a baseline for algorithm evaluation', and Section 4.5 reports that experts disagreed on about 25% of B-scan pairs under the 7-class scheme, but no summary statistic from the two independent test annotations described in Section 4.2 is given anywhere. Without this baseline, the parity claim cannot be verified: if the physician baseline is appreciably higher than the best model's 0.858 F1 or 0.692 rank correlation (Table 5), the claim is false; if it is lower, the claim is trivially true. In addition, the paper does not state how the two independent test annotations were reconciled into the ground truth used in Tables 5-7; if one annotator's labels serve as ground truth, the other annotator's labels should be reported as the physician baseline.
- [§6.4, §6.7] The external validation set from Tlemcen contains only five patients (Section 6.4), yet Tables 8-10 and Figure 19 present B-scan-level metrics and rankings without acknowledging the resulting intra-patient correlation. The bootstrap rank intervals (Section 6.7.1, Figure 21) and the McNemar pairwise tests (Section 6.7.2) operate on individual predictions, so the effective sample size is far smaller than the stated counts of 703 and 574 B-scans, and the reported confidence bands are likely over-confident. The authors should resample at the patient level or justify why slice-level independence is valid, and they should temper the domain-shift conclusions drawn from such a small external cohort.
- [§4.3, Tables 1-2] The simplified annotation scheme defines three classes (Reduced, Stable, Worsened), but Table 2 also lists an 'Other' (-1) category containing 2312 pairs, and the MIC group 6 summary (Section 5.1.8) explicitly describes predicting an 'uninterpretable' state. The manuscript never states whether these 'Other' pairs were excluded from training and testing or treated as a fourth class in the evaluation. This ambiguity directly affects the denominator of the F1 and specificity values reported in Tables 5-9 and should be resolved explicitly.
- [§6.3.3, Table 6, abstract] The Task 2 conclusion that AI 'is not yet able of predicting future evolution' is supported only by small point estimates (best quadratic weighted kappa 0.192, best rank correlation 0.210, Table 6). To make this negative claim quantitative, the authors should compare against a defined chance-level predictor (for example, majority-class prediction or random predictions) and provide confidence intervals or a test of whether the kappa values differ from zero. As written, the statement is an informal reading of the table rather than a measured outcome.
minor comments (8)
- [Abstract] The phrase 'is not yet able of predicting' should read 'is not yet able to predict'.
- [§1] The abbreviation 'neovascular AMD (AMD)' is confusing; the standard term is 'nAMD' and the acronym should be defined consistently.
- [§6.7.1] The heading 'Ranking boostraping' contains a typo, and the sentence 'following the methodology proposed by Wiesenfarth et al. Wiesenfarth et al. (2021)' contains a duplicated citation; both should be corrected.
- [§5.1.12] The passages 'Theynote that uninterpretable predictions' should be split into 'They note that uninterpretable predictions'.
- [Table 1] The class names 'PersistentReduced' and 'PersistentStable' are missing internal spaces and should be formatted consistently with the other entries.
- [Figures 8-18] In the manuscript version I received, many confusion matrices and misclassification plots are illegible (they appear as garbled character sequences); the authors must ensure high-resolution, correctly rendered figures in the final submission.
- [§6.4] The paper mentions that the Algerian cohort contains only five patients within the body text; this should be emphasized in the abstract or introduction as a prominent limitation of the cross-dataset analyses.
- [§5.1] The sentence 'To be selected as a finalist, you should have submitted...' shifts to the second person; the entire paper should use a consistent formal voice.
Circularity Check
No significant circularity: MARIO is a self-contained benchmark evaluation; the 'physician-level' parity claim rests on a missing physician baseline, which is an evidence gap rather than a reduction-by-construction.
full rationale
The paper's derivation chain is: expert annotation of consecutive OCT pairs (Section 4.2), a deterministic collapse of the 7-class scheme into three simplified classes (Section 4.3), pre-registered evaluation metrics with fixed weighting formulas (Eqs. 1-3, Section 6.1), and held-out test rankings (Tables 5-7). Team predictions are scored against expert-labeled test data; no parameter is fitted to those labels and then renamed a prediction, and the metric weights were fixed before the challenge. The sole self-citation (Quellec et al., 2019, Section 4.4) supports only a dataset-size comparison and is not load-bearing. The only self-referential element is the framing of inter-annotator agreement 'as a baseline for algorithm evaluation' (Section 4.1) combined with the abstract's claim that AI 'performs as well as a physician' on Task 1; however, the paper never reports a physician-vs-physician agreement on the test set in the ranking metrics, and the algorithm's F1 (0.858 for MIPLAB) is computed independently against the expert labels and could in principle exceed or fall short of such a baseline. That makes the parity claim an unquantified interpretation, i.e., a reporting/evidence gap, not a result equivalent to its inputs by construction. Observed limitations (approximately 25% annotator disagreement, approximately 10% unregistered volumes, small cohort) are acknowledged in Sections 4.4-4.5 and do not create a fitted-input loop. Therefore, under the rule that circularity requires quoting an actual reduction, no circular step exists.
Assumptions & free parameters
free parameters (1)
- Ranking metric weights =
final = 0.35*f_task1 + 0.65*f_task2; f_task1 = 0.5*F1 + 0.25*Rk + 0.25*Spec; f_task2 = 0.1*F1 + 0.2*Rk + 0.1*Spec +…
assumptions (3)
- domain assumption The seven manual annotation classes can be collapsed into three clinical classes (Reduced, Stable, Worsened) by the mapping in Section 4.3, and the 'Other' class is excluded from evaluation.
- domain assumption Inter-annotator agreement is a valid baseline for physician-level performance on Task 1.
- domain assumption The Tlemcen dataset of five patients is sufficient to measure population and device shift effects on rankings.
Cite this review
Pith. "Pith review of Deep Learning for Retinal Degeneration Assessment: A Comprehensive Analysis of the MARIO Challenge." pith.science (2026). https://pith.science/paper/XUC6DQKM
@misc{pith2026250602976,
author = {Pith},
title = {Pith review of: Deep Learning for Retinal Degeneration Assessment: A Comprehensive Analysis of the MARIO Challenge},
year = {2026},
howpublished = {\url{https://pith.science/paper/XUC6DQKM}},
note = {Machine review of arXiv:2506.02976}
}
read the original abstract
The MARIO challenge, held at MICCAI 2024, focused on advancing the automated detection and monitoring of age-related macular degeneration (AMD) through the analysis of optical coherence tomography (OCT) images. Designed to evaluate algorithmic performance in detecting neovascular activity changes within AMD, the challenge incorporated unique multi-modal datasets. The primary dataset, sourced from Brest, France, was used by participating teams to train and test their models. The final ranking was determined based on performance on this dataset. An auxiliary dataset from Algeria was used post-challenge to evaluate population and device shifts from submitted solutions. Two tasks were involved in the MARIO challenge. The first one was the classification of evolution between two consecutive 2D OCT B-scans. The second one was the prediction of future AMD evolution over three months for patients undergoing anti-vascular endothelial growth factor (VEGF) therapy. Thirty-five teams participated, with the top 12 finalists presenting their methods. This paper outlines the challenge's structure, tasks, data characteristics, and winning methodologies, setting a benchmark for AMD monitoring using OCT, infrared imaging, and clinical data (such as the number of visits, age, gender, etc.). The results of this challenge indicate that artificial intelligence (AI) performs as well as a physician in measuring AMD progression (Task 1) but is not yet able of predicting future evolution (Task 2).
Figures
Figures from the paper (16 more)
Reference graph
Works this paper leans on
-
[1]
N. M. Schultz, S. Bhardwaj, C. Barclay, L. Gaspar, J. Schwartz, Global burden of dry age-related macular degeneration: A targeted literature re- view, Clinical Therapeutics 43 (2021) 1792–1818. URL:https://www.sciencedirect.com/ science/article/pii/S0149291821003106. 5https://zenodo.org/records/15270469 32 doi:https://doi.org/10.1016/j.clinthera. 2021.08....
-
[27]
G. Quellec, J. Kowal, P. W. Hasler, H. P. N. Scholl, S. Zweifel, B. Konstantinos, J. E. R. Carvalho, T. Heeren, C. Egan, A. Tufail, P. M. Maloca, Feasibil- ity of support vector machine learning in age-related macular degeneration using small sample yielding sparse optical coherence tomography data, Acta Oph- thalmologica 97 (2019). URL:https://doi.org/ 1...
-
[875]
R. H. Guymer, C. M. Markey, I. L. McAllister, M. C. Gillies, A. P. Hunyor, J. J. Arnold, F. Investiga- tors, et al., Tolerating subretinal fluid in neovas- cular age-related macular degeneration treated with ranibizumab using a treat-and-extend regimen: Fluid 33 study 24-month results, Ophthalmology 126 (2019) 723–734. A. Lang, A. Carass, O. Al-Louzi, P. ...
work page 2019
-
[1266]
C. Lea, M. D. Flynn, R. Vidal, A. Reiter, G. D. Hager, Temporal convolutional networks for action segmen- tation and detection, Proceedings of the IEEE con- ference on computer vision and pattern recognition (2017) 156–165. R. T. Chen, Y . Rubanova, J. Bettencourt, D. K. Duve- naud, Neural ordinary differential equations, Ad- vances in neural information ...
-
[2019]
URL:https://arxiv.org/abs/ 1912.01703.arXiv:1912.01703. V . Andrearczyk, V . Oreiller, S. Boughdad, C. C. Le Rest, O. Tankyevych, H. Elhalawani, M. Jreige, J. O. Prior, M. Vallières, D. Visvikis, M. Hatt, A. Depeursinge, Automatic head and neck tumor segmentation and outcome prediction re- lying on fdg-pet/ct images: Findings from the second edition of th...
arXiv 2023
-
[2021]
Y . Qiu, S. Yamamoto, K. Nakashima, R. Suzuki, K. Iwata, H. Kataoka, Y . Satoh, Describing and lo- calizing multiple changes with transformers, in: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1971–1980. J. Lu, D. Batra, D. Parikh, S. Lee, Vilbert: Pretrain- ing task-agnostic visiolinguistic representations for visi...
work page 2019
-
[2022]
URL:https://arxiv.org/abs/ 2211.15733.arXiv:2211.15733. 37
-
[2464]
doi:10.3390/diagnostics13142464
URL:http:// dx.doi.org/10.3390/diagnostics13142464. doi:10.3390/diagnostics13142464. U. Schmidt-Erfurth, S. Klimscha, S. M. Waldstein, H. Bogunovi´c, A view of the current and future role of optical coherence tomography in the management of age-related macular degeneration, Eye 31 (2016) 26–44. URL:http://dx.doi.org/10.1038/eye. 2016.227. doi:10.1038/eye....
Show all 9 references
-
[3642]
Simader, M
C. Simader, M. Ritter, M. Bolz, G. G. Deák, U. Mayr- Sponer, I. Golbaz, M. Kundi, U. M. Schmidt-Erfurth, Morphologic parameters relevant for visual outcome during anti-angiogenic therapy of neovascular age- related macular degeneration, Ophthalmology 121 (2014) 1237–1245. A. R...
2014
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.