pith. sign in

arxiv: 2303.17719 · v1 · pith:WZB52G5Rnew · submitted 2023-03-30 · 💻 cs.CV · cs.LG

Why is the winner the best?

Matthias Eisenmann , Annika Reinke , Vivienn Weru , Minu Dietlinde Tizabi , Fabian Isensee , Tim J. Adler , Sharib Ali , Vincent Andrearczyk
show 117 more authors
Marc Aubreville Ujjwal Baid Spyridon Bakas Niranjan Balu Sophia Bano Jorge Bernal Sebastian Bodenstedt Alessandro Casella Veronika Cheplygina Marie Daum Marleen de Bruijne Adrien Depeursinge Reuben Dorent Jan Egger David G. Ellis Sandy Engelhardt Melanie Ganz Noha Ghatwary Gabriel Girard Patrick Godau Anubha Gupta Lasse Hansen Kanako Harada Mattias Heinrich Nicholas Heller Alessa Hering Arnaud Huaulm\'e Pierre Jannin Ali Emre Kavur Old\v{r}ich Kodym Michal Kozubek Jianning Li Hongwei Li Jun Ma Carlos Mart\'in-Isla Bjoern Menze Alison Noble Valentin Oreiller Nicolas Padoy Sarthak Pati Kelly Payette Tim R\"adsch Jonathan Rafael-Pati\~no Vivek Singh Bawa Stefanie Speidel Carole H. Sudre Kimberlin van Wijnen Martin Wagner Donglai Wei Amine Yamlahi Moi Hoon Yap Chun Yuan Maximilian Zenk Aneeq Zia David Zimmerer Dogu Baran Aydogan Binod Bhattarai Louise Bloch Raphael Br\"ungel Jihoon Cho Chanyeol Choi Qi Dou Ivan Ezhov Christoph M. Friedrich Clifton Fuller Rebati Raman Gaire Adrian Galdran \'Alvaro Garc\'ia Faura Maria Grammatikopoulou SeulGi Hong Mostafa Jahanifar Ikbeom Jang Abdolrahim Kadkhodamohammadi Inha Kang Florian Kofler Satoshi Kondo Hugo Kuijf Mingxing Li Minh Huan Luu Toma\v{z} Martin\v{c}i\v{c} Pedro Morais Mohamed A. Naser Bruno Oliveira David Owen Subeen Pang Jinah Park Sung-Hong Park Szymon P{\l}otka Elodie Puybareau Nasir Rajpoot Kanghyun Ryu Numan Saeed Adam Shephard Pengcheng Shi Dejan \v{S}tepec Ronast Subedi Guillaume Tochon Helena R. Torres Helene Urien Jo\~ao L. Vila\c{c}a Kareem Abdul Wahid Haojie Wang Jiacheng Wang Liansheng Wang Xiyue Wang Benedikt Wiestler Marek Wodzinski Fangfang Xia Juanying Xie Zhiwei Xiong Sen Yang Yanwu Yang Zixuan Zhao Klaus Maier-Hein Paul F. J\"ager Annette Kopp-Schneider Lena Maier-Hein
This is my paper
classification 💻 cs.CV cs.LG
keywords strategiescompetitionsfocusimagewhatwinningalgorithmsanalysis
0
0 comments X
read the original abstract

International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT $\unicode{x2013}$ Multitracer Multicenter Generalization

    cs.CV 2026-05 unverdicted novelty 5.0

    The autoPET3 challenge finds that leading AI models reach a mean Dice score of 0.66 for multitracer PET/CT lesion segmentation, with compositional generalization to unseen tracer-center pairs remaining an open problem...

  2. Surgical Visual Understanding (SurgVU) Dataset

    cs.CV 2025-01 unverdicted novelty 5.0

    Releases the SurgVU dataset of surgical videos and labels to enable machine learning research in surgical data science.

  3. The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT $\unicode{x2013}$ Multitracer Multicenter Generalization

    cs.CV 2026-05 unverdicted novelty 4.0

    The autoPET3 challenge finds good in-domain lesion segmentation performance in multitracer PET/CT but identifies compositional generalization to unseen tracer-center combinations as an open problem driven by volume ov...