Pith. sign in

REVIEW

Understanding metric-related pitfalls in image analysis validation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2302.01790 v4 pith:SRA7UYPJ submitted 2023-02-03 cs.CV

Understanding metric-related pitfalls in image analysis validation

Annika Reinke , Minu D. Tizabi , Michael Baumgartner , Matthias Eisenmann , Doreen Heckmann-N\"otzel , A. Emre Kavur , Tim R\"adsch , Carole H. Sudre
show 70 more authors
Laura Acion Michela Antonelli Tal Arbel Spyridon Bakas Arriel Benis Matthew Blaschko Florian Buettner M. Jorge Cardoso Veronika Cheplygina Jianxu Chen Evangelia Christodoulou Beth A. Cimini Gary S. Collins Keyvan Farahani Luciana Ferrer Adrian Galdran Bram van Ginneken Ben Glocker Patrick Godau Robert Haase Daniel A. Hashimoto Michael M. Hoffman Merel Huisman Fabian Isensee Pierre Jannin Charles E. Kahn Dagmar Kainmueller Bernhard Kainz Alexandros Karargyris Alan Karthikesalingam Hannes Kenngott Jens Kleesiek Florian Kofler Thijs Kooi Annette Kopp-Schneider Michal Kozubek Anna Kreshuk Tahsin Kurc Bennett A. Landman Geert Litjens Amin Madani Klaus Maier-Hein Anne L. Martel Peter Mattson Erik Meijering Bjoern Menze Karel G.M. Moons Henning M\"uller Brennan Nichyporuk Felix Nickel Jens Petersen Susanne M. Rafelski Nasir Rajpoot Mauricio Reyes Michael A. Riegler Nicola Rieke Julio Saez-Rodriguez Clara I. S\'anchez Shravya Shetty Maarten van Smeden Ronald M. Summers Abdel A. Taha Aleksei Tiulpin Sotirios A. Tsaftaris Ben Van Calster Ga\"el Varoquaux Manuel Wiesenfarth Ziv R. Yaniv Paul F. J\"ager Lena Maier-Hein
This is my paper
classification cs.CV
keywords analysisimagevalidationmetricspitfallsaccessiblecomprehensionindividual
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.