{"paper":{"title":"ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":[],"primary_cat":"cs.CV","authors_text":"Aaditya Baranwal, Akash Gupta, Alexander Lo, Alexandru Coca, Anh Totti Nguyen, Ansh Sharma, Brian Pulfer, Charig Yang, David I. Atkinson, Florian Langer, Gyungin Shin, Hanyi Xiong, Ioana Croitoru, Jakob D. Kunz, Jialu Tang, Jingyi Lu, Jonathan Roberts, Kai Han, Kaiqu Liang, Mikah Dang, Mohammad Reza Taesiri, Qiaochu Yang, Sam Purkis, Samuel Albanie, Samuel Roberts, Sebastian Dziadzio, Shiyang Chen, Simion-Vlad Bogolin, Steven Walton, Tianshuo Yan, Vatsal Raina, Vishaal Udandarao, Vyas Raina, Wenye Lin","submitted_at":"2025-02-13T18:59:11Z","abstract_excerpt":"Large Multimodal Models (LMMs) exhibit major shortfalls when interpreting images and, by some measures, have poorer spatial cognition than small children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by an ongoing surge of model progress. To address this, there is a pressing need for difficult benchmarks that remain relevant for longer. We take this idea to its limit by introducing ZeroBench-a lightweight visual reasoning benchmark that is entirely impossible for contemporary frontier LMMs. Our benchmark consists of 100 manual"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2502.09696","kind":"arxiv","version":2},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2502.09696/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}