{"paper":{"title":"BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models","license":"http://creativecommons.org/licenses/by-sa/4.0/","headline":"A benchmark with 18 diverse datasets shows BM25 as a robust zero-shot baseline while re-ranking models lead in performance at higher cost.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.IR","authors_text":"Abhishek Srivastava, Andreas R\\\"uckl\\'e, Iryna Gurevych, Nandan Thakur, Nils Reimers","submitted_at":"2021-04-17T23:29:55Z","abstract_excerpt":"Existing neural information retrieval (IR) models have often been studied in homogeneous and narrow settings, which has considerably limited insights into their out-of-distribution (OOD) generalization capabilities. To address this, and to facilitate researchers to broadly evaluate the effectiveness of their models, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous evaluation benchmark for information retrieval. We leverage a careful selection of 18 publicly available datasets from diverse text retrieval tasks and domains and evaluate 10 state-of-the-art retrieval systems includi"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The 18 selected datasets provide a sufficiently heterogeneous and representative sample of out-of-distribution scenarios to draw general conclusions about model generalization in information retrieval.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"BEIR is a heterogeneous zero-shot benchmark showing BM25 as a robust baseline while re-ranking and late-interaction models perform best on average at higher cost, with dense and sparse models lagging in generalization.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"A benchmark with 18 diverse datasets shows BM25 as a robust zero-shot baseline while re-ranking models lead in performance at higher cost.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"c8c5486d65b59f6abd205fc193c2546ab52f251a8c9211c93e46e83dfb64620a"},"source":{"id":"2104.08663","kind":"arxiv","version":4},"verdict":{"id":"96c48c26-5450-4830-9dd1-725c1bcf2020","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-12T14:35:26.616158Z","strongest_claim":"Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches.","one_line_summary":"BEIR is a heterogeneous zero-shot benchmark showing BM25 as a robust baseline while re-ranking and late-interaction models perform best on average at higher cost, with dense and sparse models lagging in generalization.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The 18 selected datasets provide a sufficiently heterogeneous and representative sample of out-of-distribution scenarios to draw general conclusions about model generalization in information retrieval.","pith_extraction_headline":"A benchmark with 18 diverse datasets shows BM25 as a robust zero-shot baseline while re-ranking models lead in performance at higher cost."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2104.08663/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":94,"sample":[{"doi":"","year":2019,"title":"Amin Ahmad, Noah Constant, Yinfei Yang, and Daniel Cer. 2019. ReQA: An Evaluation for End-to-End Answer Retrieval Models. InProceedings of the 2nd Workshop on Machine Reading for Question Answering, p","work_id":"5d0113a3-6e54-41bc-b4c0-6c8692485ea0","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi","work_id":"6da63d05-546b-4e55-b023-a4f551c20f18","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2021,"title":"In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pages 547–564, Online","work_id":"9df27e51-690e-449e-a906-b69d6e0b3dce","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2015,"title":"Petr Baudiš and Jan Šediv `y. 2015. Modeling of the question answering task in the yodaqa system. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 222–2","work_id":"6d514303-9ae8-4f7f-81e7-1f9ba1b352d2","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2013,"title":"Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Langu","work_id":"a75402fc-378a-4306-95ce-99b8fe6087b9","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":94,"snapshot_sha256":"b1341d9084d08791920925d03692a5cb3bcde1535bbacca8d41c47bdb9d36331","internal_anchors":2},"formal_canon":{"evidence_count":1,"snapshot_sha256":"9ebd7936b407744ce6e0bc8bbb449774aee4d74c233f913afa29e29a412518b8"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}