Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces BEIR-NL, a machine-translated Dutch version of 14 BEIR datasets, and reports that BM25 remains a competitive zero-shot baseline, outperformed only by larger dense models trained specifically for retrieval.

desk verdict Useful Dutch BEIR resource, but the abstract's BM25 claim is contradicted by the paper's own Table 3 and the translation validation is thinner than it looks. read the letter →

arxiv 2412.08329 v1 pith:54BUXJMJ submitted 2024-12-11 cs.CL

Nikolay Banar , Ehsan Lotfi , Walter Daelemans This is my paper · ORCID
classification cs.CL
keywords informationretrievalzero-shotevaluationDutchlanguageBEIRBM25densererankingmachinetranslationbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to establish that a Dutch information retrieval benchmark can be built by automatically translating the English BEIR datasets, and that this translated benchmark is usable for zero-shot evaluation—gauging retrieval models on Dutch tasks they were not trained on. It translates the 14 publicly available BEIR datasets into Dutch, releases them as BEIR-NL, and evaluates a spread of lexical, dense, and reranking models. The main empirical claim is that BM25 keyword search remains a competitive baseline on the Dutch data, clearly outperformed only by the larger dense models trained specifically for retrieval; pairing BM25 with a multilingual reranker reaches performance comparable to the best dense rankers. A back-translation check on five datasets shows a small but consistent performance drop, which the authors take as evidence that translation itself reduces benchmark quality. The authors present BEIR-NL as a public resource for Dutch IR research while noting that native Dutch gold data is still needed for full fidelity.

What carries the argument

The central object is BEIR-NL, a machine-translated Dutch mirror of the 14 publicly available datasets from the BEIR benchmark, spanning biomedical, Wikipedia, financial, scientific, argument, and question-answer retrieval tasks. The evaluation machinery is the standard BEIR zero-shot protocol: BM25 as the lexical baseline, dense bi-encoder models that score query-document pairs by cosine similarity on normalized embeddings, and cross-encoder rerankers applied to the top-100 documents retrieved by BM25, with nDCG@10 and Recall@100 as metrics. Translation is performed by a commercial LLM-based API with queries and documents translated independently, and quality is checked through a small human-annotated sample and a five-dataset back-translation control that isolates translation loss from model-language competence.

What would settle it

A concrete check would be to have native Dutch speakers write natural Dutch queries for a subset of BEIR-NL topics and compare model rankings on these queries against rankings on the machine-translated queries; if the rankings diverge substantially or if a larger human-annotated sample finds major translation errors well above the reported 2.2%, the benchmark is measuring translation artifacts rather than Dutch retrieval ability.

Watch

Extended reading notes

Core claim

The paper's discovery, stated on its own terms, is that a machine-translated benchmark can reproduce the shape of the English BEIR evaluation landscape in Dutch: on the ten overlapping datasets, BM25 reaches 35.9 average nDCG@10 against 41.9 for the original English BEIR, and the top spots go to the larger retrieval-trained dense models (notably multilingual-e5-large-instruct) and to BM25 combined with cross-encoder rerankers. The paper also establishes that translation carries a measurable cost: back-translating a five-dataset subset from Dutch to English lowers nDCG@10 by about 1.9 points for BM25 and 2.6 points for gte-multilingual-base, which the authors attribute to lexical mismatches caused by translating queries and passages independently. The overall pattern is that older sentence-embedding models trail BM25 on Dutch data, while only the new generation of retrieval-trained dense models clearly surpasses it.

Load-bearing premise

The benchmark's usefulness rests on the assumption that automatic translation preserves enough semantic fidelity that retrieval scores reflect Dutch-language competence rather than translation artifacts; the evidence offered is a 140-item human quality check and a five-dataset back-translation proxy, not native Dutch gold labels.

Editorial extensions

If this is right

  • Dutch IR models can be compared zero-shot across 14 tasks and multiple domains on a single public benchmark, filling a gap for a language with few native IR test collections.
  • For practical Dutch retrieval, BM25 followed by a multilingual reranker is a strong recipe that matches the best dense ranking models, so teams without the largest dense encoders are not at a major disadvantage.
  • Translated benchmarks carry a translation penalty: scores on BEIR-NL are several points lower than on English BEIR and drop further under back-translation, so cross-lingual numbers should not be read as exact native-language performance.
  • Evaluations on BEIR-NL need to account for training contamination, since several of the top dense models have likely seen BEIR data during training, which may inflate their zero-shot scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the translation penalty is roughly uniform across models, the model ordering found on BEIR-NL likely transfers to real Dutch IR, making the benchmark useful for model selection even if absolute scores are pessimistic.
  • The five-dataset back-translation protocol can serve as a reusable quality control for any future translated benchmark: a small or zero delta would indicate the translation pipeline is not the main source of performance loss.
  • The result that reranked BM25 matches the best dense models suggests that lexical recall in Dutch is not the bottleneck, so Dutch-specific rerankers or query-expansion methods may yield larger gains than scaling dense encoders.
  • A native Dutch gold benchmark built from the same relevance judgments would separate translation loss from model-language competence, which the paper explicitly leaves to future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces BEIR-NL, a Dutch-language zero-shot information retrieval benchmark created by automatically translating 14 publicly available BEIR datasets from English to Dutch, mainly with Gemini-1.5-flash. The authors evaluate BM25, eight multilingual dense ranking models, and three reranking models, reporting nDCG@10 and Recall@100 on all datasets plus Dutch mMARCO. They also report a small human quality check of the translations and a back-translation experiment on five datasets. The main claims are that BEIR-NL is a usable resource for Dutch IR evaluation and that BM25 remains competitive, being outperformed only by larger dense retrieval models, with BM25+reranking matching the best dense rankers.

Significance. If the benchmark holds up, this is a useful and timely resource for Dutch IR, a language that is underrepresented in retrieval evaluation. The authors release the data on Hugging Face, follow BEIR conventions, compare with BEIR-PL and BEIR, and are transparent about license inheritance and potential contamination. The evaluation is standard and the resource is likely to be reused. However, the paper's headline claims about the BM25 comparison are overstated, and the evidence for translation quality is thin; both issues need to be fixed before the paper can be relied upon as a benchmark paper.

major comments (4)
  1. [Abstract, Section 5.1, Table 3] The abstract's claim that BM25 'is only outperformed by the larger dense models trained for retrieval' is contradicted by the paper's own results. In Table 3, multilingual-e5-small (118M parameters, retrieval-finetuned, and not flagged with a contamination dagger on these datasets) outperforms BM25 on FiQA-2018 (20.39 vs 18.73), ArguAna (44.76 vs 41.76), CQADupstack (28.51 vs 27.77), and DBPedia (25.89 vs 25.46). Since this performance ranking is part of the paper's central contribution, the wording must be changed to reflect the actual pattern, for example by saying BM25 is outperformed by most or many retrieval-trained dense models, or by giving the exceptions explicitly.
  2. [Section 3.1, Section 5.3] The evidence supporting the benchmark's reliability is currently too weak for the strength of the paper's claims. The translation quality check uses only 140 items (10 per dataset) with a single annotator, and the reported 2.2% major issues corresponds to just three items; this yields a very wide confidence interval. The back-translation experiment covers only 5 of the 14 datasets and only one dense model. I recommend reporting the confidence interval for the quality estimate, ideally adding a second annotator or a larger sample, and tempering statements such as 'almost 98% of the translated samples can be trusted' so that they do not overstate the precision of the estimate.
  3. [Appendix B] The translation prompts in Appendix B instruct the model to 'Translate to English', yet Section 3.1 states that the pipeline translates from English to Dutch. If this is a typographical error it should be corrected; if the prompts were actually used as written, the resulting data would not be Dutch. Either way, the appendix must be fixed because the prompt template is essential for reproducibility.
  4. [Section 5.1, Section 6, Limitations] The paper repeatedly describes the evaluations as zero-shot even though Table 3 marks most of the top-performing dense models with a dagger indicating likely in-domain contamination, and the Limitations section acknowledges that these results may not be proper zero-shot. The text should more clearly separate contaminated from uncontaminated rows and specify that the benchmark itself is zero-shot for future models, while some of the reported numbers are not zero-shot evaluations. This distinction matters because the conclusion that larger dense models outperform BM25 rests substantially on daggered numbers.
minor comments (6)
  1. [Table 2] The 'IR Finetuned' column contains the typo 'Y es' for several models; it should read 'Yes'.
  2. [Table 6] The first column header 'BEIR' is confusing because the table reports results on original BEIR and back-translated data; renaming it to 'BEIR (EN)' would match Table 4 and improve clarity.
  3. [Section 5.2] Footnote 8 says 'Assuming a uniform BM25 performance for different languages, which is not trivial'; this is an important caveat and should be moved into the main text rather than relegated to a footnote.
  4. [References] The Hendrycks et al. reference for MMLU lacks a year and venue; please complete it. Also, several author names in the bibliography contain spacing artifacts such as 'Y ang', 'Y an', and 'T worek', which should be cleaned.
  5. [Section 3.1] The phrase 'less than 450 Euro' should include the currency symbol and, ideally, a note on the exchange rate or date, to give readers a better sense of the cost.
  6. [Table 1] The table lists mMARCO as a dataset while the text says it is not translated in this work; a parenthetical note in the caption or table would avoid confusion for readers who only look at the table.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: BEIR-NL is a translated benchmark evaluated empirically, and no claim reduces to its own inputs.

full rationale

BEIR-NL is constructed by an external translation step: the paper translates 14 public BEIR datasets into Dutch with Gemini-1.5-flash (Section 3.1) and then evaluates fixed retrieval models with standard metrics (nDCG@10, Recall@100). There is no fitted parameter, no equation whose output is defined by its input, and no load-bearing self-citation used to justify the central claim. The benchmark's validity is an empirical matter of translation fidelity, and the paper explicitly flags the relevant caveats: the 140-sample quality check (2.2% major, 14.8% minor issues) and the independent translation of queries and passages in Section 5.2, which can cause lexical mismatch. These are data-quality limitations, not circular reasoning. The back-translation experiment in Section 5.3 is a proxy for translation loss, but it does not define the benchmark in terms of its own results. The abstract's statement that BM25 'is only outperformed by the larger dense models trained for retrieval' is inconsistent with Table 3, where e5-multilingual-small (118M parameters) beats BM25 on FiQA-2018, ArguAna, CQADupstack, and DBPedia; however, an internal inconsistency or overstatement is a correctness issue, not circularity. Likewise, the limitations section's discussion of native Dutch resources, data contamination, and benchmark validity over time qualifies the benchmark's reliability without creating a definitional loop. Appendix B's prompt says 'Translate to English' while the pipeline translates to Dutch; this appears to be a typographical error, not a circular step. Overall, the paper's derivation chain is self-contained in the sense that no experimental conclusion is forced by construction, fitted input, or self-citation, and the claimed performance rankings stand or fall on the reported measurements rather than on any circular reduction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its central claims depend on domain assumptions about translation fidelity, representativeness of quality checks, and the validity of the back-translation proxy.

assumptions (4)
  • domain assumption Machine translation preserves the relevance labels of the original BEIR datasets.
    Queries and documents are translated independently, so lexical or semantic alignment may shift. This is assumed throughout and discussed in section 5.2.
  • domain assumption The sample of 140 translated items (10 per dataset) is representative of the full benchmark.
    The quality check in section 3.1 uses a single native speaker on a small sample; generalization to the full corpus is assumed without statistical justification.
  • domain assumption The back-translation performance drop is a proxy for translation quality impact.
    Section 5.3 attributes the average 1.9 and 2.6 point drops to translation quality, assuming round-trip artifacts do not dominate the loss.
  • domain assumption BM25 performance is comparable across languages for the purpose of cross-lingual comparison.
    Footnote 8 notes this assumption is 'not trivial', yet it underpins the BEIR-NL versus BEIR-PL versus BEIR comparison in Table 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language." pith.science (2026). https://pith.science/paper/54BUXJMJ

@misc{pith2026241208329,
  author       = {Pith},
  title        = {Pith review of: BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/54BUXJMJ}},
  note         = {Machine review of arXiv:2412.08329}
}
read the original abstract

Zero-shot evaluation of information retrieval (IR) models is often performed using BEIR; a large and heterogeneous benchmark composed of multiple datasets, covering different retrieval tasks across various domains. Although BEIR has become a standard benchmark for the zero-shot setup, its exclusively English content reduces its utility for underrepresented languages in IR, including Dutch. To address this limitation and encourage the development of Dutch IR models, we introduce BEIR-NL by automatically translating the publicly accessible BEIR datasets into Dutch. Using BEIR-NL, we evaluated a wide range of multilingual dense ranking and reranking models, as well as the lexical BM25 method. Our experiments show that BM25 remains a competitive baseline, and is only outperformed by the larger dense models trained for retrieval. When combined with reranking models, BM25 achieves performance on par with the best dense ranking models. In addition, we explored the impact of translation on the data by back-translating a selection of datasets to English, and observed a performance drop for both dense and lexical methods, indicating the limitations of translation for creating benchmarks. BEIR-NL is publicly available on the Hugging Face hub.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning

    cs.CL 2025-06 conditional novelty 5.0 of 10

    State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.

Reference graph

Works this paper leans on

49 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268

    arXiv 2016

  2. [2]

    Alexander Bondarenko, Maik Fr \"o be, Meriem Beloucif, Lukas Gienapp, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, et al. 2020. Overview of touch \'e 2020: argument retrieval. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association...

    work page 2020

  3. [3]

    Luiz Henrique Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, , Roberto Lotufo, and Rodrigo Nogueira. 2021. https://arxiv.org/abs/2108.13897 mmarco: A multilingual version of ms marco passage ranking dataset . Preprint, arXiv:2108.13897

    arXiv 2021

  4. [4]

    Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016. A full-text learning to rank dataset for medical information retrieval. In Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20--23, 2016. Proceedings 38, pages 716--722. Springer

    2016

  5. [5]

    Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. https://doi.org/10.18653/v1/P17-1171 Reading W ikipedia to answer open-domain questions . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870--1879, Vancouver, Canada. Association for Computational Linguistics

    doi:10.18653/v1/p17-1171 2017

  6. [6]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. https://arxiv.org/abs/2402.03216 Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation . Preprint, arXiv:2402.03216

    arXiv 2024

  7. [7]

    Mathieu Ciancone, Imene Kerboua, Marion Schaeffer, and Wissam Siblini. 2024. Mteb-french: Resources for french sentence embedding evaluation and analysis. arXiv preprint arXiv:2405.20468

    arXiv 2024

  8. [8]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

Show all 49 references
  1. [9]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  2. [10]

    Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. https://doi.org/10.18653/v1/2020.acl-main.207 SPECTER : Document-level representation learning using citation-informed transformers . In Proceedings of the 58th Annual Meeting of the Association for C...

    2020 doi

  3. [11]

    Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. Climate-fever: A dataset for verification of real-world climate claims. arXiv preprint arXiv:2012.00614

    2020 arXiv

  4. [12]

    Kenneth Enevoldsen, M \'a rton Kardos, Niklas Muennighoff, and Kristoffer Laigaard Nielbo. 2024. The scandinavian embedding benchmarks: Comprehensive assessment of multilingual and monolingual text embedding. arXiv preprint arXiv:2406.02396

    2024 arXiv

  5. [13]

    Leon Engl \"a nder, Hannah Sterz, Clifton Poth, Jonas Pfeiffer, Ilia Kuznetsov, and Iryna Gurevych. 2024. M2qa: Multi-domain multilingual question answering. arXiv preprint arXiv:2407.01091

    2024 arXiv

  6. [14]

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878--891

    2022

  7. [15]

    Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017. Dbpedia-entity v2: a test collection for entity search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in I...

    2017

  8. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations

  9. [17]

    Doris Hoogeveen, Karin M Verspoor, and Timothy Baldwin. 2015. Cqadupstack: A benchmark data set for community question-answering research. In Proceedings of the 20th Australasian document computing symposium, pages 1--8

    2015

  10. [18]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research

    2022

  11. [19]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  12. [20]

    Viet Lai, Chien Nguyen, Nghia Ngo, Thuật Nguyễn, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Met...

    2023

  13. [21]

    Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rockt \"a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://api.semanticscholar.org/CorpusID:218869575 Retrieval-augmented gener...

    2020 arXiv

  14. [22]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214--3252

    2022

  15. [23]

    Antoine Louis, Vageesh Saxena, Gijs van Dijck, and Gerasimos Spanakis. 2024. Colbert-xm: A modular multi-vector representation model for zero-shot multilingual information retrieval. arXiv preprint arXiv:2402.15059

    2024 arXiv

  16. [24]

    Macedo Maia, Siegfried Handschuh, Andr \'e Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www'18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pages 1941--1942

    2018

  17. [25]

    Zhuoyuan Mao and Tetsuji Nakagawa. 2023. https://doi.org/10.18653/v1/2023.eacl-main.138 LEALLA : Learning lightweight language-agnostic sentence embeddings with knowledge distillation . In Proceedings of the 17th Conference of the European Chapter of the Association for Comput...

    2023 doi

  18. [26]

    Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014--2037

    2023

  19. [27]

    Rafa Po \'s wiata, S awomir Dadas, and Micha Pere kiewicz. 2024. Pl-mteb: Polish massive text embedding benchmark. arXiv preprint arXiv:2405.10138

    2024 arXiv

  20. [28]

    Robertson, Steve Walker, Susan Jones, Micheline Hancock - Beaulieu, and Mike Gatford

    Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock - Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3 . In Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994 , volume 500-225 of NIST Special Publicati...

    1994

  21. [29]

    Artem Snegirev, Maria Tikhonova, Anna Maksimova, Alena Fenogenova, and Alexander Abramov. 2024. The russian-focused embedders' exploration: rumteb benchmark and russian embedding model design. arXiv preprint arXiv:2408.12503

    2024 arXiv

  22. [30]

    Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael G \"u nther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. 2024. jina-embeddings-v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409.10173

    2024 arXiv

  23. [31]

    Nandan Thakur, Nils Reimers, Andreas R \"u ckl \'e , Abhishek Srivastava, and Iryna Gurevych. 2021. https://openreview.net/forum?id=wCu6T5xFjeJ BEIR : A heterogeneous benchmark for zero-shot evaluation of information retrieval models . In Thirty-fifth Conference on Neural Info...

    2021

  24. [32]

    o hler, Ren \'e J \

    Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim K \"o hler, Ren \'e J \"a kel, et al. 2024. Towards multilingual llm evaluation for european languages. arXiv preprint arXiv:2410.08928

    2024 arXiv

  25. [33]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

    2018

  26. [34]

    J \"o rg Tiedemann and Santhosh Thottingal. 2020. Opus-mt--building open translation services for the world. In Proceedings of the 22nd annual conference of the European Association for Machine Translation, pages 479--480

    2020

  27. [35]

    Bram Vanroy. 2023. Language resources for dutch large language modelling

    2023

  28. [36]

    Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information retrieval test collection. In ACM SIGIR Forum, volume 54, pages 1--12. ACM New York, NY, USA

    2021

  29. [37]

    Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 241--251

    2018

  30. [38]

    David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.609 Fact or fiction: Verifying scientific claims . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  31. [39]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533

    2022 arXiv

  32. [40]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672

    2024 arXiv

  33. [41]

    Silvan Wehrli, Bert Arnrich, and Christopher Irrgang. 2024. German text embedding clustering benchmark. arXiv preprint arXiv:2401.02709

    2024 arXiv

  34. [42]

    Konrad Wojtasik, Kacper Wo owiec, Vadim Shishkin, Arkadiusz Janz, and Maciej Piasecki. 2024. Beir-pl: Zero shot information retrieval benchmark for the polish language. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources a...

    2024

  35. [43]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. 2023. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597

    2023 arXiv

  36. [44]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language ...

    2018

  37. [45]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4800

    2019

  38. [46]

    Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. https://arxiv.org/abs/2407.19669 mgte: Generalized long-context text representation and reranking models for ...

    2024 arXiv

  39. [47]

    Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024. Dense text retrieval based on pretrained language models: A survey. ACM Transactions on Information Systems, 42(4):1--60

    2024

  40. [48]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.