Pith. sign in

REVIEW 4 major objections 5 minor 34 references

A Comparative Study of Text Retrieval Models on DaReCzech

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Gemma2 beats all tested retrieval models for Czech-language accuracy.

desk verdict Useful first Czech retrieval benchmark, but the underspecified 100k pool and missing error bars keep the ranking from being authoritative. read the letter →

arxiv 2411.12921 v2 pith:NROAGGSP submitted 2024-11-19 cs.IR cs.AI

Jakub Stetina , Martin Fajcik , Michal Stefanik , Michal Hradis This is my paper · ORCID
classification cs.IRcs.AI
keywords informationretrievalCzechlanguageevaluationDaRedensesparsemultilingualembeddingsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to determine which off-the-shelf text retrieval models work best for Czech-language information retrieval, using the DaReCzech dataset as the test bed. It evaluates seven modern models plus a BM25 baseline on retrieval quality, speed, and index size, and asks whether translating Czech to English before retrieval helps. The central finding is that the multilingual Gemma2 embedding model achieves the highest precision and recall, with its Czech-oriented version slightly ahead of its English-oriented one, while the lexical BM25 and the dense Contriever trail the field. The paper concludes that accuracy-first Czech IR should use Gemma2, memory-constrained deployments should use SPLADE, and PLAID models offer a middle ground.

What carries the argument

The argument is carried by a controlled evaluation protocol rather than a single mathematical identity: a fixed 100,000-document index sampled from DaReCzech, a grid-searched BM25 baseline, OPUS-MT document translation for the English-condition models, truncation or non-overlapping segmentation according to each model's token limit, and a shared set of metrics (Precision@K, Recall@K, MRR, NDCG, plus index size, query latency, and top-100 overlap/Kendall correlation). This protocol makes the models directly comparable on the same Czech queries and documents, and the pairwise overlap analysis shows which models agree in their rankings.

What would settle it

Re-run the same seven models on the full DaReCzech test split, or on several independently drawn 100,000-document samples, using graded relevance instead of the '>0' binarization; if Gemma2's margin over SPLADE and PLAID narrows substantially or the ordering changes, the paper's headline ranking would not survive.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a comparative ranking: over a 100,000-document sample of DaReCzech with relevance binarized at scores above zero, Gemma2 consistently delivers the best precision and recall at k values from 20 to 100, with the Czech version slightly outperforming the English one. Contriever performs below BM25, which the paper attributes to its English-only training and single-vector representation. SPLADE and PLAID strike the efficiency-performance balance, with SPLADE producing the smallest index and PLAID offering higher precision at modest storage cost. The paper also reports that segmenting long documents does not help these models and that translating the corpus to English does not close the gap to the best Czech-native run.

Load-bearing premise

The 100,000-document sample drawn from DaReCzech is representative of the full corpus and contains the relevant documents for all test queries; if the sample misses relevant documents, recall is underestimated and model rankings could shift.

Editorial extensions

If this is right

  • Accuracy-oriented Czech search systems should adopt Gemma2 as the primary retriever when storage and compute allow, since it leads all tested models on precision, recall, MRR, and NDCG.
  • Memory-constrained Czech IR deployments should use SPLADE, which keeps the smallest per-document index among the neural models while staying competitive in precision.
  • PLAID and PLAID-X offer a practical middle ground, with the original PLAID slightly ahead in precision but requiring more storage than its multilingual variant.
  • Translating Czech queries and documents to English before retrieval is not a winning strategy: the English-conditioned Gemma2 run trails its Czech-conditioned counterpart, and other English-trained models do not overtake the Czech-native runs.
  • Segmenting long documents does not improve retrieval for the ColBERT-style models on DaReCzech, indicating that relevant content tends to sit near the document start.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's rankings are conditional on the 100,000-document sample and on the binarization of graded relevance at '>0'; a full-corpus evaluation or graded metrics could compress the gaps between Gemma2, SPLADE, and PLAID, though it is unlikely to overturn the top-bottom ordering without a different sample.
  • Because DaReCzech queries are real user queries and the corpus is web-sourced, the same ranking may transfer to other Czech web-search scenarios, but not necessarily to specialized domains such as legal or medical text, where lexicon and document structure differ.
  • A natural next experiment is to apply the same protocol to a larger Czech corpus (for example, the 100-million-pair click dataset discussed in the related work) to test whether SPLADE's compact index and competitive precision hold at the scale of tens of millions of documents.
  • The finding that Gemma2's Czech run slightly beats its English run suggests that for inflected languages, native-language retrieval may be preferable to translate-retrieve; testing this on other Slavic languages would show whether the advantage generalizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports an empirical comparison of seven off-the-shelf retrieval models (SPLADE, PLAID, PLAID-X, SimCSE, Contriever, OpenAI ADA, BGE-Multilingual-Gemma2) plus a BM25 baseline on a 100,000-document sample of the Czech DaReCzech dataset. The authors evaluate retrieval quality with precision, recall, MRR, and NDCG at several cutoffs, and supplement this with measurements of index size, query latency, and rank-list overlap. They also compare direct Czech-language retrieval against retrieval over an English translation of the corpus, using OPUS-MT for document translation. The headline finding is that Gemma2 achieves the best precision and recall, with the Czech variant slightly ahead of the English variant, while Contriever performs worst and SPLADE/PLAID offer a balance of quality and efficiency. The paper concludes with practical recommendations for Czech IR practitioners.

Significance. If the central empirical claim holds, the paper provides a useful, actionable comparison for practitioners building Czech-language retrieval systems, and it is, to my knowledge, the first such comparison of modern off-the-shelf rankers on DaReCzech. The study has clear strengths: all models are external checkpoints evaluated on an external dataset, BM25 hyperparameters are fitted on a development split rather than the test set, and the inclusion of index-size and latency measurements makes the comparison practically relevant. The rank-list overlap analysis in Figure 4 is also informative. However, the validity of the headline ranking rests on the representativeness of the self-constructed 100k-document sample and on the binarization of graded relevance labels, both of which are underspecified; the absence of any uncertainty quantification further limits how strongly the conclusions can be stated.

major comments (4)
  1. [Section 4.1] The construction of the 100,000-document sample is not specified precisely enough for the reported recall numbers to be interpretable. The text says only that "test queries along with their associated relevant documents and additional documents" were selected to form the sample; it does not state how many of the 64K test queries were used, how the "additional documents" were chosen, or whether this was a random, stratified, or convenience sample. Because recall is computed against the sample-based denominator in Eq. (2), a non-representative sample would change recall values and could alter the ordering among models. The central claim that Gemma2 is the most effective model for Czech IR is contingent on this sample being representative; the authors should either document the exact sampling procedure or restrict the claims to the sampled pool.
  2. [Section 4.1 / Appendix A.1] The binarization of DaReCzech's graded relevance scores at ">0" discards the graded annotation, and this is not a harmless simplification for NDCG. Equation (6) defines DCG with rel(k); if rel(k) is binary, NDCG can only distinguish relevant from non-relevant documents and cannot reward the placement of highly relevant documents above marginally relevant ones. Since the paper itself notes in Section 2 that DaReCzech's relevance annotations are not binary, the authors should report at least one metric computed on the original graded labels to demonstrate that the model ranking is not an artifact of binarization.
  3. [Section 4.3] The translation setup is ambiguous. Section 4.3 describes only "document-level translation" into English, but the comparison between the Czech and English versions of Gemma2 in Section 6 requires knowing whether queries were also translated. If queries remained in Czech while documents were translated, the English index is actually a cross-lingual retrieval setup, not a monolingual English one, and the conclusion that "the Czech version slightly outperformed the English one" would need to be qualified. The authors should state explicitly which components (queries, documents, or both) were translated for each model.
  4. [Section 5 / Results] No uncertainty quantification is provided for any reported metric, although the paper makes fine-grained comparative claims such as the Czech Gemma2 "slightly outperforming" the English one and SPLADE being "comparable" to PLAID-X. With a single evaluation on one 100k-document sample, these differences could lie within sampling noise. The authors should add bootstrap confidence intervals or paired significance tests over queries for the main P@k, MRR, and NDCG comparisons; without these, close model orderings should not be overinterpreted.
minor comments (5)
  1. [Abstract] The abstract contains the typo "Gemma22" instead of "Gemma2".
  2. [Section 3] The description of SPLADE says that the second version of SPLADE changes the pooling mechanism to max-pooling, but it does not explicitly state which version the evaluated checkpoint "splade-cocondenser-ensembledistil" corresponds to; this should be clarified.
  3. [Section 4.4 / Figures 1-2] The figure legends label one model as "Gemma2 (en) (trunc)" and another as "Gemma2 (trunc)" without indicating that the latter is the Czech version; this should be made explicit in the legends or captions.
  4. [Section 5 / Figure 4] The text refers to "GEMMA" in the discussion of Figure 4, while the model is called "Gemma2" elsewhere; the capitalization and naming should be made consistent.
  5. [Appendix A.1] The MRR equation (Eq. 3) is typeset incorrectly in the manuscript, with the fraction and summation symbols not rendered properly; it should be reformatted for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper is a self-contained empirical benchmark of external models on an external dataset.

full rationale

This paper performs an empirical evaluation of seven off-the-shelf retrieval models plus a BM25 baseline on the DaReCzech dataset. No theoretical derivation is attempted, so there is no chain of equations that could reduce to its own inputs. The models are external checkpoints (SPLADE, PLAID, PLAID-X, SimCSE, Contriever, OpenAI ADA, Gemma2) evaluated with standard metrics (Precision, Recall, MRR, MAP, nDCG) defined in Appendix A.1 from first principles. BM25 hyperparameters are tuned on the dev split (Section 4.2), which is standard practice and does not feed into any neural model's parameters. The only fitted component, BM25's [K1, B] = [2, 1], is a baseline calibrated on held-out development data, not a prediction derived from the tested models. The paper's conclusion that Gemma2 performs best and SPLADE/PLAID balance efficiency and performance is a direct reading of measured metrics, not a quantity defined as an input to the experiment. Self-citations are absent in a load-bearing role: the dataset citation [14] is external prior work by other authors, and no 'uniqueness theorem' or author-derived ansatz is invoked to force any conclusion. The unspecified composition of the 100,000-document sample (Section 4.1) is a validity threat to the benchmark's representativeness, but it is an experimental-design concern, not a circularity: the reported numbers are computed from the sample as described, and the rankings are not definitionally equal to any fitted parameter or cited premise. Accordingly, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper combines external models, an external dataset, and standard metrics; no new entities or fitted theory are introduced. The main choices that could bias results are the 100k sample, the >0 relevance threshold, and the OPUS-MT translation, all entered without validation.

free parameters (4)
  • BM25 K1 = 2.0
    Grid-searched on DaReCzech dev set (Section 4.2); affects only the BM25 baseline.
  • BM25 B = 1.0
    Grid-searched on DaReCzech dev set (Section 4.2); set to maximum, highlighting length normalization.
  • Relevance binarization threshold = 0
    Relevance scores above 0 are treated as relevant (Section 4.1); discards graded relevance and changes metric values.
  • Index sample size = 100,000 documents
    Chosen for computational and API cost (Section 4.1); sample construction is under-specified.
assumptions (4)
  • domain assumption OPUS-MT translation preserves relevance
    Section 4.3 uses document-level translation so English-only models can be evaluated; this assumes no meaning-altering translation errors.
  • domain assumption Important document content is at the beginning
    Section 4.4 justifies truncation without overlap by an analysis in Appendix C; if content is distributed, truncation biases results.
  • domain assumption Relevance labels can be binarized at >0 without changing model order
    Section 4.1 collapses graded relevance to binary for precision and recall; this assumes the threshold preserves ranking comparisons.
  • domain assumption The 100k document pool contains all relevant documents
    Section 4.1 indexes relevant documents for test queries plus additional documents; a missing relevant document would lower recall.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Comparative Study of Text Retrieval Models on DaReCzech." pith.science (2026). https://pith.science/paper/NROAGGSP

@misc{pith2026241112921,
  author       = {Pith},
  title        = {Pith review of: A Comparative Study of Text Retrieval Models on DaReCzech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NROAGGSP}},
  note         = {Machine review of arXiv:2411.12921}
}
read the original abstract

This article presents a comprehensive evaluation of 7 off-the-shelf document retrieval models: Splade, Plaid, Plaid-X, SimCSE, Contriever, OpenAI ADA and Gemma2 chosen to determine their performance on the Czech retrieval dataset DaReCzech. The primary objective of our experiments is to estimate the quality of modern retrieval approaches in the Czech language. Our analyses include retrieval quality, speed, and memory footprint. Secondly, we analyze whether it is better to use the model directly in Czech text, or to use machine translation into English, followed by retrieval in English. Our experiments identify the most effective option for Czech information retrieval. The findings revealed notable performance differences among the models, with Gemma22 achieving the highest precision and recall, while Contriever performing poorly. Conclusively, SPLADE and PLAID models offered a balance of efficiency and performance.

Figures

Figures reproduced from arXiv: 2411.12921 by the authors.

Figure 1
Figure 1. Comparison of Precision and Recall at different values of [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Comparison of MRR, NDCG at different values of [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Doc size, query latency in relation to P@5. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Pairwise overlap and correlation of overlapped items in top-100 responses [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: BM25 hyperparameters grid search. C ColBERT’s Token-level Focus To estimate which parts of the document are important, the study analyzed ColBERTv2, a model that uses a multi-vector approach, where each token in a document is represented by a separate vector. By examin…
Figure 6
Figure 6. Figure 6: Colbert interpretability [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 8 canonical work pages

  1. [1]

    Bai,Y.,Li,X.,Wang,G.,Zhang,C.,Shang,L.,Xu,J.,Wang,Z.,Wang,F.,Liu,Q.: Sparterm: Learning term-based sparse representation for fast text retrieval (2020), https://arxiv.org/abs/2010.00768

    arXiv 2020

  2. [2]

    Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Ti- wary, S., Wang, T.: Ms marco: A human generated machine reading comprehension dataset (2018), https://arxiv.org/abs/1611.09268

    arXiv 2018

  3. [3]

    In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J

    Bañón, M., Chen, P., Haddow, B., Heafield, K., Hoang, H., Esplà-Gomis, M., For- cada, M.L., Kamran, A., Kirefu, F., Koehn, P., Ortiz Rojas, S., Pla Sempere, L., Ramírez-Sánchez, G., Sarrías, E., Strelec, M., Thompson, B., Waites, W., Wig- gins, D., Zaragoza, J.: ParaCrawl: Web-scale acquisition of parallel corpora. In: Jurafsky, D., Chai, J., Schluter, N....

    work page 2020

  4. [4]

    Bednář, J., Náplava, J., Barančíková, P., Lisický, O.: Some like it small: Czech semantic embedding models for industry applications (2023), https://arxiv.org/ abs/2311.13921

    work page Pith review arXiv 2023

  5. [5]

    Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., Liu, Z.: Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self- knowledge distillation (2024), https://arxiv.org/abs/2402.03216

    arXiv 2024

  6. [6]

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding (2019), https://arxiv.org/abs/ 1810.04805

    arXiv 2019

  7. [7]

    Enevoldsen, K., Kardos, M., Muennighoff, N., Nielbo, K.L.: The scandinavian em- bedding benchmarks: Comprehensive assessment of multilingual and monolingual text embedding (2024), https://arxiv.org/abs/2406.02396

    arXiv 2024

  8. [8]

    Formal, T., Lassance, C., Piwowarski, B., Clinchant, S.: From distillation to hard negative sampling: Making sparse neural ir models more effective (2022), https: //arxiv.org/abs/2205.04733

    work page Pith review arXiv 2022

Show all 34 references
  1. [9]

    Formal, T., Piwowarski, B., Clinchant, S.: Splade: Sparse lexical and expansion model for first stage ranking (2021), https://arxiv.org/abs/2107.05720

    2021 arXiv

  2. [10]

    Gao, T., Yao, X., Chen, D.: Simcse: Simple contrastive learning of sentence em- beddings (2022), https://arxiv.org/abs/2104.08821

    2022 arXiv

  3. [11]

    He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning (2020), https://arxiv.org/abs/1911.05722

    2020 arXiv

  4. [12]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J.:Measuringmassivemultitasklanguageunderstanding(2021),https://arxiv.org/ abs/2009.03300

    2021 arXiv

  5. [13]

    CoRR abs/2004.12832 (2020), https: //arxiv.org/abs/2004.12832

    Khattab, O., Zaharia, M.: Colbert: Efficient and effective passage search via con- textualized late interaction over BERT. CoRR abs/2004.12832 (2020), https: //arxiv.org/abs/2004.12832

    2020 arXiv

  6. [14]

    org/abs/2112.01810

    Kocián, M., Náplava, J., Štancl, D., Kadlec, V.: Siamese bert-based model for web search relevance ranking evaluated on a new czech dataset (2021), https://arxiv. org/abs/2112.01810

    2021 arXiv

  7. [15]

    arXiv preprint arXiv:2007.03006 (2020)

    Kocmi, T., Popel, M., Bojar, O.: Announcing czeng 2.0 parallel corpus with over 2 gigawords. arXiv preprint arXiv:2007.03006 (2020)

    2020 arXiv

  8. [16]

    Koto, F., Li, H., Shatnawi, S., Doughman, J., Sadallah, A.B., Alraeesi, A., Al- mubarak, K., Alyafeai, Z., Sengupta, N., Shehata, S., Habash, N., Nakov, P., Baldwin, T.: Arabicmmlu: Assessing massive multitask language understanding in arabic (2024), https://arxiv.org/abs/2402.12840

    2024 arXiv

  9. [17]

    Technical University of Košice, D.o.E., Communication, M.: Retrieval-skquad dataset (2023), https://huggingface.co/datasets/TUKE-KEMT/retrieval-skquad, dataset for Slovak search retrieval evaluation, licensed under CC BY-NC-SA 4.0

    2023

  10. [18]

    Muennighoff, N., Tazi, N., Magne, L., Reimers, N.: Mteb: Massive text embedding benchmark (2023), https://arxiv.org/abs/2210.07316

    2023 arXiv

  11. [19]

    Möller, T., Risch, J., Pietsch, M.: Germanquad and germandpr: Improving non- english question answering and passage retrieval (2021), https://arxiv.org/abs/ 2104.12741

    2021 arXiv

  12. [20]

    In: Proceedings of the 44th European Conference on Information Retrieval (ECIR) (2022), https://arxiv.org/abs/2201.08471

    Nair, S., Yang, E., Lawrie, D., Duh, K., McNamee, P., Murray, K., Mayfield, J., Oard, D.W.: Transfer learning approaches for building cross-language dense re- trieval models. In: Proceedings of the 44th European Conference on Information Retrieval (ECIR) (2022), https://arxiv....

    2022 arXiv

  13. [21]

    arXiv preprint arXiv:2201.10005 (2022)

    Neelakantan, A., Xu, T., Puri, R., Radford, A., Han, J.M., Tworek, J., Yuan, Q., Tezak, N., Kim, J.W., Hallacy, C., et al.: Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005 (2022)

    2022 arXiv

  14. [22]

    https://platform.openai.com/docs/ models/embeddings (2023), accessed: 2024-10-29

    OpenAI: Openai ada model for retrieval. https://platform.openai.com/docs/ models/embeddings (2023), accessed: 2024-10-29

    2023

  15. [23]

    Poświata, R., Dadas, S., Perełkiewicz, M.: Pl-mteb: Polish massive text embedding benchmark (2024), https://arxiv.org/abs/2405.10138

    2024 arXiv

  16. [24]

    Foundations and Trends in Information Retrieval3, 333–389 (01 2009)

    Robertson, S., Zaragoza, H.: The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval3, 333–389 (01 2009). https://doi.org/10.1561/1500000019

    2009 doi

  17. [25]

    Santhanam, K., Khattab, O., Potts, C., Zaharia, M.: Plaid: An efficient engine for late interaction retrieval (2022), https://arxiv.org/abs/2205.09707

    2022 arXiv

  18. [26]

    Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., Zaharia, M.: Colbertv2: Effective and efficient retrieval via lightweight late interaction (2022), https:// arxiv.org/abs/2112.01488

    2022 arXiv

  19. [27]

    Team, G., Riviere, M., Pathak, S., Sessa, P.G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C.L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Gi...

    2024 arXiv

  20. [28]

    In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

    Vonásek, J., Straka, M., Krč, R., Lasonová, L., Egorova, E., Straková, J., Ná- plava, J.: Cwrczech: 100m query-document czech click dataset and its application to web relevance ranking. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development ...

    2024

  21. [29]

    In: Aletras, N., Androutsopoulos, I., Barrett, L., Goanta, C., Preotiuc- Pietro, D

    Wrzalik, M., Krechel, D.: GerDaLIR: A German dataset for legal information re- trieval. In: Aletras, N., Androutsopoulos, I., Barrett, L., Goanta, C., Preotiuc- Pietro, D. (eds.) Proceedings of the Natural Legal Language Processing Workshop

  22. [30]

    Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., Nie, J.Y.: C-pack: Packed resources for general chinese embeddings (2024), https://arxiv.org/abs/2309.07597

    2024 arXiv

  23. [31]

    In: Proceed- ings of the 46th European Conference on Information Retrieval (ECIR) (2024), https://arxiv.org/abs/2401.04810

    Yang, E., Lawrie, D., Mayfield, J., Oard, D.W., Miller, S.: Translate-distill: Learn- ing cross-language dense retrieval by translation and distillation. In: Proceed- ings of the 46th European Conference on Information Retrieval (ECIR) (2024), https://arxiv.org/abs/2401.04810

    2024 arXiv

  24. [32]

    Yüksel, A., Köksal, A., Şenel, L.K., Korhonen, A., Schütze, H.: Turkishmmlu: Measuring massive multitask language understanding in turkish (2024), https:// arxiv.org/abs/2407.12402

    2024 arXiv

  25. [33]

    Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., Lin, J.: Making a miracl: Multilingual information retrievalacrossacontinuumoflanguages(2022),https://arxiv.org/abs/2210.09984 A Evaluation Process A.1 Evaluation Metri...

    2022 arXiv

  26. [2021]

    pp. 123–128. Association for Computational Linguistics, Punta Cana, Do- minican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.nllp-1.13, https: //aclanthology.org/2021.nllp-1.13

    2021 doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.