{"id":"da989576-8358-4462-9bf0-82bcff50a5fb","arxiv_id":"2411.12921","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A benchmark on the Czech DaReCzech dataset finds Gemma2 most accurate, Contriever least accurate, and SPLADE/PLAID the best efficiency-quality trade-off.","lead":"This paper tests seven existing search and ranking models on Czech-language documents to see which one works best. It reports that a large multilingual model, Gemma2, is the most accurate, while SPLADE and PLAID give the best balance of speed, size, and accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unspecified composition of the 100k-document sample undermines the validity of the reported rankings.","rationale":"The reader's weakest assumption correctly identifies the 100k sample representativeness as the key vulnerability. The paper's description in Section 4.1 is underspecified: it says 'selecting test queries along with their associated relevant documents and additional documents,' but does not explain how the additional documents were sampled. This is load-bearing because every reported metric is computed on this pool. If the pool is not a random sample, the proportion of relevant to non-relevant documents and the difficulty of the non-relevant documents are unrepresentative of DaReCzech, which could change both absolute scores and model rankings. The binarization of relevance at >0 is a second but secondary issue. The concern does not overturn the paper's conclusion; it only means the conclusion is conditional on sample validity, which matches the reader's CONDITIONAL verdict. Therefore no verdict change is recommended.","tokens_in":10917,"tokens_out":3643,"duration_ms":41314,"concrete_test":"Reproduce the evaluation on a random sample of 100k documents from DaReCzech that includes all test-query relevant documents plus a uniformly random draw of other documents (or, where feasible, on the full corpus), and recompute P@k, R@k, MRR, and NDCG for all models. If the relative ordering of Gemma2, SPLADE, and PLAID changes, the current rankings are not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The evaluation is performed on a self-constructed 100,000-document sample of DaReCzech (Section 4.1), but the paper does not specify how the 'additional documents' were chosen. If these are not a random sample of the corpus, the pool's relevance density and difficulty differ from the real corpus, and all precision/recall/MRR/NDCG values are computed against this artificial corpus. In particular, recall denominators are defined by the sample, and the small number of non-relevant documents likely makes retrieval artificially easy; a model's ranking could change on the full corpus or on a properly random sample. The central claim that Gemma2 is best for Czech IR is therefore contingent on the unverified representativeness of the sample. Additionally, Section 4.1 states relevance scores are binarized at '>0', discarding graded relevance, which could mask ranking differences. These issues are not addressed in the paper's limitations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical comparison of seven off-the-shelf retrieval models (SPLADE, PLAID, PLAID-X, SimCSE, Contriever, OpenAI ADA, BGE-Multilingual-Gemma2) plus a BM25 baseline on a 100,000-document sample of the Czech DaReCzech dataset. The authors evaluate retrieval quality with precision, recall, MRR, and NDCG at several cutoffs, and supplement this with measurements of index size, query latency, and rank-list overlap. They also compare direct Czech-language retrieval against retrieval over an English translation of the corpus, using OPUS-MT for document translation. The headline finding is that Gemma2 achieves the best precision and recall, with the Czech variant slightly ahead of the English variant, while Contriever performs worst and SPLADE/PLAID offer a balance of quality and efficiency. The paper concludes with practical recommendations for Czech IR practitioners.","tokens_in":11099,"tokens_out":4807,"duration_ms":49778,"significance":"If the central empirical claim holds, the paper provides a useful, actionable comparison for practitioners building Czech-language retrieval systems, and it is, to my knowledge, the first such comparison of modern off-the-shelf rankers on DaReCzech. The study has clear strengths: all models are external checkpoints evaluated on an external dataset, BM25 hyperparameters are fitted on a development split rather than the test set, and the inclusion of index-size and latency measurements makes the comparison practically relevant. The rank-list overlap analysis in Figure 4 is also informative. However, the validity of the headline ranking rests on the representativeness of the self-constructed 100k-document sample and on the binarization of graded relevance labels, both of which are underspecified; the absence of any uncertainty quantification further limits how strongly the conclusions can be stated.","major_comments":[{"comment":"The construction of the 100,000-document sample is not specified precisely enough for the reported recall numbers to be interpretable. The text says only that \"test queries along with their associated relevant documents and additional documents\" were selected to form the sample; it does not state how many of the 64K test queries were used, how the \"additional documents\" were chosen, or whether this was a random, stratified, or convenience sample. Because recall is computed against the sample-based denominator in Eq. (2), a non-representative sample would change recall values and could alter the ordering among models. The central claim that Gemma2 is the most effective model for Czech IR is contingent on this sample being representative; the authors should either document the exact sampling procedure or restrict the claims to the sampled pool.","section":"Section 4.1"},{"comment":"The binarization of DaReCzech's graded relevance scores at \">0\" discards the graded annotation, and this is not a harmless simplification for NDCG. Equation (6) defines DCG with rel(k); if rel(k) is binary, NDCG can only distinguish relevant from non-relevant documents and cannot reward the placement of highly relevant documents above marginally relevant ones. Since the paper itself notes in Section 2 that DaReCzech's relevance annotations are not binary, the authors should report at least one metric computed on the original graded labels to demonstrate that the model ranking is not an artifact of binarization.","section":"Section 4.1 / Appendix A.1"},{"comment":"The translation setup is ambiguous. Section 4.3 describes only \"document-level translation\" into English, but the comparison between the Czech and English versions of Gemma2 in Section 6 requires knowing whether queries were also translated. If queries remained in Czech while documents were translated, the English index is actually a cross-lingual retrieval setup, not a monolingual English one, and the conclusion that \"the Czech version slightly outperformed the English one\" would need to be qualified. The authors should state explicitly which components (queries, documents, or both) were translated for each model.","section":"Section 4.3"},{"comment":"No uncertainty quantification is provided for any reported metric, although the paper makes fine-grained comparative claims such as the Czech Gemma2 \"slightly outperforming\" the English one and SPLADE being \"comparable\" to PLAID-X. With a single evaluation on one 100k-document sample, these differences could lie within sampling noise. The authors should add bootstrap confidence intervals or paired significance tests over queries for the main P@k, MRR, and NDCG comparisons; without these, close model orderings should not be overinterpreted.","section":"Section 5 / Results"}],"minor_comments":[{"comment":"The abstract contains the typo \"Gemma22\" instead of \"Gemma2\".","section":"Abstract"},{"comment":"The description of SPLADE says that the second version of SPLADE changes the pooling mechanism to max-pooling, but it does not explicitly state which version the evaluated checkpoint \"splade-cocondenser-ensembledistil\" corresponds to; this should be clarified.","section":"Section 3"},{"comment":"The figure legends label one model as \"Gemma2 (en) (trunc)\" and another as \"Gemma2 (trunc)\" without indicating that the latter is the Czech version; this should be made explicit in the legends or captions.","section":"Section 4.4 / Figures 1-2"},{"comment":"The text refers to \"GEMMA\" in the discussion of Figure 4, while the model is called \"Gemma2\" elsewhere; the capitalization and naming should be made consistent.","section":"Section 5 / Figure 4"},{"comment":"The MRR equation (Eq. 3) is typeset incorrectly in the manuscript, with the fraction and summation symbols not rendered properly; it should be reformatted for clarity.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"This is a purely empirical evaluation paper, and I see no circularity problem: all models are external checkpoints evaluated on an external dataset, and the BM25 hyperparameter tuning on the development split is standard. The main risk is that the underspecified 100k-document sample and the binarization of graded relevance could undermine the headline ranking if they are not corrected or qualified. These issues appear fixable within the scope of the manuscript, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a sensible, much-needed comparison of off-the-shelf retrieval models on Czech data. The headline result—Gemma2 (Czech) is best, Contriever is poor, SPLADE/PLAID are the middle ground—is believable and consistent with scaling trends. The paper's real value is practical: someone building Czech search can use this as a starting point without rerunning the whole zoo.\n\nWhat's new: an evaluation of seven models on DaReCzech, including multilingual variants, with attention to index size and latency, and a Czech-vs-English translation comparison. That's genuinely useful for the Czech IR community. The BM25 grid search on the dev set is standard and fine. The overlap/Kendall analysis is a nice extra.\n\nSoft spots: The biggest one is the construction of the 100,000-document sample. Section 4.1 says they selected test queries with relevant documents plus 'additional documents' but doesn't say how those additional documents were drawn. If they're random, fine; if they're easy negatives, the retrieval task becomes artificially easy and the rankings could shift on the full corpus. This is a real gap, and the paper doesn't flag it as a limitation. Related: relevance is binarized at >0, which throws away the graded annotations. The effect on ranking is likely modest, but it should be stated. There are no error bars or significance tests. Close pairs like Czech vs English Gemma2 are within noise. Also, the translation step is described briefly; no details on what was translated and whether the English index used identical segmentation and truncation. No code or data release makes checking these details harder.\n\nI'm not buying the stress-test claim that the unspecified sample completely undermines the rankings. The relative ordering is probably stable for the big gaps. But the close pairs and the absolute numbers should be treated as approximate. This is a solid paper with a few reporting gaps, not a flawed one.\n\nWho it's for: Czech IR practitioners who want a quick model selection guide. It deserves a serious referee; the gaps are fixable with more detail and maybe a small significance analysis. I'd recommend acceptance after minor revision.","headline":"Useful first Czech retrieval benchmark, but the underspecified 100k pool and missing error bars keep the ranking from being authoritative.","tokens_in":11569,"tokens_out":2439,"would_cite":false,"duration_ms":22253,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Gemma2 beats all tested retrieval models for Czech-language accuracy.","keywords":["information retrieval","Czech language","retrieval evaluation","DaReCzech","dense retrieval","sparse retrieval","multilingual embeddings","benchmark"],"falsifier":"Re-run the same seven models on the full DaReCzech test split, or on several independently drawn 100,000-document samples, using graded relevance instead of the '>0' binarization; if Gemma2's margin over SPLADE and PLAID narrows substantially or the ordering changes, the paper's headline ranking would not survive.","tokens_in":10760,"feed_emoji":"📊","tokens_out":7720,"duration_ms":64212,"temperature":0.7,"pith_summary":"The paper sets out to determine which off-the-shelf text retrieval models work best for Czech-language information retrieval, using the DaReCzech dataset as the test bed. It evaluates seven modern models plus a BM25 baseline on retrieval quality, speed, and index size, and asks whether translating Czech to English before retrieval helps. The central finding is that the multilingual Gemma2 embedding model achieves the highest precision and recall, with its Czech-oriented version slightly ahead of its English-oriented one, while the lexical BM25 and the dense Contriever trail the field. The paper concludes that accuracy-first Czech IR should use Gemma2, memory-constrained deployments should use SPLADE, and PLAID models offer a middle ground.","feed_headline":"Gemma2 tops Czech retrieval accuracy; SPLADE suits lean systems","feed_subtitle":"Seven off-the-shelf retrievers compared on DaReCzech: quality, speed, and index size trade-offs.","key_machinery":"The argument is carried by a controlled evaluation protocol rather than a single mathematical identity: a fixed 100,000-document index sampled from DaReCzech, a grid-searched BM25 baseline, OPUS-MT document translation for the English-condition models, truncation or non-overlapping segmentation according to each model's token limit, and a shared set of metrics (Precision@K, Recall@K, MRR, NDCG, plus index size, query latency, and top-100 overlap/Kendall correlation). This protocol makes the models directly comparable on the same Czech queries and documents, and the pairwise overlap analysis shows which models agree in their rankings.","core_discovery":"On the paper's own terms, the discovery is a comparative ranking: over a 100,000-document sample of DaReCzech with relevance binarized at scores above zero, Gemma2 consistently delivers the best precision and recall at k values from 20 to 100, with the Czech version slightly outperforming the English one. Contriever performs below BM25, which the paper attributes to its English-only training and single-vector representation. SPLADE and PLAID strike the efficiency-performance balance, with SPLADE producing the smallest index and PLAID offering higher precision at modest storage cost. The paper also reports that segmenting long documents does not help these models and that translating the corpus to English does not close the gap to the best Czech-native run.","pith_inferences":["The paper's rankings are conditional on the 100,000-document sample and on the binarization of graded relevance at '>0'; a full-corpus evaluation or graded metrics could compress the gaps between Gemma2, SPLADE, and PLAID, though it is unlikely to overturn the top-bottom ordering without a different sample.","Because DaReCzech queries are real user queries and the corpus is web-sourced, the same ranking may transfer to other Czech web-search scenarios, but not necessarily to specialized domains such as legal or medical text, where lexicon and document structure differ.","A natural next experiment is to apply the same protocol to a larger Czech corpus (for example, the 100-million-pair click dataset discussed in the related work) to test whether SPLADE's compact index and competitive precision hold at the scale of tens of millions of documents.","The finding that Gemma2's Czech run slightly beats its English run suggests that for inflected languages, native-language retrieval may be preferable to translate-retrieve; testing this on other Slavic languages would show whether the advantage generalizes."],"forward_implications":["Accuracy-oriented Czech search systems should adopt Gemma2 as the primary retriever when storage and compute allow, since it leads all tested models on precision, recall, MRR, and NDCG.","Memory-constrained Czech IR deployments should use SPLADE, which keeps the smallest per-document index among the neural models while staying competitive in precision.","PLAID and PLAID-X offer a practical middle ground, with the original PLAID slightly ahead in precision but requiring more storage than its multilingual variant.","Translating Czech queries and documents to English before retrieval is not a winning strategy: the English-conditioned Gemma2 run trails its Czech-conditioned counterpart, and other English-trained models do not overtake the Czech-native runs.","Segmenting long documents does not improve retrieval for the ColBERT-style models on DaReCzech, indicating that relevant content tends to sit near the document start."],"supporting_citations":[{"why":"Supplies the DaReCzech dataset and its query-document pairs, the evaluation ground truth.","marker":"[14]"},{"why":"Defines the BM25 lexical baseline whose hyperparameters are grid-searched on the development set.","marker":"[24]"},{"why":"Describes SPLADE, the sparse lexical-expansion model whose index size and precision are compared.","marker":"[9]"},{"why":"Describes PLAID, the late-interaction engine whose retrieval quality and latency are measured.","marker":"[25]"},{"why":"Describe PLAID-X, the multilingual ColBERT variant trained via translate-train, tested here on Czech.","marker":"[20, 31]"},{"why":"Defines Contriever, the self-supervised dense retriever whose poor performance anchors the low end of the ranking.","marker":"[10]"},{"why":"Introduces the multilingual Gemma2 embedding model that wins precision and recall in this evaluation.","marker":"[5]"},{"why":"Provides the scaling observation linking embedding size to retrieval quality, used to explain Gemma2's storage cost.","marker":"[21]"}],"fun_headline_variants":["Gemma2 tops Czech retrieval; Contriever below BM25 on DaReCzech","Czech-native Gemma2 beats translated English for retrieval","SPLADE smallest index, PLAID balances speed and precision","English translation fails to beat native Czech retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 100,000-document sample drawn from DaReCzech is representative of the full corpus and contains the relevant documents for all test queries; if the sample misses relevant documents, recall is underestimated and model rankings could shift.","fun_headline_variants_meta":{"raw":{"variants":["Gemma2 tops Czech retrieval; Contriever below BM25 on DaReCzech","Czech-native Gemma2 beats translated English for retrieval","SPLADE smallest index, PLAID balances speed and precision","English translation fails to beat native Czech retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0017,"raw_usage":{"total_tokens":6681,"prompt_tokens":845,"completion_tokens":5836,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":5775}},"tokens_in":461,"tokens_out":5836,"duration_ms":43419,"temperature":1.0,"reasoning_tokens":5775,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:02:37.745991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same seven models on the full DaReCzech test split, or on several independently drawn 100,000-document samples, using graded relevance instead of the '>0' binarization; if Gemma2's margin over SPLADE and PLAID narrows substantially or the ordering changes, the paper's headline ranking would not survive.","supporting_citations":[{"cited_title":"Siamese BERT-based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset","cited_arxiv_id":"2112.01810","evidence_quote":"Supplies the DaReCzech dataset and its query-document pairs, the evaluation ground truth."}],"review_version":1}