{"id":"c1bbb720-db44-49bb-99c5-5eb3cddfbe2a","arxiv_id":"2504.12879","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RusBEIR introduces a 17-dataset Russian IR benchmark, finds mE5-large and BGE-M3 dominate on most tasks, while BM25 stays strong on long-document retrieval.","lead":"RusBEIR is a new Russian-language benchmark for zero-shot information retrieval, built from 17 datasets. It shows that neural retrievers usually beat BM25, except on long documents where BM25 remains competitive.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BGE-M3's long-document shortfall is tested only at 2048 tokens despite an 8192 limit; the paper's input-size-constraint claim is unsupported for BGE-M3.","rationale":"The reader's weakest assumption was translation quality for the four machine-translated datasets. That is a legitimate concern, but it affects a subset of the benchmark and is checkable post-hoc; even if those datasets were noisy, the overall finding that neural models outperform BM25 on most datasets would likely stand. The truncation issue is more directly load-bearing because it undercuts one of the paper's explicitly stated central findings: that BGE-M3, alongside mE5-large, faces challenges with long-document retrieval due to input size constraints. The paper itself provides the counter-evidence: Table 2 lists BGE-M3's max input as 8192, while Section 4.2 restricts it to 2048, and Section 5 acknowledges this limitation. Thus the reported BGE-M3 scores on long documents do not measure the model's capability; they measure the chosen evaluation configuration. This is an internal inconsistency, not merely an external critique. The reader's rationale did mention 'the self-imposed 2048-token truncation for BGE models' as a partial concern, so my agreement is partial rather than full. My recommendation is UNCHANGED because the reader's verdict of CONDITIONAL already captures the need for revision; however, the condition should explicitly require rerunning BGE-M3 at its full context length (or softening the abstract claim to refer specifically to mE5's 512-token limit). The training-data overlap with USER-BGE-M3 is a secondary concern about zero-shot cleanliness, but it does not affect the headline comparison involving mE5-large and BGE-M3 as strongly.","tokens_in":13052,"tokens_out":7517,"duration_ms":67674,"concrete_test":"Re-run BGE-M3 and USER-BGE-M3 on the three longest-document datasets (wikifacts-articles, rus-NFCorpus, rus-SciFact) with max_length=8192, keeping all other settings identical, and recompute NDCG@10. Also report the share of documents per dataset that exceed 2048 tokens and are therefore truncated under the current configuration. If BGE-M3 at 8192 reaches or exceeds BM25's score on wikifacts-articles (84.28), then the central claim should be revised to attribute BGE-M3's shortfall to the chosen truncation rather than to intrinsic input-size constraints.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim states that neural models such as mE5-large and BGE-M3 'face challenges with long-document retrieval due to input size constraints.' For BGE-M3, the reported experiments do not support this conclusion. Table 2 lists BGE-M3's maximum input as 8192 tokens, but Section 4.2 says 'we use BGE models with a max-length parameter set to 2048.' Consequently, on long-document datasets such as wikifacts-articles (average document length 2535.9 tokens, Table 1), BGE-M3 is forced to truncate documents at 2048 tokens, and its NDCG@10 of 74.50 lags behind BM25's 84.28 (Table 3). Section 5 concedes this: 'the results of the BGE models presented in Table 3 were obtained with a maximum input length set to 2048. However, as indicated in Table 2, BGE models can process up to 8192 tokens, making them more suitable for full-text search in long documents.' The paper nevertheless reiterates the input-size-constraint conclusion in the abstract and Section 6. This is an internally acknowledged missing experiment: the 2048-token cap is an evaluation choice, not an intrinsic model limit. While mE5-large genuinely has a 512-token limit (Table 2), pairing it with BGE-M3 in the claim overgeneralizes. The finding that BM25 is the best single model on long-document datasets may change if BGE-M3 is allowed its full context window.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RusBEIR, a BEIR-inspired benchmark for zero-shot information retrieval in Russian, consisting of 17 datasets. The datasets come from four sources: four BEIR datasets machine-translated into Russian, Russian portions of existing multilingual datasets (mMARCO, MIRACL, XQuAD, TyDi QA), existing Russian datasets (RuBQ, Ria-News, SberQuAD, ruSciBench, ru-facts), and three newly created Wikipedia-based datasets (wikifacts-sents, wikifacts-para, wikifacts-articles) built from the same 540 annotated facts at different document granularities. The authors evaluate BM25 with a custom lemmatization-based preprocessing pipeline, several dense retrievers (LaBSE, mE5 variants, BGE-M3, USER-BGE-M3, RoSBERTa), and reranking combinations, reporting NDCG@10, MAP@10, and Recall@10. The central claims are that BM25 remains a strong baseline for full-document retrieval, that neural models outperform BM25 on most datasets, and that neural models such as mE5-large and BGE-M3 face difficulties with long-document retrieval due to input size constraints.","tokens_in":13387,"tokens_out":5999,"duration_ms":62814,"significance":"If the benchmark is valid, RusBEIR would be a valuable community resource: it addresses a real gap in Russian IR evaluation, is BEIR-format compatible, is publicly released, includes a controlled long-document setting through the three wikifacts variants, and compares a reasonable range of lexical, dense, and reranking models. The emphasis on morphological preprocessing for BM25 is also a useful contribution. However, the significance is conditional on several validation steps that the manuscript does not report: translation quality for the four translated datasets, inter-annotator agreement for the new wikifacts annotations, statistical reliability of the model rankings, and disclosure of whether the Russian-trained USER-BGE-M3 model was trained on any RusBEIR test datasets. The paper's headline claim about long-document limitations is weakened by the fact that BGE models were run at a 2048-token cap despite having an 8192-token capacity.","major_comments":[{"comment":"The abstract and conclusion claim that neural models such as mE5-large and BGE-M3 'face challenges with long-document retrieval due to input size constraints,' but this claim is not supported for BGE-M3. Section 4.2 states that BGE models were run with max-length 2048, while Table 2 lists BGE-M3's maximum input as 8192 tokens. On wikifacts-articles, whose average document length is 2535.9 tokens, this evaluation choice truncates documents and depresses BGE-M3's NDCG@10 (74.50 vs. BM25's 84.28). Section 5 itself concedes this: the BGE results were obtained with maximum input length 2048, although the models can process up to 8192 tokens. The finding that BM25 is the best single model on long-document datasets may change if BGE-M3 is evaluated with its full context window. Additionally, the sentence in Section 5 that BM25 outperforms BGE-M3 by 13 percentage points and mE5-large by 27 percentage points does not match Table 3, where the differences are 9.78 and 18.19 percentage points, respectively. The paper should either run BGE-M3 at 8192 tokens or restrict the input-size-constraint claim to mE5-large.","section":"§4.2, §5, Table 3"},{"comment":"The four translated datasets (rus-NFCorpus, rus-ArguAna, rus-SciFact, rus-SCIDOCS) are translated with Google Translate without any reported quality check. Prior analogues take validation seriously: Dutch BEIR samples and checks translations with a native speaker, and Hindi BEIR uses back-translation with Chrf++. The paper cites prior work to justify choosing Google Translate, but that does not establish that the Russian translations preserve query-document relevance. If translations contain errors that change entities, negations, or argumentative structure, the NDCG@10 scores on these four datasets do not measure Russian retrieval quality. The authors should add a human or automatic translation-quality assessment, or at least report error rates per dataset.","section":"§3.1"},{"comment":"The zero-shot evaluation claim is at risk for USER-BGE-M3, described as 'trained on the Russian datasets.' The paper does not list these training datasets or check them for overlap with RusBEIR test sets. RusBEIR includes RuBQ and Ria-News from ruMTEB, as well as SberQuAD, ruSciBench, and ru-facts, all of which are plausible training sources for a Russian embedding model. If any of these overlap with USER-BGE-M3's training data, its high scores on those datasets reflect in-distribution performance rather than zero-shot generalization. The authors should disclose the full training-data list for USER-BGE-M3 and either exclude overlapping datasets from the zero-shot interpretation or report results separately.","section":"§4.2"},{"comment":"The new wikifacts datasets are based on student annotations with relevance scores 0, 1, and 2, but the paper reports no inter-annotator agreement, no number of annotators per fact, and no adjudication procedure. Because these datasets are a primary novel contribution and are used to support the long-document retrieval conclusions, the reliability of the relevance labels needs to be established. Report an agreement measure such as Cohen's or quadratic weighted kappa on a held-out annotation subset, or otherwise justify why the labels can be treated as ground truth.","section":"§3.4"},{"comment":"The paper reports single-point NDCG@10, MAP@10, and Recall@10 values without any error bars or significance tests. Several comparative claims rest on small differences: for example, mE5-large+BGE (65.71) vs. BGE-M3+BGE (65.85) in Table 3, and many near-saturated scores on rus-XQuAD and ru-facts. Without paired bootstrap or per-query variance estimates, the stated ranking of models on individual datasets is not statistically supported. Add significance testing or report per-query confidence intervals for at least the headline comparisons.","section":"§5, Tables 3-5"}],"minor_comments":[{"comment":"The text contains numerous inline artifacts such as '/one.sup', '/two.sup', and similar footnote markers, which should be rendered as proper footnotes or superscripts.","section":"§3.1, footnote rendering"},{"comment":"The reranker column header 'BGE+BGE' in the additional metrics tables is inconsistent with 'BGE-M3+BGE' used in Table 3; use identical model names throughout.","section":"Tables 4 and 5"},{"comment":"The introduction motivates an investigation of the efficiency trade-off between lexical and dense models, but no efficiency measurements are reported, and the conclusion defers this to future work. Either add a small runtime or memory comparison or remove this promise from the introduction.","section":"§1 and §6"},{"comment":"The GitHub and HuggingFace links appear only in footnotes; they should be prominently stated in the abstract or introduction so readers can immediately access the released resources.","section":"Abstract and Introduction"},{"comment":"There is a typo, 'signifcantly', in the discussion of Hindi-BEIR results. The manuscript would also benefit from a general proofreading pass for similar minor OCR-style errors.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"The resource is a useful and timely contribution for Russian IR evaluation, and the paper's overall structure is sound. However, the central zero-shot and long-document claims currently rest on unverified assumptions about training-data overlap and evaluation configuration. I would like the authors to resolve these points before the paper is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing to know: this is a genuinely useful resource. RusBEIR is the first BEIR-style benchmark for Russian, with 17 datasets in a unified format, and the new wikifacts sets (articles/paragraphs/sentences from the same 540 facts) are a nice way to study document-length effects. The authors also did careful work adapting existing Russian datasets and translating four BEIR sets. If you do Russian IR, you will probably use this benchmark. The evaluation itself is standard and the paper is clearly written.\n\nThe strongest conclusion—that BM25 is still a competitive baseline, especially on full-document retrieval—survives scrutiny. The preprocessing results for Russian morphology are also a practical takeaway.\n\nWhere the paper stumbles is the abstract's claim that neural models like mE5-large and BGE-M3 'face challenges with long-document retrieval due to input size constraints.' That is true for mE5-large, which caps at 512 tokens. It is not supported for BGE-M3, which supports 8192 tokens but was run with a self-imposed 2048-token maximum. The authors admit this in Section 5, yet leave the overgeneralized sentence in the abstract and conclusion. On wikifacts-articles, BGE-M3 at 2048 gets 74.5 NDCG@10 versus BM25's 84.28; running at 8192 might change that result. This is a fixable but real error, and it is the paper's headline finding.\n\nTwo softer weaknesses. First, the machine-translated datasets have no translation-quality check at all—the authors rely on prior work choosing Google Translate, but never sample or validate the Russian output. BEIR-NL did a 10-item human check; this paper should at least do something comparable or add a caveat. Second, the USER-BGE-M3 model is described as trained on Russian datasets, but the paper does not list its training data or rule out overlap with RuBQ and Ria-News, which are in the benchmark. That matters for the zero-shot claim, and a short statement would settle it.\n\nMinor: no statistical significance or error bars across runs, but that is common in benchmark papers and not a dealbreaker.\n\nWho should read this: Russian-language IR researchers and anyone building multilingual retrieval benchmarks. It deserves a serious referee, not a desk reject. The benchmark contribution is real and the flaws are correctable. I would ask for revision that fixes the abstract/conclusion on BGE-M3, adds a translation-quality note, and addresses the USER-BGE-M3 overlap question.","headline":"Useful first BEIR-style Russian retrieval benchmark with new datasets, but the abstract's long-document claim about BGE-M3 is undercut by the paper's own 2048-token cap and needs revision.","tokens_in":13882,"tokens_out":2195,"would_cite":true,"duration_ms":23805,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces RusBEIR, a zero-shot information-retrieval benchmark for Russian built from 17 datasets, and shows that dense neural models lead on most tasks while BM25 still wins on full-document retrieval.","keywords":["information retrieval","benchmark","Russian language","zero-shot evaluation","BM25","dense retrieval","BEIR","datasets"],"falsifier":"Ask native Russian speakers to re-judge a sample of translated query-document pairs from rus-NFCorpus, rus-ArguAna, rus-SciFact, and rus-SCIDOCS against the original English labels; if many originally relevant pairs are no longer relevant after translation, the reported scores on those datasets need to be recomputed with validated translations.","tokens_in":12903,"feed_emoji":"🔍","tokens_out":6374,"duration_ms":54315,"temperature":0.7,"pith_summary":"The paper aims to give Russian-language information retrieval a standard evaluation resource like the English BEIR benchmark, and to find out which retrieval models actually work in Russian. It assembles 17 datasets from translated English corpora, multilingual collections, existing Russian resources, and newly annotated Wikipedia facts. Across these datasets, multilingual dense encoders such as mE5-large and BGE-M3 outperform BM25 on average, but BM25 remains the strongest single model on four long-document datasets. The paper also demonstrates that lemmatization preprocessing substantially improves lexical models in Russian. If the benchmark is sound, it becomes the common yardstick for comparing retrieval models on Russian.","feed_headline":"Dense models lead Russian IR benchmark; BM25 wins on long docs","feed_subtitle":"First large-scale Russian retrieval benchmark shows dense models lead; BM25 still wins on long documents.","key_machinery":"The load-bearing object is the RusBEIR dataset collection itself, released in the format of the standard English zero-shot benchmark BEIR, plus the new wikifacts triplet: article, paragraph, and sentence versions of the same 540 Wikipedia-based facts. These variants let the same query be tested at three document lengths, isolating the input-size limitation of neural encoders. For the lexical baseline, the preprocessing pipeline of lowercasing, punctuation removal, tokenization, PyMorphy3 lemmatization, and stop-word removal is the mechanism that makes BM25 competitive in Russian.","core_discovery":"The central discovery is that a unified zero-shot benchmark for Russian is feasible from existing and newly built resources, and that its first large-scale run produces a clear ranking: mE5-large and BGE-M3 as dense encoders, combined with a BGE reranker, achieve the best NDCG@10 on most datasets, while BM25 remains the best single model on datasets dominated by long documents. The new wikifacts datasets, built from the same 540 facts at sentence, paragraph, and article granularity, isolate document-length effects without changing the queries. The paper argues that preprocessing, especially lemmatization, materially affects lexical-model performance in Russian, which has rich morphology.","pith_inferences":["A direct validation step—back-translating the four machine-translated datasets or having native speakers re-judge relevance—would test whether translation noise changes the reported rankings; the paper does not include such a check.","The same three-granularity annotation design could be applied to other languages, creating comparable long-document test sets for cross-lingual IR studies.","RusBEIR's data could be used for fine-tuning rather than only zero-shot evaluation, potentially boosting performance on Russian IR downstream tasks.","The explicit link between document length and neural failure suggests that adaptive truncation or long-context encoders might close the gap BM25 currently enjoys."],"forward_implications":["Russian-language IR research gains a common open benchmark spanning fact-checking, question answering, citation prediction, and news retrieval.","A standard recipe for Russian retrieval will be dense first-stage retrieval with a cross-encoder reranker, since the reranked BGE-M3 and mE5-large pipelines lead the benchmark.","For full-document retrieval in Russian, lemmatized BM25 is a strong and hard-to-beat baseline, especially against encoders whose input windows are too small.","The wikifacts triplet gives other researchers a controlled testbed for studying how document length affects retrieval performance.","The benchmark's results align with translated-BEIR findings for other languages, supporting the practice of building multilingual zero-shot resources by translating English datasets."],"supporting_citations":[{"why":"It defines the zero-shot evaluation paradigm, dataset format, and baseline comparisons that RusBEIR adapts to Russian.","marker":"[24]"},{"why":"It supplies the Russian portion of mMARCO and the evidence for choosing Google Translate over the Helsinki model.","marker":"[5]"},{"why":"It is the Polish translated-BEIR predecessor that validates the approach of translating English datasets and provides a comparison point.","marker":"[27]"},{"why":"It is the BGE-M3 model whose long input window is central to achieving top results on short and mid-length datasets.","marker":"[6]"},{"why":"It is the multilingual E5 model whose 512-token input limit is used to explain retrieval failures on long documents.","marker":"[25]"},{"why":"It contributes the RuBQ and Ria-News datasets and provides the ruMTEB evaluation context for Russian embeddings.","marker":"[23]"},{"why":"It contributes the MIRACL Russian subset with human-annotated relevance judgments.","marker":"[28]"},{"why":"It is the PyMorphy3 lemmatizer used in the BM25 preprocessing pipeline that the paper credits for lexical-model gains.","marker":"[17]"}],"fun_headline_variants":["Russian IR benchmark: dense models top NDCG, BM25 rules long docs","RusBEIR: dense encoders win overall, BM25 best for long text","New Russian benchmark: neural models lead, but BM25 holds on long docs","Dense models beat BM25 in Russian IR, except on long documents","First Russian IR benchmark: neural edge overall, BM25 still strong"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The four machine-translated datasets are assumed to preserve the original English relevance judgments; if translation changes which documents actually answer a query, roughly a quarter of the benchmark's scores would not measure true retrieval quality.","fun_headline_variants_meta":{"raw":{"variants":["Russian IR benchmark: dense models top NDCG, BM25 rules long docs","RusBEIR: dense encoders win overall, BM25 best for long text","New Russian benchmark: neural models lead, but BM25 holds on long docs","Dense models beat BM25 in Russian IR, except on long documents","First Russian IR benchmark: neural edge overall, BM25 still strong"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000718,"raw_usage":{"total_tokens":3146,"prompt_tokens":786,"completion_tokens":2360,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":402,"completion_tokens_details":{"reasoning_tokens":2259}},"tokens_in":402,"tokens_out":2360,"duration_ms":15740,"temperature":1.0,"reasoning_tokens":2259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:19:38.534472+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask native Russian speakers to re-judge a sample of translated query-document pairs from rus-NFCorpus, rus-ArguAna, rus-SciFact, and rus-SCIDOCS against the original English labels; if many originally relevant pairs are no longer relevant after translation, the reported scores on those datasets need to be recomputed with validated translations.","supporting_citations":[{"cited_title":"BEIR: A Heterogeneous Benchmark f or Zero-shot Evalu- ation of Information Retrieval Models","cited_arxiv_id":null,"evidence_quote":"It defines the zero-shot evaluation paradigm, dataset format, and baseline comparisons that RusBEIR adapts to Russian."},{"cited_title":"BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Lan- guage","cited_arxiv_id":null,"evidence_quote":"It is the Polish translated-BEIR predecessor that validates the approach of translating English datasets and provides a comparison point."},{"cited_title":"The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design","cited_arxiv_id":null,"evidence_quote":"It contributes the RuBQ and Ria-News datasets and provides the ruMTEB evaluation context for Russian embeddings."},{"cited_title":"Miracl: A multilingual retrieval da taset covering 18 diverse languages","cited_arxiv_id":null,"evidence_quote":"It contributes the MIRACL Russian subset with human-annotated relevance judgments."},{"cited_title":"Morphological analyzer and generat or for Russian and Ukrainian languages","cited_arxiv_id":null,"evidence_quote":"It is the PyMorphy3 lemmatizer used in the BM25 preprocessing pipeline that the paper credits for lexical-model gains."}],"review_version":1}