{"id":"2331487d-432f-460f-9443-fdd68bc5f0ed","arxiv_id":"2505.19494","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Anveshana is a new English-to-Sanskrit cross-lingual retrieval benchmark on Srimadbhagavatam chapters, with document translation outperforming direct and query-translation approaches.","lead":"The paper introduces Anveshana, a benchmark dataset of 3,400 English queries paired with Sanskrit chapter documents from the Srimadbhagavatam, and compares three retrieval pipelines: direct, document-translation based, and query-translation based. It reports that translating Sanskrit documents to English and retrieving with BM25 gives the best results, and it makes the dataset publicly available.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold relevance in Anveshana is not independently validated: queries were authored from English translations of the paired chapters (§3.1), and §3.4 itself shows a target document only 'indirectly related' to its query.","rationale":"The reader's weakest assumption correctly identifies the single-gold-document labeling convention as the fragile part of the argument, and Section 3.4's 'indirectly related' example is a real red flag. My stress-test adds a second mechanism: because queries were written from English translations of the target chapters, the DT pipeline, which translates documents back into English before BM25 matching, can exploit English-to-English lexical overlap that DR and QT cannot. This makes the reported DT advantage potentially artifactual. However, this is not yet proven; no annotation guidelines, translation-source details, or overlap statistics are provided. The proposed re-judgment experiment would settle whether the labels are valid and whether the ranking survives. The reader already conditioned acceptance on label validity, so no verdict change is needed. The dataset may still be a useful resource, but the abstract's claim that DT methods outperform DR and QT should be treated as provisional pending label validation.","tokens_in":16440,"tokens_out":8525,"duration_ms":77467,"concrete_test":"Select 50 test queries at random. Have two Sanskrit-literate annotators, blind to the original §3.1 translation and to the official pairing, judge relevance of each query against all 334 Sanskrit chapters. Measure inter-annotator agreement and count queries with zero or more than one relevant chapter. Then have a third adjudicator resolve disagreements and recompute NDCG@10 for DT-BM25, the best DR model, and the best QT model. If the DT-BM25 margin over the best DR/QT narrows substantially (e.g., by more than 10 NDCG points) or if the original single-label judgments show low agreement, the headline ranking is an artifact of source-chapter labeling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DT outperforms DR and QT rests entirely on the 340 test queries' gold labels, but those labels were not produced by independent relevance assessment. Section 3.1 says queries were manually crafted by examining English translations of each document; the paired Sanskrit chapter was then treated as the relevant document. Section 3.4's own example is a query whose paired chapter is admitted to be only 'indirectly related' to the question. This matters for two reasons. First, if a positive label can be only loosely related, then NDCG, MAP, Recall, and Precision measure whether the system finds the source chapter, not whether it finds relevant content. Second, because the queries were written from English translations and the DT framework translates Sanskrit documents back into English for BM25 (§4.2), DT BM25 has access to the same English wording that generated the query, while DR and QT do not. A high DT score (62.46% NDCG@10) may therefore reflect lexical overlap between query and translated document rather than cross-lingual retrieval skill. The reported comparison is thus not interpretable unless the single-source relevance labels are validated independently.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Anveshana, a new English-to-Sanskrit cross-lingual information retrieval (CLIR) benchmark built from 3,400 manually authored English query–Sanskrit chapter pairs over 334 Srimadbhagavatam chapters. The authors compare three retrieval frameworks—direct retrieval (DR), document translation (DT), and query translation (QT)—using BM25, dense retrievers (Contriever, ColBERT, XLM-RoBERTa, multilingual-E5, mDPR), GPT-2, and REPLUG LSR, with both fine-tuned and zero-shot configurations. The central claim, stated in the abstract and repeated in Sections 6–8, is that DT methods outperform DR and QT, with DT-BM25 achieving the best results (e.g., NDCG@10 = 62.46% in Table 6 versus 10.74% for the best DR model and 6.86% for the best QT model). The paper also reports a zero-shot monolingual-Contriever English model as strongest among zero-shot systems.","tokens_in":16669,"tokens_out":3704,"duration_ms":33598,"significance":"If the benchmark is valid and the evaluation protocol is sound, Anveshana would be a valuable first public resource for English-to-Sanskrit CLIR, filling an acknowledged gap for a low-resource classical language. The paper deserves credit for releasing the dataset publicly, for spanning three clearly separated retrieval paradigms, and for including multiple modern retrieval models with fine-tuning. The empirical ranking is internally consistent across Tables 3–6 in the sense that DT-BM25 is always best, which is a useful sanity check. However, the significance of the central comparative claim is currently conditional on two unresolved issues: the gold relevance labels are single-source and were authored from English translations of the same chapters that DT translates back into English, and the zero-shot evaluation protocol is ambiguous about what document language is being retrieved. The dataset itself is a contribution regardless, but the paper's headline finding—that DT is superior for this task—cannot be interpreted as evidence about cross-lingual retrieval ability until these issues are addressed.","major_comments":[{"comment":"The gold relevance labels are not independently validated. Section 3.1 states that queries were manually crafted by examining English translations of each document and then pairing each query with that single document as relevant. Section 3.4 then presents a sample query whose paired Sanskrit document is admitted to be only 'indirectly related' to the query. Because NDCG, MAP, Recall, and Precision are computed against these binary labels, a high score such as DT-BM25's 62.46% NDCG@10 (Table 6) may indicate that the system recovers the source chapter rather than that it finds content relevant to the query's information need. Please provide independent relevance judgments for at least the 340 test queries (or a representative sample), report inter-annotator agreement, and re-evaluate the main comparisons using those judgments.","section":"§3.1 and §3.4"},{"comment":"The DT pipeline translates Sanskrit documents into English via Google Translate, while the queries themselves were written from English translations of those same chapters. This creates a lexical-overlap advantage for DT-BM25 that is not shared by DR or QT: the query and the translated document share vocabulary from a common English rendering of the source. The reported 62.46% NDCG@10 may therefore reflect translation-memory effects rather than cross-lingual retrieval skill. The manuscript should include a control that removes this confound, for example by evaluating DT-BM25 with queries that are paraphrases independent of the Vedabase English translation, or by measuring and reporting the query–translated-document lexical overlap and comparing against a simple 'translate document then BM25' baseline on a held-out set of queries not derived from those translations.","section":"§4.2, §3.1, Tables 3–6"},{"comment":"The zero-shot evaluation is ambiguous with respect to document language. Section 4.3 says Contriever was used for 'zero-shot retrieval of English documents,' and Section 6 reports the best zero-shot model as 'monolingual-Contriever eng.' But the task is English-query-to-Sanskrit-document retrieval. If these zero-shot models are evaluated on English translations of the Sanskrit documents, then the 'Zero-shot' rows in Tables 3–6 belong to the DT family and cannot be compared as a separate framework in support of the abstract's DT-versus-DR conclusion. Please state explicitly for every row in Tables 3–6 what document representation was indexed (original Sanskrit, English translation, or both), and, if English translations are used, re-frame those results accordingly.","section":"§4.3 and Tables 3–6 (Zero-shot rows)"},{"comment":"The text in the QT paragraph says BM25 'demonstrated a uniform performance across all metrics at approximately 2.95%,' but the tables report NDCG@3 = 5.18%, NDCG@5 = 5.94%, and NDCG@10 = 6.86% (Tables 4–6). This is a direct numerical contradiction between the results narrative and the reported data. Please correct the text to match the tables and ensure all QT numbers are reported consistently across the section.","section":"§6, QT results paragraph"},{"comment":"No variance estimates, standard errors, or significance tests are reported for any of the retrieval results. Several comparisons that are discussed as meaningful differences (e.g., ColBERT-fine-tunedDOT NDCG@10 = 40.78% versus contriever-fine-tunedDOT = 41.70%; intfloat/multilingual-e5-base DR NDCG@10 = 10.74% versus mDPR-BM35-HN1 = 4.74%) could easily be within query-level noise. Please report per-query score distributions, bootstrap confidence intervals, or a paired significance test (e.g., paired bootstrap or Wilcoxon) for the key comparisons that support the central claim.","section":"Tables 3–6"}],"minor_comments":[{"comment":"The description of the regex substitution for poetic markers (e.g., '——1.1.3——') is vague; please provide the actual preprocessing rule and give an example of a document after preprocessing.","section":"§3.2"},{"comment":"The sentence describing the scaling is garbled: 'scaled by 1√length of embeddings' should be written as 'scaled by the inverse square root of the embedding length,' and the surrounding math notation should be cleaned up.","section":"§4.1"},{"comment":"The naming of the Contriever variants is confusing: the text refers to 'mjwong/contriever-mnli fine-tuned (CONCAT and DOT)' and later to 'contriever - fine-tunedDOT' and 'contriever - fine-tuned' without defining the exact model checkpoints and pooling used; please standardize the model names and specify the configuration for each table row.","section":"§4.2 and Tables 3–6"},{"comment":"The model 'mDPR-BM35-HN1' appears in Table 3 without being defined in the text; please explain what BM35 and HN1 refer to (e.g., BM25 hard negatives, number of negatives) and how this model differs from the mDPR model described in §4.3.","section":"§4.3 and Table 3"},{"comment":"The caption states 'For k = 1, NDCG, MAP, Precision, Recall will yield same value,' which holds only because each query has exactly one gold document; please state this assumption explicitly in the caption.","section":"Table 3 caption"},{"comment":"Table 2 is described as showing a sample query and its corresponding Sanskrit document, but the excerpt in the text does not display the actual Sanskrit text; please include the full paired document snippet so readers can evaluate the relevance relationship for themselves.","section":"§3.4 and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The dataset is a useful contribution and the paper is within scope for a CLIR or NLP-resources venue. The main risk is not the construction of the data but the interpretability of the headline comparison: unless the gold labels are independently validated and the DT lexical-overlap confound is addressed, the abstract's 'DT outperforms DR and QT' claim is not established. The ambiguity in the zero-shot setup compounds this. I would be comfortable with a revised version that fixes the evaluation protocol and the numerical inconsistencies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about Anveshana: it is a genuinely new resource, 3,400 English-to-Sanskrit query-document pairs over 334 Srimadbhagavatam chapters, the first benchmark for this language pair, and it is on HuggingFace. That alone makes it worth a look for anyone doing low-resource CLIR or Sanskrit NLP. The paper also runs a reasonable three-way comparison (direct retrieval, document translation, query translation) across BM25 and several dense models, and the headline finding, DT BM25 at 62.46% NDCG@10 crushing the DR and QT approaches, is internally consistent across tables.\n\nBut the central comparison is confounded in a way the paper never addresses. Queries were manually written from the English translations of each paired chapter. Then in the DT framework, Sanskrit documents are translated back into English before retrieval. So DT BM25 is matching query text against a translated version of the very source the query was written from. That is essentially lexical overlap, not evidence of cross-lingual retrieval skill. The stress-test note is right: Section 3.4 even shows a query whose paired chapter is only 'indirectly related.' Unless the relevance labels are independently validated by someone who did not see the English translation, NDCG/MAP are measuring 'find the source chapter,' not 'find the relevant content.' This does not kill the dataset, but it does kill the cross-framework comparison as evidence for DT superiority.\n\nOther soft spots are smaller. No error bars or significance testing on 340 test queries; the Results prose says QT BM25 is 'approximately 2.95%' while Table 6 reports 6.86% NDCG@10 (the 2.95 is the k=1 value, so sloppy rather than wrong); the zero-shot setup never says whether Contriever saw English translations of Sanskrit documents or the Sanskrit originals; and no code or checkpoints are released, only the dataset.\n\nWhat the paper does well: it identifies a real gap, builds a reusable resource, and is honest about translation-model limitations. The authors deserve credit for releasing the data.\n\nWho is this for? Researchers working on low-resource CLIR, Sanskrit processing, or benchmark design. It deserves a serious referee, but the revision needs to address the label-validation problem head-on, either by adding independent relevance judgments or by reframing the task as 'source-chapter identification' rather than open-ended relevance. As is, I would recommend conditional acceptance with major revisions, not desk rejection.","headline":"A valuable new English-Sanskrit CLIR dataset whose headline DT-over-DR/QT result is confounded by labels built from the same English translations that DT uses.","tokens_in":17206,"tokens_out":2673,"would_cite":true,"duration_ms":22737,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For finding Sanskrit texts with English queries, translating the documents to English beats direct cross-lingual retrieval, the paper reports.","keywords":["cross-lingual information retrieval","Sanskrit","benchmark dataset","document translation","query translation","dense retrieval","BM25","Srimadbhagavatam"],"falsifier":"Have two independent Sanskrit scholars annotate, for each of the 340 test queries, all Srimadbhagavatam chapters that actually answer the query, then recompute NDCG@10 for DT-BM25 and the best direct-retrieval model; if DT-BM25 no longer clearly beats direct retrieval under this multi-relevance gold standard, the central claim is an artifact of single-relevance annotation.","tokens_in":16240,"feed_emoji":"📜","tokens_out":6168,"duration_ms":46849,"temperature":0.7,"pith_summary":"This paper introduces Anveshana, a benchmark of 3,400 English-language queries paired with Sanskrit chapter documents from the Srimadbhagavatam, and uses it to ask how a retrieval system should bridge English queries and Sanskrit texts. The authors compare three routes: translating the query into Sanskrit (query translation), translating the documents into English (document translation), and retrieving directly in a shared embedding space. Their central finding is that document translation is the strongest route: after machine-translating Sanskrit chapters to English, a standard BM25 ranker reaches 62.46% NDCG@10, while the best direct-retrieval model reaches 10.74% and the best query-translation model 6.86%. If this holds, it suggests that for low-resource ancient languages, the pragmatic path to cross-lingual retrieval is to invest in document translation rather than in cross-lingual dense embeddings.","feed_headline":"English queries find Sanskrit texts best after document translation","feed_subtitle":"New 3,400-pair benchmark shows BM25 on translated documents reaches 62% NDCG@10, far ahead of direct retrieval.","key_machinery":"The load-bearing object is the Anveshana dataset: 3,400 manually crafted English queries, roughly ten per document, over 334 Srimadbhagavatam chapters scraped from the Vedabase, with each query paired to a single gold chapter. Around this dataset, the paper builds a three-armed evaluation framework — Document Translation (DT), Query Translation (QT), and Direct Retrieve (DR) — with the Google Translate API as the translation engine for DT and QT. The argument is carried by comparing BM25, ColBERT, Contriever, XLM-R, multilingual-e5, mDPR, GPT-2, and REPLUG-LSR across these arms using NDCG, MAP, Recall, and Precision at k=1,3,5,10.","core_discovery":"The paper claims that in the English-query-to-Sanskrit-document setting, the dominant retrieval strategy is document translation followed by English monolingual retrieval, and specifically that BM25 on Google-translated Sanskrit chapters is far more effective than any of the evaluated dense cross-lingual or query-translation approaches. Across all test metrics and cutoffs, the DT framework outperforms DR and QT; at k=10, DT-BM25 reaches 62.46% NDCG while the best DR model (multilingual-e5-base) achieves 10.74% and the best QT model (BM25 on Sanskrit-translated queries) achieves 6.86%. The authors attribute this to the maturity of English retrieval models and the fact that translated documents preserve enough content for lexical matching, whereas direct embedding spaces and Sanskrit-side retrieval are poorly served by current multilingual models.","pith_inferences":["If the single-relevance annotation is relaxed to allow multiple relevant chapters, the reported absolute NDCG numbers would drop, but the relative ordering (DT ≫ DR ≈ QT) may be robust because lexical overlap in translated English is a stronger signal than cross-lingual semantic similarity for this corpus.","A testable extension is to run the same three-armed comparison with human-translated documents instead of Google Translate; if the DT advantage shrinks, part of the result is an artifact of translation quality rather than a property of the retrieval task.","The result may generalize to other low-resource classical languages where parallel translations exist: the paper's recipe is essentially 'use the existing translation to move the retrieval problem into a high-resource language,' which is cheaper than building new embedding spaces.","The paper notes translation-error cascading as a limitation but does not quantify how retrieval performance changes as translation quality varies, so a sensitivity analysis on translation quality would be a natural next step."],"forward_implications":["If DT+BM25 is the best recipe, then building Sanskrit CLIR systems today should prioritize high-quality document translation over training cross-lingual embedders.","The Anveshana dataset provides a standardized testbed for future English-Sanskrit retrieval research, with train/validation/test splits and negative sampling.","Fine-tuned dense models like ColBERT and Contriever on translated documents substantially beat zero-shot alternatives (e.g., 41.70% vs 30.48% NDCG@10), so task-specific fine-tuning still adds value in the DT setting.","Query translation into Sanskrit is currently a weak route; improving Sanskrit-side translation or monolingual Sanskrit retrieval would be a direct lever on QT performance.","The gap between DT and DR suggests that current multilingual embedding models do not align Sanskrit well with English for retrieval purposes."],"supporting_citations":[{"why":"Supplies the BM25 probabilistic retrieval function that achieves the top NDCG in the document-translation framework.","marker":"Robertson et al., 2009"},{"why":"Provides the Contriever model that is fine-tuned for the DT framework and used as a zero-shot English baseline.","marker":"Izacard et al., 2021"},{"why":"Provides the ColBERT late-interaction model fine-tuned on translated documents and used as a zero-shot baseline.","marker":"Khattab and Zaharia, 2020"},{"why":"Provides XLM-RoBERTa, the multilingual encoder fine-tuned for the QT and DR frameworks and used zero-shot.","marker":"Conneau et al., 2019"},{"why":"Provides the CLIRMatrix resource and the baseline neural CLIR model (mDPR) used for direct retrieval.","marker":"Sun and Duh, 2020"},{"why":"Provides the multilingual-e5-base model, which is the best-performing direct-retrieval model in the comparison.","marker":"Wang et al., 2024"},{"why":"Provides the REPLUG retrieval-augmented framework (REPLUG LSR) evaluated within document translation.","marker":"Shi et al., 2023"},{"why":"Earlier medical-domain CLIR study that found query translation superior, against which this paper's opposite DT result is implicitly compared.","marker":"Saleh and Pecina, 2020"},{"why":"Inspires the manual query-crafting procedure used to build the benchmark pairs from English translations.","marker":"Chen et al., 2017"}],"fun_headline_variants":["Translated Sanskrit texts beat neural retrieval in English-query test","English queries find Sanskrit best via document translation, not dense models","Anveshana benchmark: translate documents, then retrieve for Sanskrit IR","New dataset shows BM25 on translated Sanskrit wins over dense retrieval","For English to Sanskrit, translating documents beats query translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Each English query is assumed to be relevant to exactly one Sanskrit chapter, and the manual pairing made from English translations is assumed to carry over to the Sanskrit originals; if the correspondence is loose, every reported recall, precision, NDCG, and MAP number loses its meaning.","fun_headline_variants_meta":{"raw":{"variants":["Translated Sanskrit texts beat neural retrieval in English-query test","English queries find Sanskrit best via document translation, not dense models","Anveshana benchmark: translate documents, then retrieve for Sanskrit IR","New dataset shows BM25 on translated Sanskrit wins over dense retrieval","For English to Sanskrit, translating documents beats query translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000757,"raw_usage":{"total_tokens":3346,"prompt_tokens":912,"completion_tokens":2434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":2349}},"tokens_in":528,"tokens_out":2434,"duration_ms":17921,"temperature":1.0,"reasoning_tokens":2349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:13:06.386421+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent Sanskrit scholars annotate, for each of the 340 test queries, all Srimadbhagavatam chapters that actually answer the query, then recompute NDCG@10 for DT-BM25 and the best direct-retrieval model; if DT-BM25 no longer clearly beats direct retrieval under this multi-relevance gold standard, the central claim is an artifact of single-relevance annotation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ColBERT late-interaction model fine-tuned on translated documents and used as a zero-shot baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier medical-domain CLIR study that found query translation superior, against which this paper's opposite DT result is implicitly compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the manual query-crafting procedure used to build the benchmark pairs from English translations."}],"review_version":1}