{"id":"55fe88b0-dd43-4b65-b3a2-3ef166983a7e","arxiv_id":"2509.09459","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A three-stage data-utilization pipeline for multilingual dense retrieval, combining ensemble hard-negative mining, LLM-based filtering/generation, and monolingual topic-diverse mini-batches, improves MIRACL nDCG@10 by about 2.5 points over comparable hard-negative baselines.","lead":"This paper introduces a three-stage recipe for multilingual search: it mines hard negative examples with an ensemble of retrievers, uses a large language model to filter out false negatives and generate new ones, and builds training batches that keep languages consistent while spreading topics. The authors report modest but consistent ranking gains over strong baselines on a 16-language benchmark, suggesting the approach can improve dense retrievers without changing their arc","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Inconsistent LLM attribution: main text says Llama-3.1-70B for false-negative selection, but Appendix B.2 says GPT-4o; the reported gains may depend on which model actually ran Stage 1.","rationale":"The reader's weakest assumption was that Llama-3.1-70B accurately judges false negatives across all 16 languages, with human validation on only 5. My inspection found a more specific, potentially more damaging inconsistency: the paper contradicts itself about which LLM performs the false-negative selection. This is not a question of whether LLMs are accurate in low-resource languages; it is a question of what model was actually used. The reader did not flag this, so my agreement is partial. If the actual model is GPT-4o, then the method's reproducibility is compromised, and the central claim—that the proposed data-utilization pipeline with a fine-tuned Llama variant drives the gains—is not supported by the evidence as written. The appropriate verdict remains CONDITIONAL (as the reader set), because the issue could be a typo or leftover documentation; the condition is that the authors must clarify and, if necessary, re-run with the stated model. I chose UNCHANGED rather than REJECT because the concern, while serious, is resolvable by inspecting the code and is not yet proven to invalidate the empirical results. The concrete test would settle it definitively.","tokens_in":22225,"tokens_out":8104,"duration_ms":97360,"concrete_test":"Inspect the GitHub repository (https://github.com/miaomiao1205/xin_BDUMDR) and locate the Stage-1 false-negative selection script. Identify whether the code calls an OpenAI GPT-4o endpoint or loads Llama-3.1-70B weights (e.g., via vLLM/Transformers). If the code uses GPT-4o, the method description in Section 4.1 is inaccurate and the reliance on a proprietary model must be disclosed; if it uses Llama, Appendix B.2 is erroneous. To settle the impact, re-run Stage-1 filtering on a sample of 300 queries (100 each for sw, te, th) with the opposite model and compare filtering decisions; if the agreement rate is below 80% or downstream nDCG@10 shifts by more than 0.5, the reported results do not correspond to the described method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a three-stage data-utilization pipeline, including LLM-based false-negative selection and LLM-generated hard negatives, improves multilingual dense retrieval. The identity of the LLM performing Stage-1 selection is load-bearing because Stage 1 contributes the largest single ablation gain (Table 3, +3.7 nDCG@10 vs. removing it), and because the paper's narrative attributes this to the 'multilingual ability of LLMs' generically. Section 4.1 states: 'For hard negatives selection and generation, we employ the Llama-3.1-70B-instruct model.' Appendix B.2, however, says: 'For each hard negative sample in the candidate set, we prompt GPT-4o (2024-11-20) to select the true hard negatives.' These cannot both be correct. If GPT-4o—a proprietary model with substantially stronger multilingual instruction-following—was actually used, then the method as described is not what was evaluated, and the reported improvements may be due to GPT-4o's judgment rather than to the proposed pipeline's own contributions (e.g., the multilingual instruction fine-tuning of Llama for generation). If Llama was used, the appendix is wrong and the code must be checked. This ambiguity directly affects reproducibility and the honesty of the comparison against baseline negative-mining methods. The paper's own Limitations section acknowledges that 'the judgment of false negatives via LLMs might still be inaccurate,' but does not disclose which LLM, and the human validation in Appendix C is not tied to a model name. Until this is resolved, the magnitude and attribution of the claimed 2.5% improvement over TopK-PercPos cannot be trusted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage data-utilization pipeline for fine-tuning multilingual dense retrievers. Stage 1 builds a hard-negative candidate set by ensembling several multilingual retrievers and then filtering out false negatives with an LLM judge. Stage 2 supplements the candidate set using an instruction-tuned LLM and a positive-driven back-forward generation procedure. Stage 3 constructs mini-batches that are monolingual but topic-diverse and adds language- and topic-based weights to the contrastive objective. Experiments on MIRACL (16 languages) report an average nDCG@10 of 70.6 with a BGE backbone, which is +1.4 absolute over BGE and +2.5 over TopK-PercPos; ablations show each stage contributes roughly 2 nDCG@10 points. The paper includes code, a human-validation study, and extensive comparison tables.","tokens_in":22558,"tokens_out":4039,"duration_ms":44538,"significance":"If the results hold, the paper offers a practical data-centric alternative to architecture-level changes for multilingual dense retrieval. The controlled comparison against fine-tuned BGE with standard hard-negative mining (Table 2) is credible, and the stagewise ablations (Table 3) support the claim that each component helps. The paper also ships code and performs a small human validation of LLM judgments. However, two load-bearing issues—an inconsistent attribution of the LLM used for false-negative filtering, and an ambiguous/possibly incorrect weighted-loss formula—must be resolved before the reported gains can be fully credited. The limited language coverage of the human validation is an additional concern for the 16-language claim.","major_comments":[{"comment":"The main text states 'For hard negatives selection and generation, we employ the Llama-3.1-70B-instruct model' (§4.1), but Appendix B.2 says 'For each hard negative sample in the candidate set, we prompt GPT-4o (2024-11-20) to select the true hard negatives.' These attributions are mutually exclusive. This is not a cosmetic inconsistency: Stage 1 is the largest single ablation component (Table 3: removing Stage 1 costs 3.7 nDCG@10), and the paper's narrative attributes that gain to the 'multilingual ability of LLMs.' If GPT-4o was the model actually used, the method as described is not what was evaluated, and the gains may reflect GPT-4o's proprietary judgment rather than the proposed pipeline. Please state exactly which model performed the false-negative selection, align the appendix, and either rerun the key comparisons with the declared model or clearly disclose the GPT-4o dependence.","section":"§4.1 and Appendix B.2"},{"comment":"The final loss is written as L_final = 1/|B| Σ_{(q,d+,d−)∈B} ω(d−) × L_MR, but L_MR as defined is already a per-query contrastive loss whose denominator sums over all negatives in D−. Multiplying the entire L_MR by ω(d−) after the summation does not implement per-negative weighting; it is not equivalent to weighting each negative inside the softmax denominator. The actual objective being optimized is therefore unclear, which makes the ablation results in Table 7 and the weight analysis in Table 9 difficult to interpret. Please rewrite the loss explicitly, e.g., L = −log[ e^{sim(q,d+)} / ( e^{sim(q,d+)} + Σ_{d−} ω(d−) e^{sim(q,d−)} ) ], or state precisely how ω(d−) enters the computation.","section":"§3.5, Eq. (L_final)"},{"comment":"The false-negative filter is applied to all 16 training languages, but human validation is reported for only 5 languages (en, es, zh, hi, bn), with 100 samples each. Eleven languages, including low-resource sw, te, and th, have no validation. The paper's own Limitations section acknowledges that 'the judgment of false negatives via LLMs might still be inaccurate,' and Table 8 shows substantial variation in elimination rates across languages (e.g., 32.6% for en but 9.4% for bn). Given that Stage 1 contributes the largest performance gain, the central 16-language claim rests on the assumption that LLM judgment quality transfers to all languages. Please provide per-language validation data (or a principled argument for transfer) and, failing that, qualify the claim about low-resource languages.","section":"Appendix C and §5.1/Table 8"}],"minor_comments":[{"comment":"The notation 'C={C1, C2, ..., C12}' is confusing: C is used both for the topic count and for the set of topics. Clarify that C=12 and use a different symbol for the set.","section":"§3.5"},{"comment":"The fr column for mE5large is garbled in the typeset table ('54.562.052.9'); this should be corrected to a single number.","section":"Table 1"},{"comment":"The caption and surrounding text are inconsistent: the text says 'the first row indicates the percentage of false negative samples eliminated,' but the first row of the table is 'From Retrieval,' and the elimination percentages appear in the third row. Please align the description with the table.","section":"Table 8 caption"},{"comment":"The reference list contains a malformed entry beginning 'Multi-Linguality Multi-Functionality Multi-Granularity. M3-embedding...' with no author names; this appears to duplicate or replace the proper citation for BGE M3 (Chen et al., 2024). Please fix.","section":"References"},{"comment":"The column header 'zhmBERT' in both tables should be split into 'zh' and 'mBERT' to match the per-language columns and model rows used elsewhere.","section":"Appendix D, Tables 15 and 16"},{"comment":"Minor typos: 'Enlish MS MARCO' should be 'English MS MARCO,' and the Figure 1 caption contains 'fied rice' for 'fried rice.'","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the LLM identity inconsistency between §4.1 and Appendix B.2. If GPT-4o was actually used for false-negative selection, the comparison against hard-negative mining baselines is still potentially valid, but the paper must disclose this and should rerun or clearly caveat the results before publication. The weighted-loss ambiguity is also central because it defines the training objective. The human-validation coverage is thin but could be addressed by additional experiments or a revised claim. The paper otherwise appears well within scope and the empirical design is generally sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a competent empirical contribution to multilingual dense retrieval. The three-stage pipeline—ensemble hard-negative initialization, LLM filtering/generation, and language-consistent topic-diverse mini-batches—is genuinely new as a combination, and the controlled comparison against fine-tuned BGE with standard hard-negative mining is credible. Each ablation stage contributes roughly 2 nDCG@10 points, and the human validation on a subset of false negatives is a good faith check. No inference overhead is a real practical plus. This deserves a serious referee, but there is a load-bearing inconsistency that must be resolved first.\n\nSection 4.1 says the hard-negative selection and generation use Llama-3.1-70B-instruct. Appendix B.2 says GPT-4o (2024-11-20) was prompted to select true hard negatives. Both cannot be correct. Stage 1 is the single largest ablation component (w/o Stage 1 drops 3.7 points in Table 3), so the identity of the LLM doing the judging is not a minor implementation detail. If it was GPT-4o, the reported gains may reflect a proprietary model's multilingual judgment rather than the proposed pipeline's own contributions, and the method as described is not what was evaluated. If it was Llama, the appendix is wrong and the code must be checked. The Limitations section acknowledges possible LLM inaccuracy but does not name the model. This has to be fixed before the numbers can be trusted.\n\nThe other concerns are real but secondary. The weighted loss formula in Section 3.5 is ambiguous: applying the negative weight to the whole L_MR per triple is not the same as per-negative weighting, and the paper does not clarify. Hyperparameters (N, thresholds, alpha/beta) appear to be selected against test performance on MIRACL, which can inflate measured gains; the authors should report selection procedure or validation-based tuning. Evaluation on a single benchmark is a limitation but not a fatal one for a data-utilization recipe.\n\nWho should read this: anyone working on multilingual dense retrieval training data or hard-negative mining. It is a useful engineering recipe with clear improvements over strong baselines, provided the LLM attribution is corrected. I would not cite it in its current form because of the contradiction, but I would send it to peer review and expect a revision to address that plus the loss-formula details.","headline":"Solid empirical pipeline for multilingual dense retrieval, but the paper contradicts itself on which LLM ran the load-bearing false-negative filtering; fix that before trusting the numbers.","tokens_in":23128,"tokens_out":1354,"would_cite":false,"duration_ms":16267,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that curating hard negatives with LLM judgment and generation, plus building monolingual topic-diverse mini-batches, lifts multilingual dense retrieval by up to 2.5 nDCG@10 points over strong baselines on 16 MIRACL language","keywords":["multilingual dense retrieval","hard negative mining","false negatives","contrastive learning","LLM judgment","mini-batch construction","MIRACL","data utilization"],"falsifier":"Run the same human-validation procedure (three annotators, 100 samples) on languages outside the paper's five validated ones—e.g., Swahili, Telugu, Thai—and compute LLM-judgment agreement. If agreement with human ratings drops below 70% for any of these languages, and the 'w/o LLM judgment' ablation shows no performance drop in that language, the central claim would be unsupported for that language.","tokens_in":22089,"feed_emoji":"🌐","tokens_out":5306,"duration_ms":53017,"temperature":0.7,"pith_summary":"The paper argues that the main bottleneck in fine-tuning multilingual dense retrievers is not model architecture but the quality of negative samples and the composition of training mini-batches. It proposes a three-stage pipeline: first, it builds a candidate hard-negative set by combining rankings from multiple multilingual retrievers and uses a large language model to remove likely false negatives (documents that are actually relevant but lack annotations); second, it generates additional hard negatives by instructing an LLM to summarize positive documents and produce new queries that retrieve extra candidates; third, it assembles mini-batches that are monolingual but topic-diverse, and weights contrastive learning negatives by language and topic frequency. On the MIRACL benchmark with 16 languages, the method improves average nDCG@10 by 1.4 points over the strongest baseline, and beats the best alternative hard-negative mining strategy by 2.5 absolute points. Each stage contributes roughly 2 nDCG@10 points, with the false-negative filtering stage making the largest difference.","feed_headline":"False negatives filtered by AI boost multilingual retrieval 2.5 points","feed_subtitle":"A three-stage data pipeline—LLM filtering, negative generation, and batch restructuring—adds about 2 nDCG@10 points per stage across 16 lang","key_machinery":"The load-bearing mechanism is the three-stage data-construction pipeline. Stage 1: a multilingual retriever ensemble produces a top-k candidate list per query; an LLM scores each candidate against the query and a known positive, and only candidates rated 'irrelevant' survive as hard negatives. Stage 2: a summarization fine-tuned multilingual LLM summarizes each positive, generates a new query, retrieves fresh candidates, and tops up each query's negative set to a fixed size. Stage 3: two topic classifiers assign coarse and fine topics to each positive document; mini-batches are then sampled so all pairs share a language while topics are spread, and the contrastive loss weights negatives by l","core_discovery":"The central claim is that false negatives—relevant documents that are unlabeled and therefore treated as negatives—seriously degrade multilingual dense retrieval fine-tuning, and that an LLM can identify them well enough to filter them out. The paper shows that combining (i) LLM-based filtering of hard negatives, (ii) LLM-generated supplementary hard negatives, and (iii) mini-batches that are monolingual and topic-balanced, with language/topic-weighted negative sampling, improves retrieval accuracy across all 16 MIRACL languages, with the largest gains on low-resource languages. The paper further reports that replacing any stage with its absence reduces accuracy by about 2 nDCG@10 points, an","pith_inferences":["If LLM judgment quality is high, this pipeline could be applied to other retrieval tasks where false negatives from pseudo-relevance feedback are common, such as conversational search or cross-lingual QA; the false-negative filtering step should generalize.","The finding that same-language, multi-topic mini-batches help might extend to any multilingual contrastive learning, not just retrieval; one could test it on multilingual sentence embeddings.","The paper validates LLM false-negative judgments on only 5 languages (100 samples each); the 11 unvalidated languages, including low-resource Swahili, Telugu, and Thai, carry the real risk. A natural extension is a larger human validation across all 16 languages, or a per-language weighting that downweights LLM judgments in low-resource languages.","The generated negatives come from LLM-generated queries, so the pipeline could accumulate LLM bias; testing with different LLM families or sizes would reveal whether the gains are LLM-specific."],"forward_implications":["The method improves average nDCG@10 over strong baselines on MIRACL across 16 languages, with gains in 15 of 16 languages and slight decreases on French and Chinese relative to the backbone.","It outperforms existing hard-negative mining heuristics for every backbone tried, with a 2.5-point absolute gain over the best alternative.","Each of the three stages contributes independently: removing any one drops nDCG@10 by roughly 2 points, and removing LLM judgment is the most damaging.","The pipeline transfers across different backbone models, so it is a data-curation add-on rather than an architecture change.","The paper's analysis shows that over 20% of initial hard-negative candidates are false negatives and are removed by the LLM; the optimal initial negative set size is around 30-40 candidates."],"fun_headline_variants":["LLM filters false negatives to boost multilingual search by 2.5 points","Three-step data trick lifts multilingual retrieval across 16 languages","Kill false negatives: new pipeline adds 2 nDCG per step","LLM-powered data pipeline improves retrieval on all 16 MIRACL languages","Better negatives, better batches: multilingual retrieval +2.5 nDCG"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The pipeline assumes a large language model can reliably judge whether a candidate document is truly irrelevant to a query across all 16 training languages, but the paper checks this against human labels on only 5 languages.","fun_headline_variants_meta":{"raw":{"variants":["LLM filters false negatives to boost multilingual search by 2.5 points","Three-step data trick lifts multilingual retrieval across 16 languages","Kill false negatives: new pipeline adds 2 nDCG per step","LLM-powered data pipeline improves retrieval on all 16 MIRACL languages","Better negatives, better batches: multilingual retrieval +2.5 nDCG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001013,"raw_usage":{"total_tokens":4073,"prompt_tokens":659,"completion_tokens":3414,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":403,"completion_tokens_details":{"reasoning_tokens":3316}},"tokens_in":403,"tokens_out":3414,"duration_ms":25480,"temperature":1.0,"reasoning_tokens":3316,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:02:26.064485+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same human-validation procedure (three annotators, 100 samples) on languages outside the paper's five validated ones—e.g., Swahili, Telugu, Thai—and compute LLM-judgment agreement. If agreement with human ratings drops below 70% for any of these languages, and the 'w/o LLM judgment' ablation shows no performance drop in that language, the central claim would be unsupported for that language.","supporting_citations":[],"review_version":1}