{"id":"64bc230f-73dd-40ea-a750-ac77fa2426e0","arxiv_id":"2506.02527","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A weighted sampling strategy for negative pairs, mixing random and label-similarity-based hard negatives with synthetic data, improves multilingual retrieval from a monolingual English knowledge base.","lead":"This paper fine-tunes a language model that turns sentences into numbers, so English-only knowledge base entries can be found by questions asked in other languages, including Hinglish, which mixes Hindi and English. It reports that a weighted way of choosing which examples to contrast improves retrieval over several internal baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 31.03% MRR / 33.98% Recall@3 gains match absolute metric differences over the no-labeled-data Synthetic Data Only ablation, not relative improvements over any standard sampling baseline; against Random Negative Mining the gain is only about 2.0% MRR, so the central claim is overstated.","rationale":"I focused on the quantitative claim because the paper's stated contribution is a new sampling strategy with large gains. The exact values in the abstract can be recovered from Table 1 only as absolute differences against Synthetic Data Only, an ablation without any labeled data; they cannot be recovered as relative gains against any standard negative-sampling baseline. This is a direct correctness issue in the central claim, independent of whether the LLM-generated translations preserve intent. The reader's concern about unverified synthetic and translated labels is legitimate, but it is secondary: even with perfectly clean labels, the table shows the advantage over standard sampling is small, and Labeled Data Only is numerically nearly identical to the full method (MRR 0.6639 vs. 0.6653). The Section 3.1 statement that the data cannot be shared but the conclusions remain valid, combined with no code or public benchmark, further blocks verification. I therefore agree with the REJECT disposition, while identifying a different weakest assumption than the reader did.","tokens_in":6865,"tokens_out":11415,"duration_ms":109285,"concrete_test":"Recompute all pairwise comparisons in Table 1, expressing each as both absolute metric difference and relative improvement. Check whether 31.03% MRR and 33.98% Recall@3 arise only against the Synthetic Data Only ablation; if so, the abstract's 'compared to standard ones' wording is unsupported. To settle the substantive question, run Algorithm 1 versus Random Negative Mining on the same translated-labeled data with at least three seeds and report the distribution of MRR and Recall@3; if the improvement is not consistently beyond a few points, the weighted-sampling contribution is not established.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim of the paper is that weighted sampling 'produces performance gains compared to standard ones by up to 31.03% in MRR and up to 33.98% in Recall@3.' The reported numbers in Table 1 do not support this as a comparison to standard negative-sampling strategies. In fact, 31.03 and 33.98 are exactly the absolute metric differences between Algorithm 1 and the Synthetic Data Only ablation: 0.6653 - 0.3550 = 0.3103 and 0.7410 - 0.4012 = 0.3398. As relative gains, these are 87% and 85%, not 31% and 34%; as absolute differences, the appropriate baseline is an ablation that deliberately removes all labeled data, not a 'standard' sampling strategy. Against the standard labeled-data baselines, the gains are much smaller: Algorithm 1 improves MRR by about 2.0% over Random Negative Mining (0.6653 vs. 0.6520), 11.2% over Hard Negative Mining (0.6653 vs. 0.5982), and 18.6% over Hardest Negative Mining (0.6653 vs. 0.5610). The paper also concedes that Labeled Data Only performs comparably to Algorithm 1 (MRR 0.6639 vs. 0.6653), so the proposed method's marginal benefit over its own closest ablation is negligible, and no error bars or significance tests are reported. The headline claim is therefore misleading and the central contribution is unsupported by the paper's own table.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a contrastive fine-tuning pipeline (Algorithm 1) for retrieving entries from a monolingual English knowledge base using multilingual or code-switched queries. For each English KB query, an LLM translates it into a target language, a positive pair is formed with a same-label English query, and negatives are generated by weighted sampling based on label similarity plus one random different-label negative; synthetic Hinglish queries generated by an LLM are added as augmentation. The authors fine-tune multilingual-e5-base and evaluate on an internal Hinglish retrieval task. The abstract claims gains up to 31.03% in MRR and up to 33.98% in Recall@3 over standard sampling strategies. In Table 1, Algorithm 1 achieves MRR 0.6653 and Recall@3 0.7410, compared with Random Negative Mining (0.6520, 0.7271), Hard Negative Mining (0.5982, 0.6829), Hardest Negative Mining (0.5610, 0.6678), Labeled Data Only (0.6639, 0.7356), and Synthetic Data Only (0.3550, 0.4012).","tokens_in":7121,"tokens_out":7302,"duration_ms":66283,"significance":"If the claimed gains were valid, this would be a practically useful and language-agnostic recipe for cross-lingual and code-switched retrieval without constructing a target-language knowledge base. The paper has a clear algorithmic description, a reasonable choice of base model, and a systematic ablation structure. However, the headline result is not supported by the paper's own Table 1: the 31.03% and 33.98% figures are absolute differences against the Synthetic Data Only ablation, not relative improvements over any standard negative-sampling baseline. The comparison to Labeled Data Only shows a difference of 0.0014 MRR, which is negligible without error bars or significance tests. Because the central claim is misrepresented and the most natural baselines show only marginal gains, the contribution as stated is not established.","major_comments":[{"comment":"The headline claim that the weighted sampling strategy produces gains 'by up to 31.03% in MRR and up to 33.98% in Recall@3' is not supported by Table 1. 0.6653−0.3550 = 0.3103 and 0.7410−0.4012 = 0.3398 are the absolute differences between Algorithm 1 and the Synthetic Data Only ablation, not relative gains over any standard negative-sampling strategy. Against Random Negative Mining, the relative MRR gain is only (0.6653−0.6520)/0.6520 ≈ 2.0%, and against Hardest Negative Mining it is (0.6653−0.5610)/0.5610 ≈ 18.6%, not 31%. The abstract and the discussion in §3.5 must be corrected to report the actual comparison baselines and to distinguish absolute percentage-point differences from relative improvements.","section":"Abstract and §3.5, Table 1"},{"comment":"The comparison to Labeled Data Only undercuts the central claim. Algorithm 1 yields MRR 0.6653 and Recall@3 0.7410, while Labeled Data Only yields 0.6639 and 0.7356, respectively; the differences are 0.0014 MRR and 0.0054 Recall@3. The paper reports no error bars, no multiple seeds, and no significance tests, and §3.5 concedes that Labeled Data Only 'performs comparably.' Without statistical evidence, the assertion that the hybrid approach 'provides better protection against performance degradation' is not substantiated. At minimum, the authors should report variance across repeated fine-tuning runs and a significance test for the differences that matter.","section":"Table 1 and §3.5"},{"comment":"The training-data validity assumption is unverified. Step 3 assumes that LLM translations preserve the original English query's intent label, and Step 5 assumes that LLM-generated synthetic Hinglish queries and their 'semantically different' English counterparts are correctly labeled positives and negatives. No quality filtering, human evaluation, or noise analysis is reported for either the translated or the synthetic data. If translation or generation changes the intent, the contrastive signal is corrupted and the measured gains cannot be attributed to the weighted sampling strategy. Additionally, Step 4's hard-negative sampling does not explicitly exclude queries with the same label as the positive; only the Hardest Negative Mining description states 'excluding exact matches.' If same-label queries can be sampled as hard negatives, they are false negatives, which would bias the comparison.","section":"Algorithm 1, Steps 3–5"},{"comment":"The ablation definitions are ambiguous about whether Step 5 (synthetic data augmentation) is included in the Negative Sampling Variations. Random Negative Mining, Hard Negative Mining, and Hardest Negative Mining are described only in terms of Step 4; it is unclear whether they also use synthetic data from Step 5. If they do not, then the comparison between Algorithm 1 and these baselines conflates the sampling strategy with the synthetic-data augmentation. If they do, the paper should say so explicitly. This distinction is load-bearing for the claim that mixed hard/random negative sampling is superior.","section":"§3.2 and Table 1"}],"minor_comments":[{"comment":"The word 'denonymized' appears to be a typo; the intended meaning is likely 'de-identified' or 'anonymized.' Please clarify the anonymization procedure.","section":"§3.1"},{"comment":"The sentence 'Pure hard negative mining (0.4613) or hardest negative mining (0.4012) approaches underperform' cites values that are Top-1 accuracies from Table 1, not MRR or Recall@3 values as the surrounding argument about global embedding structure would require. Please use consistent metric labels.","section":"§3.5"},{"comment":"There are several typos, including 'optimzed' and 'constrastive'; a careful proofread is needed.","section":"§3.3"},{"comment":"The positive-to-negative ratio 1:3 is stated for k=2, but the text also describes k=0 and k=3 variations; please specify how the number of random negatives changes in those cases to maintain the stated ratio.","section":"Algorithm 1 Step 4"},{"comment":"No baseline without fine-tuning is reported. Adding the off-the-shelf multilingual-e5-base result would help the reader assess whether the proposed pipeline improves over the starting model at all.","section":"§3.2"}],"recommendation":"reject","confidential_remarks":"The paper relies on proprietary internal data and does not release code or data, which limits reproducibility. More importantly, the central numerical claim appears to be an artifact of comparing against the Synthetic Data Only ablation rather than a standard sampling baseline; the actual gains over Random Negative Mining and Labeled Data Only are very small. Even with a corrected abstract, the contribution as presented would need substantially stronger evidence, such as multiple seeds, significance tests, and a properly matched ablation set, before it could be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable, honest engineering paper with a headline claim that does not survive contact with its own Table 1. The 31.03% MRR gain in the abstract is not a relative improvement over a standard sampling baseline; it is the absolute difference between Algorithm 1 and the Synthetic Data Only ablation (0.6653 vs 0.3550). That ablation intentionally removes all labeled data, so it is not a fair baseline for the sampling strategy. Against the standard labeled-data baselines, the gains are much smaller: about 2% over Random Negative Mining, 11% over Hard Negative Mining, and 19% over Hardest Negative Mining. The stress-test note gets this exactly right, and I agree with it.\n\nWhat is genuinely new: the specific combination of label-similarity-weighted hard negatives, one random negative, and LLM-generated synthetic queries for code-switched retrieval. Each ingredient is standard, but I do not know of a prior paper that puts them together for Hinglish. The ablation design is thoughtful: separating negative-sampling variations from data-source variations lets you see that the hybrid mining matters most when compared against hardest-negative-only, and that synthetic data helps a bit on top of labeled data. Labeled Data Only nearly matches the full method (MRR 0.6639 vs 0.6653), which the authors concede; they attribute it to test-set composition. That is plausible but not demonstrated.\n\nSoft spots, in proportion. The headline overstatement is the big one, and it is load-bearing: the abstract and Section 3.5 both claim up to 31.03% MRR improvement over standard methods, which Table 1 does not support. Second, there are no error bars, no significance tests, and the evaluation is on a private, unshared dataset with no code. That matters for a paper whose central message is that the proposed method improves retrieval. Third, the language-agnostic claim rests on one language pair (Hinglish→English), so it is not yet justified. Fourth, the LLM-generated training data has no quality filtering or noise analysis; if translations or synthetic queries shift intent, the contrastive signal degrades. That is a real concern, though minor given the clean ablations.\n\nWho this is for: practitioners at companies that want to extend an existing English-only retrieval or knowledge-base stack to code-switched or low-resource queries. They will get a concrete, reproducible recipe (assuming they build their own data following Algorithm 1) and a calibrated sense of what each ingredient buys.\n\nRecommendation: send it to peer review, but expect major revision. The authors need to fix the headline numbers to the actual relative gains, add significance testing or error bars, and ideally test at least one more target language. The core method is sound and the ablation is useful; the overclaiming is fixable. I would not desk-reject this. For my own work, I would cite it as an example of hybrid negative sampling for multilingual retrieval, but note the overstated abstract.","headline":"Decent engineering recipe with a misleading headline: the 31% MRR gain is an absolute gap against a no-labeled-data ablation, not a relative gain over a standard sampling baseline, yet the hybrid approach does show real gains over hardest-negative mining.","tokens_in":839,"tokens_out":1159,"would_cite":true,"duration_ms":23123,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Weighted sampling for contrastive fine-tuning lets a multilingual embedding model retrieve English knowledge-base entries from Hinglish queries, with reported gains up to 31.03% in MRR and 33.98% in Recall@3.","keywords":["multilingual information retrieval","contrastive learning","weighted sampling","hard negative mining","code-switching","Hinglish","embedding model fine-tuning","monolingual knowledge base"],"falsifier":"Sample the translated and synthetic training pairs, have annotators or a strong classifier check whether each pair's intent label still matches the original English label, and retrain on the filtered subset; if the weighted-sampling gains shrink substantially, the reported improvements are not attributable to the sampling strategy alone.","tokens_in":6568,"feed_emoji":"🌐","tokens_out":11625,"duration_ms":97914,"temperature":0.7,"pith_summary":"This paper tries to establish that a monolingual knowledge base can power multilingual and code-switching retrieval if the embedding model is fine-tuned with the right contrastive pairs. The authors keep the knowledge base in one high-resource language (English) and use only unlabeled queries in the target language (here, Hinglish), avoiding the expensive construction of language-specific knowledge bases. Their recipe translates English queries, pairs each translation with a same-intent index query as a positive, and mixes random negatives with hard negatives sampled by label similarity, plus synthetically generated queries. On their Hinglish-to-English retrieval test, this recipe reports gains up to 31.03% in MRR and 33.98% in Recall@3 over standard sampling, and the authors argue the approach is language-agnostic. This matters because knowledge base construction is labor-intensive, so sharing one high-quality English knowledge base across languages would remove a major bottleneck for multilingual dialogue systems.","feed_headline":"Weighted sampling lifts Hinglish search by up to a third","feed_subtitle":"Label-guided contrastive training lets one English knowledge base answer Hinglish queries, with gains up to a third.","key_machinery":"The carrying mechanism is Algorithm 1's rule for building contrastive pairs from a monolingual knowledge base. For each English query, an LLM produces a target-language translation; the positive pair is that translation matched with a random same-label index query. Negatives are drawn by computing a label-similarity score $s_{ij}$ between the query's label $l_i$ and each index query's label $l_j$, then sampling $k=2$ hard negatives with probability weighted by $s_{ij}$, plus one random negative from a different label, maintaining a 1:3 positive-to-negative ratio. A final step augments the data with synthetic target-language paraphrases. This weighted distribution is what the paper credits for preserving global embedding structure while sharpening local distinctions.","core_discovery":"On its own terms, the paper discovers that a mixed-negative recipe works: one random negative combined with hard negatives weighted by label similarity trains a multilingual embedding model to place Hinglish queries near their English intent-equivalents better than pure random, pure hard, or hardest-negative sampling. The best configuration reaches Top-1 0.5450, Top-3 0.7410, Top-10 0.8842, and MRR 0.6653 on the reported test set, with Recall@1 around 0.54 on the selected multilingual backbone. It also finds that labeled translated data is indispensable: training on synthetic data alone yields MRR 0.3550, while synthetic augmentation on top of labeled data protects performance on native low-resource queries.","pith_inferences":["Replacing label-similarity weights $s_{ij}$ with embedding-similarity weights would extend the pipeline to knowledge bases without labels; the paper does not test that variant.","Since the paper observes that Labeled Data Only matches the hybrid on the current test set, a distribution-shifted Hinglish test set would be the sharper test of whether the synthetic augmentation branch contributes on native low-resource queries.","Verifying label preservation on LLM-generated pairs and filtering mismatches would reveal whether the weighted-sampling gains can be pushed higher or whether some gains are an artifact of label noise being amplified for high-similarity labels."],"forward_implications":["A monolingual English knowledge base can serve retrieval for Hinglish queries without building a separate Hinglish knowledge base, as long as unlabeled Hinglish queries are available for synthetic augmentation.","Mixing one random negative with label-similarity-weighted hard negatives outperforms pure random, pure hard, and hardest-negative sampling on the reported benchmark.","High-quality labeled translated pairs are the main driver of performance; synthetic data alone drops MRR to 0.3550, while adding synthetic data on top of labeled data yields the best Top-3, Top-10, and MRR values in the paper's comparison.","Because the pipeline only uses unlabeled target-language queries and a labeled English knowledge base, it is language-agnostic and should apply to other low-resource languages and code-switching varieties."],"supporting_citations":[{"why":"It demonstrates the importance of hard negatives for dense retrieval training, which motivates the hard-negative branch of Algorithm 1.","marker":"[24]"},{"why":"It describes an existing hard-negative mining pipeline for embedding models that the paper's label-weighted sampling is intended to improve on.","marker":"[25]"},{"why":"It shows the prior practice of using a text embedding model to score the hardest negatives, the baseline approach the weighted sampling modifies.","marker":"[16]"},{"why":"It supplies the multilingual embedding backbone used as the foundation for all fine-tuning experiments.","marker":"[17]"},{"why":"It defines the InfoNCE contrastive loss that all models are trained with.","marker":"[26]"},{"why":"It grounds Step 5 by showing that LLM-generated data can improve text embedding training.","marker":"[22]"},{"why":"It identifies the LLM used in the paper's pipeline for translation and synthetic pair generation.","marker":"[4]"}],"fun_headline_variants":["Weighted negatives lift multilingual search by up to a third","Mixed random and hard negatives improve multilingual retrieval by a third","One English KB answers Hinglish queries with up to 34% better Recall@3","Label-weighted hard negatives improve cross-lingual recall by up to 34%","Weighted contrastive sampling improves MRR by up to 31%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every LLM translation and every synthetic query preserves the intent label of the original English query, and the paper reports no quality filtering, human evaluation, or noise analysis for that generated data.","fun_headline_variants_meta":{"raw":{"variants":["Weighted negatives lift multilingual search by up to a third","Mixed random and hard negatives improve multilingual retrieval by a third","One English KB answers Hinglish queries with up to 34% better Recall@3","Label-weighted hard negatives improve cross-lingual recall by up to 34%","Weighted contrastive sampling improves MRR by up to 31%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001505,"raw_usage":{"total_tokens":5985,"prompt_tokens":847,"completion_tokens":5138,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":5042}},"tokens_in":463,"tokens_out":5138,"duration_ms":32707,"temperature":1.0,"reasoning_tokens":5042,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:21:56.402216+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample the translated and synthetic training pairs, have annotators or a strong classifier check whether each pair's intent label still matches the original English label, and retrain on the filtered subset; if the weighted-sampling gains shrink substantially, the reported improvements are not attributable to the sampling strategy alone.","supporting_citations":[{"cited_title":"Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering, 2021","cited_arxiv_id":null,"evidence_quote":"It demonstrates the importance of hard negatives for dense retrieval training, which motivates the hard-negative branch of Algorithm 1."},{"cited_title":"Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge","cited_arxiv_id":null,"evidence_quote":"It describes an existing hard-negative mining pipeline for embedding models that the paper's label-weighted sampling is intended to improve on."},{"cited_title":"Multilingual e5 text embeddings: A technical report, 2024","cited_arxiv_id":null,"evidence_quote":"It supplies the multilingual embedding backbone used as the foundation for all fine-tuning experiments."},{"cited_title":"Improving text embeddings with large language models","cited_arxiv_id":null,"evidence_quote":"It grounds Step 5 by showing that LLM-generated data can improve text embedding training."},{"cited_title":"https://www.anthropic.com/news/claude-3-family","cited_arxiv_id":null,"evidence_quote":"It identifies the LLM used in the paper's pipeline for translation and synthetic pair generation."}],"review_version":1}