{"id":"c825344d-98fe-49b9-b8ef-459628b4d0c2","arxiv_id":"2412.17364","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A contrastive learning penalty that uses synthetic queries for negative documents improves retrieval nDCG by about 4 points on average over the BGE-M3 baseline across Korean, Persian, and Hindi.","lead":"This paper proposes a new fine-tuning method for embedding models, adding a penalty to contrastive learning that keeps hard negative documents close to their own relevant queries. On a three-language retrieval benchmark, the method improves nDCG over the base model, but the gains are small and not statistically validated.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (2) as printed is anti-contrastive for the stated λ=0.1: minimizing it would maximize the InfoNCE term, so the reported gains cannot be explained by the stated loss; the code must be checked.","rationale":"The reader's verdict is REJECT, and this stress-test finds an even more basic internal inconsistency that supports that verdict. The paper's strongest claim is that the CLP loss in Eq. (2) improves retrieval by penalizing the distance between a negative document and its positive queries. But the equation as printed has the wrong sign on the standard contrastive term for the stated λ=0.1. This is an internal inconsistency, not a disagreement with consensus: independent of any question about synthetic query quality, the formula cannot behave as described. The reader's weakest assumption concerned the unshown Gemini prompt and noisy synthetic queries; that is a legitimate secondary concern, because if the synthetic positives are poor, the penalty term becomes arbitrary. However, the sign error is prior: even with ideal synthetic queries, the printed loss would be anti-contrastive. I do not attribute this to fraud or dishonesty; it may be a typographical error, but as submitted the paper does not define a working loss. The available code could settle the issue, and the concrete test is designed to do exactly that. I also note that the reported gain of 'approximately 5 points' is 3.94 points on average in Table 4 (55.95 to 59.89), which is a smaller improvement than claimed in the conclusion, though this is less important than the Eq. (2) issue. No independent verification, formal proof, or reproducible experiment is provided in the manuscript, so the empirical tables cannot override the mathematical inconsistency. The verdict REJECT remains appropriate, and no change is needed to the reader's decision.","tokens_in":5566,"tokens_out":3334,"duration_ms":33654,"concrete_test":"Clone https://github.com/CreaLabs/Enhanced-BGE-M3-with-CLP-and-MoE and inspect the training loss implementation corresponding to Eq. (2). Determine whether the coefficient on the InfoNCE term is (λ−1) or (1−λ), and whether the penalty term is normalized (e.g., averaged over H* or log-transformed). If the code uses (1−λ), then Eq. (2) is a misprint and the reported numbers are produced by a different loss than the one claimed; the claim of a novel CLP must be re-evaluated against the actual loss. If the code uses (λ−1) as printed, run a small training run on MIRACL Korean (e.g., 100 queries, 1 epoch, the same hyperparameters) and compare dev nDCG@5 to the baseline; if the printed loss is used literally, performance should collapse rather than improve, confirming that Eq. (2) is not the source of the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution is the CLP loss in Eq. (2), §3.1. As printed, L = (λ−1)·log[exp(sim(h_i,h_i^+)/τ)/Σ exp(sim(h_i,H_i)/τ)] + λ·(1 − Σ_{h*∈H*} sim(h′,h*)). With λ=0.1 (§4.1.2), the first term has coefficient −0.9. Minimizing this loss therefore drives the InfoNCE log-probability toward large values, i.e., it pushes the query embedding away from its positive and toward the negatives. That is the opposite of contrastive learning. The second term penalizes low similarity between a negative document and its synthetic positive queries, but with weight only 0.1 it cannot offset the reversed first term. Thus, unless Eq. (2) is a typo (for example, the intended coefficient is (1−λ), or the log term is subtracted), the method as defined cannot yield the nDCG improvements in Table 4. This is more fundamental than the missing prompt in Table 5: even with perfect synthetic queries, Eq. (2) would be anti-contrastive. The GitHub repository is therefore the decisive evidence: it must resolve whether the paper states the implemented loss correctly.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fine-tuning methodology for the BGE-M3 text embedding model on the MIRACL dataset for Korean, Hindi, and Persian. The method combines ANCE-style hard negative sampling, a novel Contrastive Learning Penalty (CLP) loss that uses synthetic positive queries for negative documents (generated with Gemini 1.5 Pro), and a Mixture-of-Experts (MoE) layer inserted into the model's intermediate layer. Experiments in Table 4 report an average nDCG@5 improvement from 55.95 (baseline) to 59.89 (ANCE-CLP-moe-intermediate), which the conclusion summarizes as an improvement of approximately 5 points. The paper also reports training and inference time observations and releases code and model links.","tokens_in":5845,"tokens_out":5218,"duration_ms":51206,"significance":"If the reported gains are reproducible, the paper offers a practical recipe for domain-specializing multilingual embedding models, and the CLP idea is simple enough to be tested independently on other datasets and base models. The paper does provide a concrete falsifiable claim: combining ANCE, CLP, and MoE improves MIRACL dev-set nDCG@5 by about 4 points over BGE-M3. However, the significance is currently undercut by empirical limitations: single-run results without variance, model selection and evaluation on the same dev set, a missing synthetic-query prompt, and unclear notation in the core loss equation. These issues make it impossible to judge whether the observed improvement is a real property of CLP or an artifact of the evaluation setup.","major_comments":[{"comment":"The notation h' in the penalty term λ(1 − Σ_{h*∈H*} sim(h', h*)) is not defined. The surrounding text says that the penalty should consider the distance between H_i^- and Q_i^*, but the equation shows only a single h' and has no summation over the negative documents in H_i^-. The authors must specify what h' denotes and whether the penalty is summed over all negatives. As printed, the second term is ambiguous.","section":"§3.1, Eq. (2)"},{"comment":"With λ = 0.1, the coefficient of the InfoNCE log term is (λ − 1) = −0.9, so minimizing Eq. (2) is equivalent to minimizing 0.9 × (−log InfoNCE) plus the penalty term. The objective is thus a scaled contrastive loss with a penalty, not an anti-contrastive objective, but the paper should state this relation explicitly and motivate why the CL term is weighted by 0.9 rather than using the standard weight 1.0. If the intended loss was different, the equation and the λ value should be corrected.","section":"§3.1, Eq. (2)"},{"comment":"The prompt used to generate synthetic positive queries via Gemini 1.5 Pro is referenced as 'Table 5', but the actual Table 5 shows the same MIRACL dev-set nDCG@5 values already given in Table 4; the prompt itself is missing. The synthetic queries are a load-bearing component of the CLP method, because the penalty term in Eq. (2) depends on H*. Without the exact prompt and the generated queries, the experiment cannot be reproduced and the sensitivity of the result to the synthetic data cannot be assessed.","section":"§4.2.2, Table 5"},{"comment":"The best epoch for each method is selected on the MIRACL dev set, and the same dev set is then used for the final performance comparison in Table 4. Combined with single-run training and no variance estimates, the claimed 3.94-point average improvement over the baseline could be due to dev-set overfitting rather than to the proposed methods. The paper should evaluate on the MIRACL test set, run multiple seeds, and report means and standard deviations.","section":"§4.1.2, Tables 3–4"},{"comment":"The training-time comparison is internally contradictory: the text states that CLP took 3 minutes 16 seconds per epoch while existing CL took 4 minutes 18 seconds, but then attributes the 'longer training time for CLP' to the additional similarity computation. Since the title and abstract emphasize efficiency, this contradiction must be resolved and the actual timing comparison should be stated consistently.","section":"§4.2.2"}],"minor_comments":[{"comment":"The subsection heading contains a typo: 'Funtion' should be 'Function'.","section":"§4.2.2"},{"comment":"The text says 'r is a temperature hyperparameter', but the formula uses τ; the notation should be unified.","section":"§2, Eq. (1)"},{"comment":"Table 2 reports nDCG@10 from the BGE-M3 paper, while Table 4 reports nDCG@5 for the proposed methods. The metric discrepancy should be explained or the tables should use the same cutoff.","section":"§4.1.1 vs Tables 2 and 4"},{"comment":"The sentence 'training only the intermediate layer (ANCE-CLP-intermediate) resulted in higher performance (59.45 -> 59.89) than training all parameters of the model (ANCE-CLP)' is inconsistent with Table 4, where ANCE-CLP is 58.06 and ANCE-CLP-intermediate is 59.45. The intended comparison should be clarified.","section":"§4.2.3"},{"comment":"The phrase 'while minimizing the distance between qi and p_i^+ and maximizing the distance to H_i^-' describes the standard contrastive objective, but Eq. (2) uses a cross-entropy form; the paper should align the prose with the equation.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is not ready for acceptance because the central empirical claim is supported by a single run on the dev set with the model-selection procedure on that same dev set, and a key component of the proposed method (the synthetic-query prompt) is missing from the paper. However, the CLP idea is straightforward and potentially testable; if the authors provide the exact prompt, the generated queries, a proper train/dev/test split, multiple seeds, and a corrected Eq. (2) with defined notation, the paper could become a useful empirical contribution. I would also ask the editor to verify that the linked GitHub repository contains the actual training code and the prompt, since the paper's reproducibility depends on them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the headline: this is a modest, easy-to-follow fine-tuning recipe with a small twist on contrastive loss, but the reported 5-point nDCG gain is not backed by enough evidence. The loss function in Eq. (2) is not anti-contrastive; with λ=0.1 the first term is simply (1−λ) times the standard InfoNCE loss, so it pulls the positive pair together. The real weaknesses are evidential, not algebraic.\n\nWhat's new: the CLP penalty term that encourages negative document representations to stay close to their own synthetic positive queries, plus the combination of ANCE, that penalty, and a frozen-backbone MoE intermediate layer. The code is released, which is good. The experimental design is simple enough to reproduce on a single GPU.\n\nThe empirical case is thin. All results are single-run, best-epoch-per-method on the dev set, with no standard deviations. The dev set doubles as the model-selection set, so the final numbers are selected, not predicted. The synthetic-query prompt is referenced as Table 5 but the table shows a results table instead – the actual prompt is missing, which is a reproducibility gap. There's also an internal contradiction: they say CLP took 3m16s per epoch vs CL's 4m18s, then attribute 'the longer training time for CLP' to the extra similarity step, when CLP was actually faster. These are easy to fix but they indicate the paper wasn't polished. The biggest substantive concern is that the penalty term depends entirely on Gemini-generated queries; if those are noisy, the gains may be an artifact of that particular generation pipeline.\n\nThat said, the method is coherent and the direction of improvement is consistent across languages for ANCE vs random and for CLP vs CL. The gains are small: ANCE adds ~1.5 points average over baseline, CLP adds ~0.6, MoE-intermediate adds ~1.4. The paper would be believable as a technical report if the evidence were tightened.\n\nFor peer review: I'd send it to a referee, because the loss formulation is checkable and the code exists; but I'd expect the reviewer to ask for multiple seeds, a proper dev/test split, the actual prompt, and corrected text. It's a borderline desk-reject in current form, but with revision it could be a solid short paper for an IR workshop.","headline":"A plausible small twist on contrastive fine-tuning, but the single-run, dev-set-selected evidence is too thin to support the 5-point claim; the anti-contrastive reading of Eq. (2) is a misinterpretation.","tokens_in":6359,"tokens_out":3310,"would_cite":false,"duration_ms":31096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that adding a penalty term for negative documents' own positive queries to contrastive learning, plus ANCE hard-negative sampling and a MoE layer, lifts nDCG@5 by about five points.","keywords":["contrastive learning penalty","text embedding fine-tuning","dense retrieval","Mixture of Experts","ANCE negative sampling","MIRACL","multilingual document retrieval","nDCG evaluation"],"falsifier":"Re-run the exact ANCE-CLP-moe-intermediate recipe with the synthetic positive queries replaced by human-written or otherwise independently verified queries for the same negative documents, and compare nDCG@5 on MIRACL; if the gap over the 55.95 baseline disappears or reverses, the claimed gain is an artifact of the synthetic queries rather than a property of CLP.","tokens_in":5354,"feed_emoji":"📈","tokens_out":9679,"duration_ms":84720,"temperature":0.7,"pith_summary":"This paper is trying to show that an off-the-shelf multilingual embedding model can be specialized to a document-retrieval domain with three cheap changes: picking hard negative samples with ANCE, adding a new Contrastive Learning Penalty term to the contrastive loss, and training only a Mixture-of-Experts layer in the middle of the network. On MIRACL (Korean, Hindi, Persian), the full recipe raises average nDCG@5 from 55.95 to 59.89, roughly a 5-point gain over the BGE-M3 baseline. The CLP term is the paper's main idea: instead of only pushing negative documents away from the query, it also penalizes the model when a negative document moves far from the queries that were generated as positive for that document. A reader should care because the gains come from fine-tuning on a single GPU with a modest epoch budget, which is an efficient route to better retrieval for retrieval-augmented generation systems.","feed_headline":"Contrastive penalty lifts retrieval by about 5 points","feed_subtitle":"Fine-tuning BGE-M3 with CLP, ANCE negatives, and a MoE layer raises average nDCG@5 from 55.95 to 59.89 on three languages.","key_machinery":"The load-bearing component is the Contrastive Learning Penalty in Eq. (2), which rewrites the standard contrastive loss as $$$L^{{\\text{ITER}}$}_i = (\\$\\lambda$-1)\\log\\frac{\\exp(\\mathrm{sim}(h_i,h^+_i)/\\tau)}{\\sum_{H_i\\in\\{h^+_i,H'\\}}\\exp(\\mathrm{sim}(h_i,H_i)/\\tau)} + \\$\\lambda$\\left(1-\\sum_{h^*\\in H^*}\\mathrm{sim}(h',h^*)\\right),$$ where $h'$ is the embedding of a negative document, $H^*$ contains embeddings of queries that are positive for that document, and $\\lambda$ (set to $0.1$ in experiments) controls the penalty weight. The first term is the familiar contrastive loss; the second term penalizes any increase in distance between negative documents and their own positive queries. The other two pieces are ANCE, which selects the top-10 nearest non-relevant documents as hard negatives, and a Mixture-of-Experts module inserted in the intermediate layer (the 1024-to-4096 expansion) with two experts and one expert per token, while all other parameters are frozen. The combination is what carries the reported gain, and Table 3 shows each component contributes: ANCE alone raises the average from 55.95 to 57.43, adding CLP gives 58.06, training only the intermediate layer gives 59.45, and MoE in that layer gives 59.89.","core_discovery":"On the paper's own terms, the central discovery is that the standard contrastive learning objective is incomplete: it optimizes the distance between a query and its positive and negative documents, but it ignores the distance between a negative document and the queries that should retrieve it. The authors define a Contrastive Learning Penalty, added to the usual cross-entropy contrastive loss, that grows as a negative document's embedding moves away from the embeddings of its positive queries. Because no public dataset used here provides those positive queries for negative documents, the paper synthesizes them with an LLM prompt. Combined with ANCE hard-negative sampling and a Mixture-of-Experts intermediate layer, the method reports an average nDCG@5 of 59.89 on the MIRACL dev set for Korean, Persian, and Hindi, versus 55.95 for the untuned BGE-M3 model. The authors also report that CLP is especially helpful in Persian, where standard contrastive learning dropped below baseline.","pith_inferences":["If CLP's mechanism is real, its effect should be measurable in datasets that already contain human relevance judgments per document, where the 'positive queries for negative documents' can be taken from the data instead of synthesized; a natural check is to run CLP on such a dataset and see whether the same roughly 5-point gain appears.","The intermediate-layer result deserves emphasis: adding MoE after already training only the intermediate layer adds 0.44 points (59.45 to 59.89), so a reader should not attribute the full gain to the loss alone; an ablation that isolates CLP from MoE on more languages would separate the two contributions.","Because the penalty weight $\\lambda=0.1$ appears fixed in the experiments, a natural extension is a small sweep over $\\lambda$ and over the number of generated positive queries per negative document; sensitivity there would tell whether CLP is a robust objective or a narrow tuning artifact.","The MoE inference cost (roughly 5 hours vs 3 hours to embed the Korean corpus) suggests that for deployment, routing or distillation might recover most of the gain at lower cost."],"forward_implications":["With all three components, average nDCG@5 on MIRACL rises from 55.95 (BGE-M3) to 59.89, so a roughly 5-point gain is available from fine-tuning alone.","CLP recovers Persian, where ordinary contrastive learning with ANCE negatives had fallen from 51.13 to 50.88; CLP moves it up to 52.39.","Training only the intermediate layer (freezing everything else) outperforms training all parameters, so parameter-efficient fine-tuning may be sufficient for this task.","The full recipe trains on a single GPU with at most three epochs; CLP changes per-epoch training time on the Korean set, and the MoE model takes about 5 hours to embed the Korean corpus versus about 3 hours for the baseline.","The gains are measured on dev nDCG@5, so the paper's direct corollary is that embedding quality for retrieval, not just classification, improves under this recipe."],"supporting_citations":[{"why":"Supplies BGE M3-Embedding, the pre-trained multilingual model that the fine-tuning recipe starts from and the baseline it must beat.","marker":"(Chen et al., 2024)"},{"why":"Provides ANCE, the hard-negative sampling method used to select the top-10 informative negative documents.","marker":"(Xiong et al., 2020)"},{"why":"Defines the contrastive cross-entropy objective that CLP extends with its penalty term.","marker":"(Chen et al., 2020)"},{"why":"Supplies the Mixture-of-Experts architecture applied to the intermediate layer.","marker":"(Gupta et al., 2022)"},{"why":"Provides the MIRACL multilingual document-retrieval dataset, its languages, and the dev-set nDCG evaluation used in the experiments.","marker":"(Zhang et al., 2022)"}],"fun_headline_variants":["Contrastive penalty boosts retrieval by 5 nDCG points","Penalty targets a blind spot in contrastive learning","CLP penalty lifts retrieval five points on MIRACL","Fine-tuning with CLP gains 5 points in retrieval","New penalty fixes contrastive learning's blind spot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The penalty requires positive queries for every negative document; the benchmark used for evaluation does not contain them, so the paper manufactures those queries with a large language model, and if those synthetic queries are not representative of real user queries, the penalty term could be training toward arbitrary directions and the gain could vanish outside this setup.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive penalty boosts retrieval by 5 nDCG points","Penalty targets a blind spot in contrastive learning","CLP penalty lifts retrieval five points on MIRACL","Fine-tuning with CLP gains 5 points in retrieval","New penalty fixes contrastive learning's blind spot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1529,"prompt_tokens":921,"completion_tokens":608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":527}},"tokens_in":537,"tokens_out":608,"duration_ms":6263,"temperature":1.0,"reasoning_tokens":527,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:32:30.888658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact ANCE-CLP-moe-intermediate recipe with the synthetic positive queries replaced by human-written or otherwise independently verified queries for the same negative documents, and compare nDCG@5 on MIRACL; if the gap over the 55.95 baseline disappears or reverses, the claimed gain is an artifact of the synthetic queries rather than a property of CLP.","supporting_citations":[],"review_version":1}