{"id":"f9047ccc-a6a3-406c-a006-c575cd4f45c3","arxiv_id":"2506.02020","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A hard-negative gradient amplifier improves multimodal contrastive embedding training, achieving 72.5 average on MMEB, but it is a heuristic reweighting rather than a theoretical advance.","lead":"This paper proposes a training tweak for multimodal AI embedding models: amplify the learning signal from 'hard' negative examples, defined as images or texts that are confusingly similar to the query. The authors report top scores on the MMEB benchmark and release code, but the gains over existing methods are small and partly rely on a proprietary model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 ties UniME on the same backbone and the only controlled EGA ablation uses unreleased QQMM, so the central claim that EGA improves embedding learning is not yet supported.","rationale":"The reader's CONDITIONAL verdict is appropriate, so I do not move it. The gradient analysis in Section 3.2 is correct, but the causal claim depends on empirical support, and the current support is undercut by three facts: Table 1's LLaVA-OneVision result ties UniME and is worse on OOD and GRD; Table 1 lacks a same-recipe baseline row; and the only controlled ablation uses the unreleased QQMM model with a single seed and fixed alpha. The concrete test directly fills the missing baseline using the authors' own protocol and released code, so it can settle the concern without relying on theoretical debate about whether the modified gradient is conservative. The lack of a corresponding scalar loss remains an open issue, but it is secondary to the missing controlled experiment.","tokens_in":9320,"tokens_out":16541,"duration_ms":171659,"concrete_test":"Using the released code, train LLaVA-OneVision-7B on MMEB-train with the paper's exact protocol (2,000 steps, batch 1,024, tau=0.02) for three seeds each of (a) standard InfoNCE and (b) EGA with alpha=20.0, and report mean +/- std on overall and OOD subsets of the 36 MMEB eval sets. If EGA's gain over the same-recipe baseline is within seed noise (or below 0.5 overall), the central claim that EGA improves embedding learning fails, and the SOTA claim in Table 1 is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is causal: EGA, not the backbone or training recipe, improves MMEB scores. The gradient decomposition in Eq.3-4 is correct, but Table 1 contains no same-recipe baseline trained by the authors on LLaVA-OneVision; the comparison rows are published numbers from other training runs. In that table, Ours (LLaVA-OneVision) averages 70.7, exactly tied with UniME (70.7), and is worse on GRD (90.4 vs 90.9) and OOD (65.6 vs 65.8). The abstract's 'state-of-the-art' claim and Section 4.2's 'particularly clear improvements on OOD' therefore do not hold against the strongest prior same-backbone method. The only controlled ablation (Table 2) is on QQMM, a proprietary model with no released architecture or training details: baseline 70.4, +EGA 72.2, +EGA+RS-H 72.5, with no error bars, no seed variance, and no sensitivity analysis for alpha=20.0. If that QQMM run is not reproducible, the method's contribution reduces to a numerical tie. The paper also does not show that inserting reweighted probabilities into the InfoNCE gradient is the gradient of any loss, which makes the missing controlled experiment the decisive evidence rather than a theoretical guarantee.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Explicit Gradient Amplifier (EGA), a method that modifies the gradient of the InfoNCE loss during contrastive training by reweighting the negative-sample probabilities with a hardness score h_i = exp(alpha * (q·x_i - q·x_+)), then substituting these reweighted values for the original softmax probabilities in the gradient computation. The authors argue that this explicitly amplifies the contribution of hard negatives and leads to more discriminative multi-modal embeddings. They evaluate on the MMEB benchmark using LLaVA-OneVision-7B and a proprietary QQMM model, reporting an overall score of 70.7 for LLaVA-OneVision and 72.5 for QQMM. A controlled ablation on QQMM shows an improvement from 70.4 baseline to 72.2 with EGA and 72.5 with the additional relative-similarity hardness (RS-H) score.","tokens_in":9620,"tokens_out":4841,"duration_ms":52446,"significance":"If the reported gains are reproducible, EGA is a simple, computationally lightweight, and potentially general technique for hard-negative exploitation in contrastive training of MLLM-based embedding models. The paper's strengths include a correct (albeit elementary) gradient derivation of InfoNCE, a plug-and-play design that is easy to implement, and evaluation on a public benchmark with released code. The principal weakness is the evidence base: the only controlled ablation uses a proprietary backbone whose details are not released, the same-backbone comparison with prior work is an exact tie rather than a state-of-the-art result, and no error bars or sensitivity analyses are provided. The central claim that EGA improves embedding learning therefore rests on a single, unreproducible run, and the theoretical analysis does not supply a substitute because the modified gradient is not the gradient of any stated loss.","major_comments":[{"comment":"The abstract and Section 4.2 claim that the LLaVA-OneVision model 'achieves state-of-the-art performance' and shows 'particularly clear improvements on OOD', but Table 1 shows Ours (LLaVA-OneVision) averaging 70.7, exactly matching UniME (70.7), and being lower on GRD (90.4 vs 90.9) and OOD (65.6 vs 65.8). These claims are not supported by the table. Please either provide a controlled same-recipe comparison showing a real improvement or revise the claims to reflect parity with UniME.","section":"§4.2, Table 1"},{"comment":"The only controlled ablation of EGA is performed with QQMM, a proprietary model whose architecture and training details are not publicly described. The table reports no error bars, no seed variance, and no sensitivity analysis for alpha=20.0, so it is unclear whether the +1.8 and +0.3 point differences are statistically meaningful or reproducible. Add a controlled ablation on the publicly available LLaVA-OneVision backbone (baseline vs +EGA vs +EGA+RS-H) with multiple seeds, and report standard deviations and an alpha sweep.","section":"§4.3, Table 2"},{"comment":"The text states that the reweighted values are 'normalized across all negative samples to ensure they form a valid probability distribution', but Eq. (7) does not produce a probability distribution. Specifically, sum_i \\tilde{p}_i = (sum_i p_i) * (sum_i \\hat{p}_i) / (sum_i \\hat{p}_i) = sum_i p_i = 1 - p^+, which is less than 1 in general. Either correct the normalization so that the \\tilde{p}_i sum to 1, or explicitly state that the total negative mass is intentionally preserved rather than re-normalized to a distribution.","section":"§3.3, Eq. (7)"},{"comment":"The modified gradient is obtained by inserting reweighted probabilities into the InfoNCE gradient, but it is not shown to be the gradient of any loss function. Since the theoretical contribution of the paper is the gradient analysis, this gap makes the empirical evidence the decisive support for the method. Please either show that the modified gradient is a valid descent direction for a well-defined objective, or compare EGA against the equivalent loss-level reweighting (e.g., LLaVE's hardness weighting) to isolate the effect of gradient-level modulation.","section":"§3.3, Eqs. (3)-(7)"}],"minor_comments":[{"comment":"The phrase 'consistently outperforms both VLM2Vec and LLaVE' is fine, but 'state-of-the-art' and 'particularly clear improvements on OOD' should be tempered because the table shows parity with UniME and a small OOD deficit.","section":"Abstract and §4.2"},{"comment":"The hardness score uses dot products x·x_i and x·x_+, while the text refers to 'cosine similarity'. Please clarify whether the embeddings are L2-normalized so that dot products equal cosine similarities, and make the notation consistent.","section":"§3.3, Eq. (5)"},{"comment":"The definition of the weighting matrix W as P - I requires a clear index convention. If P_{i,j} = \\tilde{p}_j for row i, the diagonal terms are W_{i,i} = \\tilde{p}_i - 1, which are negative and will push the positive pair together; the notation in the bullet list should be explicit about which index corresponds to the query and which to the target.","section":"§3.3, Step 2"},{"comment":"The implementation details omit the optimizer, learning rate schedule, and warmup steps. These are needed for reproducibility, especially given that the only controlled ablation uses an unreleased model.","section":"§4.1"},{"comment":"There is a typo in the caption: 'is integerated' should be 'is integrated'.","section":"Figure 2"},{"comment":"Reference [19] appears to have the author order garbled ('Shafiq Rayhan Joty Caiming Xiong Yingbo Zhou Semih Yavuz Rui Meng, Ye Liu' is not a standard citation format). Please check the metadata for this entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim currently rests on a proprietary QQMM ablation with no public details, while the public-backbone comparison is a tie with UniME. This combination makes the contribution difficult to verify. If the authors can provide a controlled, seeded ablation on LLaVA-OneVision and correct the normalization issue in Eq. (7), the paper could become a solid empirical contribution. The current version, however, oversells its results and would need substantial additional experiments to justify acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clean, well-written paper with a correct but elementary derivation and a sensible heuristic. The relative-similarity hardness score (RS-H) is a genuine variant over LLaVE's absolute similarity, and the ablation on QQMM shows a real jump (70.4 to 72.2 to 72.5). If that result is reproducible, the method is a cheap plug-in that helps contrastive embedding training. But the paper's headline claim—state-of-the-art on the same LLaVA-OneVision backbone—is not supported by Table 1: Ours (LLaVA-OneVision) averages 70.7, exactly tied with UniME, and worse on GRD and OOD. The abstract and Section 4.2 say \"particularly clear improvements on OOD,\" which is the opposite of what the table shows (65.6 vs 65.8). That is an overstatement.\n\nThe only controlled ablation uses QQMM, a proprietary model with no architecture or training details, no error bars, no seed variance, and no alpha sensitivity. The paper also does not show that injecting reweighted probabilities into the gradient corresponds to any loss, so we are left with an empirical claim that stands or falls on that single QQMM run. Code is released for the LLaVA version, but that row is a numerical tie, so there is no controlled demonstration of EGA's benefit on a public backbone.\n\nWhere the paper does well: the gradient decomposition in Eq.3-4 is correct, the writing is clear, the plug-and-play framing is honest, and the RS-H margin idea is a reasonable contribution, even if a small variant. The authors also concede \"on par with UniME\" in Section 4.2, though the abstract does not match that concession.\n\nWhat I would want before accepting: a same-recipe baseline and EGA on LLaVA-OneVision with multiple seeds, alpha sensitivity over a range, and ideally a direct comparison to LLaVE's weighting. Without that, the paper reads as a strong workshop submission rather than a confident SOTA claim. I would still send it to peer review, but I would push for these additions. The idea is worth engaging with; the current evidence is not yet convincing.","headline":"Tidy gradient analysis and a plausible hardness heuristic, but the SOTA claim is undercut by a numerical tie with UniME on the same backbone and an ablation that relies on an unreleased QQMM model.","tokens_in":10125,"tokens_out":1918,"would_cite":false,"duration_ms":20531,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that reweighting InfoNCE gradients by a relative-similarity hardness score improves multi-modal embeddings, and reports a top MMEB score of 72.5 with the QQMM model.","keywords":["multi-modal embeddings","contrastive learning","InfoNCE","hard negative mining","gradient reweighting","relative similarity","MLLM","MMEB"],"falsifier":"Train the same LLaVA-OneVision-7B setup on MMEB with EGA at several values of $\\alpha$ (for example 1, 5, 20, and 50) and on at least one other backbone; if the gains over the InfoNCE baseline disappear for $\\alpha$ away from 20 or do not transfer across backbones, the claim that relative-similarity amplification is the mechanism is weakened. A sharper check is to search numerically for any loss function whose gradient equals the EGA gradient from Eq.3 and Eq.4; if none exists, EGA is a gradient heuristic, not a principled objective.","tokens_in":1671,"feed_emoji":"🧲","tokens_out":6884,"duration_ms":124665,"temperature":0.7,"pith_summary":"This paper argues that in contrastive learning for multi-modal embeddings, a negative sample is hard not when its similarity to the query is absolutely high but when it is close to the positive sample's similarity. It introduces the Explicit Gradient Amplifier (EGA), a module that reweights the softmax probabilities in the InfoNCE gradient using a relative-similarity hardness score, so hard negatives exert a stronger push. With the LLaVA-OneVision-7B backbone, EGA matches or improves over prior MLLM embedding methods on the MMEB benchmark, and with the authors' QQMM model it reaches an average score of 72.5 and the top leaderboard position. A sympathetic reader would care because EGA is a drop-in, parameter-light change to the training objective that could transfer to any MLLM-based retrieval system.","feed_headline":"Relative similarity reweighting tops MMEB leaderboard at 72.5","feed_subtitle":"A relative-similarity hardness score reweights InfoNCE gradients, pushing MMEB average to 72.5 on the QQMM backbone.","key_machinery":"The central object is the Explicit Gradient Amplifier (EGA), a module that takes query and target embeddings, computes a hardness score for each negative given by $h_i^- = e^{\\alpha(q\\cdot x_i^- - q\\cdot x^+)}$, multiplies the softmax probabilities $p_i^-$ by these scores, and renormalizes the products before using them to form the query and target gradients. The relative-similarity form of the hardness score carries the argument: a negative is marked as hard only when its similarity to the query is close to the positive's, so a moderately similar negative is easy when the positive is very similar and hard when the positive is weakly similar. EGA then injects the amplified probabilities into the InfoNCE gradient formulas, amplifying hard-negative contributions without changing the model architecture or the forward embedding computation.","core_discovery":"The central claim is that the InfoNCE gradient with respect to the query decomposes into a weighted sum of differences between negative embeddings and the positive embedding, where the weight on each negative is the softmax probability $p_i^-$ that the query is classified as that negative. Therefore, the learning contribution of each negative is directly controlled by these probabilities, and amplifying the probabilities of hard negatives should yield more discriminative embeddings. EGA does this by computing a hardness score $h_i^- = e^{\\alpha(q\\cdot x_i^- - q\\cdot x^+)}$ for each negative, multiplying each $p_i^-$ by its hardness score, renormalizing the products across negatives while preserving the total probability mass, and using the amplified probabilities in the gradient formulas. With the same LLaVA-OneVision-7B backbone, the paper reports state-of-the-art MMEB performance relative to VLM2Vec, LLaVE, and UniME, and with its QQMM backbone it reports an average score of 72.5 and the top rank on the MMEB leaderboard.","pith_inferences":["A testable extension is to apply the same gradient reweighting to unimodal contrastive learning, since the InfoNCE gradient analysis is modality-agnostic; if the mechanism is general, EGA should improve text-only or image-only retrieval benchmarks as well.","The top-rank result with QQMM is confounded by the backbone change, so an apples-to-apples comparison of EGA with other hard-negative methods on QQMM would isolate the module's contribution from the base model's strength.","One could check whether the optimal exponent $\\alpha$ needs to anneal over training, since the hardness distribution changes as similarities sharpen; a fixed $\\alpha=20$ may be near-optimal only for this particular backbone and batch protocol.","The paper does not show that the amplified gradient corresponds to the gradient of any actual loss function, so a useful follow-up would be to search for a loss whose gradient matches EGA; if none exists, the method should be understood as a gradient heuristic rather than a principled objective."],"forward_implications":["On LLaVA-OneVision-7B, EGA matches or improves over VLM2Vec, LLaVE, and UniME on MMEB under the same training protocol, with particularly clear gains reported on out-of-distribution datasets.","With the QQMM backbone, the approach reports an average MMEB score of 72.5 and the top position on the leaderboard.","The ablations attribute roughly 1.8 points of average MMEB improvement to EGA itself and additional gains to the relative-similarity hardness score, particularly on retrieval and grounding tasks.","Because EGA is computed purely from embeddings with basic tensor operations, it adds negligible overhead and can be plugged into any MLLM-based embedding model without architectural changes."],"supporting_citations":[{"why":"Supplies the InfoNCE loss whose gradient decomposition is the basis for EGA.","marker":"[14]"},{"why":"Defines the hardness-by-query-similarity weighting that EGA replaces with a relative-similarity score.","marker":"[17]"},{"why":"Supplies the MMEB benchmark, the training protocol, and the VLM2Vec baseline.","marker":"[22]"},{"why":"GradCache is the mechanism that allows the large batch size producing many in-batch hard negatives.","marker":"[18]"},{"why":"UniME ranks negatives by absolute similarity, providing the comparison that motivates the relative hardness score.","marker":"[24]"},{"why":"LLaVA-OneVision-7B is the backbone on which the main comparisons are run.","marker":"[11]"}],"fun_headline_variants":["Hard-negative gradient amplification achieves top MMEB results","Explicit gradient amplifier leads MMEB at 72.5","Reweighting InfoNCE negatives improves multi-modal embeddings","Hard-negative gradient scaling yields sharper embeddings","EGA hard-negative weights hit MMEB top benchmark"],"cache_read_input_tokens":12288,"weakest_assumption_plain":"The load-bearing premise is that replacing the true classification probabilities in the InfoNCE gradient with a sharpened, renormalized version, while leaving the positive gradient unchanged, is a valid optimization signal that truly improves embeddings rather than just distorting the gradient.","fun_headline_variants_meta":{"raw":{"variants":["Hard-negative gradient amplification achieves top MMEB results","Explicit gradient amplifier leads MMEB at 72.5","Reweighting InfoNCE negatives improves multi-modal embeddings","Hard-negative gradient scaling yields sharper embeddings","EGA hard-negative weights hit MMEB top benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000451,"raw_usage":{"total_tokens":2335,"prompt_tokens":1071,"completion_tokens":1264,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":687,"completion_tokens_details":{"reasoning_tokens":1187}},"tokens_in":687,"tokens_out":1264,"duration_ms":13907,"temperature":1.0,"reasoning_tokens":1187,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:11:54.993759+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same LLaVA-OneVision-7B setup on MMEB with EGA at several values of $\\alpha$ (for example 1, 5, 20, and 50) and on at least one other backbone; if the gains over the InfoNCE baseline disappear for $\\alpha$ away from 20 or do not transfer across backbones, the claim that relative-similarity amplification is the mechanism is weakened. A sharper check is to search numerically for any loss function whose gradient equals the EGA gradient from Eq.3 and Eq.4; if none exists, EGA is a gradient heuristic, not a principled objective.","supporting_citations":[{"cited_title":"Vlm2vec: Training vision-language models for massive multimodal embedding tasks","cited_arxiv_id":null,"evidence_quote":"Supplies the MMEB benchmark, the training protocol, and the VLM2Vec baseline."},{"cited_title":"Scaling deep contrastive learning batch size under memory limited setup","cited_arxiv_id":null,"evidence_quote":"GradCache is the mechanism that allows the large batch size producing many in-batch hard negatives."}],"review_version":1}