{"id":"00fc2ac5-f908-473d-938e-c8e63dbb8048","arxiv_id":"2505.15576","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AHNPL improves compositional reasoning in VLMs by generating image-side hard negatives from text embedding shifts and adding adaptive contrastive losses.","lead":"This paper trains vision-language models to notice small semantic differences by creating tricky image-side examples: it shifts an image's embedding in the direction that separates a correct caption from an incorrect one. The method, AHNPL, reports higher accuracy on ARO, VALSE, and SugarCrepe benchmarks than previous hard-negative contrastive approaches.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central mechanism rests on an unvalidated linear transfer of text-embedding differences to image embeddings (Eq. 2); if that transfer fails, L_visual_neg (Eq. 5) trains against arbitrary directions rather than genuine visual hard negatives.","rationale":"The paper is clear and its empirical results are internally consistent, but the central novelty rests on an unverified geometric assumption: that a difference of two text embeddings in CLIP space is a valid semantic displacement to apply to an image embedding. The reader's weakest assumption identifies exactly this point, and I agree. The concern is not that Eq. (2) contradicts known theory; it is that the paper gives no independent check, and the provided visualization in Table 5 is insufficient because the distances are arithmetic consequences of the construction. Since the method is a fine-tuning recipe with consistent benchmark gains and promised code, outright rejection is not warranted. However, the mechanism claim is not yet supported: the ablation does not isolate the visual-perturbation term, and no error bars or multiple-seed results are provided. A conditional acceptance with the requested validation is appropriate, so I keep the reader's CONDITIONAL verdict unchanged.","tokens_in":12488,"tokens_out":4963,"duration_ms":47024,"concrete_test":"Run AHNPL with the visual-perturbation term removed (set Ihs empty or use only L_textual_neg), keeping all other losses and the same COCO fine-tuning, and compare Table 1. If ARO/VALSE/SugarCrepe averages do not drop materially, Eq. (2) is not the source of the gains. In parallel, validate the construct: for 200 COCO training captions, generate real negative images from the hard-negative texts with a text-to-image model, compute their CLIP image embeddings, and measure whether e_Iorig+(e_Tneg-e_Torig) is closer to the real negative-image embedding than to e_Iorig; if not, the linear transfer in Eq. (2) is not a valid image-domain hard negative.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (2) defines e_Ineg = e_Iorig + (e_Tneg - e_Torig). The entire visual-perturbation contribution, and specifically L_visual_neg in Eq. (5), depends on this vector addition producing an embedding that behaves like a real hard-negative image: close to the original image in overall content but specifically displaced toward the semantics of the negative text. The paper provides no evidence for this. Table 5 only shows cosine distances among computed embeddings; because e_Ineg is defined as a sum, these distances move in the expected direction by construction and do not establish that the resulting vector lies on or near the image-embedding manifold, nor that it is more similar to the negative text than to the positive text in a way that matches an actual image. CLIP's image and text encoders map into a shared space, but that space is not a single Euclidean semantic space where text differences are valid image displacements; there is no invertibility or linearity guarantee. If Eq. (2) fails, the visual negative loss trains the image encoder to repel arbitrary directions that may not correspond to any real image, so the reported VALSE/SugarCrepe/ARO gains could come from regularization or text-side losses rather than the claimed visual-perturbation mechanism. The ablation in Table 3 does not isolate Eq. (2): the 'negatives' row includes the full hard-negative pipeline, so the contribution of the visual perturbation alone is untested. This is the load-bearing concern because the paper's novelty and its stated reason for outperforming text-only hard-negative methods (e.g., CE-CLIP) is precisely this cross-modal displacement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AHNPL, a fine-tuning method for CLIP on compositional reasoning (CR) benchmarks. It generates textual hard negatives via POS-based noun swapping and RoBERTa-based masked infilling, then creates synthetic visual hard negatives by adding the text-embedding difference to the original image embedding (Eq. 2). Training adds two auxiliary losses to the standard contrastive loss: a multimodal hard negative loss (Eqs. 5-7) and a dynamic margin loss (Eqs. 8-11). The authors report state-of-the-art average accuracy on VALSE (75.9% vs. 72.5% for CE-CLIP), with gains also on ARO and SugarCrepe (Tables 1-2). Ablations (Table 3) and a small embedding-distance visualization (Table 5) are provided.","tokens_in":12864,"tokens_out":3871,"duration_ms":37583,"significance":"If the reported gains are robust, AHNPL offers a simple and reproducible recipe for improving compositional reasoning in CLIP-style models, with code release and evaluation on standard benchmarks being clear strengths. The central novelty--transferring text-derived semantic shifts into the image embedding space--is conceptually attractive but is also the main risk: the paper does not demonstrate that the synthetic vectors produced by Eq. (2) behave like genuine image hard negatives. The empirical claims are currently supported by single-run results without error bars, and the ablation table does not isolate the visual perturbation mechanism. These issues are addressable and do not, in my view, invalidate the approach, but they need to be resolved before the paper's central claim can be accepted.","major_comments":[{"comment":"The load-bearing assumption is that e_Ineg = e_Iorig + (e_Tneg - e_Torig) yields an embedding that behaves like a real hard-negative image. The paper does not validate that this vector lies on or near the image-embedding manifold, nor that it is more similar to the negative text than to the positive text in a way matching an actual image. Table 5 only reports cosine distances among computed embeddings, and because e_Ineg is defined as a sum, those distances move as expected by construction. If Eq. (2) does not hold, L_visual_neg in Eq. (5) trains the image encoder to repel arbitrary directions that may not correspond to any real image, so the reported gains could come from regularization or the text-side losses rather than the claimed visual-perturbation mechanism. Please provide evidence such as decoding nearest images, measuring retrieval consistency with real hard-negative images, or showing that the synthetic embeddings are indistinguishable from genuine image negatives in downstream behavior.","section":"Section 3.2, Eq. (2)"},{"comment":"The ablation does not isolate the visual perturbation of Eq. (2). The row labeled \"negatives\" includes the combined L_neg of Eq. (7), which already contains L_visual_neg; there is no condition with only textual hard negatives and no visual perturbation, and no condition with the visual negative loss removed while keeping the textual hard negatives. As a result, the contribution of the visual perturbation to the final result is untested. Please add ablations that factor out Eq. (5) separately from Eq. (6), and also report the effect of replacing the synthetic visual negatives with actual images drawn as hard negatives.","section":"Table 3, ablation study"},{"comment":"All results appear to be from a single training run, with no error bars, no multiple-seed reporting, and no significance tests. The headline VALSE gain over CE-CLIP is 3.4 percentage points, and several subtask differences are much smaller than that; without variance information the claim of a new state of the art is not statistically supported. Please report mean and standard deviation over at least three seeds, and ideally paired significance tests for the main benchmark comparisons.","section":"Tables 1-3, Section 4.3"},{"comment":"The loss definitions in Eqs. (5) and (6) are written with an ambiguous fraction: the displayed \"-log(1 / sum exp(...))\" is algebraically equivalent to log(sum exp(...)), which is a soft-plus-like penalty that pushes similarities down, but the notation should be clarified. More importantly, the losses use text-text and image-image similarities (S(I, In), S(T, Tn)) that are not defined by Eq. (3), which only defines text-image similarity; please specify the exact similarity computation used for within-modality pairs and whether a temperature is applied.","section":"Section 3.3, Eqs. (5)-(6)"}],"minor_comments":[{"comment":"The VALSE column headers are garbled in the typeset table (e.g., \"A vgExistence\", \"Av g\", \"Foil-it! A vgquantifiers\"), making it hard to read which sub-benchmark corresponds to which score. Please reformat the table with clear column separators.","section":"Table 1"},{"comment":"The \"+++\" symbol in the visual-perturbation panel is unexplained; it appears to denote vector addition in embedding space, but a one-line caption definition would help.","section":"Figure 2"},{"comment":"Table 5 presents \"cosine distances\" but the numbers shown are sparse and it is not clear which pairs they correspond to, nor whether higher or lower values are intended as evidence. Please label each distance and state explicitly which comparison supports the claim.","section":"Section 4.6, Table 5"},{"comment":"The sentence \"The first type involves swapping two nouns in a sentence to generate hard negatives\" is followed by a sentence fragment describing the purpose; please merge these into a complete paragraph.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer vision venue and is not methodologically unsound in principle, but the central visual-perturbation mechanism is currently supported only by an unvalidated linear-transfer assumption. The absence of multiple-seed reporting and the non-isolating ablation make the empirical SOTA claim fragile. If the authors can validate Eq. (2) (e.g., via nearest-neighbor decoding, retrieval comparisons, or an ablation that removes the synthetic visual negatives), and add variance estimates, the paper could become acceptable. I would not recommend rejection at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a genuinely new mechanism—synthesizing image-side hard negatives by adding a text-embedding difference to an image embedding—and the combined loss recipe is not in the cited literature. The gains on three benchmarks are consistent, and the ablations do show each loss contributes. But the central assumption (Eq. 2) is unvalidated, the ablation never isolates the visual perturbation, and there are no error bars or multi-seed runs. As it stands, I can't tell whether the improvement comes from the mechanism claimed or from the text-side losses and regularization.\n\nWhat's new: existing methods use text-side negatives or cross-modal ranking; this paper transfers a text shift into image space and trains a visual negative loss. The dynamic margin that adjusts based on the model's own previous-step similarity is also a reasonable addition. The method is simple to implement and the paper is clearly written.\n\nThe soft spots: The stress-test note is on target. Eq. (2) assumes text-embedding differences are valid displacements for image embeddings in CLIP space. The paper doesn't show the resulting vectors decode to anything image-like, nor that they are closer to real images matching the negative text. Table 5 reports cosine distances that are guaranteed to move in the expected direction by construction, so it's not evidence. The ablation in Table 3 has a 'negatives' row that includes both visual and textual negatives, so removing the visual perturbation alone is never tested. Without that ablation, the contribution of the paper's claimed novelty is unmeasured.\n\nAlso missing: error bars, multiple seeds, and a clear statement about the MSCOCO training/evaluation overlap with SugarCrepe, which shares COCO images. The CREPE citation is wrong—the referenced paper is about question answering, not the compositional benchmark.\n\nThe paper is not incoherent; the mechanism might work. But the evidence is not there yet. I'd send it to a serious referee, but the author's response would need to add a validation study of the synthetic embeddings and an ablation that isolates the visual perturbation. For a reader, treat the VALSE improvement as promising but unproven.","headline":"A new idea for synthesizing visual hard negatives via text-embedding displacement, but the core assumption is unvalidated and the experiments lack key controls—worth review, not yet trustworthy.","tokens_in":13312,"tokens_out":3285,"would_cite":false,"duration_ms":28400,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Text deviations become image negatives to sharpen VLM compositional reasoning","keywords":["compositional reasoning","vision-language models","hard negative mining","contrastive learning","visual perturbation","CLIP fine-tuning","adaptive margin loss","VALSE benchmark"],"falsifier":"Compute cosine similarity between the generated negative image embedding and the hard negative text before and after training; if the perturbation does not shift the image embedding toward the negative text semantics, or if replacing the deviation embedding with a random unit vector reproduces the reported gains, the visual perturbation mechanism is not the source of the improvement.","tokens_in":12259,"feed_emoji":"🧩","tokens_out":5773,"duration_ms":47471,"temperature":0.7,"pith_summary":"This paper claims that compositionality failures in vision-language models can be reduced by translating hard-negative captions into the visual embedding domain. The proposed method, AHNPL, takes the difference between a positive caption embedding and a hard-negative caption embedding, adds that difference to the image embedding, and uses the resulting vector as a hard negative image during fine-tuning. Two auxiliary losses then separate hard negatives within each modality and adaptively increase the margin for difficult positive and negative pairs. On the VALSE benchmark the method reaches 75.9% average accuracy, surpassing the previous hard-negative method CE-CLIP by 3.4 points, with gains also on ARO and SugarCrepe. If the gains hold, it offers a recipe for improving compositional reasoning without collecting new images.","feed_headline":"VALSE score hits 75.9% via text-to-image hard negatives","feed_subtitle":"Turning caption differences into image-embedding perturbations helps CLIP tell swapped objects and attributes apart.","key_machinery":"The load-bearing identity is the linear embedding transfer $e_{I_{neg}} = e_{I_{orig}} + (e_{T_{neg}} - e_{T_{orig}})$, which the paper calls visual perturbation with semantic shift. It converts a text-side hard negative into a synthetic image-side hard negative by reusing the CLIP text embedding difference as a displacement in image embedding space. The dynamic margin threshold $M_t^n = \\frac{1}{|B|}\\sum_{(I,T)\\in B} (S^{t-1}(I,T) - S^{t-1}(I,T_n))$ is the adaptive component that changes contrastive pressure based on sample difficulty.","core_discovery":"AHNPL's central claim is that the image encoder, not just the text encoder, needs hard negatives during fine-tuning, and that such image negatives can be synthesized directly in CLIP embedding space. The deviation embedding $\\Delta e = e_{T_{neg}} - e_{T_{orig}}$ is added to the original image embedding to form $e_{I_{neg}} = e_{I_{orig}} + \\Delta e$, giving the image encoder a training signal that mirrors the semantic shift already present in text. On top of this, a multimodal hard negative loss pushes generated visual negatives and textual negatives away from positives, and a dynamic margin loss adjusts the margin using the previous step's similarity gap, focusing training on the hardest pairs. The paper reports state-of-the-art results on VALSE and improvements over CE-CLIP on ARO and SugarCrepe, attributing the gains to better visual discrimination and difficulty-aware alignment.","pith_inferences":["Because Eq. (2) assumes linear arithmetic in CLIP space, the same trick could transfer to other joint embedding models only if their geometry is similarly linear; the paper does not test this.","The synthetic image negatives are never decoded or validated as real images, so a natural extension is to check whether they correspond to semantically altered images, for example by retrieval or generation.","The dynamic margin uses the previous training step's similarity gap; with small batches or high learning rates, this lag could cause oscillation. A sensitivity analysis would clarify the operating range.","The negative-generation step is limited to POS-based swaps and RoBERTa infilling; pairing AHNPL with LLM-generated negatives, as in SugarCrepe, could broaden hard-negative diversity and likely push the method further."],"forward_implications":["On VALSE, the method's 75.9% average surpasses CE-CLIP's 72.5%, with the largest margins on the quantifiers and clean subtasks.","On ARO, relation and attribute accuracy improve over CE-CLIP (83.8 vs 83.0 and 77.0 vs 76.4), showing better relationship and attribute understanding.","On SugarCrepe, object-level swap and add tasks rise by 3.5 and 4.8 points over CE-CLIP, indicating reduced language bias in favor of visual cues.","Ablations show that each component—hard negatives, multimodal hard negative loss, and dynamic margin loss—adds a positive increment, and the full combination gives the best results.","The method needs only existing captions to synthesize visual negatives, so it can fine-tune an image encoder without additional image collection."],"supporting_citations":[{"why":"Supplies the pretrained CLIP model whose embedding space is used for both text deviations and image perturbation.","marker":"[Radford et al., 2021]"},{"why":"Introduced hard-negative fine-tuning for compositional reasoning and contributes the ARO benchmark as well as the NegCLIP baseline.","marker":"[Yuksekgonul et al., 2022]"},{"why":"CE-CLIP, the intra-modal hard-negative method that AHNPL compares against and surpasses on VALSE.","marker":"[Zhang et al., 2024b]"},{"why":"Contributes the VALSE benchmark that measures the headline 75.9% result.","marker":"[Parcalabescu et al., 2022]"},{"why":"Contributes the debiased SugarCrepe benchmark used to show the method reduces language bias.","marker":"[Hsieh et al., 2023]"},{"why":"RoBERTa is used to fill masked words and generate textual hard negatives.","marker":"[Liu et al., 2019]"},{"why":"Spacy is used for part-of-speech parsing to construct swap-based hard negatives.","marker":"[Honnibal, 2017]"},{"why":"MSCOCO is the fine-tuning dataset for all training runs.","marker":"[Lin et al., 2014]"}],"fun_headline_variants":["Text-to-image negatives boost VLM reasoning to 75.9% VALSE","VALSE 75.9% via text-derived hard image negatives","Image-side hard negatives from text shifts improve CLIP reasoning","Dynamic margins and image negatives sharpen VLM compositional skills"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the assumption that adding the text-to-text semantic difference to an image embedding produces a legitimate hard-negative image embedding, rather than a meaningless point in embedding space.","fun_headline_variants_meta":{"raw":{"variants":["Text-to-image negatives boost VLM reasoning to 75.9% VALSE","VALSE 75.9% via text-derived hard image negatives","Image-side hard negatives from text shifts improve CLIP reasoning","Dynamic margins and image negatives sharpen VLM compositional skills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00058,"raw_usage":{"total_tokens":2741,"prompt_tokens":962,"completion_tokens":1779,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1706}},"tokens_in":578,"tokens_out":1779,"duration_ms":13013,"temperature":1.0,"reasoning_tokens":1706,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:13:54.155680+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute cosine similarity between the generated negative image embedding and the hard negative text before and after training; if the perturbation does not shift the image embedding toward the negative text semantics, or if replacing the deviation embedding with a random unit vector reproduces the reported gains, the visual perturbation mechanism is not the source of the improvement.","supporting_citations":[{"cited_title":"Learning transferable visual models from nat- ural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained CLIP model whose embedding space is used for both text deviations and image perturbation."},{"cited_title":"Valse: A task-independent benchmark for vision and language models centered on linguistic phe- nomena","cited_arxiv_id":null,"evidence_quote":"Contributes the VALSE benchmark that measures the headline 75.9% result."},{"cited_title":"Sugar- crepe: fixing hackable benchmarks for vision-language compositionality","cited_arxiv_id":null,"evidence_quote":"Contributes the debiased SugarCrepe benchmark used to show the method reduces language bias."},{"cited_title":"spacy 2: Natural lan- guage understanding with bloom embeddings, convolu- tional neural networks and incremental parsing.To appear,","cited_arxiv_id":null,"evidence_quote":"Spacy is used for part-of-speech parsing to construct swap-based hard negatives."}],"review_version":1}