{"id":"f6dcbe3d-c2ff-458a-9d54-cd59be131b06","arxiv_id":"2507.19064","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"NEAT adapts VLMs to negation by tuning only text-encoder normalization layers with three losses: entropy refinement, reversed-caption contrastive learning, and textual debiasing.","lead":"This paper introduces NEAT, a test-time adaptation method that updates only layer normalization parameters in a vision-language model's text encoder to improve understanding of negated captions. It reports matching or beating expensive post-training methods on negation benchmarks while using far fewer trainable parameters and only unlabeled test data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (7)'s hardest-negative selection is internally inconsistent: the chosen sample has minimal similarity, so the logit gap is non-positive and the stated metric-loss equivalence fails; if the least-similar image is semantically unrelated, L_sr can corrupt cross-modal structure.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing point: the hardest visual sample for L_sr may be the semantically wrong contrastive target. My read sharpens this into a concrete internal inconsistency in Eq. (7). The selected sample \\hat v_i^- is defined as the visual embedding with minimal similarity to \\bar t_i, so the logit gap (s_neg - s_pos)/τ2 is non-positive at selection time; the stated approximation -log(1+e^x) ≈ -x requires x to be large and positive, so the claimed equivalence to a simple metric loss does not hold in the regime where the method operates. Moreover, the least-similar image is not generally the image that the reversed caption affirms, so the loss can push reversed captions toward arbitrary unrelated images rather than toward semantically complementary ones. This matters because the ablation attributes much of the MCQ improvement to L_sr, and the reversed-negation metric in Table VI is directly optimized by L_sr and L_tri, making the headline generalization numbers less probative. I do not think this overturns the paper: the method is broadly evaluated, code is promised, and even a flawed derivation can still yield a workable heuristic. The reader's CONDITIONAL verdict already requires clarification of this mechanism, so no change to the verdict is needed. The concrete test would resolve whether the concern lands: if random or semantically matched negatives perform as well as the least-similar rule, the claimed hard-negative mechanism is not the source of the gains; if the least-similar rule is both necessary and semantically appropriate, the concern is answered.","tokens_in":22226,"tokens_out":11769,"duration_ms":128046,"concrete_test":"Using the released code, instrument the first adaptation batches on MS-COCO Retrieval-Neg. (1) Record s_pos= \\bar t_i · \\hat v_i and s_neg= \\bar t_i · \\hat v_i^- at initialization and across epochs; if (s_neg - s_pos)/τ2 is non-positive at initialization, the Eq. (7) approximation is invalid in the regime where training starts. (2) Ablate the negative-selection rule, holding L_ent and L_tri fixed: (a) least-similar image (current), (b) most-similar image, (c) random image, (d) an image selected by a semantic rule (highest score for the affirmed concept and lowest for the negated concept after LLM decomposition). Compare COCO MCQ-Neg accuracy and CIFAR10 reversed-negation error. If (c) or (d) matches or exceeds (a), the 'hardest-negative' mechanism is not the source of the gains; if (a) is both necessary and semantically appropriate, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"L_sr is the load-bearing component: the ablation in Table IX shows MCQ accuracy rises from 28.69 with L_ent only to 45.91 when L_sr is added, so the central generalization claim depends on it. Two linked problems affect Eq. (7). First, the 'equivalence' to a metric loss uses -log(1+e^x) ≈ -x, which is only accurate for x large and positive. Here x=(s_neg - s_pos)/τ2, with \\hat v_i^- chosen as the visual sample having minimal similarity to \\bar t_i; because \\hat v_i is in the comparison set, s_neg≤s_pos, so x≤0 and the approximation is invalid at initialization and in the regime where the selection criterion holds. The exact loss has vanishing gradient w.r.t. the selected sample when x is negative, while the linearized loss has constant gradient; the two objectives are not equivalent. Second, and more substantively, the least-similar image is unlikely to be the semantically correct target for \\bar t_i. For a reversed caption such as 'a person is in sight, no dog in the grass', the correct image contains a person and no dog; the least-similar CLIP embedding is more likely an image containing neither concept (or a dog image, due to bag-of-words similarity). Minimizing Eq. (7) then pulls \\bar t_i toward that arbitrary image and away from its paired image, potentially encoding spurious associations. Because the reversed-negation error (Table VI, 1.36%) is exactly the quantity L_sr and L_tri are designed to minimize, this concern is not softened by the transfer results; it threatens the interpretation of the gains as negation understanding rather than fitting to the chosen hard-negative heuristic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NEAT, a test-time adaptation method that updates only LayerNorm parameters of the text encoder of a vision-language model in order to improve negation understanding. The method uses an LLM to split each negation-conditioned caption into positive and negative components, selects visual candidates through a penalty-augmented similarity, and optimizes three objectives: refined entropy (L_ent), a semantics-reversion contrastive loss (L_sr), and a textual dual-concept debiasing loss (L_tri). Experiments on COCO, VOC2007, MSR-VTT, CheXpert, and nine image classification datasets report consistent gains over CLIP, NegCLIP, and BLIP baselines, and competitive or superior performance relative to post-training methods that use millions of image-text pairs, while updating only a tiny fraction of parameters. The paper also reports transfer of the adapted normalization layers to unseen datasets and to a medical foundation model.","tokens_in":22510,"tokens_out":10227,"duration_ms":100328,"significance":"If the empirical results are taken at face value, the paper identifies a practical and relatively underexplored problem, proposes a lightweight adaptation procedure, and validates it across image, video, and medical benchmarks with released code. The finding that updating only LayerNorm parameters of the text encoder can produce large negation-understanding gains and transfer across datasets would be a useful contribution to the test-time adaptation and vision-language communities. However, the theoretical justification of the central loss in Eq. (7) is not sound as written, the negative-sample selection is under-specified, and one of the headline transfer metrics (reversed negation-conditioned error, Table VI) is directly optimized by the method; these issues currently prevent me from fully endorsing the paper's claims.","major_comments":[{"comment":"The derivation of the simplified metric form in Eq. (7) uses the approximation -log(1+exp(x)) ≈ -x, which is accurate only for x large and positive. With \\hat v_i^- chosen as the visual sample of minimal similarity to \\bar t_i, and with \\hat v_i included in the comparison set, we have x = (\\bar t_i · \\hat v_i^- - \\bar t_i · \\hat v_i)/τ2 ≤ 0, so the approximation is invalid exactly in the intended operating regime. In this regime the exact loss has derivative -sigmoid(x) with respect to x, with magnitude at most 1/2, whereas the linearized objective has constant gradient, so the two objectives are not equivalent. Because Table IX shows that L_sr is the component that raises MCQ accuracy from 28.69 to 45.91, the paper should state clearly which form is implemented (the exact log-sum-exp loss or the linearized metric loss) and justify the choice; if the linearized form is used, the claimed relationship to the negative InfoNCE in Eq. (6) does not hold.","section":"IV-B2, Fig. 4"},{"comment":"The selection rule for \\hat v_i^- is under-specified and semantically questionable. The paper calls \\hat v_i^- the 'hardest' visual sample but then defines it as the one with minimal similarity to \\bar t_i, which is the opposite of the standard hardest-negative convention; it is also not stated whether \\hat v_i itself is included in the candidate set, how large the candidate set is, or how ties are broken. More substantively, for a reversed caption such as 'a person in sight, no dog in the grass', the least-similar CLIP embedding is unlikely to be the semantically correct contrastive target, and minimizing Eq. (7) can pull \\bar t_i toward an arbitrary unrelated image while pushing it away from the paired image. I request a precise definition of the candidate set, statistics on the selected samples (for example, their semantic categories and similarity ranks), and an ablation that replaces the minimal-similarity sample with a standard hardest negative drawn from the same batch.","section":"IV-B2, Fig. 4"},{"comment":"The reversed negation-conditioned error reported in Table VI is not an independent measure of generalization. The second term of L_tri and the L_sr loss are explicitly designed to separate the reversed negation-conditioned caption from the negation-conditioned caption and from the paired image, so the near-perfect RNC error of 1.36% after adaptation on MS-COCO is partly a self-fulfilling optimization target rather than evidence of transfer. The generalization claims should be supported primarily by metrics that are not directly optimized, such as Retrieval-Neg on MSR-VTT, MCQ-Neg on VOC2007, and affirmative classification tasks, and the RNC numbers should be accompanied by evaluation on held-out negation distributions with non-template structure.","section":"V-C, Table VI"},{"comment":"In Eq. (8), the term (2 - \\|\\hat t_i - \\bar t_i\\|_2) is described as encouraging maximum distance between \\hat t_i and \\bar t_i, but the expression is unbounded below as the distance grows, and the scale of this term relative to the first term depends on the units of the embedding norm. If a squared distance was intended, the notation should be \\|\\cdot\\|_2^2, and the boundedness and interaction with λ should be clarified; otherwise the optimization may be dominated by a small number of large-distance examples.","section":"IV-B3, Eq. (8)"}],"minor_comments":[{"comment":"The abstract states 'less than 0.01% of trainable parameters', while the introduction states '0.014‰' of trainable parameters and '0.36‰' of the data; these numbers should be reconciled.","section":"I and Abstract"},{"comment":"The text says 'From Fig. V' when referring to the results shown in Table V; the reference should be corrected.","section":"V-B1"},{"comment":"The dataset name 'Caltch101' should be spelled 'Caltech101'.","section":"Table VI"},{"comment":"The method name is spelled inconsistently as 'ConCLIP' in Table II and 'CoNCLIP' in Section II-A and the references; one spelling should be used throughout.","section":"II-A and Table II"},{"comment":"The quantity S(\\hat t_i, v_j) is called a similarity but is not guaranteed to be non-negative; it would be clearer to describe it as a scoring function rather than a similarity.","section":"IV-B1, Eq. (4)"},{"comment":"Equation (2) sums over N_T while Eq. (5) sums over the batch size B; the batch-level normalization and the role of N_T should be stated explicitly.","section":"IV-B1, Eqs. (2) and (5)"}],"recommendation":"major_revision","confidential_remarks":"I recommend requesting a major revision rather than rejection because the empirical contribution is substantial and the main concerns are addressable with clarifications and additional experiments. In particular, the authors should be asked to report which form of Eq. (7) is actually optimized in their code, to specify and justify the candidate-selection procedure for \\hat v_i^-, and to de-emphasize the reversed negation-conditioned error as a transfer metric unless it is evaluated on distributions that were not directly optimized. The editor may also wish to ask for the exact candidate-selection implementation to verify whether the 'hardest' sample is selected with or without the paired image in the candidate set."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper before trusting the Eq. (7) derivation, because the linearized loss does not do what the text says. The empirical work, on the other hand, is broad and mostly convincing. The genuinely new piece is the combination of LLM-based negation decomposition with test-time adaptation of LayerNorm parameters, and the results across COCO, MSR-VTT, CheXpert, and nine classification sets are consistent. That is worth a serious look.\n\nWhere it does well: the idea is practical—updating only normalization layers is genuinely cheap—and the comparisons include both post-training (NegFull, NegCLIP) and TTA baselines. The transfer experiments (adapted on COCO, tested on VOC2007 and on template-based tasks) are the right kind of evidence. The ablation shows each of the three losses contributes, and L_sr is clearly the load-bearing one.\n\nThe soft spot is Eq. (7). The paper writes the reversed-contrastive loss as a negative InfoNCE and then simplifies it to a metric loss using -log(1+e^x) ≈ -x. That approximation is only accurate for large positive x. But the 'hardest visual sample' is defined as the one with minimal similarity to the reversed caption, so x = (s_neg - s_pos)/tau2 is ≤ 0 from the start. In that regime the exact loss is between -log 2 and 0, while the linear surrogate is non-negative; the gradients even point in opposite directions for the selected negative. So the implemented loss is not equivalent to the stated objective, and the surrogate pulls the reversed caption toward the least-similar image—which is semantically arbitrary. Because the ablation shows L_sr accounts for most of the MCQ gain, this is not a footnote; it needs to be fixed or clearly justified.\n\nSmaller issues: the 'zero-shot MCQ-Neg' numbers are transductive (the model adapts on the same data), no error bars or repeated runs are reported, the LLM prompt is not given, and the reversed-negation error rate in Table VI is the direct target of L_sr and L_tri, so that result is partly by construction.\n\nNone of these kill the paper. The empirical pattern is consistent, the resource claim is believable, and the problem is real. I would send this to a serious referee and ask for major revision focused on Eq. (7), plus a few missing reproducibility details.","headline":"The empirical story is solid, but the derivation of L_sr in Eq. (7) is internally inconsistent and needs to be fixed before publication.","tokens_in":23134,"tokens_out":4217,"would_cite":true,"duration_ms":39545,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A test-time tweak to normalization layers gives vision-language models a grasp of negation.","keywords":["negation understanding","vision-language models","test-time adaptation","LayerNorm adaptation","dual-concept shift","contrastive learning","cross-modal retrieval","CLIP"],"falsifier":"Run NEAT on a negation benchmark while replacing the hardest-negative selection in the reversion loss with a randomly chosen sample from the same batch; if retrieval and multiple-choice performance do not drop, the hardest-negative mechanism is not doing the claimed work, and the geometric diagnosis would need revision.","tokens_in":21946,"feed_emoji":"🚫","tokens_out":5280,"duration_ms":51351,"temperature":0.7,"pith_summary":"This paper argues that vision-language models (VLMs) fail at negation not mainly because negation is rare in their training data, but because the embedding distributions of affirmative, negation-conditioned, and reversed negation-conditioned text are misaligned: the model puts semantically consistent affirmative and negated captions far apart, while placing semantically opposite captions close together. It proposes NEAT, a test-time adaptation method that updates only the LayerNorm parameters of the text encoder to pull these distributions into line. With less than 0.01% of trainable parameters and only unlabeled test data, NEAT matches or beats post-training methods that fine-tune on tens of millions of negation-labelled pairs, and the adapted layers transfer to unseen datasets and to other vision-language models. A sympathetic reader would care because the method points to a cheap, data-free route to a capability that otherwise requires massive curated data collection.","feed_headline":"Teaching CLIP to understand 'no' with a tiny parameter update","feed_subtitle":"Updating only the text encoder's normalization layers closes the gap between affirmative and negated queries.","key_machinery":"The load-bearing object is the dual-concept shift diagnosis, made quantitative with Maximum Mean Discrepancy on CIFAR10 features: normal versus negation-conditioned text embeddings show MMD = 1.57, while negation-conditioned versus reversed text embeddings show MMD = 0.08. The adaptation machinery is NEAT's combined objective, a sum of three losses over the text encoder's LayerNorm parameters: refined entropy minimization on a negative-penalised retrieval score, a reversion loss that reduces (for each reversed caption) to a simple metric gap between similarity to its paired image and similarity to the single hardest negative visual sample, and a triplet loss on textual embeddings. The only non-distributional ingredient is an LLM that parses each negated caption into affirmative sub-captions and generates the reversed caption; the actual learned parameters are just the scale and shift of normalization layers.","core_discovery":"Negation understanding in a VLM can be repaired at inference time by correcting two distributional errors measured in the text encoder's embedding space: the affinity gap between an affirmative caption and its negation-conditioned variant (which share meaning), and the false affinity between a negation-conditioned caption and its reversed counterpart (which are opposites). NEAT decomposes each negated caption into a positive and a negative part with an LLM, uses the negative part to re-rank candidate visual matches, then optimizes three objectives on the LayerNorm parameters of the text encoder: an entropy loss over refined text–image matches, a semantics-reversion loss that treats the reversed caption as a hard negative against its paired image, and a textual loss that pulls negation-conditioned text toward its affirmative base while pushing it away from its reversed form. The paper shows empirically that this adaptation improves retrieval and multiple-choice accuracy on negation benchmarks across images, videos, and chest X-rays, and that the modified normalization layers transfer to new negation tasks and to different base models.","pith_inferences":["The same distribution-shift repair could plausibly apply to other closed-class linguistic operators, such as quantifiers, temporal markers, or modality words, where VLMs behave like bag-of-words models; the paper only demonstrates negation, but the mechanism of aligning consistent paraphrases and separating semantic opposites is operator-agnostic.","The method's reliance on an LLM to separate positive and negative components and build reversed captions means its ceiling is partly set by the LLM's parse quality; a testable extension would measure NEAT's sensitivity to parse errors or replace the LLM with a deterministic parser.","The reversion loss selects a single hardest negative visual sample; a natural variant would be to sample a small set of hardest negatives, which might retain the anti-clustering benefit while reducing sensitivity to one mis-selected anchor."],"forward_implications":["Negation capability can be added to an existing VLM without collecting or labelling negation data; adaptation runs on the unlabeled query set at deployment time.","Model updates are tiny (normalization parameters only), so adaptation is cheap enough to run per client or per domain, and the adapted layers can be shipped as a small patch or swapped between models.","Because the method targets distribution geometry rather than memorised negation patterns, its gains transfer to negation queries and benchmarks the model never saw during adaptation.","Post-training pipelines that generate tens of millions of negated captions may be unnecessary for many applications; a test-time correction can reach comparable accuracy with a fraction of the data and parameters."],"supporting_citations":[{"why":"Supplies the negation benchmarks (NegBench) and the observation that VLMs fail at negation; NEAT is evaluated on these tasks.","marker":"[2]"},{"why":"The base vision-language model (CLIP) whose embedding-space dual-concept shifts are measured and corrected.","marker":"[35]"},{"why":"Provides the refined candidate selection strategy NEAT borrows and serves as a test-time adaptation baseline it must beat.","marker":"[25]"},{"why":"The InfoNCE objective whose negative form underlies the derivation of the semantics-reversion loss.","marker":"[32]"},{"why":"NegCLIP, one of the source models NEAT adapts and a post-training baseline for comparison.","marker":"[47]"},{"why":"The LLM (Llama-3-8B) used to decompose negated captions into affirmative parts and generate reversed captions.","marker":"[9]"}],"fun_headline_variants":["Fix VLMs' negation blindness by tuning just normalization layers","NEAT: Test-time adaptation for negation understanding in VLMs","Adjusting LayerNorm at inference repairs negation in VLMs","Tiny tweak to text encoder handles 'not' in vision-language models","Test-time LayerNorm updates teach CLIP to grasp negation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the single 'hardest' visual sample for a reversed caption, meaning the image whose embedding is least similar to that caption, is a semantically sound contrastive target, so that pushing the reversed caption away from it teaches the model to separate opposites; if that sample is actually an unrelated image, the loss pulls the representation toward noise.","fun_headline_variants_meta":{"raw":{"variants":["Fix VLMs' negation blindness by tuning just normalization layers","NEAT: Test-time adaptation for negation understanding in VLMs","Adjusting LayerNorm at inference repairs negation in VLMs","Tiny tweak to text encoder handles 'not' in vision-language models","Test-time LayerNorm updates teach CLIP to grasp negation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000759,"raw_usage":{"total_tokens":3397,"prompt_tokens":994,"completion_tokens":2403,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2317}},"tokens_in":610,"tokens_out":2403,"duration_ms":17747,"temperature":1.0,"reasoning_tokens":2317,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:01:44.012386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run NEAT on a negation benchmark while replacing the hardest-negative selection in the reversion loss with a randomly chosen sample from the same batch; if retrieval and multiple-choice performance do not drop, the hardest-negative mechanism is not doing the claimed work, and the geometric diagnosis would need revision.","supporting_citations":[{"cited_title":"Test-time Adaptation for Cross-modal Retrieval with Query Shift","cited_arxiv_id":"2410.15624","evidence_quote":"Provides the refined candidate selection strategy NEAT borrows and serves as a test-time adaptation baseline it must beat."},{"cited_title":"The llama 3 herd of models","cited_arxiv_id":null,"evidence_quote":"The LLM (Llama-3-8B) used to decompose negated captions into affirmative parts and generate reversed captions."}],"review_version":2}