{"id":"9c90bf40-3730-4da2-9af9-c590bf8e7842","arxiv_id":"2412.16232","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new benchmark and inference-aware evaluator measure how textual updates strengthen or weaken image-text entailment, with a reward-driven loop that improves generated updates.","lead":"This paper introduces Defeasible Visual Entailment, a task in which a textual update can strengthen or weaken whether an image supports a hypothesis, and builds a benchmark from existing NLI and image-caption datasets. It also proposes a learned evaluator that scores update strength and uses that score as a reward to refine vision-language model outputs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scalar evaluator's only training signal is the unvalidated ordering in Eq. (7); if it fails for a material fraction of DVE triplets, both the metric and the reward loop built on it lose their foundation.","rationale":"The reader's weakest-assumption analysis identifies Eq. (7)'s pairwise ordering as the central unvalidated premise, and I agree that this is the most load-bearing concern. The entire contribution rests on the evaluator score: it is both the proposed metric and the reward signal for optimization. If the ordering assumption fails on a nontrivial fraction of DVE samples, the scalar score is not a faithful measure of update-induced entailment change, and the optimization loop is chasing a miscalibrated target.\n\nI considered whether the circularity of using the same evaluator as reward and evaluation metric is a stronger concern. It is real, but Table 7 does provide an independent human annotation of the generated updates, so the circularity is partial rather than complete. That said, the Table 7 differences are small and no statistical significance is reported, so it does not fully rescue the optimization claim. I therefore keep the reader's conditional verdict rather than moving to reject: the concern is addressable with a targeted human-ordering study, and the paper already contains substantial positive evidence, notably the high human correlation in Table 2 and the reference-free nature of the metric.\n\nThe concrete test I propose directly settles whether the Eq. (7) ordering holds in the multimodal setting, which is the load-bearing assumption. It is feasible with a modest annotation effort and would either confirm the evaluator's foundation or show that it needs re-training with additional constraints (for example, including an image-only baseline score in the contrastive loss).","tokens_in":16418,"tokens_out":5203,"duration_ms":48980,"concrete_test":"Sample 200 triplets from the DVE training set (100 strengtheners, 100 weakeners). For each, ask at least three annotators to rate the likelihood of the hypothesis under three conditions: image alone, image plus original caption, and image plus update (or, equivalently, ask whether the update makes H more or less likely than the image alone and whether the caption does so). Compute the violation rate: for strengtheners, the fraction where human-rated strength(update) <= human-rated strength(caption); for weakeners, the fraction where human-rated strength(update) >= human-rated strength(caption), treating 'image alone' as the neutral anchor. If the violation rate exceeds roughly 10%, the Eq. (7) contrastive signal is unreliable; a stronger check would then retrain the evaluator only on confirmed-order samples and re-run the Table 2 correlation to see whether the claimed advantage persists.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The inference-aware evaluator is trained with the pairwise contrastive loss in Eq. (7), which assumes that for every DVE triplet, entailment strength satisfies s(strengthener) > s(caption) > s(weakener). The paper calls this ordering 'evident' and provides no validation on the actual DVE data. This is the load-bearing step because the evaluator's scalar score is the sole measure used for both the headline correlation results (Table 2) and as the reward in reward-driven update optimization.\n\nThe ordering was inherited from δ-NLI, where updates were collected relative to a text premise T. In DVE, the premise is the image I, and the caption T is merely a text description of I. An image is not informationally equivalent to its caption: it can contain additional evidence relevant to H, and it can fail to depict details asserted in T. Therefore H's likelihood given I alone need not equal H's likelihood given T, and the caption is not guaranteed to be a neutral midpoint. If a strengthener is already entailed by the image alone, or a weakener is inconsistent with what the image actually shows, the assumed ordering can flip. The paper reports no human check of this ordering on the DVE training distribution.\n\nTable 2 shows that the final evaluator correlates with human judgments on GPT-4o and LLaVA-generated updates, which is genuine supporting evidence, but it does not verify the per-sample ordering used to train the scalar score; a model can have good average correlation while systematically misordering a meaningful subset. Moreover, the optimization improvement in Table 3 is measured primarily by the same evaluator used as the reward, so it is partly circular. The independent human annotations in Table 7 show only tiny differences (strengthener: 4.6001 vs. 4.6650; weakener: 1.4680 vs. 1.4335) with no significance test reported, which is too weak to rescue the loop on its own.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Defeasible Visual Entailment (DVE), a multimodal extension of defeasible NLI in which a textual update either strengthens or weakens the entailment between an image premise and a text hypothesis. The authors construct a DVE benchmark by replacing the text premises of δ-NLI with Flickr30k images, define classification and generation tasks, and propose an inference-aware evaluator trained with a pairwise contrastive loss and a classification loss. They also propose a reward-driven update optimization loop that uses this evaluator as a critic to refine LVLM-generated updates. Experiments report that the proposed evaluator correlates strongly with human judgments on GPT-4o and LLaVA-generated updates (Table 2), and that GPT-4o optimized with the proposed loop scores higher than unoptimized GPT-4o on the proposed metric (Table 3). Human evaluation of generated updates is also reported in the appendix (Table 7).","tokens_in":16763,"tokens_out":4648,"duration_ms":40783,"significance":"If the construction assumptions hold, the paper would make three useful contributions: a new multimodal defeasible reasoning benchmark, a reference-free evaluator that captures update-induced changes in entailment strength, and a reward-driven optimization method for update generation. The human-correlation results in Table 2 are genuinely encouraging: r=0.8262 for GPT-4o and r=0.7733 for LLaVA-1.5 are far above the comparison metrics, and the inter-annotator agreement (Fleiss' κ=80.4%) lends credibility to the evaluation protocol. The authors also release code and data, which supports reproducibility. However, the central validity of both the benchmark and the evaluator rests on an unvalidated ordering assumption in Eq. (7), and the optimization result in Table 3 is partly circular because the same evaluator is used as both the reward and the headline metric. These issues are addressable with additional validation and statistical reporting, so the work warrants a major revision rather than rejection.","major_comments":[{"comment":"The dataset construction replaces the text premise T in each δ-NLI pair with its corresponding Flickr30k image I, but the strengthener/weakener labels and the neutral caption T are inherited without any human validation that they remain valid for the image. This is load-bearing because every downstream claim—the classification benchmark, the evaluator training signal in Eq. (7), and the optimization target—depends on the assumption that an update that strengthens or weakens the text premise T also strengthens or weakens the image premise I. An image is not informationally equivalent to its caption: it may contain additional evidence relevant to the hypothesis, or it may fail to depict details asserted in T. Please report a human validation study on a random sample of DVE triplets, measuring what fraction of inherited labels are judged valid and whether the caption is indeed perceived as a neutral midpoint between the strengthener and weakener for the image.","section":"Dataset Construction"},{"comment":"The pairwise contrastive loss assumes that for every training triplet, s(strengthener) > s(caption) > s(weakener), and the paper calls this ordering 'evident' without testing it on the DVE data. This is the sole training signal for the entailment strength head, so if the ordering fails on a material fraction of triplets, the learned scalar score is not a reliable measure of update-induced entailment change, and both Table 2 and the optimization loop built on this score are compromised. The stress-test concern about this point is justified. Please provide a direct validation of the ordering, for example by having human annotators compare the entailment strength of (strengthener, I, H), (caption, I, H), and (weakener, I, H) on a sample of DVE triplets, or by reporting an alternative diagnostic that shows the ordering holds. If the ordering only holds on a subset, the loss in Eq. (7) should be modified to account for that.","section":"Eq. (7), Inference-aware Evaluator"},{"comment":"The reported improvement of GPT-4o (Optimized) over GPT-4o in Table 3 is evaluated with the same inference-aware evaluator that serves as the reward signal in the optimization loop, so the gain on the 'Ours' column is partly by construction. The human evaluation in Table 7 mitigates this concern, but the human-score differences are small (4.6650 vs. 4.6001 for strengtheners; 1.4335 vs. 1.4680 for weakeners) and no significance tests, confidence intervals, or per-item breakdowns are reported. Please report whether the human-score improvements are statistically significant, and ideally provide human evaluation on an independent sample of updates generated outside the optimization loop. Without this, the claim that reward-driven optimization 'significantly enhances' update quality is not fully supported.","section":"Table 3 and Figure 4, Reward-driven Update Optimization"}],"minor_comments":[{"comment":"The section heading 'Task Defination' contains a typo; it should read 'Task Definition'.","section":"General"},{"comment":"The sentence preceding Eq. (7) contains a duplicated 'the' ('the the entailment strength'); please correct it.","section":"Eq. (7)"},{"comment":"The table header 'Weakner' is a typo for 'Weakener'.","section":"Table 3"},{"comment":"The table caption says 'on VDI', but the paper's task is DVE; please correct the acronym.","section":"Table 2"},{"comment":"The dimensions in Eq. (8) are inconsistent: Wc is described as being in R^{d1+d2} but for a two-class output it should be in R^{2×(d1+d2)}, and bc ∈ R^2 is inconsistent with a scalar sigmoid output. Please clarify whether the output is two independent sigmoid logits or a softmax over two classes.","section":"Eq. (8)"},{"comment":"The threshold/round ablation in Table 6 reports only point estimates with no variance across seeds or a no-optimization baseline; reporting mean±std over multiple seeds would make the choice of η and M more convincing.","section":"Ablation for threshold and repetition"},{"comment":"The writing-style robustness analysis uses only three strengtheners and three weakeners for a single image; this is anecdotal. Either expand the evaluation or explicitly frame it as a qualitative illustration rather than evidence of robustness.","section":"Effect from Writing Styles"},{"comment":"With α=0.9, the classification loss receives nine times the weight of the pairwise contrastive loss. Since the paper emphasizes pairwise contrastive learning as the key mechanism for learning entailment strength, please include an ablation over α or at least justify the chosen value.","section":"Eq. (10)"},{"comment":"The data/code URL in the footnote contains a raw space ('https://github.com/skywalkerzhang/Defeasible Visual Entailment'); use a URL-encoded or hyphenated form so the link is accessible.","section":"Supplementary Material"},{"comment":"Table 7 reports human annotation scores but does not describe who the annotators were, whether they are the same as those in the main human evaluation, or what instruction/scale was used for this second annotation. Please add this information.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The core ideas are timely and the human-correlation evidence is strong, but the manuscript currently lacks validation of the label-transfer assumption in the dataset and of the ordering assumption in Eq. (7). These are not just presentation issues; they are load-bearing for the benchmark and evaluator. I would be willing to accept a revised version that provides the requested validation and significance testing, or that appropriately weakens the corresponding claims if the validation is only partial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this paper introduces a genuinely new task—Defeasible Visual Entailment (DVE)—with a benchmark, a reference-free evaluator, and a reward-driven optimization loop. The evaluator correlates with human judgments far better than existing metrics (r≈0.83 for GPT-4o, 0.77 for LLaVA), which is the strongest evidence in the paper. That alone makes it worth a serious look.\n\nWhat's actually new: no prior work applies defeasible inference to the visual modality with strengtheners and weakeners. The benchmark is built by replacing SNLI text premises with their Flickr30k images and attaching δ-NLI updates, which is a cost-effective construction even if it inherits some noise. The evaluator is well-motivated—standard text-metric correlations are near zero or negative, so there is a real gap. The human annotation is reasonably careful (Fleiss κ ≈ 0.80), and code/data are released.\n\nThe main soft spot is the training signal for the evaluator. Equation (7) assumes the entailment strength of a strengthener exceeds that of the caption, which in turn exceeds that of a weakener. This ordering is inherited from text-only δ-NLI, where the premise was the caption itself. In DVE the premise is the image, and the caption is merely a description; the image may contain extra evidence or omit details, so the ordering is not guaranteed. The paper calls it 'evident' but never validates it on actual DVE triplets. Table 2's human correlation is on model-generated updates, which is indirect support but not a per-sample check of the training ordering. A modest human validation of the ordering would close this gap.\n\nA second, milder concern is circularity in the optimization loop: Table 3 shows GPT-4o (Optimized) scoring higher on the evaluator, but that same evaluator is the reward. The independent human annotations in Table 7 do show improvement (strengthener 4.60→4.67; weakener 1.47→1.43), but the differences are tiny and no significance test is reported, so the claim that reward optimization genuinely improves quality is weaker than the paper's language suggests. The direction is consistent, though.\n\nMinor issues: calling the training 'unsupervised' is inaccurate—the contrastive loss uses the strengthener/weakener labels, so it is supervised ranking. Also, α=0.9 means the classification loss dominates, leaving the strength head with a small weight; that is not necessarily bad but worth noting.\n\nOverall, this is a solid, novel benchmark and metric paper. The central concerns are real but addressable, and the human correlation evidence means the evaluator is not just fitting its own assumptions. I would send this to peer review, expect heavy but bounded revision, and likely cite it as the DVE reference.","headline":"New DVE benchmark and evaluator are genuinely useful; the unvalidated ordering assumption in Eq. (7) is the main soft spot, but it is addressable.","tokens_in":17327,"tokens_out":3295,"would_cite":true,"duration_ms":27363,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces Defeasible Visual Entailment, a benchmark in which an extra textual update can strengthen or weaken the entailment between an image and a hypothesis, and proposes an inference-aware evaluator whose scores track human…","keywords":["defeasible visual entailment","visual entailment benchmark","multimodal reasoning","reference-free evaluation","pairwise contrastive learning","reward-driven optimization","large vision-language models","entailment strength"],"falsifier":"Collect human ratings of entailment strength for (image, hypothesis, caption) triples alongside the existing strengthener/weakener labels; if the mean human rating of strengtheners is not above the caption rating, or of weakeners not below it, on a meaningful fraction of a random DVE sample, then the pairwise contrastive signal (Eq. 7) enforces an ordering that does not hold and the evaluator score is not a trustworthy measure of update-induced change.","tokens_in":16228,"feed_emoji":"🖼️","tokens_out":7178,"duration_ms":47523,"temperature":0.7,"pith_summary":"Defeasible Visual Entailment (DVE) asks a model to take an image and a hypothesis, receive an additional textual update, and decide whether that update strengthens or weakens the claim. The paper builds the first DVE benchmark from existing resources—Flickr30k images, SNLI captions, and δ-NLI updates—and argues that standard generation metrics such as BLEU, ROUGE, BERTScore, and CLIPScore cannot capture the change in entailment strength that an update produces. To fix this, the paper trains a reference-free evaluator, without human strength labels, by contrasting the scores of strengtheners, captions, and weakeners. On a human-annotated test set the evaluator's scores correlate with human judgment (Pearson r = 0.826 for GPT-4o updates) far better than existing metrics, and the same score is then used as a critique signal to iteratively refine updates produced by large vision-language models, improving their measured quality. If the evaluator is right, it gives the field a cheap way to measure a kind of reasoning change that lexical-overlap metrics miss.","feed_headline":"New evaluator captures how extra info changes visual entailment","feed_subtitle":"The inference-aware score beats BLEU, ROUGE, and CLIP at judging whether updates strengthen or weaken image claims.","key_machinery":"The central object is the inference-aware evaluator, a scalar entailment-strength head s = W_s m + b_s applied to a multimodal representation m formed by concatenating a ResNet-50 image embedding with the BERT [CLS] embedding of the (hypothesis, update) text pair. It is trained with a pairwise contrastive loss (Eq. 7) built on the asserted ordering that every strengthener's score exceeds the score of the original caption, and every weakener's score falls below it, together with a cross-entropy classification loss over update type. This score does double duty: it serves as the reference-free evaluation metric for generated updates and as the critique signal that drives the iterative refinement loop of the reward-driven optimization method.","core_discovery":"The authors' central claim is that the effect of a textual update on a visual entailment can be captured by a single scalar strength score, and that this score can be learned without human scoring. The proposed inference-aware evaluator encodes the image with ResNet-50, encodes the (hypothesis, update) pair with BERT, concatenates the two, and maps the result to a score. Training uses two signals: a pairwise contrastive loss that, for each sample, pushes the strengthener's score above the caption's score and the weakener's below it (Eq. 7), and a cross-entropy classification loss that distinguishes strengtheners from weakeners. The authors report that this metric outperforms all compared metrics across Pearson, Spearman, and Kendall correlations with human judgments (e.g., r = 0.8262 for GPT-4o-generated updates vs. 0.1068 for CLIPScore), and that using the evaluator as a critique in a reward-driven refinement loop makes GPT-4o produce stronger strengtheners and weaker weakeners than GPT-4o alone (Table 3).","pith_inferences":["If the evaluator generalizes, the same architecture could transfer to other multimodal editing tasks—image caption correction, counterfactual image explanations, or VQA clarification—wherever a new sentence changes the relation between an image and another text.","The pairwise ordering assumption in Eq. (7) is the load-bearing premise; a natural extension is to validate it on a small human-rated probe set before trusting the metric in new domains.","The benchmark inherits the distribution of SNLI and δ-NLI, so the 'visual' defeasibility is mostly textual commonsense applied to images; a dataset with updates specifically written against image content would test whether the metric generalizes to genuinely visual uncertainty.","A testable extension is to use the evaluator's scalar score as a dense reward for RL fine-tuning of LVLMs, rather than the iterative critique-refinement loop used here."],"forward_implications":["DVE supplies a benchmark on which models can be tested for fine-grained multimodal reasoning: 93k training triplets with balanced strengthener/weakener pairs over more than 9k images.","The evaluator gives a reference-free way to grade generated updates, so DVE generation no longer needs exhaustive reference sets or human scoring.","The reward-driven optimization loop shows that a learned strength score can act as a reward signal to improve LVLM outputs without additional human feedback.","Classification results expose a large gap between closed-source GPT-4o (81.76%) and open-source LVLMs (31-52% in zero-shot), suggesting current open models lack the reasoning needed for defeasible visual inference.","The weakener scores for almost all models are near zero or positive, indicating that most LVLMs default to captioning rather than generating genuine weakeners; only GPT-4o-based models produce effective ones."],"supporting_citations":[{"why":"Supplies the δ-NLI dataset whose updates and neutral premise-hypothesis pairs form the core of the DVE benchmark.","marker":"Rudinger et al. 2020"},{"why":"Supplies Flickr30k images that replace text premises in the DVE construction.","marker":"Young et al. 2014"},{"why":"Supplies the SNLI captions and hypotheses from which premise-hypothesis pairs are drawn.","marker":"Bowman et al. 2015"},{"why":"Defines the Visual Entailment task that DVE extends with defeasible updates.","marker":"Xie et al. 2019"},{"why":"CLIPScore is a reference-free baseline that the new evaluator must beat in correlation with human judgments.","marker":"Hessel et al. 2021"},{"why":"ROUGE-L is one of the lexical-overlap baselines shown to fail on DVE updates.","marker":"Lin 2004"},{"why":"Provides the contrastive learning framing that motivates the pairwise contrastive loss.","marker":"Chen et al. 2020b"},{"why":"Provides BERT, used to encode the hypothesis-update text pair.","marker":"Devlin et al. 2019"},{"why":"Provides ResNet-50, used to extract visual embeddings for the evaluator.","marker":"He et al. 2016"}],"fun_headline_variants":["New scorer captures how extra info shifts visual claims","Defeasible visual entailment: benchmark and inference-aware evaluator","Reward-driven loop makes updates better at altering image entailment","Metric beats BLEU, ROUGE, and CLIP on update strength","Single learned score predicts human judgment on updates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluator's training assumes that for every triplet, a strengthener's entailment strength is higher than the original caption's and a weakener's is lower, and this ordering is taken as evident rather than checked against human judgments.","fun_headline_variants_meta":{"raw":{"variants":["New scorer captures how extra info shifts visual claims","Defeasible visual entailment: benchmark and inference-aware evaluator","Reward-driven loop makes updates better at altering image entailment","Metric beats BLEU, ROUGE, and CLIP on update strength","Single learned score predicts human judgment on updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1532,"prompt_tokens":942,"completion_tokens":590,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":506}},"tokens_in":558,"tokens_out":590,"duration_ms":5141,"temperature":1.0,"reasoning_tokens":506,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:12:08.145299+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect human ratings of entailment strength for (image, hypothesis, caption) triples alongside the existing strengthener/weakener labels; if the mean human rating of strengtheners is not above the caption rating, or of weakeners not below it, on a meaningful fraction of a random DVE sample, then the pairwise contrastive signal (Eq. 7) enforces an ordering that does not hold and the evaluator score is not a trustworthy measure of update-induced change.","supporting_citations":[{"cited_title":"D.; Bhagavatula, C.; Forbes, M.; Bras, R","cited_arxiv_id":null,"evidence_quote":"Supplies the δ-NLI dataset whose updates and neutral premise-hypothesis pairs form the core of the DVE benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Flickr30k images that replace text premises in the DVE construction."},{"cited_title":"R.; Angeli, G.; Potts, C.; and Manning, C","cited_arxiv_id":null,"evidence_quote":"Supplies the SNLI captions and hypotheses from which premise-hypothesis pairs are drawn."},{"cited_title":"L.; and Choi, Y","cited_arxiv_id":null,"evidence_quote":"CLIPScore is a reference-free baseline that the new evaluator must beat in correlation with human judgments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BERT, used to encode the hypothesis-update text pair."}],"review_version":1}