Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization

Liqiang Jing; Vibhav Gogate; Yue Zhang

REVIEW 3 major objections 10 minor 56 references

Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization

T0 review · 3 major / 10 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces Defeasible Visual Entailment, a benchmark in which an extra textual update can strengthen or weaken the entailment between an image and a hypothesis, and proposes an inference-aware evaluator whose scores track human…

desk verdict New DVE benchmark and evaluator are genuinely useful; the unvalidated ordering assumption in Eq. (7) is the main soft spot, but it is addressable. read the letter →

arxiv 2412.16232 v3 pith:4YBVXS7L submitted 2024-12-19 cs.CV cs.AIcs.LG

Yue Zhang , Liqiang Jing , Vibhav Gogate This is my paper · ORCID

classification cs.CVcs.AIcs.LG

keywords defeasiblevisualentailmentbenchmarkmultimodalreasoningreference-freeevaluationpairwisecontrastivelearningreward-drivenoptimizationlargevision-languagemodelsstrength

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Defeasible Visual Entailment (DVE) asks a model to take an image and a hypothesis, receive an additional textual update, and decide whether that update strengthens or weakens the claim. The paper builds the first DVE benchmark from existing resources—Flickr30k images, SNLI captions, and δ-NLI updates—and argues that standard generation metrics such as BLEU, ROUGE, BERTScore, and CLIPScore cannot capture the change in entailment strength that an update produces. To fix this, the paper trains a reference-free evaluator, without human strength labels, by contrasting the scores of strengtheners, captions, and weakeners. On a human-annotated test set the evaluator's scores correlate with human judgment (Pearson r = 0.826 for GPT-4o updates) far better than existing metrics, and the same score is then used as a critique signal to iteratively refine updates produced by large vision-language models, improving their measured quality. If the evaluator is right, it gives the field a cheap way to measure a kind of reasoning change that lexical-overlap metrics miss.

What carries the argument

The central object is the inference-aware evaluator, a scalar entailment-strength head s = W_s m + b_s applied to a multimodal representation m formed by concatenating a ResNet-50 image embedding with the BERT [CLS] embedding of the (hypothesis, update) text pair. It is trained with a pairwise contrastive loss (Eq. 7) built on the asserted ordering that every strengthener's score exceeds the score of the original caption, and every weakener's score falls below it, together with a cross-entropy classification loss over update type. This score does double duty: it serves as the reference-free evaluation metric for generated updates and as the critique signal that drives the iterative refinement loop of the reward-driven optimization method.

What would settle it

Collect human ratings of entailment strength for (image, hypothesis, caption) triples alongside the existing strengthener/weakener labels; if the mean human rating of strengtheners is not above the caption rating, or of weakeners not below it, on a meaningful fraction of a random DVE sample, then the pairwise contrastive signal (Eq. 7) enforces an ordering that does not hold and the evaluator score is not a trustworthy measure of update-induced change.

Watch

Extended reading notes

Core claim

The authors' central claim is that the effect of a textual update on a visual entailment can be captured by a single scalar strength score, and that this score can be learned without human scoring. The proposed inference-aware evaluator encodes the image with ResNet-50, encodes the (hypothesis, update) pair with BERT, concatenates the two, and maps the result to a score. Training uses two signals: a pairwise contrastive loss that, for each sample, pushes the strengthener's score above the caption's score and the weakener's below it (Eq. 7), and a cross-entropy classification loss that distinguishes strengtheners from weakeners. The authors report that this metric outperforms all compared metrics across Pearson, Spearman, and Kendall correlations with human judgments (e.g., r = 0.8262 for GPT-4o-generated updates vs. 0.1068 for CLIPScore), and that using the evaluator as a critique in a reward-driven refinement loop makes GPT-4o produce stronger strengtheners and weaker weakeners than GPT-4o alone (Table 3).

Load-bearing premise

The evaluator's training assumes that for every triplet, a strengthener's entailment strength is higher than the original caption's and a weakener's is lower, and this ordering is taken as evident rather than checked against human judgments.

Editorial extensions

If this is right

DVE supplies a benchmark on which models can be tested for fine-grained multimodal reasoning: 93k training triplets with balanced strengthener/weakener pairs over more than 9k images.
The evaluator gives a reference-free way to grade generated updates, so DVE generation no longer needs exhaustive reference sets or human scoring.
The reward-driven optimization loop shows that a learned strength score can act as a reward signal to improve LVLM outputs without additional human feedback.
Classification results expose a large gap between closed-source GPT-4o (81.76%) and open-source LVLMs (31-52% in zero-shot), suggesting current open models lack the reasoning needed for defeasible visual inference.
The weakener scores for almost all models are near zero or positive, indicating that most LVLMs default to captioning rather than generating genuine weakeners; only GPT-4o-based models produce effective ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the evaluator generalizes, the same architecture could transfer to other multimodal editing tasks—image caption correction, counterfactual image explanations, or VQA clarification—wherever a new sentence changes the relation between an image and another text.
The pairwise ordering assumption in Eq. (7) is the load-bearing premise; a natural extension is to validate it on a small human-rated probe set before trusting the metric in new domains.
The benchmark inherits the distribution of SNLI and δ-NLI, so the 'visual' defeasibility is mostly textual commonsense applied to images; a dataset with updates specifically written against image content would test whether the metric generalizes to genuinely visual uncertainty.
A testable extension is to use the evaluator's scalar score as a dense reward for RL fine-tuning of LVLMs, rather than the iterative critique-refinement loop used here.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this paper introduces a genuinely new task—Defeasible Visual Entailment (DVE)—with a benchmark, a reference-free evaluator, and a reward-driven optimization loop. The evaluator correlates with human judgments far better than existing metrics (r≈0.83 for GPT-4o, 0.77 for LLaVA), which is the strongest evidence in the paper. That alone makes it worth a serious look.

What's actually new: no prior work applies defeasible inference to the visual modality with strengtheners and weakeners. The benchmark is built by replacing SNLI text premises with their Flickr30k images and attaching δ-NLI updates, which is a cost-effective construction even if it inherits some noise. The evaluator is well-motivated—standard text-metric correlations are near zero or negative, so there is a real gap. The human annotation is reasonably careful (Fleiss κ ≈ 0.80), and code/data are released.

The main soft spot is the training signal for the evaluator. Equation (7) assumes the entailment strength of a strengthener exceeds that of the caption, which in turn exceeds that of a weakener. This ordering is inherited from text-only δ-NLI, where the premise was the caption itself. In DVE the premise is the image, and the caption is merely a description; the image may contain extra evidence or omit details, so the ordering is not guaranteed. The paper calls it 'evident' but never validates it on actual DVE triplets. Table 2's human correlation is on model-generated updates, which is indirect support but not a per-sample check of the training ordering. A modest human validation of the ordering would close this gap.

A second, milder concern is circularity in the optimization loop: Table 3 shows GPT-4o (Optimized) scoring higher on the evaluator, but that same evaluator is the reward. The independent human annotations in Table 7 do show improvement (strengthener 4.60→4.67; weakener 1.47→1.43), but the differences are tiny and no significance test is reported, so the claim that reward optimization genuinely improves quality is weaker than the paper's language suggests. The direction is consistent, though.

Minor issues: calling the training 'unsupervised' is inaccurate—the contrastive loss uses the strengthener/weakener labels, so it is supervised ranking. Also, α=0.9 means the classification loss dominates, leaving the strength head with a small weight; that is not necessarily bad but worth noting.

Overall, this is a solid, novel benchmark and metric paper. The central concerns are real but addressable, and the human correlation evidence means the evaluator is not just fitting its own assumptions. I would send this to peer review, expect heavy but bounded revision, and likely cite it as the DVE reference.

Referee Report

3 major / 10 minor

Summary. The paper introduces Defeasible Visual Entailment (DVE), a multimodal extension of defeasible NLI in which a textual update either strengthens or weakens the entailment between an image premise and a text hypothesis. The authors construct a DVE benchmark by replacing the text premises of δ-NLI with Flickr30k images, define classification and generation tasks, and propose an inference-aware evaluator trained with a pairwise contrastive loss and a classification loss. They also propose a reward-driven update optimization loop that uses this evaluator as a critic to refine LVLM-generated updates. Experiments report that the proposed evaluator correlates strongly with human judgments on GPT-4o and LLaVA-generated updates (Table 2), and that GPT-4o optimized with the proposed loop scores higher than unoptimized GPT-4o on the proposed metric (Table 3). Human evaluation of generated updates is also reported in the appendix (Table 7).

Significance. If the construction assumptions hold, the paper would make three useful contributions: a new multimodal defeasible reasoning benchmark, a reference-free evaluator that captures update-induced changes in entailment strength, and a reward-driven optimization method for update generation. The human-correlation results in Table 2 are genuinely encouraging: r=0.8262 for GPT-4o and r=0.7733 for LLaVA-1.5 are far above the comparison metrics, and the inter-annotator agreement (Fleiss' κ=80.4%) lends credibility to the evaluation protocol. The authors also release code and data, which supports reproducibility. However, the central validity of both the benchmark and the evaluator rests on an unvalidated ordering assumption in Eq. (7), and the optimization result in Table 3 is partly circular because the same evaluator is used as both the reward and the headline metric. These issues are addressable with additional validation and statistical reporting, so the work warrants a major revision rather than rejection.

major comments (3)

[Dataset Construction] The dataset construction replaces the text premise T in each δ-NLI pair with its corresponding Flickr30k image I, but the strengthener/weakener labels and the neutral caption T are inherited without any human validation that they remain valid for the image. This is load-bearing because every downstream claim—the classification benchmark, the evaluator training signal in Eq. (7), and the optimization target—depends on the assumption that an update that strengthens or weakens the text premise T also strengthens or weakens the image premise I. An image is not informationally equivalent to its caption: it may contain additional evidence relevant to the hypothesis, or it may fail to depict details asserted in T. Please report a human validation study on a random sample of DVE triplets, measuring what fraction of inherited labels are judged valid and whether the caption is indeed perceived as a neutral midpoint between the strengthener and weakener for the image.
[Eq. (7), Inference-aware Evaluator] The pairwise contrastive loss assumes that for every training triplet, s(strengthener) > s(caption) > s(weakener), and the paper calls this ordering 'evident' without testing it on the DVE data. This is the sole training signal for the entailment strength head, so if the ordering fails on a material fraction of triplets, the learned scalar score is not a reliable measure of update-induced entailment change, and both Table 2 and the optimization loop built on this score are compromised. The stress-test concern about this point is justified. Please provide a direct validation of the ordering, for example by having human annotators compare the entailment strength of (strengthener, I, H), (caption, I, H), and (weakener, I, H) on a sample of DVE triplets, or by reporting an alternative diagnostic that shows the ordering holds. If the ordering only holds on a subset, the loss in Eq. (7) should be modified to account for that.
[Table 3 and Figure 4, Reward-driven Update Optimization] The reported improvement of GPT-4o (Optimized) over GPT-4o in Table 3 is evaluated with the same inference-aware evaluator that serves as the reward signal in the optimization loop, so the gain on the 'Ours' column is partly by construction. The human evaluation in Table 7 mitigates this concern, but the human-score differences are small (4.6650 vs. 4.6001 for strengtheners; 1.4335 vs. 1.4680 for weakeners) and no significance tests, confidence intervals, or per-item breakdowns are reported. Please report whether the human-score improvements are statistically significant, and ideally provide human evaluation on an independent sample of updates generated outside the optimization loop. Without this, the claim that reward-driven optimization 'significantly enhances' update quality is not fully supported.

minor comments (10)

[General] The section heading 'Task Defination' contains a typo; it should read 'Task Definition'.
[Eq. (7)] The sentence preceding Eq. (7) contains a duplicated 'the' ('the the entailment strength'); please correct it.
[Table 3] The table header 'Weakner' is a typo for 'Weakener'.
[Table 2] The table caption says 'on VDI', but the paper's task is DVE; please correct the acronym.
[Eq. (8)] The dimensions in Eq. (8) are inconsistent: Wc is described as being in R^{d1+d2} but for a two-class output it should be in R^{2×(d1+d2)}, and bc ∈ R^2 is inconsistent with a scalar sigmoid output. Please clarify whether the output is two independent sigmoid logits or a softmax over two classes.
[Ablation for threshold and repetition] The threshold/round ablation in Table 6 reports only point estimates with no variance across seeds or a no-optimization baseline; reporting mean±std over multiple seeds would make the choice of η and M more convincing.
[Effect from Writing Styles] The writing-style robustness analysis uses only three strengtheners and three weakeners for a single image; this is anecdotal. Either expand the evaluation or explicitly frame it as a qualitative illustration rather than evidence of robustness.
[Eq. (10)] With α=0.9, the classification loss receives nine times the weight of the pairwise contrastive loss. Since the paper emphasizes pairwise contrastive learning as the key mechanism for learning entailment strength, please include an ablation over α or at least justify the chosen value.
[Supplementary Material] The data/code URL in the footnote contains a raw space ('https://github.com/skywalkerzhang/Defeasible Visual Entailment'); use a URL-encoded or hyphenated form so the link is accessible.
[Table 7] Table 7 reports human annotation scores but does not describe who the annotators were, whether they are the same as those in the main human evaluation, or what instruction/scale was used for this second annotation. Please add this information.

Circularity Check

1 steps flagged · score 6.0 of 10

Reward-driven optimization is rewarded and evaluated by the same learned evaluator; its reported improvement is partly by construction, while the evaluator and benchmark claims retain independent human-correlation support.

fitted input called prediction [Reward-driven Update Optimization (Critique/Refinement) and Results and Analysis, Generation Task, Table 3]
"Our inference-aware evaluator serves as the critique, assessing the entailment strength of the generated updates. If the critique assigns a low score, we proceed to the next step (i.e., Refinement) to improve the response. ... This process is repeated until the model produces a high-quality update (as defined in the critique step) or until the loop reaches a maximum iteration count of M. ... GPT-4o (Optimized), performs even better than GPT-4o alone."

The optimization loop's accept/refine criterion is the evaluator's own score (threshold eta; refine until a 'high-quality update' is produced). Table 3 then cites the same evaluator ('Ours') as evidence that 'GPT-4o (Optimized), performs even better than GPT-4o alone.' Thus the reported improvement on that metric is partly manufactured by the optimization procedure: it selects outputs that score higher under the very metric used for evaluation. The independent human annotation in Table 7 gives only a marginal strengthener improvement (4.6650 vs 4.6001) and a weakener regression (1.4335 vs 1.4680), so the headline optimization claim is not confirmed outside the reward loop.

full rationale

The main evaluator claim is not circular: the inference-aware evaluator is trained on DVE labels/orderings and then tested against external human annotations on GPT-4o and LLaVA outputs (Table 2; r=0.8262 vs GPT-4o), so it has independent empirical content. Eq. (7)'s 'evident' ordering is an unvalidated assumption and a correctness risk, but it is an input hypothesis rather than a derivation step that reduces to the conclusion. The one genuine circularity is the reward-driven optimization evaluation: the optimization objective and the headline evaluation metric are the same learned score, so the Table 3 gain of GPT-4o (Optimized) over GPT-4o under 'Ours' is in part by construction. This affects contribution 3 only; the benchmark and evaluator correlation results remain independently grounded. No load-bearing self-citations or imported uniqueness theorems appear.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claims rely on an unvalidated transfer of defeasibility labels from text premises to images, an unverified pairwise ordering as the sole training signal for the strength score, and a reward loop whose primary metric is the model being optimized.

free parameters (4)

loss weight alpha = 0.9
Eq. (10) balances pairwise contrastive loss and classification loss; chosen by hand and not swept in the main paper.
critique threshold eta = 1
In Reward-driven Update Optimization, updates with strengthener score below eta or weakener score above -eta are sent for refinement; chosen by hand.
max refinement rounds M = 3
Stops the refinement loop; chosen by hand.
evaluator heads (Ws, bs, Wc, bc) = learned on DVE train split
The scalar strength score and classification output are produced by learned linear heads on concatenated ResNet50 and BERT features; the metric is therefore an empirical model, not a fixed formula.

assumptions (4)

domain assumption Delta-NLI strengthener and weakener labels remain valid when the text premise is replaced by the corresponding Flickr30k image.
Used throughout Dataset Construction and for training the evaluator. No validation that the image supports the same entailment relation as the caption.
ad hoc to paper For every training triplet, strength(strengthener) > strength(caption) > strength(weakener).
Eq. (7) builds the pairwise contrastive loss on this ordering, described as 'evident'; no empirical check is reported.
domain assumption Human 5-point scores are a valid ground truth for update effect.
Fleiss kappa 80.4% is reported, but the scores are used as ground truth for correlation without further calibration.
domain assumption Pretrained ResNet50 and BERT features are sufficient for visual entailment strength.
The evaluator's architecture fixes these encoders and does not ablate alternative backbones.

invented entities (1)

entailment strength score s independent evidence
purpose: Serves as the reference-free evaluator of how an update changes the likelihood of a hypothesis given an image, and as the reward signal for update optimization.
The score is a learned latent scalar without a fixed meaning; its external validity rests on the reported human correlations and is only partially confirmed for the optimization loop.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization." pith.science (2026). https://pith.science/paper/4YBVXS7L

@misc{pith2026241216232,
  author       = {Pith},
  title        = {Pith review of: Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4YBVXS7L}},
  note         = {Machine review of arXiv:2412.16232}
}

read the original abstract

We introduce a new task called Defeasible Visual Entailment (DVE), where the goal is to allow the modification of the entailment relationship between an image premise and a text hypothesis based on an additional update. While this concept is well-established in Natural Language Inference, it remains unexplored in visual entailment. At a high level, DVE enables models to refine their initial interpretations, leading to improved accuracy and reliability in various applications such as detecting misleading information in images, enhancing visual question answering, and refining decision-making processes in autonomous systems. Existing metrics do not adequately capture the change in the entailment relationship brought by updates. To address this, we propose a novel inference-aware evaluator designed to capture changes in entailment strength induced by updates, using pairwise contrastive learning and categorical information learning. Additionally, we introduce a reward-driven update optimization method to further enhance the quality of updates generated by multimodal models. Experimental results demonstrate the effectiveness of our proposed evaluator and optimization method.

Figures

Figures reproduced from arXiv: 2412.16232 by the authors.

**Figure 2.** The workflow of generating the DVE dataset by [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** The architecture of our Inference-aware Evaluator, including three modules: Multimodal Embedding, Feature Fusion, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: An overview of Reward-driven Update Optimization, which includes three steps: Initial Response Generation, Cri [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 6.** Figure 6: Prompt used for Zero-shot Baselines in the Gener [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

**Figure 5.** Figure 5: Prompt used for the Classification Task across [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 7.** Figure 7: Prompt used for the Optimized Method [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: An example of evaluator scoring for different up [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

**Figure 9.** Figure 9: An example of the reward-driven update optimiza [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 45 canonical work pages

[1]

L.; and Parikh, D

Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Zitnick, C. L.; and Parikh, D. 2015. VQA: Visual Question Answering. In ICCV , 2425--2433. IEEE Computer Society

work page 2015
[2]

Bos, J.; and Markert, K. 2005. Recognising Textual Entailment with Logical Inference. In HLT/EMNLP , 628--635. The Association for Computational Linguistics

work page 2005
[3]

R.; Angeli, G.; Potts, C.; and Manning, C

Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In EMNLP , 632--642. The Association for Computational Linguistics

work page 2015
[4]

Chen, T.; Jiang, Z.; Poliak, A.; Sakaguchi, K.; and Durme, B. V. 2020 a . Uncertain Natural Language Inference. In ACL , 8772--8779. Association for Computational Linguistics

work page 2020
[5]

Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. E. 2020 b . A Simple Framework for Contrastive Learning of Visual Representations. In ICML , 1597--1607. PMLR

work page 2020
[6]

E.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J

Chen, Y.; Li, L.; Yu, L.; Kholy, A. E.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2020 c . UNITER: UNiversal Image-TExt Representation Learning. In ECCV , 104--120. Springer

work page 2020
[7]

Cui, S.; Milikic, L.; Feng, Y.; Ismayilzada, M.; Paul, D.; Bosselut, A.; and Faltings, B. 2024. \( \) -CAUSAL: Exploring Defeasibility in Causal Reasoning. CoRR, abs/2401.03183

work page Pith review arXiv 2024
[8]

Dagan, I.; Glickman, O.; and Magnini, B. 2005. The PASCAL Recognising Textual Entailment Challenge. In MLCW , volume 3944, 177--190. Springer

work page 2005

Show all 56 references

[9]

Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S. C. H. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In NeurIPS

2023
[10]

Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT , 4171--4186. Association for Computational Linguistics

2019
[11]

V.; Williams, C

Everingham, M.; Gool, L. V.; Williams, C. K. I.; Winn, J. M.; and Zisserman, A. 2010. The Pascal Visual Object Classes (VOC) Challenge. IJCV, 88: 303--338

2010
[12]

Gong, T.; Lyu, C.; Zhang, S.; Wang, Y.; Zheng, M.; Zhao, Q.; Liu, K.; Zhang, W.; Luo, P.; and Chen, K. 2023. MultiModal-GPT: A Vision and Language Model for Dialogue with Humans. CoRR, abs/2305.04790

2023 arXiv
[13]

Goyal, Y.; Khot, T.; Summers - Stay, D.; Batra, D.; and Parikh, D. 2017. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In CVPR , 6325--6334. IEEE Computer Society

2017
[14]

Han, X.; You, Q.; Liu, Y.; Chen, W.; Zheng, H.; Mrini, K.; Lin, X.; Wang, Y.; Zhai, B.; Yuan, J.; Wang, H.; and Yang, H. 2023. InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models

2023
[15]

He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In CVPR . IEEE Computer Society

2016
[16]

L.; and Choi, Y

Hessel, J.; Holtzman, A.; Forbes, M.; Bras, R. L.; and Choi, Y. 2021. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In EMNLP . Association for Computational Linguistics

2021
[17]

L.; and Girshick, R

Johnson, J.; Hariharan, B.; van der Maaten, L.; Fei - Fei, L.; Zitnick, C. L.; and Girshick, R. B. 2017. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In CVPR , 1988--1997. IEEE Computer Society

2017
[18]

Kazemi, M.; Yuan, Q.; Bhatia, D.; Kim, N.; Xu, X.; Imbrasaite, V.; and Ramachandran, D. 2023. BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information. In NeurIPS 2023

2023
[19]

Kim, W.; Son, B.; and Kim, I. 2021. ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. In ICML , volume 139 of Proceedings of Machine Learning Research, 5583--5594. PMLR

2021
[20]

P.; and Ba, J

Kingma, D. P.; and Ba, J. 2015. Adam: A Method for Stochastic Optimization. In ICLR

2015
[21]

Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images. Technical report, Toronto, ON, Canada

2009
[22]

Y.; Shen, M.; Zhou, W.; Zhou, P.; Bhagavatula, C.; Choi, Y.; and Ren, X

Lin, B. Y.; Shen, M.; Zhou, W.; Zhou, P.; Bhagavatula, C.; Choi, Y.; and Ren, X. 2020. CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning. In AKBC

2020
[23]

Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74--81

2004
[24]

J.; Hays, J.; Perona, P.; Ramanan, D.; Doll \' a r, P.; and Zitnick, C

Lin, T.; Maire, M.; Belongie, S. J.; Hays, J.; Perona, P.; Ramanan, D.; Doll \' a r, P.; and Zitnick, C. L. 2014. Microsoft COCO: Common Objects in Context. In ECCV 2014 , 740--755. Springer

2014
[25]

Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. In NeurIPS

2023
[26]

Lu, P.; Qiu, L.; Chen, J.; Xia, T.; Zhao, Y.; Zhang, W.; Yu, Z.; Liang, X.; and Zhu, S. 2021. IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning. In NeurIPS

2021
[27]

MacCartney, B.; and Manning, C. D. 2009. An extended model of natural logic. In IWCS , 140--156. Association for Computational Linguistics

2009
[28]

Mathew, M.; Bagal, V.; Tito, R.; Karatzas, D.; Valveny, E.; and Jawahar, C. V. 2022. InfographicVQA. In WACV , 2582--2591. IEEE

2022
[29]

B.; and Lapata, M

Narayan, S.; Cohen, S. B.; and Lapata, M. 2018. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In EMNLP, 1797--1807. Association for Computational Linguistics

2018
[30]

Nie, Y.; Williams, A.; Dinan, E.; Bansal, M.; Weston, J.; and Kiela, D. 2020. Adversarial NLI: A New Benchmark for Natural Language Understanding. In ACL , 4885--4901. Association for Computational Linguistics

2020
[31]

Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In ACL, 311--318. ACL

2002
[32]

Pavlick, E.; and Kwiatkowski, T. 2019. Inherent Disagreements in Human Textual Inferences. TACL, 7: 677--694

2019
[33]

W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML , volume 139, 8748--8763. PMLR

2021
[34]

Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. SQuAD: 100, 000+ Questions for Machine Comprehension of Text. In EMNLP , 2383--2392. The Association for Computational Linguistics

2016
[35]

D.; Bhagavatula, C.; Forbes, M.; Bras, R

Rudinger, R.; Shwartz, V.; Hwang, J. D.; Bhagavatula, C.; Forbes, M.; Bras, R. L.; Smith, N. A.; and Choi, Y. 2020. Thinking Like a Skeptic: Defeasible Inference in Natural Language. In EMNLP , 4661--4675. Association for Computational Linguistics

2020
[36]

S.; Berg, A

Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M. S.; Berg, A. C.; and Fei - Fei, L. 2015. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115: 211--252

2015
[37]

Sakaguchi, K.; and Durme, B. V. 2018. Efficient Online Scalar Annotation with Bounded Support. In ACL , 208--218. Association for Computational Linguistics

2018
[38]

Sidorov, O.; Hu, R.; Rohrbach, M.; and Singh, A. 2020. TextCaps: A Dataset for Image Captioning with Reading Comprehension. In ECCV , 742--758. Springer

2020
[39]

Singh, A.; Hu, R.; Goswami, V.; Couairon, G.; Galuba, W.; Rohrbach, M.; and Kiela, D. 2022. FLAVA: A Foundational Language And Vision Alignment Model. In CVPR , 15617--15629. IEEE

2022
[40]

Talman, A.; C elikkanat, H.; Virpioja, S.; Heinonen, M.; and Tiedemann, J. 2023. Uncertainty-Aware Natural Language Inference with Stochastic Weight Averaging. In Alum \" a e, T.; and Fishel, M., eds., NoDaLiDa, 358--365. University of Tartu Library

2023
[41]

Thrush, T.; Jiang, R.; Bartolo, M.; Singh, A.; Williams, A.; Kiela, D.; and Ross, C. 2022. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. In CVPR , 5228--5238. IEEE

2022
[42]

Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2019. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In ICLR . OpenReview.net

2019
[43]

Wang, P.; Yang, A.; Men, R.; Lin, J.; Bai, S.; Li, Z.; Ma, J.; Zhou, C.; Zhou, J.; and Yang, H. 2022. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework. In ICML , 23318--23340. PMLR

2022
[44]

Williams, A.; Nangia, N.; and Bowman, S. R. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In NAACL-HLT , 1112--1122. Association for Computational Linguistics

2018
[45]

Wu, R.; Ma, X.; Li, Q.; Wang, W.; Zhang, Z.; Zhu, S.; and Wang, Y. 2023. Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World. CoRR, abs/2310.10207

2023 arXiv
[46]

Xie, N.; Lai, F.; Doran, D.; and Kadav, A. 2019. Visual Entailment: A Novel Task for Fine-Grained Image Understanding. CoRR, abs/1901.06706

2019 arXiv
[47]

Ye, Q.; Xu, H.; Xu, G.; Ye, J.; Yan, M.; Zhou, Y.; Wang, J.; Hu, A.; Shi, P.; Shi, Y.; Li, C.; Xu, Y.; Chen, H.; Tian, J.; Qi, Q.; Zhang, J.; and Huang, F. 2023. mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality. CoRR, abs/2304.14178

2023 arXiv
[48]

Young, P.; Lai, A.; Hodosh, M.; and Hockenmaier, J. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2: 67--78

2014
[49]

Yu, J.; Wang, Z.; Vasudevan, V.; Yeung, L.; Seyedhosseini, M.; and Wu, Y. 2022. CoCa: Contrastive Captioners are Image-Text Foundation Models. TMLR

2022
[50]

Zellers, R.; Bisk, Y.; Schwartz, R.; and Choi, Y. 2018. SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In EMNLP, 93--104. Association for Computational Linguistics

2018
[51]

Zhang, S.; Rudinger, R.; Duh, K.; and Durme, B. V. 2017. Ordinal Common-sense Inference. TACL, 5: 379--395

2017
[52]

Q.; and Artzi, Y

Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2020. BERTScore: Evaluating Text Generation with BERT . In ICLR . OpenReview.net

2020
[53]

Zhou, B.; Zhao, H.; Puig, X.; Fidler, S.; Barriuso, A.; and Torralba, A. 2017. Scene Parsing through ADE20K Dataset. In CVPR , 5122--5130. IEEE Computer Society

2017
[54]

Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. CoRR, abs/2304.10592

2023 arXiv
[55]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
[56]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

L.; and Parikh, D

Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Zitnick, C. L.; and Parikh, D. 2015. VQA: Visual Question Answering. In ICCV , 2425--2433. IEEE Computer Society

work page 2015

[2] [2]

Bos, J.; and Markert, K. 2005. Recognising Textual Entailment with Logical Inference. In HLT/EMNLP , 628--635. The Association for Computational Linguistics

work page 2005

[3] [3]

R.; Angeli, G.; Potts, C.; and Manning, C

Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In EMNLP , 632--642. The Association for Computational Linguistics

work page 2015

[4] [4]

Chen, T.; Jiang, Z.; Poliak, A.; Sakaguchi, K.; and Durme, B. V. 2020 a . Uncertain Natural Language Inference. In ACL , 8772--8779. Association for Computational Linguistics

work page 2020

[5] [5]

Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. E. 2020 b . A Simple Framework for Contrastive Learning of Visual Representations. In ICML , 1597--1607. PMLR

work page 2020

[6] [6]

E.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J

Chen, Y.; Li, L.; Yu, L.; Kholy, A. E.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2020 c . UNITER: UNiversal Image-TExt Representation Learning. In ECCV , 104--120. Springer

work page 2020

[7] [7]

Cui, S.; Milikic, L.; Feng, Y.; Ismayilzada, M.; Paul, D.; Bosselut, A.; and Faltings, B. 2024. \( \) -CAUSAL: Exploring Defeasibility in Causal Reasoning. CoRR, abs/2401.03183

work page Pith review arXiv 2024

[8] [8]

Dagan, I.; Glickman, O.; and Magnini, B. 2005. The PASCAL Recognising Textual Entailment Challenge. In MLCW , volume 3944, 177--190. Springer

work page 2005