Pith. sign in

REVIEW 3 major objections 5 minor 57 references

Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces BlackSwanSuite, a benchmark of 15,469 video-reasoning questions, and reports that state-of-the-art vision-language models lag human performance by up to 32 percentage points when reasoning about unexpected events.

desk verdict A genuinely useful video reasoning benchmark whose headline human-model gaps are probably right in direction but softer than reported due to single-annotator labels and a max-of-two human baseline. read the letter →

arxiv 2412.05725 v2 pith:QNC7TQRX submitted 2024-12-07 cs.CV cs.AI

Aditya Chinchure , Sahithya Ravi , Raymond Ng , Vered Shwartz , Boyang Li , Leonid Sigal This is my paper · ORCID
classification cs.CVcs.AI
keywords videoreasoningbenchmarkabductivedefeasiblevision-languagemodelsunexpectedeventscommonsensemultiple-choicequestionansweringyes/novalidation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BlackSwanSuite is a new benchmark for testing whether vision-language models can reason about unexpected events rather than recall patterns from training data. Each of 1,655 videos is split into before, main-event, and after segments, and the benchmark asks models to forecast what happens next, infer a hidden event from its surrounding context (abductive reasoning), and revise or confirm hypotheses when new video evidence appears (defeasible reasoning). Across 15,469 questions in multiple-choice, yes/no, and generative formats, the paper finds that the best evaluated models, including GPT-4o and Gemini 1.5 Pro, trail human experts by up to 32 percentage points on yes/no tasks and by about 25 points on abductive multiple-choice questions. The point of the benchmark is that atypical events are exactly where statistical recall fails, so the gap measures genuine reasoning limits rather than memorization.

What carries the argument

The load-bearing mechanism is the three-part narrative decomposition of each video into $V_{\text{pre}}$, $V_{\text{main}}$, and $V_{\text{post}}$, combined with a staged annotation process. Annotators first see only $V_{\text{pre}}$ and propose what happens next; then they see $V_{\text{post}}$ and mark those hypotheses valid or invalid while writing abductive explanations for the hidden $V_{\text{main}}$; finally they see the full video and again validate or invalidate each hypothesis. Those validation labels become the ground truth for yes/no questions, the invalidated hypotheses become multiple-choice distractors, and the final explanations become the generative references. This structure turns defeasible reasoning into a concrete measurable operation: each hypothesis carries a timestamped defeasibility status that changes as visual evidence increases.

What would settle it

Re-annotate a random sample of, say, 200 Detective–Y/N questions with five independent annotators each and measure agreement with the original single-annotator labels; if agreement is low or if a majority human vote moves the labels, the reported 32-point gap would need to be recomputed. A simpler check: run the human baseline with randomly chosen single annotators instead of the best of two experts; if the human score drops materially, the headline gap would be inflated.

Watch

Extended reading notes

Core claim

The paper's central claim is that state-of-the-art VLMs cannot reliably perform abductive and defeasible reasoning about expectation-violating video events. To demonstrate this, the authors built BlackSwanSuite from 1,655 YouTube fail videos, each manually divided into pre-event, main event, and post-event parts, and annotated so that hypotheses proposed from earlier parts are later validated or invalidated once more video is shown. On the resulting tasks—Forecaster, Detective, and Reporter—the best closed-source models score 60.1–79.3% on multiple-choice and yes/no questions, while human experts score 85.3–95.3%, a gap of up to 31.9 points. The paper further shows that supplying human-written perception and comprehension descriptions improves one open-source model's Detective accuracy by up to 10 points, and that chain-of-thought prompting helps Detective but hurts Reporter, evidence that the bottleneck is distributed across perception, comprehension, and reasoning rather than a single component.

Load-bearing premise

The ground truth labels for the 1,655 videos rest on a single annotator per video, with quality validation performed on only 60 videos, so the benchmark is only as reliable as that annotator's judgment of which hypotheses are valid.

Editorial extensions

If this is right

  • If the reported gaps hold, current production VLMs are not reliable enough for decisions that depend on revising an initial interpretation of a scene when new evidence appears, such as driving or surveillance.
  • The perception/comprehension ablation indicates that a large share of the failure is not in the language model but in grounding: giving models human-written descriptions of what is visible moves Detective accuracy by up to 10 points.
  • Chain-of-thought prompting is not a universal fix: it improves Detective multiple-choice accuracy but lowers Reporter accuracy, so its benefit depends on how much of the evidence is already visible.
  • On questions where even annotators could not guess the event until the video ended, model accuracy drops by up to 10.1 points, suggesting that the most surprising events are exactly the ones current models handle worst.
  • The generative evaluations show models often produce generic captions and miss the specific unexpected action, indicating the discriminative gaps reflect a broader understanding deficit, not just option selection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the reported human baseline is the maximum of two expert annotators, the published gap is an upper-bound estimate; a single average human may score lower, so the true gap could be smaller.
  • Editorial inference: the same three-part design could be applied to other event types, such as sports blunders, magic tricks, or animal behavior, to test whether the model-human gap is specific to the video source or reflects a general limitation in revising beliefs.
  • Editorial inference: the benchmark's yes/no defeasible questions resemble the belief-update steps needed in collaborative or assistive systems, so an immediate testable extension would be to measure whether models that pass here also revise a stated plan when given contradictory visual evidence.
  • Editorial inference: because some multiple-choice distractors were machine-generated from captions and edited by an LLM, part of the MCQ difficulty could come from stylistic confounds; re-running the best models on a fully human-authored subset would isolate reasoning from language-prior effects.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces BlackSwanSuite, a video reasoning benchmark built from 1,655 videos from the Oops! test set. Three tasks are defined — Forecaster (predict the future from the pre-event), Detective (abduce the hidden main event and validate hypotheses), and Reporter (describe the full video and validate hypotheses with full context) — with generative, multiple-choice, and yes/no variants totaling 15,469 questions. Six vision-language models (GPT-4o, Gemini 1.5 Pro, VideoChat2, VideoLLaMA 2, VILA-1.5, LLaVA-Video) are evaluated, and the headline finding is that the best models lag humans by 21–32 percentage points on MCQ and yes/no tasks, with additional analyses of perception/comprehension substitution, chain-of-thought prompting, and a hard subset. The data, validation/test splits, and leaderboard are publicly released.

Significance. If its measurement concerns are resolved, BlackSwanSuite fills a real gap: it targets abductive and defeasible reasoning about unexpected events, areas not covered by existing video QA benchmarks, and it provides a suite of tasks that decompose reasoning into prediction, abduction, and defeasible update. The paper is transparent about its data collection, reports a quality check, releases a leaderboard, and includes a useful diagnostic experiment (§7.1) showing that supplying human perceptual and comprehension annotations improves LLaVA-Video's MCQ accuracy. The potential value is high, but the core quantitative claim — the magnitude of model-human gaps — is not yet fully supported because the ground-truth labels rest on single-annotator judgments and the human baseline is constructed as a maximum over two annotators, as detailed in the major comments.

major comments (3)
  1. [4.2, B.2, 7.3] The ground-truth validity labels for the Y/N tasks and the correct/incorrect status of MCQ options come from a single annotator per video (Section 4.2), and the only reported quality check covers 60 of 1,655 videos with average correctness/depth/grammar ratings, not per-label agreement (Appendix B.2). Appendix B.2 itself records that annotators sometimes "marked a description that could have been valid as invalid (or vice versa)", and no inter-annotator agreement statistic is reported anywhere. Because a model that selects an alternative but equally plausible explanation is scored wrong, the reported 21–32% gaps cannot be distinguished from annotation ambiguity. Please report per-label agreement statistics (e.g., Cohen's kappa on validation decisions for a random sample of videos) and either re-annotate with multiple annotators or provide a multi-annotator subset on which the model-human gaps are recomputed.
  2. [F.1] The human baseline is the maximum of two student annotators on 100–150 questions per task variant (Appendix F.1), which is an upper bound that inflates the headline gap; the main text (Section 5.2) states "a human expert 150 questions for each task variant" while Appendix F.1 says 100 for MCQ and 150 for Y/N, a direct inconsistency. The abstract's "up to 32%" gap in Reporter–Y/N is computed against this maximum baseline. Please report mean and median human accuracy over the two annotators alongside the max, provide confidence intervals for the sampled questions, and reconcile the stated number of baseline questions.
  3. [7.3] The hard subset in Section 7.3 is defined by the same single annotator's validity judgments — videos for which all Detective hypotheses were marked invalid in Reporter — so the hard/easy partition and the resulting 4.7–10.1 percentage point drops (Table 8) are not independent of the annotation noise documented in Appendix B.2. Please validate the partition with a second annotator or an alternative predictability measure (e.g., accuracy of human Forecasters on Vpre), or at least report the size of the hard subset and the overlap between annotators on the invalid-valid boundary.
minor comments (5)
  1. [4.3] The sentence "The Y/N variants for Forecaster (Detective) include each hypothesis proposed in step 1 (step 2)" is confusing because Forecaster has no Y/N task according to Table 1; please rephrase to describe the two Y/N variants without the nested parentheticals.
  2. [Tables 2, 13, 14] The captions of Tables 2, 13, and 14 say "Forecaster and Detective" but the tables report Detective and Reporter results; please correct the captions.
  3. [5.2, F.1] The description of the human evaluators is inconsistent: Section 5.2 says "a human expert", while Appendix F.1 says two students; please clarify who the human benchmark participants were and how they were recruited.
  4. [6.1.1, 7.1, 7.2] No error bars or significance tests are reported for the model-human accuracy differences or for the perception/comprehension gains in Table 6 and the CoT gains in Table 7; given the small 100–150 item samples, please add confidence intervals or a significance test.
  5. [References and text] There are several typos and LaTeX artifacts: reference [2] contains "Jastrzundefinedbski", Section 8 has "crucial step in models" (missing article), and Figure 1 uses "✅ explanation valid / ❌ explanation invalid" without defining the symbols in the figure caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; benchmark construction and evaluations are self-contained.

full rationale

BlackSwanSuite is a benchmark and evaluation paper rather than a derivation. The central claims—that VLMs underperform humans by up to 32% on abductive and defeasible video reasoning—are empirical measurements against newly collected annotations, not consequences of fitted parameters or of definitions that presuppose the conclusion. The annotation pipeline (Section 4.2) uses a single annotator per video, and the data-quality check covers 60 of 1,655 videos (Appendix B.2), but these are validity concerns about label reliability, not circularity in the sense of an input being renamed as an output. The 'hard subset' (Section 7.3) is defined by annotator invalidation patterns and then measured separately, so the reported accuracy drop is an empirical finding rather than a forced identity. The LLM-Match metric uses Llama 3.1 8B to rate outputs of other LLMs; this is a known evaluation choice and is not used to derive the headline gaps. The paper's citations to prior NLP work on abduction and defeasibility, including works co-authored by one of the present authors (Refs. [26], [31]), are background context and are not load-bearing for any uniqueness claim or derivation. No equation is equated to itself, no fitted parameter is repackaged as a prediction, and no self-citation is used to forbid alternatives. The reported gaps may be affected by annotation variance and by taking the max of two human annotators (Appendix F.1), but these are measurement choices, not circular reasoning.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity depends on the Oops! event localization, the heuristic video splitting, the reliability of single-annotator labels, and the choice of human baseline. No free parameters are fitted and no new entities are postulated. The four axioms listed are the load-bearing premises that the reported measurements rely on.

assumptions (4)
  • domain assumption Videos from the Oops! test set contain a single unexpected event with reliable localization annotations.
    Section 4.1. The task definitions and the split into Vpre, Vmain, and Vpost rest on the correctness of Oops! event localization.
  • domain assumption The automatic splitter and heuristic cut at 0.8*t produce a Vpre that does not reveal the event, a Vmain that contains it, and a Vpost that shows only the outcome.
    Algorithm 1, Appendix B.1. If the split leaks the event into Vpre or hides it from Vmain, the Forecaster and Detective tasks lose their intended meaning.
  • domain assumption A single annotator's responses and validation decisions are reliable ground truth.
    Section 4.2. Each video is annotated by one qualified worker, with quality validation on only 60 of 1,655 videos. Systematic annotation errors would bias all reported accuracies.
  • ad hoc to paper The human baseline is defined as the maximum score of two student annotators.
    Appendix F.1. This choice inflates the reported human performance and directly affects the headline gap of up to 32%. It is an upper bound, not a typical human score.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events." pith.science (2026). https://pith.science/paper/QNC7TQRX

@misc{pith2026241205725,
  author       = {Pith},
  title        = {Pith review of: Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QNC7TQRX}},
  note         = {Machine review of arXiv:2412.05725}
}
read the original abstract

The commonsense reasoning capabilities of vision-language models (VLMs), especially in abductive reasoning and defeasible reasoning, remain poorly understood. Most benchmarks focus on typical visual scenarios, making it difficult to discern whether model performance stems from keen perception and reasoning skills, or reliance on pure statistical recall. We argue that by focusing on atypical events in videos, clearer insights can be gained on the core capabilities of VLMs. Explaining and understanding such out-of-distribution events requires models to extend beyond basic pattern recognition and regurgitation of their prior knowledge. To this end, we introduce BlackSwanSuite, a benchmark for evaluating VLMs' ability to reason about unexpected events through abductive and defeasible tasks. Our tasks artificially limit the amount of visual information provided to models while questioning them about hidden unexpected events, or provide new visual information that could change an existing hypothesis about the event. We curate a comprehensive benchmark suite comprising over 3,800 MCQ, 4,900 generative and 6,700 yes/no questions, spanning 1,655 videos. After extensively evaluating various state-of-the-art VLMs, including GPT-4o and Gemini 1.5 Pro, as well as open-source VLMs such as LLaVA-Video, we find significant performance gaps of up to 32% from humans on these tasks. Our findings reveal key limitations in current VLMs, emphasizing the need for enhanced model architectures and training strategies. Our data and leaderboard is available at blackswan.cs.ubc.ca.

Figures

Figures reproduced from arXiv: 2412.05725 by the authors.

Figure 1
Figure 1. BlackSwanSuite. Our benchmark involves three tasks: i) Forecaster evaluates a model’s ability to hypothesize future events. ii) Detective involves abductive reasoning by explaining the hidden event, and defeasible reasoning by validating existing hypotheses. iii) Reporter again tests defeasability and the model’s ability to describe the unexpected event. and defeasible reasoning capabilities of models, as these even… view at source ↗
Figure 2
Figure 2. BlackSwanSuite contains 1655 videos from variety of topics, as depicted above. Tasks: Generative MCQ Yes/No Forecaster 1,655 - - Detective 1,655 2,415 4,917 Reporter 1,655 1,410 1,762 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Qualitative results on MCQ and Y/N variants. In the video, a man swings a pillow at the Christmas tree, causing ornaments to fly towards the lady. Examples (a), (b), (c) and (d) are task questions from our dataset. Model Automatic Human Rating CLIP LLM-M C T D V Closed Source GPT-4o 0.78 2.08 3.41 3.28 3.53 0.78 Gemini 1.5 Pro 0.73 2.14 3.05 3.10 3.45 0.65 Open Source VideoChat2 0.68 1.91 3.60 2.90 3.20 0.75 VideoLL… view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Data Collection Process. We start by filtering videos that adhere to our dataset requirements, such that they can be split into Vpre, Vmain and Vpost. With 10% of data, we collect an￾notations to select the best annotators. With these annotators, we collect the full da…
Figure 5
Figure 5. Figure 5: Length of Videos. The median video length is 8.83 seconds. Only a small number of videos are outliers, with 29 of them being longer than 25 seconds. provided will all parts of the video along with annotations from all three steps. We provide them with the UI in [PITH_…
Figure 6
Figure 6. Figure 6: Qualitative results on Gen variants. Due to space constrains, only one sample from each model is shown. Depending on the model, we may vary the prompt in small ways. For example, we may use the default system instruction for each model, or ask models to return an an￾sw…
Figure 7
Figure 7. Figure 7: Perception and Comprehension with LLaVA-Video. (a) shows a case where human-written Perception helps. (b) shows a case where both human-written Comprehension helps. 6 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: CoT Examples with GPT-4o. (a) has a case where GPT-4o shows good step-by-step reasoning to arrive at the correct answer. (b) shows that GPT-4o sometimes relies of preconceived notions in its reasoning steps, and therefore makes a mistake. 9 [PITH_FULL_IMAGE:figures/fu…
Figure 9
Figure 9. Figure 9: Black Swan Annotation Template 10 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 9
Figure 9. Figure 9: (Continued) Black Swan Annotation Template [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 9
Figure 9. Figure 9: (Continued) Black Swan Annotation Template [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 9
Figure 9. Figure 9: (Continued) Black Swan Annotation Template [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 9
Figure 9. Figure 9: (Continued) Black Swan Annotation Template [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 9
Figure 9. Figure 9: (Continued) Black Swan Annotation Template [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Template for Dataset Validation. 16 [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Human Evaluation for Generative Tasks 17 [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 11
Figure 11. Figure 11: (Continued) Human Evaluation for Generative Tasks [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 11
Figure 11. Figure 11: (Continued) Human Evaluation for Generative Tasks [PITH_FULL_IMAGE:figures/full_fig_p029_11.png]
Figure 12
Figure 12. Figure 12: Sample template for Human Baseline for the generative variant. 20 [PITH_FULL_IMAGE:figures/full_fig_p030_12.png]
Figure 13
Figure 13. Figure 13: Sample template for Human Baseline for the MCQ variant. 21 [PITH_FULL_IMAGE:figures/full_fig_p031_13.png]
Figure 14
Figure 14. Figure 14: Sample template for Human Baseline for the Y/N variant. 22 [PITH_FULL_IMAGE:figures/full_fig_p032_14.png]
Figure 15
Figure 15. Figure 15: Analysis: Collection of Human Perception [PITH_FULL_IMAGE:figures/full_fig_p033_15.png]
Figure 15
Figure 15. Figure 15: (Continued) Analysis: Collection of Human Perception [PITH_FULL_IMAGE:figures/full_fig_p034_15.png]
Figure 15
Figure 15. Figure 15: (Continued) Analysis: Collection of Human Perception [PITH_FULL_IMAGE:figures/full_fig_p035_15.png]
Figure 16
Figure 16. Figure 16: Analysis: Collection of Human Comprehension [PITH_FULL_IMAGE:figures/full_fig_p036_16.png]
Figure 16
Figure 16. Figure 16: (Continued) Analysis: Collection of Human Comprehension [PITH_FULL_IMAGE:figures/full_fig_p037_16.png]
Figure 16
Figure 16. Figure 16: (Continued) Analysis: Collection of Human Comprehension [PITH_FULL_IMAGE:figures/full_fig_p038_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 47 canonical work pages

  1. [1]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, 2015. 1

    work page 2015

  2. [2]

    Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien

    Devansh Arpit, Stanisław Jastrzundefinedbski, Nicolas Bal- las, David Krueger, Emmanuel Bengio, Maxinder S. Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at mem- orization in deep networks. In Proceedings of the 34th In- ternational Conference on Machine Learning - Volume 70 , page 233–242. ...

    work page 2017

  3. [3]

    Abductive common- sense reasoning

    Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Yih, and Yejin Choi. Abductive common- sense reasoning. ArXiv, abs/1908.05739, 2019. 2

    arXiv 1908

  4. [4]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 1, 5, 4

    arXiv 2024

  5. [5]

    δ-causal: Exploring defeasibility in causal reasoning

    Shaobo Cui, Lazar Milikic, Yiyang Feng, Mete Ismayilzada, Debjit Paul, Antoine Bosselut, and Boi Faltings. δ-causal: Exploring defeasibility in causal reasoning. In Annual Meet- ing of the Association for Computational Linguistics , 2024. 3

    work page 2024

  6. [6]

    A graph enhanced bert model for event prediction

    Li Du, Xiao Ding, Yue Zhang, Ting Liu, and Bing Qin. A graph enhanced bert model for event prediction. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2628–2638, 2022. 2

    work page 2022

  7. [7]

    Oops! predicting unintentional action in video

    Dave Epstein, Boyuan Chen, and Carl V ondrick. Oops! predicting unintentional action in video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 919–929, 2020. 4, 1, 8

    work page 2020

  8. [8]

    Read- ing books is great, but not if you are driving! visually grounded reasoning about defeasible commonsense norms

    Seungju Han, Junhyeok Kim, Jack Hessel, Liwei Jiang, Ji- wan Chung, Yejin Son, Yejin Choi, and Youngjae Yu. Read- ing books is great, but not if you are driving! visually grounded reasoning about defeasible commonsense norms. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 894–914, 2023. 3

    work page 2023

Show all 57 references
  1. [9]

    Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi

    Jack Hessel, Jena D. Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. The abduction of sherlock holmes: A dataset for visual abductive reasoning. In European Conference on Computer Vision, 2022. 2

    2022

  2. [10]

    Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gun- hee Kim

    Y . Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gun- hee Kim. Tgif-qa: Toward spatio-temporal reasoning in vi- sual question answering. 2017 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 1359– 1367, 2017. 3

    2017

  3. [11]

    Sgd on neural networks learns functions of increas- ing complexity

    Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Ben- jamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increas- ing complexity. In Advances in Neural Information Process- ing Systems. Curran Associates, Inc., 2019. 1

    2019

  4. [12]

    From representa- tion to reasoning: Towards both evidence and commonsense reasoning for video question-answering

    Jiangtong Li, Li Niu, and Liqing Zhang. From representa- tion to reasoning: Towards both evidence and commonsense reasoning for video question-answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3

    2022

  5. [13]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  6. [14]

    Visual abductive reasoning

    Chen Liang, Wenguan Wang, Tianfei Zhou, and Yi Yang. Visual abductive reasoning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15544–15554, 2022. 2

    2022

  7. [15]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. ArXiv preprint, 2023. 1

    2023

  8. [16]

    Vila: Efficient video-language alignment for video question answering

    Ming C Lin and Shan Yang. Vila: Efficient video-language alignment for video question answering. 2024. 5, 4

    2024

  9. [17]

    Revisiting commonsense reasoning in machine translation: Training, evaluation and challenge

    Xuebo Liu, Yutong Wang, Derek F Wong, Runzhe Zhan, Liangxuan Yu, and Min Zhang. Revisiting commonsense reasoning in machine translation: Training, evaluation and challenge. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    2023

  10. [18]

    Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, and Eduard H. Hovy. Think about it! improving defeasible reasoning by first modeling the ques- tion scenario. In Conference on Empirical Methods in Natu- ral Language Processing, 2021. 3

    2021

  11. [19]

    Openeqa: Em- bodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Sil- wal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batr...

    2024

  12. [20]

    Scalable end-to-end autonomous vehicle testing via rare-event simulation

    Matthew O’Kelly, Aman Sinha, Hongseok Namkoong, John Duchi, and Russ Tedrake. Scalable end-to-end autonomous vehicle testing via rare-event simulation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, page 9849–9860, Red Hook, NY , USA,

  13. [21]

    GPT-4o system card, 2024

    OpenAI. GPT-4o system card, 2024. 4, 5 9

    2024

  14. [22]

    Hobbs, Michael C

    Ekaterina Ovchinnikova, Niloofar Montazeri, Theodore Alexandrov, Jerry R. Hobbs, Michael C. McCord, and Rutu Mulkar-Mehta. Abductive reasoning with a large knowledge base for discourse processing. In International Conference on Computational Semantics, 2011. 2

    2011

  15. [23]

    Visualcomet: Reasoning about the dynamic context of a still image

    Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visualcomet: Reasoning about the dynamic context of a still image. In In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 1

    2020

  16. [24]

    Generating hypothetical events for abductive inference

    Debjit Paul and Anette Frank. Generating hypothetical events for abductive inference. In Proceedings of* SEM 2021: The Tenth Joint Conference on Lexical and Compu- tational Semantics, pages 67–77, 2021. 2

    2021

  17. [25]

    Collected papers of charles sanders peirce

    Charles Sanders Peirce. Collected papers of charles sanders peirce. Harvard University Press, 1974. 2

    1974

  18. [26]

    Back to the future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning

    Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhaga- vatula, Jena D Hwang, Ronan Le Bras, Antoine Bosse- lut, and Yejin Choi. Back to the future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning. In Proceedings of the 2020 Confer- e...

    2020

  19. [27]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 5

    2021

  20. [28]

    On the spectral bias of neural networks

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference on Ma- chine Learning, pages 5301–5310. PMLR, 2019. 1

  21. [29]

    What makes it ok to set a fire? iterative self-distillation of con- texts and rationales for disambiguating defeasible social and moral situations

    Kavel Rao, Liwei Jiang, Valentina Pyatkin, Yuling Gu, Niket Tandon, Nouha Dziri, Faeze Brahman, and Yejin Choi. What makes it ok to set a fire? iterative self-distillation of con- texts and rationales for disambiguating defeasible social and moral situations. In Conference on ...

    2023

  22. [30]

    A logic for default reasoning

    Raymond Reiter. A logic for default reasoning. Artificial intelligence, 13(1-2):81–132, 1980. 2

    1980

  23. [31]

    Thinking like a skeptic: Defeasible inference in natural language

    Rachel Rudinger, Vered Shwartz, Jena D Hwang, Chan- dra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A Smith, and Yejin Choi. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages...

    2020

  24. [32]

    Movieqa: Understanding stories in movies through question- answering, 2016

    Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question- answering, 2016. 3

    2016

  25. [33]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 5, 4

    2024 arXiv

  26. [34]

    The llama 3 herd of models, 2024

    Llama3 Team. The llama 3 herd of models, 2024. 5

    2024

  27. [35]

    Advsim: Generating safety-critical scenar- ios for self-driving vehicles

    Jingkang Wang, Ava Pun, James Tu, Sivabalan Mani- vasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. Advsim: Generating safety-critical scenar- ios for self-driving vehicles. Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1

    2021

  28. [36]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. 8

    2023

  29. [37]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9772–9781,

    2021

  30. [38]

    Funqa: Towards surprising video comprehension

    Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuan- han Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. ArXiv, abs/2306.14899, 2023. 3

    2023 arXiv

  31. [39]

    Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xi- angnan He, and Yueting Zhuang

    D. Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xi- angnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017. 1

    2017

  32. [40]

    Tenenbaum

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Ji- ajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and rea- soning. ArXiv, abs/1910.01442, 2019. 3

    1910 arXiv

  33. [41]

    Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao

    Zhou Yu, D. Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. ArXiv, abs/1906.02467, 2019. 3

    1906 arXiv

  34. [42]

    From recognition to cognition: Visual commonsense reason- ing

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reason- ing. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16- 20, 2019, 2019. 1, 3

    2019

  35. [43]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. 1, 4, 5

    2024

  36. [44]

    Videoabc: A real-world video dataset for ab- ductive visual reasoning

    Wenliang Zhao, Yongming Rao, Yansong Tang, Jie Zhou, and Jiwen Lu. Videoabc: A real-world video dataset for ab- ductive visual reasoning. IEEE Transactions on Image Pro- cessing, 31:6048–6061, 2022. 2

    2022

  37. [45]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264,

    arXiv

  38. [46]

    Hwang, Swabha Swayamdipta, and Maarten Sap

    Xuhui Zhou, Haojie Zhu, Akhila Yerukola, Thomas David- son, Jena D. Hwang, Swabha Swayamdipta, and Maarten Sap. Cobra frames: Contextual reasoning about effects and harms of offensive statements.ArXiv, abs/2306.01985, 2023. 3 10 Black Swan: Abductive and Defeasible Video Reaso...

    2023 arXiv

  39. [48]

    Correctness (5 Point Scale) Check whether the answer correctly describes what could have happened in part 2 based on part 1

  40. [49]

    Depth and Thoughtfulness (5 Point Scale) How much depth in reasoning is covered in the description? Is there sufficient reasoning to support the description?

  41. [50]

    Level of Detail (5 Point Scale) Does the description have sufficient level of detail to easily identify and differenti- ate between people/objects and actions?

  42. [51]

    We use the same location, age and education filters as used above while collecting annotations

    Visual Contradiction (Yes/No) Does the description fol- low the content in the video, or contradict the contents of the video? A contradiction happens when the description is impossible, given the video. We use the same location, age and education filters as used above while c...

  43. [52]

    We use the GPT-4o model using the OpenAI Batch API3

    OpenAI GPT-4o [21] GPT-4o was an important choice for our video-based task because, apart from leading all kinds of VL benchmarks, it is OpenAI’s first model that is capable of accepting up to 50 pictures as input in the prompt. We use the GPT-4o model using the OpenAI Batch A...

    2024

  44. [53]

    This model was accessed between Oc- tober 20th and November 14th, 2024

    Google Gemini 1.5 Pro [33] We chose to use Gemini 1.5 Pro as it is the most capable VL model that can natively accept video input. This model was accessed between Oc- tober 20th and November 14th, 2024. When asking ques- tions, we directly upload the entire video clip. In the ...

    2024

  45. [54]

    We evaluate the latest VideoChat2 HD (Huggingface: videochat2 hd mistral 7b stage4) model

    VideoChat2 [13] VideoChat2 showed stronger detail and contextual understanding than models prior to it, and even beat GPT-4V at the time of its release on multiple VL tasks. We evaluate the latest VideoChat2 HD (Huggingface: videochat2 hd mistral 7b stage4) model. When asking ...

  46. [55]

    Though audio is not a requirement for this task, there may be cases where it can reveal important information about the events in the video

    VideoLLaMA 2 [4] VideoLLaMA 2 is capable of un- derstanding both visual and audio signals. Though audio is not a requirement for this task, there may be cases where it can reveal important information about the events in the video. We use the VideoLLaMA2.1-7B-16F model. Again,...

  47. [56]

    We use the Llama-3-VILA1.5-8b-Fix model

    VILA [16] VILA 1.5 is the latest in the VILA series of models, one of the first models to support multi-image un- derstanding. We use the Llama-3-VILA1.5-8b-Fix model. We uniformly sample 5 frames for each part of the video. 3https://platform.openai.com/docs/guides/batch/ overview

  48. [57]

    leading to an unpredictable outcome

    LLaV A-Video [43]LLaV A was first introduced as a mul- timodal model with performance rivaling GPT-4. Follow- ing the release of LLaV A 1.5 and LLaV A-NeXT, LLaV A- Video is the most advanced version of this model, specif- ically trained on a new 178K video dataset. Specifical...

  49. [2018]

    Curran Associates Inc. 1

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.