Pith. sign in

REVIEW 4 major objections 4 minor 75 references

Admitting Ignorance Helps the Video Question Answering Models to Answer

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training VideoQA models to admit ignorance on intervened questions breaks spurious question-answer correlations and improves accuracy on multiple benchmarks.

desk verdict A simple, model-agnostic training trick for VideoQA—make models say 'unknown' on intervened questions—gives consistent gains across datasets, but missing error bars and an unquantified label-noise assumption keep it from being a clean accept. read the letter →

arxiv 2501.08771 v2 pith:YZISEAHW submitted 2025-01-15 cs.CV

Haopeng Li , Tom Drummond , Mingming Gong , Mohammed Bennamoun , Qiuhong Ke This is my paper · ORCID
classification cs.CV
keywords videoquestionansweringspuriouscorrelationsadmittingignoranceinterventioncurriculumlearningmodel-agnosticdebiasingmulti-choiceQAopen-ended
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that VideoQA models should be trained to admit ignorance on deliberately broken video-question pairs, rather than always predicting an answer. The authors argue that standard training encourages models to exploit spurious correlations between question text and answer, especially when video-text alignment is imperfect. By intervening on questions—swapping in another pair's question or altering crucial words—and forcing the model to answer "unknown" or "not given," the model is pushed to rely on the joint video-question representation. On six benchmarks, including TGIF-Action, TGIF-Transition, NExT-QA, MSVD-QA, and MSRVTT-QA, the framework improves accuracy by about 1 to 2 points over strong pretrained baselines, with no extra inference cost.

What carries the argument

The load-bearing mechanism is the question intervention paired with an "admit ignorance" training signal. Displacement (global replacement) creates easy inconsistency, teaching coarse video-question correspondence; perturbation (local word replacement) detects subtle mismatches, teaching fine-grained alignment. On the answer side, multi-choice tasks add a "not given" option mixed into a freshly sampled option set, while open-ended tasks add a separate logit trained with soft labels proportional to the semantic distance $d = D(Q, Q')$ between the original and intervened question. An ignorance-diminishing curriculum schedule $p(e) = \frac{p_r}{E^2}(e-E)^2$ balances the two learning goals.

What would settle it

Measure the fraction of intervened video-question pairs that still admit a valid answer on a given dataset, using human annotation or an oracle model; if that fraction is not tiny, retraining the same base model with this recipe should produce little or no improvement.

Watch

Extended reading notes

Core claim

The central claim is that breaking the spurious question-to-answer path forces a VideoQA model to learn more robust video-text alignment, and that this can be achieved purely through data intervention and a modified training objective. The paper introduces two question interventions: displacement, which replaces a question with one from a different video-question pair, and perturbation, which changes only crucial words such as subjects, adjectives, or numbers. The model is trained to say "not given" (multi-choice) or activate an ignorance logit (open-ended) on intervened pairs, and to answer normally on original pairs. A curriculum that gradually decreases the intervention probability from $p_r$ to zero lets the model first learn to detect inconsistency and then focus on correct answering. The paper reports consistent gains when the recipe is applied to InternVideo, HQGA, All-in-one, and VideoLLaMA2, and even to image QA models, with improvements such as +1.9 on TGIF-Action and +1.9 on NExT-QA.

Load-bearing premise

The method's labels for intervened pairs assume that a displaced or perturbed question almost never has a valid answer in the video, and the paper asserts such cases are "extremely infrequent".

Editorial extensions

If this is right

  • Any existing VideoQA model can be retrained with this data-only recipe and see similar gains, since no architectural change is required.
  • The framework transfers to image QA, suggesting the principle applies beyond video.
  • Because the intervention probability decays to zero, training time and inference cost stay identical to the base fine-tuning.
  • Gains are largest on multi-choice datasets where spurious question-answer correlations are strongest, offering a diagnostic for shortcut reliance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The "admit ignorance" signal acts as a regularizer on the question encoder, and could be repurposed as a calibration or uncertainty measure at test time, although the paper does not propose this.
  • If applied to non-template, natural-language questions, the assumption that intervened pairs are unanswerable is weaker, so the method may need semantic-distance thresholds tuned per dataset.
  • The approach could be combined with video-side interventions to break both video-answer and question-answer shortcuts simultaneously.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a training framework for VideoQA in which the model is trained to admit ignorance when presented with an intervened question. Two interventions are used: displacement (replacing the question with one from another video-question pair) and perturbation (replacing crucial words in the question). For multi-choice QA, a "not given" option is added; for open-ended QA, an extra "unknown" logit is introduced with a soft loss weighted by the semantic distance between original and intervened questions. A curriculum schedule anneals the intervention probability to zero during training. The framework is applied to InternVideo, All-in-one, HQGA, and VideoLLaMA2, with reported accuracy gains of roughly 0.5 to 2.4 points on TGIF-Action, TGIF-Transition, NExT-QA, TGIF-FrameQA, MSVD-QA, MSRVTT-QA, and two ImageQA datasets.

Significance. If the reported gains hold, this is a simple, model-agnostic, and cost-free-at-inference recipe for improving VideoQA accuracy by discouraging question-answer shortcuts. The paper's strengths include consistent improvements across multiple dataset types, multiple base models (including a modern LLM-based model), controlled comparisons against naive text augmentation, and ablations on curriculum schedules. The main deficit is evidentiary: no error bars or multiple seeds are reported, the central unanswerability assumption for intervened pairs is not quantified, and an important semantic-distance threshold is not reported, leaving the mechanism claim only partially supported.

major comments (4)
  1. [Tables I and II, Section V-B/V-C] The headline improvements (e.g., +1.9 on TGIF-Action, +1.7 on TGIF-Transition, +1.1 on MSRVTT-QA) are reported as single numbers without error bars or multiple seeds. Since several gains are around 1 point or less, the reader cannot assess whether these differences are statistically meaningful. Please report the mean and standard deviation over at least three runs with different seeds, or provide a statistical significance test.
  2. [Section IV-B and Section IV-C] The load-bearing premise is that an intervened video-question pair (V,Q') is almost always unanswerable, so 'not given' or 'unknown' is the correct label. The justification in Section IV-B only covers template-based TGIF questions where an adjective uniquely identifies a person. It does not cover displacement, where a question from another video can accidentally match the current video, nor Polyjuice perturbations of free-form questions in NExT-QA, MSVD-QA, and MSRVTT-QA, where the changed entity, action, or number may actually be present. In MCQA the original correct answer is explicitly kept in the option set (Section IV-C), so any such mislabeled intervention trains the model to reject a valid answer in favor of 'not given'. Please quantify the mislabeling rate with a human study on a stratified sample of held-out interventions (by dataset and intervention type), and report the semantic-distance threshold used in Section V-A1 to separate augmentations from interventions.
  3. [Section V-D, Table III] The 'Is the Model Admitting Ignorance?' experiment evaluates on the same intervention distribution used during training, so it largely verifies that the training objective was learned rather than testing whether 'not given' corresponds to genuine unanswerability. On TGIF-Action, InternVideo-L admits ignorance on only 50.0% of displacements and 49.6% of perturbations; without a human-annotated answerability baseline for these same cases, these numbers are ambiguous. Please provide an answerability oracle or a human study for a sample of the test interventions to interpret Table III.
  4. [Section V-A2, Eq. (14), Table III] The initial intervention probability p_r and the displacement ratio are selected based on validation accuracy (Fig. 4), and the activation threshold used to define a successful 'ignorance' admission in Table III is not reported. This leaves open the possibility that the reported gains depend on per-dataset tuning, especially because the OEQA loss in Eq. (14) uses the continuous semantic distance d that itself depends on the unreported Sentence-BERT threshold. Please report the chosen values per dataset, the activation threshold, and a sensitivity analysis across seeds.
minor comments (4)
  1. [Fig. 2 caption] The caption reads 'The casual graph of VideoQA'; it should be 'causal graph'.
  2. [Section V-A1] The dataset name 'MSRTT-QA' is a typo; it should be 'MSRVTT-QA'.
  3. [Section V-A2, Eq. (15)] The schedule p(e) = (p_r/E^2)(e-E)^2 does not equal p_r at e=1 for finite E; the text says the probability 'decreases from p_r to 0'. Please clarify whether the schedule is intended to start near p_r or define the epoch index differently.
  4. [Section V-D] For the OEQA 'admit ignorance' evaluation, the threshold on the last logit is mentioned but its value and selection procedure are not given; please specify it in the experimental settings.

Circularity Check

1 steps flagged · score 2.0 of 10

Central accuracy improvements are externally grounded; only the Table III ignorance-capability evaluation is self-confirmatory by construction.

  1. fitted input called prediction [Section V-D ('Is the Model Admitting Ignorance?'), Table III; cf. Eq. (14) and Section IV-C.]
    "To assess whether the models trained with our framework are capable of acknowledging their ignorance when presented with intervened questions, we apply interventions to the questions in the testing set, similar to the training phase, and evaluate the predictions. Specifically, for multi-choice VideoQA, we expect the models to choose 'not given'."

    The training objective already hard-codes this behavior: Eq. (14) drives the ignorance head p_i toward the intervention indicator/soft distance d, and Section IV-C forces MCQA models to select the added 'not given' option for every intervened pair. Testing on the same intervention construction ('similar to the training phase') and scoring success as choosing 'not given' (or exceeding an activation threshold) therefore measures whether the training target was learned, not an independent behavioral consequence. It does not validate the key accuracy claim, which is separately supported by held-out original test splits.

full rationale

The paper's principal claim—that question-intervention training improves VideoQA accuracy—is tested on held-out original test splits: Table I compares AIQA with InternVideo* (+1.1 to +1.9 across six datasets) and Table II shows consistent gains across HQGA, All-in-one, and InternVideo. These results are not derivable from the training objective by construction: test questions are unchanged, the 'not given' option is kept in testing, and the paper reports rare 'not given' selections (0.6%/0.4% on TGIF-Action/TGIF-Transition). The only self-confirmatory component is Table III, which checks whether the model reproduces the trained behavior—predict 'not given'/'unknown' for intervened questions—on the same intervention distribution; this is a sanity check of optimization rather than independent evidence for the debiasing mechanism. The paper's one self-citation ([23]) appears only in related work and is not load-bearing. The 'extremely infrequent' assertion about answerable interventions (Section IV-B) is a correctness/assumption risk rather than a circular step, and hyperparameter tuning on validation (Section V-A2) is standard practice. No uniqueness theorem, no fitted constant recycled into the external evaluation, and no renaming of a known result is present.

Assumptions & free parameters 4 free parameters · 3 assumptions · 1 invented entities

The central method relies on several validation-tuned hyperparameters (intervention probability, displacement ratio, semantic-distance threshold) and one invented output category. It does not introduce new physical entities. The key domain assumption is that interventions reliably make video-question pairs unanswerable.

free parameters (4)
  • Initial intervention probability p_r = Not reported per dataset; 0.3 is recommended as a starting point
    Controls how often questions are intervened during training and is tuned on validation accuracy (Section V-A2, Equation 15, Figure 4a).
  • Displacement-to-perturbation ratio = Not reported per dataset; 1:1 is recommended as a starting point
    Controls the mix of global and local question interventions and is tuned on validation accuracy (Section V-A2, Figure 4b).
  • Semantic distance threshold for perturbation vs augmentation = Not reported
    Determines whether a perturbed question is treated as an intervention with an 'unknown' target or as an augmentation that keeps the original answer; Section IV-B says 'We then set a threshold' without giving its value.
  • OEQA ignorance activation threshold = Not reported
    Used only for the Table III evaluation, where an activation above a chosen threshold counts as successfully admitting ignorance; Section V-D says 'We set a threshold' without specifying it.
assumptions (3)
  • domain assumption For a question altered by displacement or perturbation, the correct answer for the paired video is always, or almost always, 'unknown'/'not given'.
    The whole training target depends on intervened video-question pairs being unanswerable. The paper acknowledges possible failures but calls them 'extremely infrequent' in Section IV-B; this is the load-bearing premise that could invalidate the training labels if false.
  • domain assumption The datasets are structured enough that crucial words in questions can be identified and replaced, and Polyjuice perturbations of free-form questions preserve the intended semantic change.
    Section V-A1 relies on template-based question formats for manual replacement and on Polyjuice with selected control codes for free-form questions. If the perturbations are invalid or change meaning only slightly, the soft-label target d becomes unreliable.
  • domain assumption VideoQA models trained with standard cross-entropy exhibit spurious question-answer correlations, and reducing reliance on them improves generalization.
    This is the assumed causal edge Q -> A in the graph of Section IV-A (Figure 2). It is supported by cited prior work, but if it were false, the method could not produce consistent gains.
invented entities (1)
  • 'not given' option (MCQA) and extra 'unknown' logit (OEQA)
    purpose: Gives the model an explicit output class for admitting ignorance when faced with an intervened video-question pair.
    This is an added output category, not a physical entity, so it has no falsifiable handle outside the paper. Whether it is the right target depends on the assumption that intervened pairs are truly unanswerable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Admitting Ignorance Helps the Video Question Answering Models to Answer." pith.science (2026). https://pith.science/paper/YZISEAHW

@misc{pith2026250108771,
  author       = {Pith},
  title        = {Pith review of: Admitting Ignorance Helps the Video Question Answering Models to Answer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZISEAHW}},
  note         = {Machine review of arXiv:2501.08771}
}
read the original abstract

Significant progress has been made in the field of video question answering (VideoQA) thanks to deep learning and large-scale pretraining. Despite the presence of sophisticated model structures and powerful video-text foundation models, most existing methods focus solely on maximizing the correlation between answers and video-question pairs during training. We argue that these models often establish shortcuts, resulting in spurious correlations between questions and answers, especially when the alignment between video and text data is suboptimal. To address these spurious correlations, we propose a novel training framework in which the model is compelled to acknowledge its ignorance when presented with an intervened question, rather than making guesses solely based on superficial question-answer correlations. We introduce methodologies for intervening in questions, utilizing techniques such as displacement and perturbation, and design frameworks for the model to admit its lack of knowledge in both multi-choice VideoQA and open-ended settings. In practice, we integrate a state-of-the-art model into our framework to validate its effectiveness. The results clearly demonstrate that our framework can significantly enhance the performance of VideoQA models with minimal structural modifications.

Figures

Figures reproduced from arXiv: 2501.08771 by the authors.

Figure 1
Figure 1. The difference between conventional VideoQA and our [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The casual graph of VideoQA. 2. This causal graph helps to visualize the cause-and-effect connections between four important elements: the input video V , the input question Q, the causal multimodal feature C, and the ground-truth answer A. Specifically, a detailed breakdown of the causal relationships depicted in the graph is as follows: • V → C ← Q: The causal multimodal feature C is determined by the combination … view at source ↗
Figure 3
Figure 3. Admitting ignorance for multi-choice VideoQA: a “not [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The impact of the initial intervention probability [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 56 canonical work pages

  1. [1]

    Bilinear attention networks,

    J.-H. Kim, J. Jun, and B.-T. Zhang, “Bilinear attention networks,” Advances in neural information processing systems , vol. 31, 2018

    work page 2018

  2. [2]

    Attend what you need: Motion-appearance synergistic networks for video question answering,

    A. Seo, G.-C. Kang, J. Park, and B.-T. Zhang, “Attend what you need: Motion-appearance synergistic networks for video question answering,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 2021, pp. 6167–6177

    work page 2021

  3. [3]

    Video as conditional graph hierarchy for multi-granular question answering

    J. Xiao, A. Yao, Z. Liu, Y . Li, W. Ji, and T.-S. Chua, “Video as conditional graph hierarchy for multi-granular question answering.” AAAI, 2022

    work page 2022

  4. [4]

    Merlot: Multimodal neural script knowledge models,

    R. Zellers, X. Lu, J. Hessel, Y . Yu, J. S. Park, J. Cao, A. Farhadi, and Y . Choi, “Merlot: Multimodal neural script knowledge models,” Ad- vances in Neural Information Processing Systems , vol. 34, pp. 23 634– 23 651, 2021

    work page 2021

  5. [5]

    Violet: End-to-end video-language transformers with masked visual- token modeling,

    T.-J. Fu, L. Li, Z. Gan, K. Lin, W. Y . Wang, L. Wang, and Z. Liu, “Violet: End-to-end video-language transformers with masked visual- token modeling,” arXiv preprint arXiv:2111.12681 , 2021

    arXiv 2021

  6. [6]

    X 2-vlm: All-in-one pre-trained model for vision-language tasks,

    Y . Zeng, X. Zhang, H. Li, J. Wang, J. Zhang, and W. Zhou, “X 2-vlm: All-in-one pre-trained model for vision-language tasks,” arXiv preprint arXiv:2211.12402, 2022

    arXiv 2022

  7. [7]

    Internvideo: General video foundation models via gen- erative and discriminative learning,

    Y . Wang, K. Li, Y . Li, Y . He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y . Liu, Z. Wang et al., “Internvideo: General video foundation models via gen- erative and discriminative learning,” arXiv preprint arXiv:2212.03191 , 2022

    arXiv 2022

  8. [8]

    All in one: Exploring unified video-language pre-training,

    J. Wang, Y . Ge, R. Yan, Y . Ge, K. Q. Lin, S. Tsutsui, X. Lin, G. Cai, J. Wu, Y . Shan et al. , “All in one: Exploring unified video-language pre-training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 6598–6608

    work page 2023

Show all 75 references
  1. [9]

    Invariant grounding for video question answering,

    Y . Li, X. Wang, J. Xiao, W. Ji, and T.-S. Chua, “Invariant grounding for video question answering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 2928–2937

    2022

  2. [10]

    Equivariant and invariant grounding for video question answering,

    Y . Li, X. Wang, J. Xiao, and T.-S. Chua, “Equivariant and invariant grounding for video question answering,” in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 4714–4722

    2022

  3. [11]

    Transformer-empowered invariant grounding for video question answering,

    Y . Li, X. Wang, J. Xiao, W. Ji, and T.-S. Chua, “Transformer-empowered invariant grounding for video question answering,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023

    2023

  4. [12]

    Discovering spatio- temporal rationales for video question answering,

    Y . Li, J. Xiao, C. Feng, X. Wang, and T.-S. Chua, “Discovering spatio- temporal rationales for video question answering,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 13 869–13 878

    2023

  5. [13]

    Adversarial vqa: A new benchmark for evaluating the robustness of vqa models,

    L. Li, J. Lei, Z. Gan, and J. Liu, “Adversarial vqa: A new benchmark for evaluating the robustness of vqa models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 2042–2051

    2021

  6. [14]

    Discovering the real association: Multimodal causal reasoning in video question answering,

    C. Zang, H. Wang, M. Pei, and W. Liang, “Discovering the real association: Multimodal causal reasoning in video question answering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19 027–19 036

    2023

  7. [15]

    Coun- terfactual vqa: A cause-effect look at language bias,

    Y . Niu, K. Tang, H. Zhang, Z. Lu, X.-S. Hua, and J.-R. Wen, “Coun- terfactual vqa: A cause-effect look at language bias,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 12 700–12 710. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. ...

    2021

  8. [16]

    Beyond question- based biases: Assessing multimodal shortcut learning in visual question answering,

    C. Dancette, R. Cadene, D. Teney, and M. Cord, “Beyond question- based biases: Assessing multimodal shortcut learning in visual question answering,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1574–1583

    2021

  9. [17]

    Roses are red, violets are blue... but should vqa expect them to?

    C. Kervadec, G. Antipov, M. Baccouche, and C. Wolf, “Roses are red, violets are blue... but should vqa expect them to?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 2776–2785

    2021

  10. [18]

    Human-adversarial visual question answer- ing,

    S. Sheng, A. Singh, V . Goswami, J. Magana, T. Thrush, W. Galuba, D. Parikh, and D. Kiela, “Human-adversarial visual question answer- ing,” Advances in Neural Information Processing Systems , vol. 34, pp. 20 346–20 359, 2021

    2021

  11. [19]

    A survey on curriculum learning,

    X. Wang, Y . Chen, and W. Zhu, “A survey on curriculum learning,”IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 9, pp. 4555–4576, 2021

    2021

  12. [20]

    Curriculum learning: A survey,

    P. Soviany, R. T. Ionescu, P. Rota, and N. Sebe, “Curriculum learning: A survey,” International Journal of Computer Vision , vol. 130, no. 6, pp. 1526–1565, 2022

    2022

  13. [21]

    Tgif-qa: Toward spatio- temporal reasoning in visual question answering,

    Y . Jang, Y . Song, Y . Yu, Y . Kim, and G. Kim, “Tgif-qa: Toward spatio- temporal reasoning in visual question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 2758–2766

    2017

  14. [22]

    Revisiting the

    S. Buch, C. Eyzaguirre, A. Gaidon, J. Wu, L. Fei-Fei, and J. C. Niebles, “Revisiting the” video” in video-language understanding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2917–2927

    2022

  15. [23]

    Answering from sure to uncertain: Uncertainty-aware curriculum learning for video question answering,

    H. Li, Q. Ke, M. Gong, and T. Drummond, “Answering from sure to uncertain: Uncertainty-aware curriculum learning for video question answering,” arXiv preprint arXiv:2401.01510 , 2024

    2024

  16. [24]

    Question-guided erasing-based spatiotemporal attention learning for video question answering,

    F. Liu, J. Liu, R. Hong, and H. Lu, “Question-guided erasing-based spatiotemporal attention learning for video question answering,” IEEE Transactions on Neural Networks and Learning Systems , vol. 34, no. 3, pp. 1367–1379, 2021

    2021

  17. [25]

    Memory augmented deep recurrent neural network for video question answering,

    C. Yin, J. Tang, Z. Xu, and Y . Wang, “Memory augmented deep recurrent neural network for video question answering,” IEEE transactions on neural networks and learning systems , vol. 31, no. 9, pp. 3159–3167, 2019

    2019

  18. [26]

    Knowledge-routed visual question reasoning: Challenges for deep representation embedding,

    Q. Cao, B. Li, X. Liang, K. Wang, and L. Lin, “Knowledge-routed visual question reasoning: Challenges for deep representation embedding,” IEEE Transactions on Neural Networks and Learning Systems , vol. 33, no. 7, pp. 2758–2767, 2021

    2021

  19. [27]

    Multitask learning for visual question answering,

    J. Ma, J. Liu, Q. Lin, B. Wu, Y . Wang, and Y . You, “Multitask learning for visual question answering,” IEEE Transactions on neural networks and learning systems , vol. 34, no. 3, pp. 1380–1394, 2021

    2021

  20. [28]

    Bilinear graph networks for visual ques- tion answering,

    D. Guo, C. Xu, and D. Tao, “Bilinear graph networks for visual ques- tion answering,” IEEE Transactions on neural networks and learning systems, vol. 34, no. 2, pp. 1023–1034, 2021

    2021

  21. [29]

    Bilateral cross-modality graph matching attention for feature fusion in visual question answering,

    J. Cao, X. Qin, S. Zhao, and J. Shen, “Bilateral cross-modality graph matching attention for feature fusion in visual question answering,”IEEE Transactions on Neural Networks and Learning Systems , 2022

    2022

  22. [30]

    Bridging the cross- modality semantic gap in visual question answering,

    B. Wang, Y . Ma, X. Li, J. Gao, Y . Hu, and B. Yin, “Bridging the cross- modality semantic gap in visual question answering,” IEEE Transactions on Neural Networks and Learning Systems , 2024

    2024

  23. [31]

    Latent attention network with position perception for visual question answering,

    J. Zhang, X. Liu, and Z. Wang, “Latent attention network with position perception for visual question answering,” IEEE Transactions on Neural Networks and Learning Systems , 2024

    2024

  24. [32]

    Webly supervised knowledge-embedded model for visual reasoning,

    W. Zheng, L. Yan, W. Zhang, and F.-Y . Wang, “Webly supervised knowledge-embedded model for visual reasoning,” IEEE Transactions on Neural Networks and Learning Systems , 2023

    2023

  25. [33]

    Uncovering the temporal context for video question answering,

    L. Zhu, Z. Xu, Y . Yang, and A. G. Hauptmann, “Uncovering the temporal context for video question answering,” International Journal of Computer Vision , vol. 124, pp. 409–421, 2017

    2017

  26. [34]

    Video question answering via gradually refined attention over appearance and motion,

    D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y . Zhuang, “Video question answering via gradually refined attention over appearance and motion,” in Proceedings of the 25th ACM international conference on Multimedia, 2017, pp. 1645–1653

    2017

  27. [35]

    Divide and conquer: Question-guided spatio-temporal contextual attention for video question answering,

    J. Jiang, Z. Chen, H. Lin, X. Zhao, and Y . Gao, “Divide and conquer: Question-guided spatio-temporal contextual attention for video question answering,” in Proceedings of the AAAI Conference on Artificial Intel- ligence, vol. 34, no. 07, 2020, pp. 11 101–11 108

    2020

  28. [36]

    Video question answering with spatio-temporal reasoning,

    Y . Jang, Y . Song, C. D. Kim, Y . Yu, Y . Kim, and G. Kim, “Video question answering with spatio-temporal reasoning,” International Journal of Computer Vision, vol. 127, no. 10, pp. 1385–1412, 2019

    2019

  29. [37]

    Dualvgr: A dual-visual graph reasoning unit for video question answering,

    J. Wang, B.-K. Bao, and C. Xu, “Dualvgr: A dual-visual graph reasoning unit for video question answering,” IEEE Transactions on Multimedia , vol. 24, pp. 3369–3380, 2021

    2021

  30. [38]

    Bridge to answer: Structure-aware graph interaction network for video question answering,

    J. Park, J. Lee, and K. Sohn, “Bridge to answer: Structure-aware graph interaction network for video question answering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15 526–15 535

    2021

  31. [39]

    Motion-appearance co-memory networks for video question answering,

    J. Gao, R. Ge, K. Chen, and R. Nevatia, “Motion-appearance co-memory networks for video question answering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 6576–6585

    2018

  32. [40]

    Heterogeneous memory enhanced multimodal attention model for video question answering,

    C. Fan, X. Zhang, S. Zhang, W. Wang, C. Zhang, and H. Huang, “Heterogeneous memory enhanced multimodal attention model for video question answering,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 1999–2007

    2019

  33. [41]

    Hierarchical conditional relation networks for video question answering,

    T. M. Le, V . Le, S. Venkatesh, and T. Tran, “Hierarchical conditional relation networks for video question answering,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9972–9981

    2020

  34. [42]

    Verbs in action: Improving verb understanding in video-language models,

    L. Momeni, M. Caron, A. Nagrani, A. Zisserman, and C. Schmid, “Verbs in action: Improving verb understanding in video-language models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 579–15 591

    2023

  35. [43]

    When and why vision-language models behave like bags-of-words, and what to do about it?

    M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou, “When and why vision-language models behave like bags-of-words, and what to do about it?” in The Eleventh International Conference on Learning Representations, 2022

    2022

  36. [44]

    Teaching structured vision & language concepts to vision & language models,

    S. Doveh, A. Arbelle, S. Harary, E. Schwartz, R. Herzig, R. Giryes, R. Feris, R. Panda, S. Ullman, and L. Karlinsky, “Teaching structured vision & language concepts to vision & language models,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2023

  37. [45]

    Rubi: Reducing unimodal biases for visual question answering,

    R. Cadene, C. Dancette, M. Cord, D. Parikh et al. , “Rubi: Reducing unimodal biases for visual question answering,” Advances in neural information processing systems , vol. 32, 2019

    2019

  38. [46]

    Don’t just assume; look and answer: Overcoming priors for visual question answering,

    A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi, “Don’t just assume; look and answer: Overcoming priors for visual question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4971–4980

    2018

  39. [47]

    Overcoming language priors in visual question answering with adversarial regularization,

    S. Ramakrishnan, A. Agrawal, and S. Lee, “Overcoming language priors in visual question answering with adversarial regularization,” Advances in Neural Information Processing Systems , vol. 31, 2018

    2018

  40. [48]

    Reliable visual question answering: Abstain rather than answer incorrectly,

    S. Whitehead, S. Petryk, V . Shakib, J. Gonzalez, T. Darrell, A. Rohrbach, and M. Rohrbach, “Reliable visual question answering: Abstain rather than answer incorrectly,” in European Conference on Computer Vision . Springer, 2022, pp. 148–166

    2022

  41. [49]

    Addressing failure prediction by learning model confidence,

    C. Corbi `ere, N. Thome, A. Bar-Hen, M. Cord, and P. P´erez, “Addressing failure prediction by learning model confidence,” Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  42. [50]

    Combating label noise in deep learning using absten- tion,

    S. Thulasidasan, T. Bhattacharya, J. Bilmes, G. Chennupati, and J. Mohd-Yusof, “Combating label noise in deep learning using absten- tion,” arXiv preprint arXiv:1905.10964 , 2019

    1905 arXiv

  43. [51]

    The art of abstention: Selective prediction and error regularization for natural language processing,

    J. Xin, R. Tang, Y . Yu, and J. Lin, “The art of abstention: Selective prediction and error regularization for natural language processing,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference o...

    2021

  44. [52]

    On the foundations of noise-free selective classifi- cation

    R. El-Yaniv et al., “On the foundations of noise-free selective classifi- cation.” Journal of Machine Learning Research , vol. 11, no. 5, 2010

    2010

  45. [53]

    Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings,

    N. Varshney, S. Mishra, and C. Baral, “Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings,” arXiv preprint arXiv:2203.00211 , 2022

    2022 arXiv

  46. [54]

    Selectivenet: A deep neural network with an integrated reject option,

    Y . Geifman and R. El-Yaniv, “Selectivenet: A deep neural network with an integrated reject option,” in International conference on machine learning. PMLR, 2019, pp. 2151–2159

    2019

  47. [55]

    Selective question answering under domain shift,

    A. Kamath, R. Jia, and P. Liang, “Selective question answering under domain shift,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020, pp. 5684–5696

    2020

  48. [56]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  49. [57]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  50. [58]

    Quo vadis, action recognition? a new model and the kinetics dataset,

    J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 6299–6308. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13

    2017

  51. [59]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips,

    A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic, “Howto100m: Learning a text-video embedding by watching hundred million narrated video clips,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 2630–2640

    2019

  52. [60]

    Ava: A video dataset of spatio-temporally localized atomic visual actions,

    C. Gu, C. Sun, D. A. Ross, C. V ondrick, C. Pantofaru, Y . Li, S. Vi- jayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar et al. , “Ava: A video dataset of spatio-temporally localized atomic visual actions,” in Proceedings of the IEEE conference on computer vision and pattern...

    2018

  53. [61]

    Frozen in time: A joint video and image encoder for end-to-end retrieval,

    M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1728–1738

    2021

  54. [62]

    Next-qa: Next phase of question-answering to explaining temporal actions,

    J. Xiao, X. Shang, A. Yao, and T.-S. Chua, “Next-qa: Next phase of question-answering to explaining temporal actions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 9777–9786

    2021

  55. [63]

    Hierarchical object-oriented spatio-temporal reasoning for video question answering,

    L. H. Dang, T. M. Le, V . Le, and T. Tran, “Hierarchical object-oriented spatio-temporal reasoning for video question answering,” arXiv preprint arXiv:2106.13432, 2021

    2021 arXiv

  56. [64]

    Less is more: Clipbert for video-and-language learning via sparse sampling,

    J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu, “Less is more: Clipbert for video-and-language learning via sparse sampling,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 7331–7341

    2021

  57. [65]

    Causal inference in natural language processing: Estimation, prediction, interpretation and beyond,

    A. Feder, K. A. Keith, E. Manzoor, R. Pryzant, D. Sridhar, Z. Wood- Doughty, J. Eisenstein, J. Grimmer, R. Reichart, M. E. Roberts et al. , “Causal inference in natural language processing: Estimation, prediction, interpretation and beyond,” Transactions of the Association for...

    2022

  58. [66]

    Docogen: Domain counterfactual generation for low resource domain adaptation,

    N. Calderon, E. Ben-David, A. Feder, and R. Reichart, “Docogen: Domain counterfactual generation for low resource domain adaptation,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022, pp. 7727– 7746

    2022

  59. [67]

    Polyjuice: Au- tomated, general-purpose counterfactual generation,

    T. Wu, M. T. Ribeiro, J. Heer, and D. S. Weld, “Polyjuice: Au- tomated, general-purpose counterfactual generation,” arXiv preprint arXiv:2101.00288, vol. 1, no. 2, 2021

    2021 arXiv

  60. [68]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

    2019

  61. [69]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019. [Online]. Available: http: //arxiv.org/...

    2019 arXiv

  62. [70]

    Stacked attention networks for image question answering,

    Z. Yang, X. He, J. Gao, L. Deng, and A. Smola, “Stacked attention networks for image question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 21– 29

    2016

  63. [71]

    Multimodal compact bilinear pooling for visual question answering and visual grounding,

    A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach, “Multimodal compact bilinear pooling for visual question answering and visual grounding,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , 2016, pp. 457– 468

    2016

  64. [72]

    Vqa: Visual question answering,

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” inProceedings of the IEEE international conference on computer vision , 2015, pp. 2425–2433

    2015

  65. [73]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 6904–6913

    2017

  66. [74]

    Mvbench: A comprehensive multi-modal video understanding benchmark,

    K. Li, Y . Wang, Y . He, Y . Li, Y . Wang, Y . Liu, Z. Wang, J. Xu, G. Chen, P. Luo et al., “Mvbench: A comprehensive multi-modal video understanding benchmark,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 22 195–22 206

    2024

  67. [75]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms,

    Z. Cheng, S. Leng, H. Zhang, Y . Xin, X. Li, G. Chen, Y . Zhu, W. Zhang, Z. Luo, D. Zhao et al. , “Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms,” arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.