{"id":"37fa96f8-2474-46f8-a1c3-f4303a8eba61","arxiv_id":"2501.08771","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training VideoQA models to answer 'unknown' on deliberately altered questions improves accuracy by one to two points on six benchmarks and on image QA.","lead":"This paper teaches video question-answering models to say 'unknown' when the question has been altered, so they stop guessing answers from wording alone. The method improves accuracy by one to two points on standard VideoQA benchmarks and also transfers to image question answering.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's premise that intervened question-video pairs are almost always unanswerable is unquantified, especially for Polyjuice-perturbed and displaced free-form questions; if wrong labels are frequent, the reported gains may not reflect a general debiasing mechanism.","rationale":"Read in good faith: the paper's framework is clear, model-agnostic, and the reported gains are consistent across base models. The strongest claim is an empirical one, and it would be supported if a single clean mechanism—training on correctly labeled unanswerable pairs—improves test accuracy. The load-bearing assumption is that the constructed labels are correct. The paper acknowledges the risk but does not quantify it; the argument in Section IV-B is tailored to template questions with unique adjectives and does not cover displacement or Polyjuice perturbations of free-form questions. A wrong label is more than benign noise in MCQA because the option set deliberately includes the original correct answer, so the model is taught to reject a correct answer. Table III's roughly 50% 'admit ignorance' rate on TGIF-Action held-out interventions makes the unanswerability assumption empirically questionable (or at least not demonstrated). The reader's weakest assumption was identical, so I agree. The lack of error bars is a real secondary concern—single-run +1-point deltas can be within noise—but I do not put it ahead of the label-correctness issue because even a statistically significant gain would not establish the mechanism if the labels are unreliable. A human audit of sampled interventions would settle the primary concern. Given the concern is checkable and the paper has supporting ablations, the appropriate verdict remains CONDITIONAL; this pass does not change the reader's verdict.","tokens_in":20651,"tokens_out":8149,"duration_ms":85940,"concrete_test":"Run a human-annotation audit: sample 200 displacement and 200 perturbation examples per dataset from the training distributions used in Tables I-II, and have at least three annotators judge whether the intervened question is answerable from the video and, for MCQA, whether the original answer remains correct. Report inter-annotator agreement and the label-error rate. If the label-error rate exceeds, say, 5% on any dataset, the 'extremely infrequent' premise in Section IV-B is falsified, and the authors should re-run the main comparison with only verified unanswerable interventions (or with corrected labels) to see whether the gains persist. Also report the semantic-distance threshold used to keep augmentations as original-answer examples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central training signal assumes that every intervened pair (V,Q') is genuinely unanswerable, so 'unknown'/'not given' is the correct label. Section IV-B states this may fail but asserts it is 'extremely infrequent,' and the supporting argument only covers template-based TGIF questions with distinguishing adjectives. It does not cover displacement (a question from another video can still match the current video), nor Polyjuice perturbations of free-form questions in NExT-QA/MSVD/MSRVTT, where the perturbed entity/action/number may actually be present. The semantic-distance threshold used to separate augmentations from interventions (Section V-A1) is not reported, so the fraction of mislabeled examples cannot be audited. This is especially consequential for MCQA, because the option set retains the original correct answer (Section IV-C): if the intervened question is answerable, the model is explicitly trained to reject a valid answer in favor of 'not given.' Table III is consistent with this worry: on TGIF-Action, InternVideo-L admits ignorance on only 50.0% of displacements and 49.6% of perturbations, so roughly half of held-out interventions are not confidently rejected; without a human baseline for true answerability, this leaves open that many interventions are not actually unanswerable. If the label-error rate is material, the reported +1.1-2.4 point gains may reflect a noisy-label regularizer or dataset-specific quirks rather than the claimed debiasing mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training framework for VideoQA in which the model is trained to admit ignorance when presented with an intervened question. Two interventions are used: displacement (replacing the question with one from another video-question pair) and perturbation (replacing crucial words in the question). For multi-choice QA, a \"not given\" option is added; for open-ended QA, an extra \"unknown\" logit is introduced with a soft loss weighted by the semantic distance between original and intervened questions. A curriculum schedule anneals the intervention probability to zero during training. The framework is applied to InternVideo, All-in-one, HQGA, and VideoLLaMA2, with reported accuracy gains of roughly 0.5 to 2.4 points on TGIF-Action, TGIF-Transition, NExT-QA, TGIF-FrameQA, MSVD-QA, MSRVTT-QA, and two ImageQA datasets.","tokens_in":20993,"tokens_out":4896,"duration_ms":49759,"significance":"If the reported gains hold, this is a simple, model-agnostic, and cost-free-at-inference recipe for improving VideoQA accuracy by discouraging question-answer shortcuts. The paper's strengths include consistent improvements across multiple dataset types, multiple base models (including a modern LLM-based model), controlled comparisons against naive text augmentation, and ablations on curriculum schedules. The main deficit is evidentiary: no error bars or multiple seeds are reported, the central unanswerability assumption for intervened pairs is not quantified, and an important semantic-distance threshold is not reported, leaving the mechanism claim only partially supported.","major_comments":[{"comment":"The headline improvements (e.g., +1.9 on TGIF-Action, +1.7 on TGIF-Transition, +1.1 on MSRVTT-QA) are reported as single numbers without error bars or multiple seeds. Since several gains are around 1 point or less, the reader cannot assess whether these differences are statistically meaningful. Please report the mean and standard deviation over at least three runs with different seeds, or provide a statistical significance test.","section":"Tables I and II, Section V-B/V-C"},{"comment":"The load-bearing premise is that an intervened video-question pair (V,Q') is almost always unanswerable, so 'not given' or 'unknown' is the correct label. The justification in Section IV-B only covers template-based TGIF questions where an adjective uniquely identifies a person. It does not cover displacement, where a question from another video can accidentally match the current video, nor Polyjuice perturbations of free-form questions in NExT-QA, MSVD-QA, and MSRVTT-QA, where the changed entity, action, or number may actually be present. In MCQA the original correct answer is explicitly kept in the option set (Section IV-C), so any such mislabeled intervention trains the model to reject a valid answer in favor of 'not given'. Please quantify the mislabeling rate with a human study on a stratified sample of held-out interventions (by dataset and intervention type), and report the semantic-distance threshold used in Section V-A1 to separate augmentations from interventions.","section":"Section IV-B and Section IV-C"},{"comment":"The 'Is the Model Admitting Ignorance?' experiment evaluates on the same intervention distribution used during training, so it largely verifies that the training objective was learned rather than testing whether 'not given' corresponds to genuine unanswerability. On TGIF-Action, InternVideo-L admits ignorance on only 50.0% of displacements and 49.6% of perturbations; without a human-annotated answerability baseline for these same cases, these numbers are ambiguous. Please provide an answerability oracle or a human study for a sample of the test interventions to interpret Table III.","section":"Section V-D, Table III"},{"comment":"The initial intervention probability p_r and the displacement ratio are selected based on validation accuracy (Fig. 4), and the activation threshold used to define a successful 'ignorance' admission in Table III is not reported. This leaves open the possibility that the reported gains depend on per-dataset tuning, especially because the OEQA loss in Eq. (14) uses the continuous semantic distance d that itself depends on the unreported Sentence-BERT threshold. Please report the chosen values per dataset, the activation threshold, and a sensitivity analysis across seeds.","section":"Section V-A2, Eq. (14), Table III"}],"minor_comments":[{"comment":"The caption reads 'The casual graph of VideoQA'; it should be 'causal graph'.","section":"Fig. 2 caption"},{"comment":"The dataset name 'MSRTT-QA' is a typo; it should be 'MSRVTT-QA'.","section":"Section V-A1"},{"comment":"The schedule p(e) = (p_r/E^2)(e-E)^2 does not equal p_r at e=1 for finite E; the text says the probability 'decreases from p_r to 0'. Please clarify whether the schedule is intended to start near p_r or define the epoch index differently.","section":"Section V-A2, Eq. (15)"},{"comment":"For the OEQA 'admit ignorance' evaluation, the threshold on the last logit is mentioned but its value and selection procedure are not given; please specify it in the experimental settings.","section":"Section V-D"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the proposed idea is attractive, but I would not accept it in its current form. The central risk is that the intervention labels may be noisy for free-form and displaced questions, and the current evidence does not rule out that the gains come from a noisy-label regularizer rather than the claimed debiasing mechanism. A human answerability study, error bars, and disclosure of the semantic-distance threshold would substantially strengthen the paper. I would also encourage the editor to ask the authors to release the code and the exact intervention-generation configuration, since the reported improvements are modest and the current text does not provide enough detail to reproduce them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean empirical paper with a genuinely different intervention target. Prior debiasing work in VideoQA intervenes on videos (IGV, EIGV, TIGV) or answers (MCR); this one intervenes on questions and trains the model to output an explicit ignorance signal with soft labels and a curriculum. That combination is new as far as the cited literature goes, and it works: AIQA beats InternVideo by +1.1 to +1.9 across six datasets, and the gains replicate on HQGA and several All-in-one variants, plus ImageQA. The method is simple, model-agnostic, adds no inference cost, and the ablations are reasonably thorough—fixed vs. dynamic intervention schedules, displacement ratio, option-set design, and a comparison with naive text augmentation.\n\nThe soft spots are real but mostly addressable. There are no error bars or multiple seeds anywhere; the improvements are around one to two points, and with per-dataset hyperparameters (p_r, displacement ratio) chosen on validation, it's hard to rule out that some of that is tuning noise. The semantic-distance threshold that separates augmentation from intervention isn't reported, so the actual intervention distribution can't be audited. And no code.\n\nThe bigger conceptual worry is the label assumption. The paper trains on the belief that every intervened pair (V, Q') is genuinely unanswerable, so 'not given'/'unknown' is the right answer. That is plausible for template-based TGIF questions with distinguishing adjectives, but much less obviously true for displaced questions (a question from another video can still match the current video) or for Polyjuice-perturbed free-form questions where the swapped entity/action/number may actually be present. The paper states these cases are 'extremely infrequent' but only supports that for the template case, and Table III hints the problem is not negligible: InternVideo-L admits ignorance on only ~50% of perturbations in TGIF-Action. If the label-error rate is material, the gains may come from a noisy-label regularizer rather than the claimed debiasing mechanism. That doesn't sink the paper—the main accuracy claims are on original held-out splits and would survive even under that alternative explanation—but the interpretation needs tightening, and a human-annotated or model-based check of intervention answerability would settle it.\n\nWho should read this: anyone working on VideoQA debiasing or robust training for vision-language models. It deserves a serious referee: the idea is novel, the experiments are broad, and the weaknesses are fixable. I'd recommend sending it to review, and asking the authors for seeds, code, the threshold value, and a quantitative estimate of mislabeled interventions.","headline":"A simple, model-agnostic training trick for VideoQA—make models say 'unknown' on intervened questions—gives consistent gains across datasets, but missing error bars and an unquantified label-noise assumption keep it from being a clean accept.","tokens_in":21517,"tokens_out":2814,"would_cite":true,"duration_ms":27725,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training VideoQA models to admit ignorance on intervened questions breaks spurious question-answer correlations and improves accuracy on multiple benchmarks.","keywords":["video question answering","spurious correlations","admitting ignorance","question intervention","curriculum learning","model-agnostic debiasing","multi-choice VideoQA","open-ended VideoQA"],"falsifier":"Measure the fraction of intervened video-question pairs that still admit a valid answer on a given dataset, using human annotation or an oracle model; if that fraction is not tiny, retraining the same base model with this recipe should produce little or no improvement.","tokens_in":20478,"feed_emoji":"🎥","tokens_out":4684,"duration_ms":41022,"temperature":0.7,"pith_summary":"This paper proposes that VideoQA models should be trained to admit ignorance on deliberately broken video-question pairs, rather than always predicting an answer. The authors argue that standard training encourages models to exploit spurious correlations between question text and answer, especially when video-text alignment is imperfect. By intervening on questions—swapping in another pair's question or altering crucial words—and forcing the model to answer \"unknown\" or \"not given,\" the model is pushed to rely on the joint video-question representation. On six benchmarks, including TGIF-Action, TGIF-Transition, NExT-QA, MSVD-QA, and MSRVTT-QA, the framework improves accuracy by about 1 to 2 points over strong pretrained baselines, with no extra inference cost.","feed_headline":"Training models to say \"I don't know\" lifts VideoQA by ~2 points","feed_subtitle":"Swapping or perturbing questions during training breaks question-answer shortcuts, improving accuracy on six benchmarks with no inference…","key_machinery":"The load-bearing mechanism is the question intervention paired with an \"admit ignorance\" training signal. Displacement (global replacement) creates easy inconsistency, teaching coarse video-question correspondence; perturbation (local word replacement) detects subtle mismatches, teaching fine-grained alignment. On the answer side, multi-choice tasks add a \"not given\" option mixed into a freshly sampled option set, while open-ended tasks add a separate logit trained with soft labels proportional to the semantic distance $d = D(Q, Q')$ between the original and intervened question. An ignorance-diminishing curriculum schedule $p(e) = \\frac{p_r}{E^2}(e-E)^2$ balances the two learning goals.","core_discovery":"The central claim is that breaking the spurious question-to-answer path forces a VideoQA model to learn more robust video-text alignment, and that this can be achieved purely through data intervention and a modified training objective. The paper introduces two question interventions: displacement, which replaces a question with one from a different video-question pair, and perturbation, which changes only crucial words such as subjects, adjectives, or numbers. The model is trained to say \"not given\" (multi-choice) or activate an ignorance logit (open-ended) on intervened pairs, and to answer normally on original pairs. A curriculum that gradually decreases the intervention probability from $p_r$ to zero lets the model first learn to detect inconsistency and then focus on correct answering. The paper reports consistent gains when the recipe is applied to InternVideo, HQGA, All-in-one, and VideoLLaMA2, and even to image QA models, with improvements such as +1.9 on TGIF-Action and +1.9 on NExT-QA.","pith_inferences":["The \"admit ignorance\" signal acts as a regularizer on the question encoder, and could be repurposed as a calibration or uncertainty measure at test time, although the paper does not propose this.","If applied to non-template, natural-language questions, the assumption that intervened pairs are unanswerable is weaker, so the method may need semantic-distance thresholds tuned per dataset.","The approach could be combined with video-side interventions to break both video-answer and question-answer shortcuts simultaneously."],"forward_implications":["Any existing VideoQA model can be retrained with this data-only recipe and see similar gains, since no architectural change is required.","The framework transfers to image QA, suggesting the principle applies beyond video.","Because the intervention probability decays to zero, training time and inference cost stay identical to the base fine-tuning.","Gains are largest on multi-choice datasets where spurious question-answer correlations are strongest, offering a diagnostic for shortcut reliance."],"supporting_citations":[{"why":"InternVideo, the pretrained video-text foundation model used as the primary base model that the framework is applied to and improved upon.","marker":"[7]"},{"why":"IGV, the causal-grounded VideoQA framework that breaks video-answer spurious correlations, providing the causal graph and a key comparison baseline.","marker":"[9]"},{"why":"TGIF-QA, source of the TGIF-Action, TGIF-Transition, and TGIF-FrameQA datasets used in evaluation.","marker":"[21]"},{"why":"MSVD-QA and MSRVTT-QA, open-ended VideoQA datasets used for evaluation.","marker":"[34]"},{"why":"NExT-QA, multi-choice VideoQA dataset used for evaluation.","marker":"[62]"},{"why":"Polyjuice, the model used to generate local perturbations for free-form questions.","marker":"[67]"},{"why":"Sentence-BERT, which computes the semantic distance D(Q,Q') used for soft ignorance labels and thresholds.","marker":"[69]"},{"why":"VideoLLaMA2, the LLM-based VideoQA model demonstrating that the framework generalizes to modern architectures.","marker":"[75]"}],"fun_headline_variants":["Saying 'I don't know' boosts VideoQA by 2 points","Teach VideoQA to admit ignorance, gain 2 points","Question interventions break shortcuts, lift VideoQA","Admitting ignorance improves VideoQA accuracy","Forcing VideoQA to say 'not given' gains 2 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's labels for intervened pairs assume that a displaced or perturbed question almost never has a valid answer in the video, and the paper asserts such cases are \"extremely infrequent\".","fun_headline_variants_meta":{"raw":{"variants":["Saying 'I don't know' boosts VideoQA by 2 points","Teach VideoQA to admit ignorance, gain 2 points","Question interventions break shortcuts, lift VideoQA","Admitting ignorance improves VideoQA accuracy","Forcing VideoQA to say 'not given' gains 2 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1258,"prompt_tokens":941,"completion_tokens":317,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":236}},"tokens_in":557,"tokens_out":317,"duration_ms":4035,"temperature":1.0,"reasoning_tokens":236,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:18:04.380793+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the fraction of intervened video-question pairs that still admit a valid answer on a given dataset, using human annotation or an oracle model; if that fraction is not tiny, retraining the same base model with this recipe should produce little or no improvement.","supporting_citations":[{"cited_title":"Invariant grounding for video question answering,","cited_arxiv_id":null,"evidence_quote":"IGV, the causal-grounded VideoQA framework that breaks video-answer spurious correlations, providing the causal graph and a key comparison baseline."},{"cited_title":"Tgif-qa: Toward spatio- temporal reasoning in visual question answering,","cited_arxiv_id":null,"evidence_quote":"TGIF-QA, source of the TGIF-Action, TGIF-Transition, and TGIF-FrameQA datasets used in evaluation."},{"cited_title":"Video question answering via gradually refined attention over appearance and motion,","cited_arxiv_id":null,"evidence_quote":"MSVD-QA and MSRVTT-QA, open-ended VideoQA datasets used for evaluation."},{"cited_title":"Next-qa: Next phase of question-answering to explaining temporal actions,","cited_arxiv_id":null,"evidence_quote":"NExT-QA, multi-choice VideoQA dataset used for evaluation."}],"review_version":1}