REVIEW 3 major objections 5 minor 57 references
FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read FADE shows that an 8B open video model can be trained to independently discover and explain counterfactual events as textual guidance fades, retaining 90.4% accuracy on open-ended questions versus GPT-5.6's 48.1%.
desk verdict FADE identifies a real evaluation weakness and a plausible fix, but the unsupported semantic verifier and fine-tuned-vs-zero-shot comparison make the headline numbers unverifiable as reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a three-level fading textual-anchor curriculum over the same video: MCQ with question and options, OQA with question only, and captioning with a generic instruction. The RL reward is prefix-gated so a success at a weaker-anchor level counts only if all stronger levels succeeded, paired with a direction reward that requires the model to call the counterfactual video anomalous and its factual counterpart normal, plus a format reward; these are normalized by GDPO's group statistics. The SFT stage supplies the evidence backbone: a localized evidence projection pulls response hidden states toward the annotated anomalous interval's visual prototype, and response-conditioned evidence re-grounding (RCER) distills the interval's content and temporal distribution into whole-video retrieval from response tokens. The named mechanism is the fading anchor itself, and the shared sharp declines across models in the paper's anchor-fading profiles are the diagnosis it exposes.
What would settle it
Replace the unspecified semantic verifier with a different published verifier or with human raters on DualityVidQA-test and IPV-Bench, and check whether FADE's OQA and captioning retention (90.4% and 67.4%) and its lead over GPT-5.6 survive; as a complementary test, give the model only caption prompts on videos whose anomaly types never appear in its training set and see whether it still localizes and describes them.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that the gap between MCQ and open-ended performance on counterfactual video benchmarks is not fixed: it can be closed by training. Specifically, FADE establishes that an 8B open Video-MLLM can internally encode the decisive visual evidence and keep using it when instance-specific textual anchors (question plus options) are absent, with strict paired accuracy of 84.6 (MCQ), 76.5 (OQA), and 57.0 (captioning) on DualityVidQA-test 'Both'—retentions of 90.4% and 67.4%—and 91.2/78.6/60.2 on IPV-Bench, exceeding GPT-5.6 in all three formats. The authors attribute this to an evidence-first curriculum: Stage I makes responses recover the anomalous interval's content and temporal location from the whole video, and Stage II rewards correct counterfactual judgments even as textual guidance fades away, including on paired factual videos to prevent an always-counterfactual shortcut.
Load-bearing premise
Every OQA and captioning score in the paper is decided by an unnamed 'frozen semantic verifier' and a caption 'discovery check' whose model, prompts, and thresholds are never specified, so if that verifier rewards wording close to the original correct option, the reported retention gains could reflect the verifier's expectations rather than genuine open-ended discovery.
Editorial extensions
If this is right
- MCQ-only counterfactual benchmarks systematically overestimate Video-MLLM understanding, since high scores can come from verifying text-specified hypotheses.
- An 8B open model trained with evidence-internalized SFT and fading-anchor RL can serve as a strong baseline for unconstrained counterfactual discovery, beating closed-source GPT-5.6 without scaled-up parameters.
- The evaluation protocol can be applied to any existing public MCQ benchmark, producing an anchor-fading profile that separates discovery capability from text-guided verification.
- The sharp decline in GPT-5.6, Gemini-2.5 Pro, and Qwen3-VL as anchors fade indicates that textual-anchor dependence is widespread across open and closed models, not a quirk of one architecture.
- SFT, RL, and progressive fading are complementary: ablations show that removing any one of them widens the gap between MCQ and open-ended performance.
Reading between the lines
- Editorial inference: the same anchor-fading protocol could be used to diagnose robustness in other open-ended video tasks, such as generic anomaly detection or temporal grounding, by measuring how much performance drops when the query is stripped of instance-specific clues.
- Editorial inference: because the semantic verifier is unnamed and its thresholds undisclosed, cross-model comparisons on OQA and captioning are hard to interpret; a public, fixed verifier would make retention numbers comparable across future papers.
- Editorial inference: FADE is trained only on DualityVidQA data with a single 8B base model, so a strong stress test is whether the discovery skill transfers to entirely new anomaly categories, unseen video styles, and other model sizes or architectures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FADE, a two-stage training framework for counterfactual video understanding: evidence-internalized supervised fine-tuning (SFT), which grounds responses in annotated counterfactual evidence via an evidence projector and response-conditioned evidence re-grounding, followed by fading-anchor reinforcement learning (RL) that progressively removes question and option anchors through a scaffold-and-consolidate curriculum. The authors also propose an evaluation protocol that reformulates existing MCQ benchmarks into aligned MCQ, OQA, and captioning tasks. Experiments on DualityVidQA-test and IPV-Bench report state-of-the-art strict paired scores, with FADE (based on Qwen3-VL-8B) achieving 84.6 MCQ, 76.5 OQA, and 57.0 captioning on DualityVidQA-test, retaining 90.4% and 67.4% of its MCQ performance versus GPT-5.6's 48.1% and 30.7%.
Significance. The problem is well motivated: exposing the gap between passive verification and active discovery is an important diagnostic for video MLLMs. The proposed training paradigm is simple and plausible, and the ablation study in Table 2 supports the complementary contributions of SFT, RL, and progressive fading. The IPV-Bench transfer result is encouraging because it reduces overfitting concerns. However, the headline OQA and captioning numbers depend on an unspecified semantic verifier that also appears to be used as the RL reward oracle, so the central claim is not yet independently checkable.
major comments (3)
- [Evaluation Protocol (Eq. 14) and Stage II (Eq. 10)] The OQA and captioning scores in Table 1 are computed by J_OQA and J_CAP, described only as 'a frozen semantic verifier' and as a check that 'the target event is independently discovered and correctly described.' The verifier model, prompts, thresholds, and matching rules are not specified. This is load-bearing because every non-MCQ result in the paper (DualityVidQA OQA 76.5, captioning 57.0; IPV-Bench OQA 78.6, captioning 60.2; and the retention ratios 90.4%/67.4% versus 48.1%/30.7%) passes through this verifier. The same level-ℓ verifier outcomes c_{i,k}^{(ℓ)} are already used as the progress reward in Eq. (10); if the evaluation verifier is the same object as the reward verifier, then the headline metric is the training objective, and the comparison to GPT-5.6 is not an independent evaluation. Please name the verifier, provide its prompts and thresholds, release it with the evaluation code, and re-run the main comparisons with an independent judge.
- [Experimental Setup / Table 1] FADE is fine-tuned on the DualityVidQA training split and evaluated on DualityVidQA-test, while GPT-5.6 and the other baselines are evaluated zero-shot. The abstract's claim that FADE 'outperforms GPT-5.6' is therefore not apples-to-apples on DualityVidQA-test, because the comparison includes the advantage of having been trained on the same benchmark's training data. The IPV-Bench transfer result is more convincing and should be emphasized; for DualityVidQA-test, please add a fine-tuned baseline trained with the same data under a standard recipe, or clearly restrict the claim to the transfer setting.
- [Training Data] The OQA and captioning training references are generated by Qwen3.6-Plus 'using each question and correct option as semantic context.' Since the open-ended evaluation verifier is also semantic and the generated references are derived from the original options, a verifier that rewards phrasings close to the original option or to the reference distribution will systematically favor FADE over models not trained on these references. Please provide evidence that J_CAP and J_OQA are insensitive to option-derived phrasing, for example by reporting a human-annotated sample on both benchmarks or by measuring the verifier's agreement with human judgments on a held-out subset.
minor comments (5)
- [Table 1] The DualityVidQA columns 'Real, CF, Both' are repeated twice, and the IPV-Bench sub-table is visually confusing; please split the table into separate sub-tables for each benchmark.
- [Figure 4] The localization score s_t used in the ECR definition is not defined; please state how it is extracted from the model (for example, attention weights, gradients, or generated timestamps).
- [Eq. (4)] The notation Attn(Q, H_i,S_i, H_i,S_i) is used without a definition; please specify the attention mechanism and the role of the K learnable queries.
- [Table 1] The main tables report single runs without error bars or significance tests; given the stochasticity of sampling-based RL and LLM evaluation, please add multiple seeds or variance estimates for at least the FADE and GPT-5.6 comparisons.
- [Table 2] The 'w/o Prog. RL' variant is described only as jointly training on MCQ, OpenQA, and captioning with identical data and update steps; please specify the mixing ratio across formats so the ablation is reproducible.
Circularity Check
No significant circularity: FADE's training objectives and the reported evaluation scores are not identical by construction, and no claim reduces to a fitted value or a self-citation chain.
full rationale
The paper's derivation chain is self-contained and does not exhibit an equation-level reduction of a prediction to an input. Stage I optimizes NLL plus evidence-alignment losses (Eqs. 1-7) using privileged temporal annotations; these are training objectives, not reported predictions. Stage II maximizes a prefix-gated progress reward (Eq. 10), paired-direction reward (Eq. 11), and format reward via GDPO; the evaluation protocol (Eq. 14) reports the mean of per-sample verifier outcomes. The one transparency gap is that J_OQA and J_CAP are not specified: the paper only says J_OQA is 'a frozen semantic verifier' and J_CAP 'checks whether the target event is independently discovered and correctly described', while Stage II uses 'task-adapted semantic verification' at the same levels and with the same notation c^(l). If the Eq. 10 verifier and Eq. 14 verifier are literally the same object, the OQA/captioning scores would coincide with the RL reward, and the statement that the protocol 'does not participate in FADE optimization' would be misleading. However, the paper asserts the protocol is separate, and the shared notation alone is not a demonstrated identity; under the hard rule requiring an exhibited reduction, this is a reproducibility/fairness concern rather than a circular step. DualityVidQA has overlapping authors with FADE, but it is a public benchmark and the claims do not reduce to a self-citation. No uniqueness theorem is imported and no ansatz is smuggled via citation.
Assumptions & free parameters
free parameters (6)
- lambda_p (projection loss weight) =
not reported
- lambda_e (RCER loss weight) =
not reported
- alpha_1, alpha_2, alpha_3 (RCER alignment weights) =
not reported
- K (number of span-guided queries) =
not reported
- Reward weights w_prog, w_pair, w_fmt =
not reported
- LoRA rank and learning rates =
rank 8; LRs 5e-5 (SFT), 1e-6 (RL)
assumptions (5)
- domain assumption Annotated counterfactual intervals I_i in DualityVidQA correspond to decisive visual evidence.
- domain assumption Qwen3.6-Plus-generated OQA answers and captions are accurate training targets.
- domain assumption Semantic verifiers for OQA and captioning reliably recognize correct target-event identification.
- domain assumption Factual and counterfactual video pairs differ only in the counterfactual event.
- domain assumption Qwen3-VL-8B's response states can carry recoverable visual evidence through the evidence projector.
Cite this review
Pith. "Pith review of FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding." pith.science (2026). https://pith.science/paper/HCSKAR2Q
@misc{pith2026260810764,
author = {Pith},
title = {Pith review of: FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding},
year = {2026},
howpublished = {\url{https://pith.science/paper/HCSKAR2Q}},
note = {Machine review of arXiv:2608.10764}
}
read the original abstract
Counterfactual video understanding evaluates whether models grasp physical and commonsense regularities. However, existing multiple-choice question (MCQ) benchmarks inadvertently leak target events through their questions and candidate options. This reduces the core challenge from active discovery to text-guided verification. In this paper, we present FADE, an effective training framework for counterfactual discovery and explanation. Our method is built on an evidence-first, two-stage training paradigm. First, evidence-internalized supervised fine-tuning grounds the model's predictions in decisive visual anomalies. Second, we apply a fading-anchor reinforcement learning strategy that progressively removes textual guidance, compelling the model to independently discover and explain evidence. To rigorously evaluate this capability, we also introduce an effective pipeline that converts existing MCQ datasets into aligned MCQ, open-ended question answering (OQA), and captioning tasks without requiring additional data curation. Our simple approach yields strong results. Using Qwen3-VL-8B as the baseline, FADE achieves state-of-the-art strict paired scores across all three tasks on DualityVidQA-test and IPV-Bench, outperforming GPT-5.6. In specific, when transitioning from constrained MCQs to unconstrained OQA and captioning, our model demonstrates remarkable robustness. Its performance retention is 90.4% and 67.4% on DualityVidQA-test-substantially higher than the 48.1% and 30.7% retained by GPT-5.6. We hope this simple framework can serve as a solid baseline for future research in unconstrained counterfactual video understanding.
Figures
Reference graph
Works this paper leans on
-
[1]
Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education
Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)
-
[2]
Classification Problem Solving
Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence
-
[3]
, title =
Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =
1980
-
[4]
New Ways to Make Microcircuits Smaller---Duplicate Entry
Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science
-
[5]
Clancey and Glenn Rennels , abstract =
Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =
-
[6]
and Rennels, Glenn R
Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies
-
[7]
Poligon: A System for Parallel Problem Solving
Rice, James. Poligon: A System for Parallel Problem Solving
-
[8]
Transfer of Rule-Based Expertise through a Tutorial Dialogue
Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue
Show all 57 references
-
[9]
The Engineering of Qualitative Models
Clancey, William J. The Engineering of Qualitative Models
-
[10]
2023 , eprint=
Attention Is All You Need , author=. 2023 , eprint=
2023
-
[11]
Pluto: The 'Other' Red Planet
NASA. Pluto: The 'Other' Red Planet
-
[12]
International Conference on Learning Representations , volume=
Physbench: Benchmarking and enhancing vision-language models for physical world understanding , author=. International Conference on Learning Representations , volume=
-
[13]
IEEE Transactions on Circuits and Systems for Video Technology , year=
Video understanding with large language models: A survey , author=. IEEE Transactions on Circuits and Systems for Video Technology , year=
-
[14]
arXiv preprint arXiv:2606.16993 , year=
DreamX-World 1.0: A General-Purpose Interactive World Model , author=. arXiv preprint arXiv:2606.16993 , year=
-
[15]
arXiv preprint arXiv:1910.01442 , year=
Clevrer: Collision events for video representation and reasoning , author=. arXiv preprint arXiv:1910.01442 , year=
1910 arXiv
-
[16]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Next-qa: Next phase of question-answering to explaining temporal actions , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[17]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Quantiphy: A quantitative benchmark evaluating physical reasoning abilities of vision-language models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[18]
arXiv preprint arXiv:2503.14378 , year=
Impossible videos , author=. arXiv preprint arXiv:2503.14378 , year=
-
[19]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Towards visual discrimination and reasoning of real-world physical dynamics: Physics-grounded anomaly detection , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[20]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[21]
arXiv preprint arXiv:2511.19923 , year=
CounterVQA: Evaluating and Improving Counterfactual Reasoning in Vision-Language Models for Video Understanding , author=. arXiv preprint arXiv:2511.19923 , year=
-
[22]
arXiv preprint arXiv:2510.07550 , year=
Travl: A recipe for making video-language models better judges of physics implausibility , author=. arXiv preprint arXiv:2510.07550 , year=
-
[23]
Findings of the Association for Computational Linguistics: ACL 2025 , pages=
Reasoning is all you need for video generalization: A counterfactual benchmark with sub-question evaluation , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=
2025
-
[24]
arXiv preprint arXiv:2511.21631 , year=
Qwen3-vl technical report , author=. arXiv preprint arXiv:2511.21631 , year=
-
[25]
arXiv preprint arXiv:1803.07616 , year=
Intphys: A framework and benchmark for visual intuitive physics reasoning , author=. arXiv preprint arXiv:1803.07616 , year=
-
[26]
arXiv preprint arXiv:2506.09849 , year=
Intphys 2: Benchmarking intuitive physics understanding in complex synthetic environments , author=. arXiv preprint arXiv:2506.09849 , year=
-
[27]
arXiv preprint arXiv:1909.12000 , year=
Cophy: Counterfactual learning of physical dynamics , author=. arXiv preprint arXiv:1909.12000 , year=
1909 arXiv
-
[28]
Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
X-voe: Measuring explanatory violation of expectation in physical events , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
-
[29]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[30]
Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
Don't just assume; look and answer: Overcoming priors for visual question answering , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
-
[31]
Advances in neural information processing systems , volume=
Rubi: Reducing unimodal biases for visual question answering , author=. Advances in neural information processing systems , volume=
-
[32]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Counterfactual vqa: A cause-effect look at language bias , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[33]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Invariant grounding for video question answering , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[34]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Can i trust your answer? visually grounded video question answering , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[35]
arXiv preprint arXiv:2506.09987 , year=
A shortcut-aware video-qa benchmark for physical understanding via minimal video pairs , author=. arXiv preprint arXiv:2506.09987 , year=
-
[36]
arXiv preprint arXiv:2603.29616 , year=
Video-oasis: Rethinking evaluation of video understanding , author=. arXiv preprint arXiv:2603.29616 , year=
-
[37]
arXiv preprint arXiv:2410.10818 , year=
Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models , author=. arXiv preprint arXiv:2410.10818 , year=
-
[38]
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
Addressing blind guessing: Calibration of selection bias in multiple-choice question answering by video language models , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[39]
arXiv preprint arXiv:2410.03290 , year=
Grounded-videollm: Sharpening fine-grained temporal grounding in video large language models , author=. arXiv preprint arXiv:2410.03290 , year=
-
[40]
Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
Enrich and Detect: Video Temporal Grounding with Multimodal LLMs , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
-
[41]
arXiv preprint arXiv:2402.03300 , year=
Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=
-
[42]
Advances in Neural Information Processing Systems , volume=
Video-r1: Reinforcing video reasoning in mllms , author=. Advances in Neural Information Processing Systems , volume=
-
[43]
arXiv preprint arXiv:2504.06958 , year=
Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning , author=. arXiv preprint arXiv:2504.06958 , year=
-
[44]
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track) , pages=
RAVEN: Robust advertisement video violation temporal grounding via reinforcement reasoning , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track) , pages=
-
[45]
Advances in Neural Information Processing Systems , volume=
Time-r1: Post-training large vision language model for temporal video grounding , author=. Advances in Neural Information Processing Systems , volume=
-
[46]
arXiv preprint arXiv:2601.05242 , year=
Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization , author=. arXiv preprint arXiv:2601.05242 , year=
-
[47]
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=. 2024 , url=
2024
-
[48]
2024 , eprint=
SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning , author=. 2024 , eprint=
2024
-
[49]
arXiv preprint arXiv:2601.03267 , year=
Openai gpt-5 system card , author=. arXiv preprint arXiv:2601.03267 , year=
-
[50]
arXiv preprint arXiv:2410.21276 , year=
Gpt-4o system card , author=. arXiv preprint arXiv:2410.21276 , year=
-
[51]
arXiv preprint arXiv:2303.08774 , year=
Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=
-
[52]
arXiv preprint arXiv:2507.06261 , year=
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=
-
[53]
5-VL Technical Report , author=
Qwen2. 5-VL Technical Report , author=. arXiv e-prints , pages=
-
[54]
5: Advancing open-source multimodal models in versatility, reasoning, and efficiency , author=
Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency , author=. arXiv preprint arXiv:2508.18265 , year=
-
[55]
arXiv preprint arXiv:2504.07491 , year=
Kimi-vl technical report , author=. arXiv preprint arXiv:2504.07491 , year=
-
[56]
2026 , month = apr, howpublished =
2026
-
[57]
2026 , month = jul, howpublished =
2026
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.