{"id":"d892c819-0677-426b-aeef-1aa8890be8d7","arxiv_id":"2608.10764","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FADE trains a video MLLM with evidence-internalized SFT plus fading-anchor RL, preserving counterfactual judgment accuracy when MCQ guidance is removed, with 90.4% and 67.4% retention on OQA and captioning on DualityVidQA-test.","lead":"FADE is a two-stage training method that teaches video AI to spot impossible events without being told what to look for, by first grounding answers in visual anomalies and then gradually removing question and multiple-choice guidance. On two benchmarks it keeps much more of its accuracy when questions are stripped away than strong commercial models such as GPT-5.6.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on an unnamed semantic verifier for OQA/captioning that also appears to be the RL reward oracle (Eqs. 10 and 14), so the reported SOTA and retention gains are not independently checkable.","rationale":"The reader's weakest_assumption is the most load-bearing concern, and I agree with it. The paper's central contribution is the anchor-fading evaluation protocol plus FADE's training; the headline claim is that FADE outperforms GPT-5.6 in OQA and captioning. Every non-MCQ number in Table 1 and Table 2 passes through J_OQA/J_CAP (Eq. 14), and the same c_i,k^(ℓ) structure appears in the progress reward (Eq. 10). If the verifier used for evaluation is the same as the one used for RL rewards, then the evaluation is not an external measure of discovery; it is a measure of whether the model matches the verifier's preferred phrasing. The paper's SFT references are generated by Qwen3.6-Plus using the question and correct option as context, so a verifier that rewards semantic similarity to those references would favor FADE by construction. This is a concrete, addressable reproducibility gap, not a claim of fraud. I would not change the CONDITIONAL verdict: the framework is plausible, the ablations are internally consistent, and the ECR analysis in Fig. 4 provides some independent evidence that SFT improves temporal localization. But the OQA/captioning SOTA claim cannot be accepted without releasing the verifier and re-running the evaluation with an independent judge. This is exactly the condition the reader already set.","tokens_in":12844,"tokens_out":4090,"duration_ms":41507,"concrete_test":"Release the verifier implementation (model identity, prompts, thresholds, and the discovery-check matching rule) and have an independent third party re-score the saved outputs of FADE, GPT-5.6, and the Qwen3-VL-8B base on DualityVidQA-test and IPV-Bench using an alternative verifier—for example, a different LLM judge with a pre-registered rubric or human-annotated target-event checklists—and compare per-sample and aggregate OQA/captioning scores; if the alternative verifier changes FADE's OQA/captioning scores or retention gaps by more than a few points, the headline SOTA claim is verifier-dependent. Also state explicitly whether the same verifier was used as the RL reward in Eq. 10.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All non-MCQ results—DualityVidQA OQA 76.5, captioning 57.0, IPV-Bench 78.6/60.2, and the retention ratios that distinguish FADE from GPT-5.6—are produced by J_OQA and J_CAP in Eq. 14. The paper only says J_OQA is 'a frozen semantic verifier' and J_CAP 'checks whether the target event is independently discovered and correctly described'; it does not name the verifier model, give prompts, thresholds, matching rules, or the event checklist used for discovery. This matters doubly because Stage II already uses level-ℓ verifier outcomes c_i,k^(ℓ) in the progress reward (Eq. 10). If the evaluation verifier is the same object as the reward verifier, then the headline metric is the training objective, and the 'independent evaluation protocol' is not independent of FADE's optimization. It would also explain why FADE retains 90.4%/67.4% while GPT-5.6 falls to 48.1%/30.7%: a verifier that rewards phrasing close to the original option (or to the Qwen3.6-Plus references generated from question + correct option) would systematically favor FADE's trained output distribution. The central claim—that an 8B model outperforms GPT-5.6 at open-ended discovery—therefore cannot be assessed until the verifier is specified and re-run independently.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FADE, a two-stage training framework for counterfactual video understanding: evidence-internalized supervised fine-tuning (SFT), which grounds responses in annotated counterfactual evidence via an evidence projector and response-conditioned evidence re-grounding, followed by fading-anchor reinforcement learning (RL) that progressively removes question and option anchors through a scaffold-and-consolidate curriculum. The authors also propose an evaluation protocol that reformulates existing MCQ benchmarks into aligned MCQ, OQA, and captioning tasks. Experiments on DualityVidQA-test and IPV-Bench report state-of-the-art strict paired scores, with FADE (based on Qwen3-VL-8B) achieving 84.6 MCQ, 76.5 OQA, and 57.0 captioning on DualityVidQA-test, retaining 90.4% and 67.4% of its MCQ performance versus GPT-5.6's 48.1% and 30.7%.","tokens_in":13182,"tokens_out":6763,"duration_ms":66283,"significance":"The problem is well motivated: exposing the gap between passive verification and active discovery is an important diagnostic for video MLLMs. The proposed training paradigm is simple and plausible, and the ablation study in Table 2 supports the complementary contributions of SFT, RL, and progressive fading. The IPV-Bench transfer result is encouraging because it reduces overfitting concerns. However, the headline OQA and captioning numbers depend on an unspecified semantic verifier that also appears to be used as the RL reward oracle, so the central claim is not yet independently checkable.","major_comments":[{"comment":"The OQA and captioning scores in Table 1 are computed by J_OQA and J_CAP, described only as 'a frozen semantic verifier' and as a check that 'the target event is independently discovered and correctly described.' The verifier model, prompts, thresholds, and matching rules are not specified. This is load-bearing because every non-MCQ result in the paper (DualityVidQA OQA 76.5, captioning 57.0; IPV-Bench OQA 78.6, captioning 60.2; and the retention ratios 90.4%/67.4% versus 48.1%/30.7%) passes through this verifier. The same level-ℓ verifier outcomes c_{i,k}^{(ℓ)} are already used as the progress reward in Eq. (10); if the evaluation verifier is the same object as the reward verifier, then the headline metric is the training objective, and the comparison to GPT-5.6 is not an independent evaluation. Please name the verifier, provide its prompts and thresholds, release it with the evaluation code, and re-run the main comparisons with an independent judge.","section":"Evaluation Protocol (Eq. 14) and Stage II (Eq. 10)"},{"comment":"FADE is fine-tuned on the DualityVidQA training split and evaluated on DualityVidQA-test, while GPT-5.6 and the other baselines are evaluated zero-shot. The abstract's claim that FADE 'outperforms GPT-5.6' is therefore not apples-to-apples on DualityVidQA-test, because the comparison includes the advantage of having been trained on the same benchmark's training data. The IPV-Bench transfer result is more convincing and should be emphasized; for DualityVidQA-test, please add a fine-tuned baseline trained with the same data under a standard recipe, or clearly restrict the claim to the transfer setting.","section":"Experimental Setup / Table 1"},{"comment":"The OQA and captioning training references are generated by Qwen3.6-Plus 'using each question and correct option as semantic context.' Since the open-ended evaluation verifier is also semantic and the generated references are derived from the original options, a verifier that rewards phrasings close to the original option or to the reference distribution will systematically favor FADE over models not trained on these references. Please provide evidence that J_CAP and J_OQA are insensitive to option-derived phrasing, for example by reporting a human-annotated sample on both benchmarks or by measuring the verifier's agreement with human judgments on a held-out subset.","section":"Training Data"}],"minor_comments":[{"comment":"The DualityVidQA columns 'Real, CF, Both' are repeated twice, and the IPV-Bench sub-table is visually confusing; please split the table into separate sub-tables for each benchmark.","section":"Table 1"},{"comment":"The localization score s_t used in the ECR definition is not defined; please state how it is extracted from the model (for example, attention weights, gradients, or generated timestamps).","section":"Figure 4"},{"comment":"The notation Attn(Q, H_i,S_i, H_i,S_i) is used without a definition; please specify the attention mechanism and the role of the K learnable queries.","section":"Eq. (4)"},{"comment":"The main tables report single runs without error bars or significance tests; given the stochasticity of sampling-based RL and LLM evaluation, please add multiple seeds or variance estimates for at least the FADE and GPT-5.6 comparisons.","section":"Table 1"},{"comment":"The 'w/o Prog. RL' variant is described only as jointly training on MCQ, OpenQA, and captioning with identical data and update steps; please specify the mixing ratio across formats so the ablation is reproducible.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The verifier omission is serious enough that I considered rejection; however, the training design and ablations are promising, and the issue is fixable by specifying the verifier and re-running with an independent evaluator. I recommend major revision rather than rejection. Please also ensure the GPT-5.6 comparison is framed with the fine-tuning caveat."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the FADE paper. The core observation is right: MCQ counterfactual benchmarks leak the target event through the question and options, so high MCQ scores can be passive verification of a suggested hypothesis rather than open-ended discovery. The paper's anchor-fading protocol—same video, then question-only, then generic caption instruction—is a simple and effective diagnostic. Figure 2 shows every model they test drops sharply as anchors fade, which is a genuinely useful result.\n\nThe training framework is also reasonable. Evidence-internalized SFT with the projection and re-grounding losses is a nontrivial extension of the DualityVidQA recipe, and the fading-anchor RL with scaffold-and-consolidate curricula is a clean idea. The ablations in Table 2 support the complementary roles of the stages, and the IPV-Bench transfer is reassuring.\n\nThe soft spots are real, though. The open-ended evaluation rests entirely on an unspecified 'frozen semantic verifier' and 'discovery check.' The paper doesn't name the verifier model, prompts, thresholds, or matching rules. That matters doubly because the RL progress reward in Eq. 10 uses verifier outcomes at each level. If the evaluation verifier is the same object as the reward verifier, then the headline metric is effectively the training objective, and the retention gains over GPT-5.6 could mostly reflect verifier alignment rather than visual discovery. The paper claims the protocol is separate from optimization, but without identifying the verifier, that claim isn't checkable.\n\nAlso, FADE is fine-tuned on the DualityVidQA training split while GPT-5.6 and the other baselines are evaluated zero-shot. That's an apples-to-oranges comparison. It doesn't invalidate the diagnosis, but it does undercut 'outperforming GPT-5.6' as a claim. And there are no error bars or significance tests; for a method whose gains are large that's a minor issue, but it should be addressed.\n\nNone of this is fatal to the central thesis. The evaluation protocol and the training recipe are both worth building on. A serious referee should ask for the verifier to be specified and re-run, code and data released, baselines either tuned or the claim softened, and some confidence intervals. With those fixes, the paper could be a solid contribution.\n\nWorth sending to a capable reviewer.","headline":"FADE identifies a real evaluation weakness and a plausible fix, but the unsupported semantic verifier and fine-tuned-vs-zero-shot comparison make the headline numbers unverifiable as reported.","tokens_in":13725,"tokens_out":4394,"would_cite":true,"duration_ms":41866,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FADE shows that an 8B open video model can be trained to independently discover and explain counterfactual events as textual guidance fades, retaining 90.4% accuracy on open-ended questions versus GPT-5.6's 48.1%.","keywords":["counterfactual video understanding","textual anchoring","active discovery","evidence-internalized SFT","fading-anchor reinforcement learning","anchor-fading evaluation","video multimodal LLM","open-ended video QA"],"falsifier":"Replace the unspecified semantic verifier with a different published verifier or with human raters on DualityVidQA-test and IPV-Bench, and check whether FADE's OQA and captioning retention (90.4% and 67.4%) and its lead over GPT-5.6 survive; as a complementary test, give the model only caption prompts on videos whose anomaly types never appear in its training set and see whether it still localizes and describes them.","tokens_in":12653,"feed_emoji":"🎬","tokens_out":8765,"duration_ms":78217,"temperature":0.7,"pith_summary":"Counterfactual video benchmarks that use multiple-choice questions give away the answer: the question and candidate options tell the model what to look for, so high scores can reflect text-guided verification rather than genuine discovery of physical or commonsense violations. FADE is a training framework that targets this gap with two stages: evidence-internalized supervised fine-tuning grounds predictions in the decisive visual anomaly, and fading-anchor reinforcement learning progressively removes textual guidance during training so the model has to find and explain the anomaly on its own. To measure this capability, the paper converts existing MCQ benchmarks into aligned MCQ, open-ended QA, and captioning evaluations. On DualityVidQA-test and IPV-Bench, the FADE-trained Qwen3-VL-8B outperforms GPT-5.6 on strict paired scores in all three formats, and its accuracy retention as anchors fade is 90.4% and 67.4% versus GPT-5.6's 48.1% and 30.7%. The broader claim is that current Video-MLLMs are often passive verifiers, and active discovery is a trainable capability even in an 8B open model.","feed_headline":"Fading Q&A cues lets an 8B video model find anomalies alone","feed_subtitle":"Removing question and options, FADE keeps 90% of MCQ accuracy; GPT-5.6 keeps only 48%.","key_machinery":"The mechanism is a three-level fading textual-anchor curriculum over the same video: MCQ with question and options, OQA with question only, and captioning with a generic instruction. The RL reward is prefix-gated so a success at a weaker-anchor level counts only if all stronger levels succeeded, paired with a direction reward that requires the model to call the counterfactual video anomalous and its factual counterpart normal, plus a format reward; these are normalized by GDPO's group statistics. The SFT stage supplies the evidence backbone: a localized evidence projection pulls response hidden states toward the annotated anomalous interval's visual prototype, and response-conditioned evidence re-grounding (RCER) distills the interval's content and temporal distribution into whole-video retrieval from response tokens. The named mechanism is the fading anchor itself, and the shared sharp declines across models in the paper's anchor-fading profiles are the diagnosis it exposes.","core_discovery":"On its own terms, the paper's central claim is that the gap between MCQ and open-ended performance on counterfactual video benchmarks is not fixed: it can be closed by training. Specifically, FADE establishes that an 8B open Video-MLLM can internally encode the decisive visual evidence and keep using it when instance-specific textual anchors (question plus options) are absent, with strict paired accuracy of 84.6 (MCQ), 76.5 (OQA), and 57.0 (captioning) on DualityVidQA-test 'Both'—retentions of 90.4% and 67.4%—and 91.2/78.6/60.2 on IPV-Bench, exceeding GPT-5.6 in all three formats. The authors attribute this to an evidence-first curriculum: Stage I makes responses recover the anomalous interval's content and temporal location from the whole video, and Stage II rewards correct counterfactual judgments even as textual guidance fades away, including on paired factual videos to prevent an always-counterfactual shortcut.","pith_inferences":["Editorial inference: the same anchor-fading protocol could be used to diagnose robustness in other open-ended video tasks, such as generic anomaly detection or temporal grounding, by measuring how much performance drops when the query is stripped of instance-specific clues.","Editorial inference: because the semantic verifier is unnamed and its thresholds undisclosed, cross-model comparisons on OQA and captioning are hard to interpret; a public, fixed verifier would make retention numbers comparable across future papers.","Editorial inference: FADE is trained only on DualityVidQA data with a single 8B base model, so a strong stress test is whether the discovery skill transfers to entirely new anomaly categories, unseen video styles, and other model sizes or architectures."],"forward_implications":["MCQ-only counterfactual benchmarks systematically overestimate Video-MLLM understanding, since high scores can come from verifying text-specified hypotheses.","An 8B open model trained with evidence-internalized SFT and fading-anchor RL can serve as a strong baseline for unconstrained counterfactual discovery, beating closed-source GPT-5.6 without scaled-up parameters.","The evaluation protocol can be applied to any existing public MCQ benchmark, producing an anchor-fading profile that separates discovery capability from text-guided verification.","The sharp decline in GPT-5.6, Gemini-2.5 Pro, and Qwen3-VL as anchors fade indicates that textual-anchor dependence is widespread across open and closed models, not a quirk of one architecture.","SFT, RL, and progressive fading are complementary: ablations show that removing any one of them widens the gap between MCQ and open-ended performance."],"supporting_citations":[{"why":"Provides DualityVidQA, the primary benchmark and training split, plus the DNA-Training baseline that FADE must beat.","marker":"Huang et al. 2026"},{"why":"Provides IPV-Bench, the second evaluation benchmark containing only counterfactual videos.","marker":"Bai, Ci, and Shou 2025"},{"why":"Defines Qwen3-VL-8B, the base model whose weights and architecture FADE starts from.","marker":"Bai et al. 2025a"},{"why":"Documents GPT-5.6, the strongest closed-source comparison whose scores FADE exceeds.","marker":"Singh et al. 2025"},{"why":"Introduces GDPO, the group reward normalization used to combine progress, pair, and format rewards in Stage II.","marker":"Liu et al. 2026"},{"why":"Supplies Qwen3.6-Plus, used to generate the OQA answers and captions that supervise the fading-anchor RL references.","marker":"Qwen Team 2026"}],"fun_headline_variants":["FADE training lets an 8B model keep 90% of MCQ accuracy in open-ended video QA","Fading question cues: 8B video model retains 90% of multiple-choice skill","Evidence-first curriculum beats GPT-5.6 on unconstrained counterfactual video QA","FADE: anomaly discovery without question clues, 8B model beats GPT-5.6","An 8B model discovers counterfactual anomalies when Q&A hints fade away"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every OQA and captioning score in the paper is decided by an unnamed 'frozen semantic verifier' and a caption 'discovery check' whose model, prompts, and thresholds are never specified, so if that verifier rewards wording close to the original correct option, the reported retention gains could reflect the verifier's expectations rather than genuine open-ended discovery.","fun_headline_variants_meta":{"raw":{"variants":["FADE training lets an 8B model keep 90% of MCQ accuracy in open-ended video QA","Fading question cues: 8B video model retains 90% of multiple-choice skill","Evidence-first curriculum beats GPT-5.6 on unconstrained counterfactual video QA","FADE: anomaly discovery without question clues, 8B model beats GPT-5.6","An 8B model discovers counterfactual anomalies when Q&A hints fade away"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1560,"prompt_tokens":1062,"completion_tokens":498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":381}},"tokens_in":678,"tokens_out":498,"duration_ms":5109,"temperature":1.0,"reasoning_tokens":381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:51:21.987358+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the unspecified semantic verifier with a different published verifier or with human raters on DualityVidQA-test and IPV-Bench, and check whether FADE's OQA and captioning retention (90.4% and 67.4%) and its lead over GPT-5.6 survive; as a complementary test, give the model only caption prompts on videos whose anomaly types never appear in its training set and see whether it still localizes and describes them.","supporting_citations":[],"review_version":1}