{"id":"913ebc93-0d0d-4088-b4e3-dd61b53faf95","arxiv_id":"2606.24797","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"EG-VQA benchmark with temporal evidence annotations reveals Video-LLM grounding failures and EG-Reasoner achieves better evidence localization via supervision.","lead":"EG-VQA is a benchmark of 2,067 videos and 11,838 QA pairs that requires models to localize supporting temporal evidence for each answer. EG-Reasoner is a model trained with explicit evidence supervision that improves grounding over other open-source systems.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"EG-F1 validity hinges on whether human evidence annotations are exhaustive and unambiguous for each QA pair","rationale":"The reader's weakest assumption directly identifies the annotation-to-metric validity step that the entire evaluation pipeline depends on. No other internal inconsistency (e.g., metric definition or model architecture) is visible from the abstract; the annotation quality is the single point that, if it fails, collapses both the discrepancy claim and the supervision benefit claim. Full-text details on annotation protocol would be the natural next check, but the concern remains load-bearing until those details are verified.","tokens_in":1782,"tokens_out":397,"duration_ms":9418,"concrete_test":"Compute Cohen's kappa (or equivalent) on a 10% random sample of QA pairs re-annotated by two independent annotators; also measure the fraction of QA pairs where at least one additional temporal segment is proposed by a third annotator. If kappa < 0.7 or >15% of pairs receive extra segments, re-evaluate all models on the augmented annotations and check whether the proprietary-model gap and EG-Reasoner gains remain statistically significant.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (discrepancy between answer correctness and faithful localization, plus gains from explicit supervision) rests on EG-F1 being a faithful measure of grounding quality. This requires that the 11,838 fine-grained temporal annotations are both complete (no missing relevant segments) and unambiguous (clear boundaries and semantics). If annotators systematically omit subtle but necessary evidence or disagree on boundaries, then low EG-F1 scores for proprietary models could reflect annotation incompleteness rather than model failure, and the reported gains for EG-Reasoner could be artifacts of training to the same annotation distribution. The abstract states the annotations exist but supplies no inter-annotator agreement, coverage statistics, or adjudication protocol; without those, the metric-to-claim link is unsecured.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces EG-VQA, a benchmark with 2,067 videos and 11,838 QA pairs each annotated with fine-grained temporal evidence segments. It defines the EG-F1 metric to jointly assess answer correctness and evidence localization against these annotations. Experiments indicate that even strong proprietary Video-LLMs show low EG-F1 despite potentially correct answers, revealing a gap in faithful grounding; the authors propose EG-Reasoner trained with explicit evidence supervision, reporting SOTA results among open-source models especially on counterfactual reasoning tasks.","tokens_in":1921,"tokens_out":512,"duration_ms":20695,"significance":"If the annotations prove reliable, the work is significant for shifting VideoQA evaluation from answer-only metrics to verifiable evidence grounding, demonstrating that scaling alone is insufficient and that explicit supervision yields measurable gains. The new benchmark and EG-F1 metric provide concrete tools for future interpretable Video-LLM development.","major_comments":[{"comment":"The central claim that proprietary models exhibit a fundamental discrepancy between answer correctness and faithful evidence localization (and that EG-Reasoner closes this gap) depends on EG-F1 being a valid measure of grounding quality. This in turn requires the 11,838 human annotations to be exhaustive and unambiguous. The manuscript provides no inter-annotator agreement statistics, coverage analysis, or adjudication protocol in the dataset construction section, leaving open the possibility that low EG-F1 scores reflect annotation incompleteness rather than model failure.","section":"Dataset construction / annotation protocol"},{"comment":"Table reporting main results (likely Table 2 or 3): without reported statistical tests, variance across runs, or breakdown of EG-F1 components (temporal overlap vs. semantic consistency), it is unclear whether the reported gains for EG-Reasoner over baselines are robust or whether the discrepancy for proprietary models is statistically significant.","section":"Experimental results"}],"minor_comments":[{"comment":"The abstract states 'particularly pronounced gains are observed on reasoning-intensive tasks' but does not quantify the differential improvement; a per-task breakdown would strengthen the claim.","section":"Abstract"},{"comment":"Notation for EG-F1 is introduced without an explicit formula in the main text; including the precise definition (e.g., how temporal IoU and semantic match are combined) would improve reproducibility.","section":"Metric definition"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their insightful comments, which highlight important aspects of dataset validation and experimental analysis. We address each major comment below and will revise the manuscript accordingly.","responses":[{"response":"We agree that the absence of inter-annotator agreement (IAA) statistics and detailed annotation protocols in the manuscript is a limitation. In the revised manuscript, we will add a dedicated subsection on the annotation process, including IAA scores computed using appropriate metrics (e.g., temporal IoU agreement), coverage analysis of evidence segments, and the adjudication protocol used for resolving disagreements. This will strengthen the validity of EG-F1 as a measure of grounding quality.","revision_made":"yes","referee_comment":"[Dataset construction / annotation protocol] The central claim that proprietary models exhibit a fundamental discrepancy between answer correctness and faithful evidence localization (and that EG-Reasoner closes this gap) depends on EG-F1 being a valid measure of grounding quality. This in turn requires the 11,838 human annotations to be exhaustive and unambiguous. The manuscript provides no inter-annotator agreement statistics, coverage analysis, or adjudication protocol in the dataset construction section, leaving open the possibility that low EG-F1 scores reflect annotation incompleteness rather than model failure."},{"response":"We acknowledge that additional statistical analysis would improve the clarity of the results. In the revised version, we will include variance across multiple runs (e.g., standard deviations), perform statistical significance tests where appropriate, and provide a component-wise breakdown of EG-F1 to separate temporal overlap from semantic consistency. This will allow readers to better assess the robustness of the reported improvements.","revision_made":"yes","referee_comment":"[Experimental results] Table reporting main results (likely Table 2 or 3): without reported statistical tests, variance across runs, or breakdown of EG-F1 components (temporal overlap vs. semantic consistency), it is unclear whether the reported gains for EG-Reasoner over baselines are robust or whether the discrepancy for proprietary models is statistically significant."}],"tokens_in":1430,"tokens_out":404,"duration_ms":20917,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"EG-VQA constructs a benchmark of 2,067 videos and 11,838 QA pairs, each with fine-grained temporal evidence annotations, introduces the EG-F1 metric that combines temporal overlap with semantic match, and trains EG-Reasoner under explicit supervision on those annotations.\n\nThis is new. Prior VideoQA work mostly stops at answer accuracy, so requiring models to output and be scored on the supporting video segments addresses a real evaluation gap. The experiments show proprietary models often produce correct answers while failing to localize the right evidence, and the supervised model improves results, with bigger lifts on counterfactual questions.\n\nThe paper does the basic job cleanly: it defines the task, builds the data, and runs the comparisons against both open and closed models.\n\nThe soft spot is exactly the one in the stress-test note. The central claims about a discrepancy and the value of supervision depend on the annotations being complete and unambiguous. Without reported inter-annotator agreement, coverage statistics, or an adjudication protocol, low EG-F1 scores could reflect missing evidence segments rather than model failure, and the reported gains could be partly annotation-specific. The abstract supplies none of those details, so the metric-to-claim link stays unsecured until the full methods section is checked.\n\nThis is for researchers working on Video-LLMs who want evaluation that goes beyond surface accuracy. It deserves peer review because the benchmark idea is useful and the experimental setup is straightforward, even if the annotation reliability needs explicit verification.","headline":"EG-VQA adds a benchmark and metric for evidence grounding in VideoQA, but the claims rest on unverified annotation quality.","tokens_in":2395,"tokens_out":370,"would_cite":false,"duration_ms":14643,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Video question answering models often produce correct answers while failing to identify the supporting video segments.","keywords":["Video Question Answering","Video-LLMs","Evidence Grounding","Temporal Localization","Benchmark Construction","EG-F1 Metric","Counterfactual Reasoning"],"falsifier":"A model that scores high on answer accuracy but low on EG-F1 when its predicted evidence intervals are compared against the ground-truth annotations would directly contradict the claimed discrepancy.","tokens_in":2680,"feed_emoji":"📹","tokens_out":611,"duration_ms":8517,"temperature":0.7,"pith_summary":"The paper builds EG-VQA, a benchmark of 2,067 videos and 11,838 QA pairs where each question comes with explicit temporal evidence annotations. It introduces the EG-F1 metric that scores both temporal overlap and semantic match between predicted and ground-truth evidence. Experiments show that strong proprietary Video-LLMs exhibit large gaps between answer accuracy and evidence localization. The authors then train EG-Reasoner with direct supervision on the evidence annotations and report state-of-the-art open-source results that approach proprietary performance, especially on counterfactual reasoning questions.","feed_headline":"Video QA models answer correctly but ground answers in wrong segments","feed_subtitle":"EG-VQA benchmark with temporal annotations reveals the gap; explicit evidence training narrows it for open-source models.","key_machinery":"EG-VQA benchmark containing fine-grained temporal evidence annotations per QA pair, scored by the Evidence-Grounded F1 (EG-F1) metric that combines temporal alignment and semantic consistency.","core_discovery":"Even strong proprietary Video-LLMs exhibit a significant discrepancy between answer correctness and faithful localization of temporal evidence. The EG-VQA benchmark requires models to jointly reason and localize evidence, evaluated by the EG-F1 metric that jointly measures temporal alignment and semantic consistency. Training EG-Reasoner with explicit evidence supervision produces state-of-the-art open-source performance and competitiveness with proprietary systems, with the largest gains on reasoning-intensive tasks.","pith_inferences":["Future VideoQA training pipelines may need to treat evidence localization as a primary objective rather than an afterthought.","The same annotation and metric approach could be applied to other temporal reasoning domains such as action anticipation or event prediction.","Explicit evidence supervision might allow automated detection of cases where a model hallucinates supporting video content."],"forward_implications":["Scaling model size alone is insufficient for robust video understanding.","Structured evidence supervision improves both reliability and interpretability of VideoQA outputs.","Gains from evidence training are especially large on counterfactual and other reasoning-intensive questions.","Open-source models can reach performance levels competitive with proprietary systems when trained with explicit evidence labels."],"fun_headline_variants":["Video-LLMs answer correctly but localize evidence poorly","EG-VQA benchmark reveals VideoQA evidence grounding failures","Strong models show gap between correct answers and right evidence","EG-Reasoner improves Video-LLM evidence localization via supervision"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The fine-grained evidence annotations accurately and comprehensively represent the supporting temporal evidence for each QA pair.","fun_headline_variants_meta":{"raw":{"variants":["Video-LLMs answer correctly but localize evidence poorly","EG-VQA benchmark reveals VideoQA evidence grounding failures","Strong models show gap between correct answers and right evidence","EG-Reasoner improves Video-LLM evidence localization via supervision"]},"model":"grok-4.3","cost_usd":0.003078,"raw_usage":{"total_tokens":1615,"prompt_tokens":714,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":30778000,"prompt_tokens_details":{"text_tokens":714,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":838,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":714,"tokens_out":63,"duration_ms":6250,"temperature":1.0,"reasoning_tokens":838,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T00:13:33.720507+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A model that scores high on answer accuracy but low on EG-F1 when its predicted evidence intervals are compared against the ground-truth annotations would directly contradict the claimed discrepancy.","supporting_citations":[],"review_version":1}