{"id":"5103671c-e96f-423f-b23f-fa97b80dd9f4","arxiv_id":"2507.09313","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ProactiveVideoQA is a benchmark for proactive video question answering, and the proposed PAUC metric jointly scores response timing and content, claiming better alignment with human preferences.","lead":"This paper introduces ProactiveVideoQA, a benchmark that tests whether video AI models can volunteer information on their own during playback, and PAUC, a metric that rewards both early and correct responses. The authors report that PAUC matches human preferences better than text-only metrics, but the human study is partly circular and the numbers are noisy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §6.2 human-preference validation is circular: annotators are instructed to prefer earlier, ground-truth-aligned responses, so Table 4 cannot independently establish that PAUC tracks user experience rather than re-stating its design assumption.","rationale":"I read the paper as proposing a benchmark plus a time-aware metric, with the central empirical claim that PAUC agrees better with human preferences than a timing-insensitive baseline (Table 4). The benchmark construction is a genuine contribution, but the claim is only as strong as the human preference signal. The §6.2 instructions explicitly tell annotators to prefer responses that capture ground-truth information at earlier timestamps; this is nearly identical to PAUC's objective (area under the time-correctness curve, Eq. 1). Hence Table 4's ω=0.5 advantage over ω=1 may simply reflect that the raters were asked to encode the same timing preference the metric encodes, not that PAUC tracks natural user experience. The low human-human kappa and the paper's own admission that the task is cumbersome and unnatural corroborate the fragility. The reader's conditional verdict already captures this concern; I agree with that verdict and see no need to move it. A neutral re-study is the decisive test, and if it fails, the validation claim would need to be withdrawn.","tokens_in":13909,"tokens_out":5980,"duration_ms":70171,"concrete_test":"Run a preregistered, neutral preference study on the same 350 turn-level pairs used in Table 4 (or a fresh sample from the same benchmark) in which naive participants are simply asked 'which assistant would you rather have while watching this video?' with no mention of timestamps, ground-truth reply spans, or 'earlier is better'. Then compute Cohen's kappa between these neutral preferences and PAUC at ω=0.5 and at ω=1. If neutral-preference agreement no longer favors ω=0.5 over ω=1, the circularity concern is confirmed and the central claim fails; if the agreement pattern persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—PAUC agrees better with human preferences than a timing-ignoring metric—rests entirely on the user study in §6.2. The annotators are told to 'assume the role of users seeking timely and accurate information' and to judge 'which model prediction better captures the information present in the ground truth at earlier timestamps, while also considering text quality.' That is effectively the definition of PAUC (Eq. 1, §3): reward correctness accumulated early within (t_start, t_end) and penalize late or incorrect content. The study therefore compares PAUC against human judgments that have already incorporated PAUC's 'earlier is better' assumption. The ω=1 vs. ω=0.5 comparison in Table 4 is not neutral evidence for timing sensitivity, since the human instructions preload that sensitivity. The low human-human kappa (0.31–0.59) and the paper's own admission in §6.1 that the annotation task is 'cumbersome and unnatural' also indicate that the elicited preferences are not stable, naturalistic user preferences. Because the human instructions are the only independent preference signal, Table 4 cannot arbitrate between PAUC and a time-agnostic baseline. The benchmark itself remains a useful contribution, but the headline validation of PAUC does not withstand this concern without a neutral re-study.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ProactiveVideoQA, a benchmark for evaluating video LLMs in proactive interaction settings, where the model must autonomously decide when to respond during video playback. The benchmark comprises four tasks (web video QA, egocentric video QA, TV-series QA, and anomaly detection) with multi-turn open-ended annotations. The authors also propose PAUC (Proactive Area Under Curve), a metric that combines response timing and content correctness into an area-under-the-curve score, with a hyperparameter ω to trade off timeliness against correctness. The paper reports extensive experiments on proprietary and open-source models, as well as a human preference study claiming PAUC agrees better with human judgments than a timing-agnostic baseline.","tokens_in":14164,"tokens_out":3470,"duration_ms":40663,"significance":"The benchmark is a genuine and useful contribution: it is the first to target proactive interaction with multi-turn, open-ended QA across diverse video domains, and it fills a real gap in evaluation methodology for streaming/proactive video LLMs. The authors also provide thorough experimental comparisons across many models, including both offline-adapted and purpose-built proactive systems. The PAUC metric is a reasonable first proposal for scoring temporal response quality, and the paper is transparent about the difficulty of human annotation in this setting. However, the central validation claim—that PAUC aligns with human preferences better than timing-agnostic metrics—rests on a human study whose instructions embed PAUC's own design assumption, which substantially weakens the headline result. The benchmark alone is solid enough to justify publication if the metric-validation claim is either fixed with a neutral study or appropriately downgraded.","major_comments":[{"comment":"The human-preference validation is circular: annotators are explicitly instructed to 'assume the role of users seeking timely and accurate information' and to judge 'which model prediction better captures the information present in the ground truth at earlier timestamps, while also considering text quality.' This is precisely the quantity PAUC (Eq. 1) rewards—early accumulation of correct content within the ground-truth reply timespan. Table 4 therefore compares PAUC against a criterion that already incorporates PAUC's 'earlier is better' assumption, so it cannot independently establish that PAUC tracks user experience; it is instead a consistency check between PAUC and its own design principle. A neutral validation would need preference elicitation from users who are not told to prioritize timing or ground-truth alignment (e.g., free-viewing or task-completion setups), or at least a post-hoc analysis showing that the agreement holds without the timing-heavy instruction.","section":"§6.2 and Eq. (1)"},{"comment":"Equation (1) contains a notation inconsistency: the denominator uses (q_end − q_start) while the text and surrounding definitions define the ground-truth reply timespan as (t_start, t_end). This is not merely cosmetic; it makes the exact formula ambiguous for reproduction. Please reconcile the symbols and ensure the numerator and denominator use the same variables (presumably t_start and t_end). In addition, the text says the initial point (t_start, 0.5) is added to make the polyline continuous, but the numerator's first term (τ1 − t_start) × 0.5 is not explicitly tied to this point in the prose; clarifying this connection would improve readability.","section":"§3, Eq. (1)"},{"comment":"The hyperparameter ω is introduced with a recommended default of 0.5, but no sensitivity analysis or principled selection is provided. Table 3 reports results for ω = 0, 0.5, 1, yet the human-agreement comparison in Table 4 only evaluates ω = 1 versus ω = 0.5. Consequently, the claim that PAUC 'better captures human preferences' is contingent on an arbitrary default; a different ω could reverse or weaken the conclusion. Additionally, the semantics of ω are counterintuitive: the text states that ω = 0 corresponds to 'timeliness is very important' (unshifted x-coordinates), while ω = 1 collapses all timestamps to t_start, completely removing timing. This inversion should be explained more clearly or the parameter should be renamed/reparameterized to avoid confusion.","section":"§3.1 and Table 4"},{"comment":"PAUC does not penalize duplicate or repeated responses. Because s_p is computed on the accumulated set of responses, repeating the same content does not lower s_p, but it still adds a time interval to the curve and can increase the area under the curve. Table 5 shows that the proactive models (MMDuet, VideoLLM-Online) have very high duplicate rates (81–99%), and the paper itself concludes in §6.1 (3) that these models 'tend to repeat previously generated content, leading to lower response quality.' This creates a bias: PAUC can reward a model that responds early with low-quality content and then repeats it, which inflates its score relative to a model that responds later with better content. The metric should either discount repeated information or treat identical consecutive responses as not contributing new area, so that the scoring is consistent with the paper's own qualitative finding.","section":"§3, §6.1, Table 5"},{"comment":"The human-validation sample is small and heavily filtered: only 100 turns per task (50 for [VAD]) are included, with criteria requiring both models to have responded within the reply span, at least one model to have responded in more than one round, and both to have at least one response with PAUC > 0. This filtering removes easy cases where metrics may agree and concentrates on borderline, difficult comparisons, which could overstate or understate the agreement. The reported inter-annotator kappa values (0.31–0.59) are low, and the claim that PAUC 'approaches the level of agreement observed between human annotators' is made without confidence intervals or significance tests. Please report statistical uncertainty and, ideally, re-run the preference study on an unfiltered random sample to assess the robustness of the agreement findings.","section":"§6.2"}],"minor_comments":[{"comment":"The phrase 'users increasingly expect multimodal systems to be more initiative' should be corrected to 'more proactive' or 'more initiative-taking.'","section":"Abstract and §1"},{"comment":"The denominator in Eq. (1) should read (t_end − t_start) × S, not (q_end − q_start) × S; the subscript 'q' appears to be a leftover from a draft with different variable names.","section":"Eq. (1)"},{"comment":"The observation that human annotators perform poorly because the task is 'cumbersome and unnatural' raises a question about construct validity: if the annotation protocol is unnatural for humans, does the benchmark measure a realistic proactive interaction paradigm? Please discuss this limitation explicitly near the human-performance results.","section":"§6.1"},{"comment":"The row for ProactiveVideoQA uses an asterisk under the 'Multi-Answer' column, but it is unclear whether 'Open-Ended' is also marked; the legend says '*' means true for some sub-tasks, which is ambiguous for a benchmark that is claimed to be fully open-ended. Clarify the table entries and legend.","section":"Table 2"},{"comment":"There are several typos, e.g., 'recommand' should be 'recommend' in §3.1, and some references use inconsistent formatting ('ArXiv' vs 'arXiv'). Please proofread the final version.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and PAUC are interesting and likely publishable after the validation concern is addressed. The circularity in §6.2 is the main blocker: it directly affects the paper's central claim. If the authors cannot run a neutral user study, they should at minimum soften the claim to describe PAUC as consistent with a timing-preferring preference model rather than as validated against independent human preferences. The notation and duplicate-response issues are also important for reproducibility and fairness of the metric. I would be willing to look at a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the short version. The paper contributes a useful benchmark for proactive video QA and a time-aware metric that captures an important dimension most evaluations ignore. The benchmark reuses four existing datasets into multi-turn open-ended proactive tasks, and the experiments cover a wide range of offline and proactive models plus human performance. PAUC is a simple, intuitive construction: area under a time-score curve, rewarding early correct content and penalizing late or wrong output. The paper is also transparent about its assumptions.\n\nWhere it gets weak is Section 6.2. The human-preference study instructs annotators to judge which model prediction 'better captures the information present in the ground truth at earlier timestamps' - which is essentially the definition of PAUC. Table 4 therefore cannot independently validate that PAUC tracks user experience; it restates the design assumption. The low kappa (0.31-0.59), the filtering of examples, and the annotators finding the task 'cumbersome and unnatural' make the preference signal even shakier. There is also a notation mismatch in Eq. (1) (q_end vs t_end), omega = 0.5 is picked without a reported sweep, and duplicate responses are not penalized even though Table 5 shows they are pervasive. These are fixable, but the headline validation needs a neutral re-study.\n\nStill, the benchmark and the metric are worth having. The adaptation of offline models to proactive settings is pragmatic, and the cross-model comparison gives the community a starting point. The central claim about human agreement should be read as unproven rather than wrong.\n\nWho should read this: researchers working on streaming or proactive video LLMs, and anyone building evaluation metrics for time-sensitive interaction. It deserves a serious referee, because the benchmark will likely be used. My recommendation: send to review, but require substantial revision of the human study and the metric details before accepting.","headline":"A useful new benchmark and a time-aware metric for proactive video QA, but the headline claim that PAUC matches human preferences rests on a circular user study.","tokens_in":733,"tokens_out":1482,"would_cite":true,"duration_ms":34734,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces the first proactive video QA benchmark and proposes PAUC, a metric that combines response timing with correctness and agrees with human preferences better than timing-agnostic metrics.","keywords":["proactive interaction","video question answering","evaluation metric","area under curve","video large language models","response timing","open-ended multi-turn QA","benchmark"],"falsifier":"Run a preference experiment that holds content quality fixed and varies only response timestamp: present users with two transcripts of identical accumulated answers, one delivered immediately when the event appears and one delayed until the end of the window, without telling them that timeliness matters. If a large majority does not prefer the earlier delivery, PAUC's time-axis weighting is not justified. A second check is to re-annotate the benchmark's ground-truth reply windows independently and measure how much PAUC scores and model rankings change; if small window perturbations flip rankings, the metric's reliance on exact spans is fragile.","tokens_in":13716,"feed_emoji":"⏱️","tokens_out":8536,"duration_ms":90742,"temperature":0.7,"pith_summary":"ProactiveVideoQA is a benchmark for the setting in which a video assistant is given a question at the start and must decide on its own when to answer as the video plays; the assistant may respond several times, and each response is judged on both what it says and when it is delivered. The paper's central claim is that this timing dimension is essential to user experience and that existing video QA benchmarks and metrics, which evaluate only final textual content, miss it. To support that claim, the authors propose PAUC, which computes the area under a time-versus-correctness curve for each ground-truth reply turn, and show in a human preference study that PAUC agrees with human judgments better than a correctness-only baseline and approaches human inter-annotator agreement. The benchmark covers web videos, egocentric footage, TV series, and surveillance anomaly detection, with multi-turn open-ended answers and speech input on the TV task. The authors report that current models, including proactive ones, are far from solving the timing problem: replies are often duplicated or late, and adapted offline models frequently outperform dedicated proactive models.","feed_headline":"Video AI scores now count when answers arrive, not only content","feed_subtitle":"PAUC folds response timing into correctness, and human judges prefer its rankings over content-only scores","key_machinery":"The central object is the PAUC score. For a ground-truth reply turn $g$ with ideal timespan $(t_{\\rm start}, t_{\\rm end})$, take the model's responses $\\mathit{pred}_1,\\dots,\\mathit{pred}_P$ at timestamps $\\tau_1<\\cdots<\\tau_P$ inside the span. A judge, GPT-4.1 in this paper, is given the question, the gold answer, and the accumulated responses up to each $\\tau_p$, and returns a correctness score $s_p\\in\\{0,1,2\\}$. PAUC draws the polyline through $(\\tau_p,s_p)$, seeds it with $(t_{\\rm start},0.5)$ to encode that silence is better than a wrong answer, closes it at $(t_{\\rm end},s_P)$, and divides the area under the polyline by the maximal area $(t_{\\rm end}-t_{\\rm start})\\times 2$. The hyperparameter $\\omega\\in[0,1]$ shifts timestamps left as $\\tau'_p=t_{\\rm start}+(1-\\omega)(\\tau_p-t_{\\rm start})$, so $\\omega=0$ keeps real time and $\\omega=1$ collapses to content-only scoring; $\\omega=0.5$ is the default. Per-turn scores are averaged over all ground-truth turns in a video.","core_discovery":"On the authors' terms, the discovery is that proactive video question answering can be measured, and the right measure is a curve, not a number: for each ground-truth reply with an ideal time span, PAUC plots the accumulated correctness of the model's responses at the timestamps they occur, then takes the normalized area under that polyline as the score for the turn. Because an early correct answer raises the curve sooner, and an early wrong answer contaminates the accumulated input for every later judge call, the area encodes both timing and content in one value. The human study is the decisive evidence: with the timeliness weight $\\omega=0.5$, PAUC's pairwise agreement with human preferences is higher than the $\\omega=1$ content-only variant, and in the [WEB], [TV], and [VAD] tasks it approaches or reaches the agreement between two independent human annotators. This is put forward as evidence that PAUC is a more faithful measure of user experience in proactive interaction than metrics that throw away timestamps.","pith_inferences":["A testable extension of PAUC's logic is that the value of a correct response may not be additive over time: users might be interrupted or annoyed by repeated partial answers, so a curve that penalizes interruption events could track user experience even better than the current area-under-curve score.","The same area-under-curve construction could transfer to any event-window task with evolving textual output, such as live captioning, real-time monitoring alerts, or assistant notifications, by swapping in a task-specific correctness judge.","The paper's ground-truth reply windows come from content annotations; an alternative ground truth derived from when users actually interrupt to ask or search could validate or revise those windows.","The result that chunk-adapted offline models often beat dedicated proactive models suggests a training hypothesis: timing may be best learned as a separate lightweight decision module on top of a strong offline video interpreter, rather than through end-to-end proactive training."],"forward_implications":["If PAUC is accepted, any evaluation of proactive video assistants must report response timestamps, because two systems with identical final text can differ substantially in user experience.","Current proactive models such as MMDuet and VideoLLM-Online are shown to suffer from duplicate, late, or absent responses; this should push training toward redundancy suppression and timing decisions, not just content accuracy.","Offline models adapted by chunking can beat dedicated proactive models on this benchmark, indicating that proactive capability is not automatically delivered by architecture choice.","The PAUC framework's judge can be replaced with BLEU, CIDEr, or accuracy, so the metric extends beyond LLM judges and can be tuned to a deployment's speed-versus-completeness trade-off via $\\omega$.","The four-task design gives separate spokes: web QA, egocentric assistant, TV-series understanding, and surveillance monitoring, already covering representative deployment surfaces for proactive interaction."],"supporting_citations":[{"why":"Supplies the user journey map design idea that PAUC turns into a time-score curve.","marker":"Huo et al., 2023"},{"why":"VideoLLM-Online is one of the two fully open-sourced proactive models benchmarked.","marker":"Chen et al., 2024a"},{"why":"MMDuet is the other proactive baseline and also the QA-generation pipeline used for the egocentric task.","marker":"Wang et al., 2024"},{"why":"Provides the Shot2Story videos and annotations forming the [WEB] task.","marker":"Han et al., 2023"},{"why":"Ego4D Goal-step supplies the egocentric videos and dense descriptions for [EGO].","marker":"Song et al., 2023"},{"why":"TVQA supplies the TV-series videos, questions, answers, and timespans for [TV].","marker":"Lei et al., 2018"},{"why":"UCF-Crime supplies the surveillance videos and anomaly timespans for [VAD].","marker":"Sultani et al., 2018"},{"why":"StreamingBench is a competing streaming benchmark that does not let the model choose when to answer, motivating ProactiveVideoQA.","marker":"Lin et al., 2024"},{"why":"OmniMMI is a partially proactive benchmark whose limitations on multi-turn open-ended interaction motivate this design.","marker":"Wang et al., 2025"},{"why":"BLEU is named as a drop-in replacement for the LLM judge inside PAUC's scoring axis.","marker":"Papineni et al., 2002"}],"fun_headline_variants":["New metric PAUC scores video AI by when answers arrive, not just what","Timing now counts: PAUC metric aligns better with human preference for video QA","Video QA benchmark: PAUC measures when and what, beating content-only scores","PAUC: first metric for proactive video QA that rewards timely answers","For video AI, timing is scoring: PAUC outperforms old metrics in user agreement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each ground-truth answer has a true ideal reply window and that receiving correct information earlier is always better for the user; if the window annotations or the 'earlier is better' preference model are wrong, PAUC's scores can diverge from real user experience despite passing its own human-study check.","fun_headline_variants_meta":{"raw":{"variants":["New metric PAUC scores video AI by when answers arrive, not just what","Timing now counts: PAUC metric aligns better with human preference for video QA","Video QA benchmark: PAUC measures when and what, beating content-only scores","PAUC: first metric for proactive video QA that rewards timely answers","For video AI, timing is scoring: PAUC outperforms old metrics in user agreement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000718,"raw_usage":{"total_tokens":3222,"prompt_tokens":939,"completion_tokens":2283,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2181}},"tokens_in":555,"tokens_out":2283,"duration_ms":16208,"temperature":1.0,"reasoning_tokens":2181,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:58:16.001114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a preference experiment that holds content quality fixed and varies only response timestamp: present users with two transcripts of identical accumulated answers, one delivered immediately when the event appears and one delayed until the end of the window, without telling them that timeliness matters. If a large majority does not prefer the earlier delivery, PAUC's time-axis weighting is not justified. A second check is to re-annotate the benchmark's ground-truth reply windows independently and measure how much PAUC scores and model rankings change; if small window perturbations flip rankings, the metric's reliance on exact spans is fragile.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ego4D Goal-step supplies the egocentric videos and dense descriptions for [EGO]."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TVQA supplies the TV-series videos, questions, answers, and timespans for [TV]."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UCF-Crime supplies the surveillance videos and anomaly timespans for [VAD]."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BLEU is named as a drop-in replacement for the LLM judge inside PAUC's scoring axis."}],"review_version":1}