Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

A single streaming model can control both when to speak and what to say, outperforming prior real-time and proactive baselines on both axes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:05 UTC pith:Q4YHJ2EF

load-bearing objection Real engineering and a useful dataset, but the proactive timing claim rests on synthetic per-second labels made by evenly spreading ASR words, so the headline F1/TimeDiff numbers should not be taken at face value. the 3 major comments →

arxiv 2603.03447 v3 pith:Q4YHJ2EF submitted 2026-03-03 cs.CV

Proact-VL: A Proactive VideoLLM for Real-Time AI Companions

Weicai Yan , Yuhong Dai , Qi Ran , Haodong Li , Wang Lin , Tao Jin , Xing Xie , Hao Liao
show 1 more author
Jianxun Lian
This is my paper
classification cs.CV
keywords proactive video LLMreal-time streamingresponse timinggame commentarychunk-wise inferencespeaking decisionlive gaming benchmarkAI companion
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to prove that a video-language model can be turned into a real-time AI companion that decides for itself when to speak, at one-second granularity, and still produces commentary that reads as human. To show this, the authors build the Live Gaming Benchmark, train Proact-VL on hundreds of hours of gameplay commentary and guidance, and measure both text quality and response timing. Their claim is that Proact-VL beats commercial models like GPT-4o and existing real-time or proactive baselines on both dimensions at once, while retaining general video understanding. If true, this closes the gap between 'smart at answering questions about video' and 'companionable in a live stream.'

Core claim

On the paper's own terms, the central discovery is that proactivity timing and content quality do not need to be traded off: a purely lightweight trigger — a sigmoid head reading a semantically empty <|FLAG|> token's hidden state each second — combined with a transition-weighted classification loss and a stability regularizer lets a streaming VideoLLM decide when to speak and say something short and relevant. Evaluated on their Live Gaming Benchmark, Proact-VL reports the best overall text quality (CC 49.23, LiveU 6.52, FinalQ 5.03) and the best proactive F1 (64.87) while keeping TimeDiff at 1.71, outperforming GPT-4o, Gemini 2.5 Pro, and both real-time and proactive baselines.

What carries the argument

The load-bearing mechanism is the decide-then-generate loop: each second the model ingests a video chunk, history, and optional query, and appends a special decision token <|FLAG|> whose hidden state is mapped through a small gated MLP and sigmoid to a speaking probability; if the probability crosses threshold τ, the assistant generates a one-second clip, otherwise it emits a fixed silence placeholder. Two losses carry the timing skill: a transition-smoothed binary cross-entropy that up-weights rare silence-to-speech switches (γ=5), and a stability regularizer that penalises jitter within states and matches the average speaking rate to the human baseline. A dual-cache sliding window with an

Load-bearing premise

The per-second ground-truth speak/silence labels come from spreading ASR words evenly across one-second bins with no human check that those bins match when a real commentator would speak; if that synthetic target is off, the trigger is trained and scored against the wrong clock.

What would settle it

Take a set of gameplay clips, have human annotators mark the seconds where a commentator should say something, recompute F1 and TimeDiff for Proact-VL and the baselines using these human labels, and compare with the results computed from the synthetic bin labels; if F1 drops by more than a few points or TimeDiff rises, the proactive advantage is an artifact of label construction. A simpler check: build a dummy that speaks at the same overall rate but at random times, and see how close it comes to Proact-VL's reported F1 on the synthetic labels.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A single VideoLLM can handle always-on interaction — watching a stream, deciding whether to comment, and generating short speech — without a separate triggering model or explicit user prompts.
  • The reported F1/TimeDiff numbers imply the trigger aligns with human commentary density closely enough to be usable in live companion products.
  • Because general benchmarks hold up (Video-MME even improves), proactive fine-tuning need not sacrifice the model's underlying video understanding.
  • The framework gives the community a 561-hour, 12-game dataset and two test benchmarks (clip-level and streaming) for reproducible development of proactive real-time agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My read: the per-second labels are synthetic — words are spread evenly into one-second bins — so the trigger may actually be learning 'when humans tend to talk' rather than 'when there is something worthy of commentary'; the two could diverge in quiet but critical moments.
  • I would expect the same chunk+FLAG recipe to transfer beyond games to any always-on narration task (sports, lectures, surveillance), since nothing in the method is game-specific; a cheap experiment would be retraining on non-gaming live video and measuring F1 against human-annotated response moments.
  • The transition-weighted loss is the piece I'd steal: it treats speaking/silence as a sequence with rare switches, a more realistic prior for social interaction than i.i.d. per-second classification.
  • A natural stress test: deploy Proact-VL on a high-FPS, high-resolution stream (the paper processes 2 FPS and 420p) and see whether timing quality collapses when transient visual cues are added.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Proact-VL, a video LLM framework for real-time proactive AI companions in gaming scenarios. It constructs a 561-hour Live Gaming Dataset with three settings (solo commentary, co-commentary, user guidance), and a benchmark with in-domain and out-of-domain test sets. The method combines a chunk-wise streaming input schema, a lightweight gated response head that decides when to speak at each second, and a multi-term loss (language modeling, transition-weighted classification, stability regularization). The paper reports that Proact-VL outperforms commercial, proactive, and real-time baselines on text quality (CC, LiveU, FinalQ) and response timing (TimeDiff, PAUC, F1) on the author-built benchmark, while preserving general video understanding on MVBench, Video-MME, and LongVideoBench. Additional analyses cover ablations, threshold/window sensitivity, LLM-judge robustness, a small user study, and a human alignment study for text-quality metrics.

Significance. If the reported results hold, the paper makes a useful contribution: a concrete instantiation of a proactive, real-time VideoLLM in a high-stakes live-commentary setting, with a large dataset, a clean chunk-wise design, and extensive experiments. The framework addresses a real gap—most prior streaming models either lack proactive triggering or produce long, high-latency responses. The paper also ships ablations, judge-robustness checks, and a human-alignment study, which are strengths. The main significance, however, is conditional on the validity of the per-second speaking-label construction used for both training and evaluation. Because the central proactivity claims are measured against synthetic labels derived from an even-spreading heuristic, the contribution's core evidence is not yet fully convincing.

major comments (3)
  1. [Appendix B.1, Sec. 4.3, Sec. 5.1 / C.1] The per-second ground-truth speaking labels y_t are constructed by evenly distributing ASR words into one-second bins (Appendix B.1). These same synthetic labels supervise the response head via L_cls and L_reg (Sec. 4.3) and are used to compute TimeDiff, PAUC, and F1 (Sec. 5.1, C.1). No human-annotated response-timing labels are provided, and no validation against word-level timestamps or manual annotations is reported. If real commentators do not speak evenly within segments—e.g., they pause, cluster words, or speak only in part of a segment—both training and evaluation are aligned with an artificial target. The headline gains in Table 2 (e.g., overall F1 64.87 vs. 56.05 for the best real-time baseline) could then reflect differential fit to this heuristic rather than genuine proactive timing quality. The human-alignment analysis in Appendix L evaluates text-quality metrics (LiveU/Final
  2. [Sec. 5.1, Tables 1–6, Appendix J] The main results reported in Secs. 5.2–5.4 use a response threshold of τ=0.3, while all other analyses and ablations (including the loss ablation in Table 6, the prompt ablation in Table 17, and the threshold analysis in Appendix J) use τ=0.5. No validation-set selection protocol for τ is described. Because τ directly controls the precision/recall trade-off (Appendix J shows F1 decreases monotonically with τ while CC varies), using one threshold for headline results and a different threshold for ablations makes cross-table comparisons and the reported 'best' outcomes difficult to interpret. The authors should specify how τ was chosen (e.g., on a held-out validation split) and report main results at the same threshold as ablations, or justify the divergence.
  3. [Appendix C.1, Appendix L] The PAUC metric uses an LLM judge (GPT-5.1) to score response quality within ground-truth intervals, and the text-quality metrics (CC, LiveU, FinalQ) are also LLM-judged. The human-alignment study in Appendix L validates LiveU/FinalQ against human ratings and reports good pairwise agreement, but it does not evaluate timing labels or the TimeDiff/F1 metrics. In particular, the PAUC judge prompt asks whether the predicted answer 'covers the key points of the ground truth answer', which is a content-coverage assessment, not a verification that the ground-truth timing labels match human annotator judgments. The paper should include human timing annotations (or at least a manual subset) to validate that the synthetic y_t correspond to when a human commentator would actually speak, and to confirm that the reported F1/TimeDiff improvements reflect better proactivity rather than better fit to th
minor comments (5)
  1. [Sec. 3.1] The paragraph after 'Video Data Collection' contains a long string of corrupted Unicode escapes ('/uni00000036/uni00000052/...'). This appears to be an encoding artifact and should be removed or replaced with the intended text.
  2. [Sec. 4.1] The text references 'Figure 4' for the architecture but the actual figure appears later; Figure 6 is cited in Sec. 4.1 and Sec. 4.2. Please check cross-references; Figure 4 seems to be the overview and Figure 6 the ChatML template, but the in-text citations are inconsistent.
  3. [Appendix C.1, Eq. (9)] The refined TimeDiff formula introduces a penalty coefficient α but no value or sensitivity analysis is provided. Please state the value used or note that it is set to a default.
  4. [Sec. 5.7 / Appendix I] Table 18 reports MVBench 'Overall' as 66.3 for the base model and 64.7 for Proact-VL, but the text in Sec. 5.7 says 'comparable' while the table shows a −1.6 drop. Please clarify whether the base model's baseline is Qwen3-VL or a slightly different number, and be more precise about the magnitude of the regression.
  5. [Appendix F] The user study is small (30 clips, 15 evaluators, 3 annotations per pair) and only reports win rates without confidence intervals or inter-annotator agreement statistics beyond 'three different evaluators'. Reporting Cohen's kappa or a CI would strengthen this section.

Circularity Check

0 steps flagged

No significant circularity: the proactive timing and quality claims are evaluated on held-out clips with external benchmarks and human checks; the synthetic-label concern is a validity limitation, not a circular reduction.

full rationale

The paper's derivation chain does not contain a step in which an output reduces to an input by construction. The per-second speaking labels in Appendix B.1 ('Suppose the segment spans t seconds and its caption contains n words. We distribute the words into t one-second bins as evenly as possible') are a fixed supervision target; the response head is trained on them via L_cls/L_reg (§4.3), while TimeDiff/F1/PAUC are computed against held-out clip labels (§C.1). This is ordinary supervised evaluation, not a fitted parameter renamed as a prediction: the model can and does fail to match the labels (e.g., ablations in Table 6, failure cases in §K.4), so the reported F1/TimeDiff are empirical outcomes. The paper also provides external grounding that does not depend on the authors' timing labels: Ego4D and Black Myth: Wukong transfer tests (§5.3), general video benchmarks MVBench/Video-MME/LongVideoBench (Appendix I), a human user study (Appendix F), and human alignment analysis (Appendix L). No load-bearing argument rests on a self-citation: the cited proactive/streaming baselines are independent prior work. The synthetic even-split label construction is a legitimate validity/measurement concern—it may not reflect true human response timing—but that is a limitation of the benchmark, not circularity in the paper's derivation.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The central empirical claim rests on constructed per-second labels and LLM-judged evaluation. The only tuned free parameters are the response threshold, loss weight, transition weight, chunk duration, and window size; no new physical entities are postulated.

free parameters (5)
  • response threshold τ = 0.3 for main results; 0.5 for all other analyses and ablations
    Controls the speaking decision; the main tables use a different value from the ablations, and no validation-set selection procedure is reported (Sec. 5.1, Appendix J.1).
  • response-loss weight α = 0.2
    Hand-set hyperparameter balancing L_main and L_resp (Sec. 5.1).
  • transition weight γ = 5
    Set to the measured transition:persistence ratio of roughly 1:5 in the training set (Sec. 5.1).
  • per-second chunk duration = 1 second
    Base granularity of the streaming schema; chosen by design rather than derived (Sec. 4.1).
  • KV-cache window size W = 16384–24576 recommended
    Selected from a hyperparameter sweep over window sizes (Appendix J.2); affects streaming stability and quality.
axioms (3)
  • domain assumption Evenly distributing ASR words into one-second bins yields valid ground-truth speaking labels y_t.
    Appendix B.1 constructs per-second captions by distributing n words into t bins as evenly as possible; no human timing annotation is used. The whole proactive training and F1/TimeDiff evaluation depend on this assumption.
  • domain assumption LLM-as-a-judge scores (GPT-5.1 and Gemini-based) are valid proxies for commentary quality.
    LiveU, FinalQ, and CC win-rate are computed by LLM judges (Appendix C.2). Appendix L acknowledges method-specific calibration bias, e.g., the judge overestimates Proact-VL by +0.82 on average and underestimates LiveStar.
  • standard math RoPE rotation additivity R(a)R(b)=R(a+b) and reverse-RoPE exactly re-base KV-cache positions.
    Appendix A.3 derives the exact position shift using standard properties of 2D rotation matrices; this is a standard mathematical result correctly applied.

pith-pipeline@v1.3.0-alltime-deepseek · 35996 in / 12251 out tokens · 127481 ms · 2026-08-02T19:05:36.867352+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Proact-VL: A Proactive VideoLLM for Real-Time AI Companions." pith.science (2026). https://pith.science/paper/Q4YHJ2EF

@misc{pith2026260303447,
  author       = {Pith},
  title        = {Pith review of: Proact-VL: A Proactive VideoLLM for Real-Time AI Companions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q4YHJ2EF}},
  note         = {Machine review of arXiv:2603.03447}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Proactive and real-time interactive experiences are essential for human-like AI companions, yet face three key challenges: (1) achieving low-latency inference under continuous streaming inputs, (2) autonomously deciding when to respond, and (3) controlling both quality and quantity of generated content to meet real-time constraints. In this work, we instantiate AI companions through two gaming scenarios, commentator and guide, selected for their suitability for automatic evaluation. We introduce the Live Gaming Benchmark, a large-scale dataset with three representative scenarios: solo commentary, co-commentary, and user guidance, and present Proact-VL, a general framework that shapes multimodal language models into proactive, real-time interactive agents capable of human-like environment perception and interaction. Extensive experiments show Proact-VL achieves superior response latency and quality while maintaining strong video understanding capabilities, demonstrating its practicality for real-time interactive applications.

Figures

Figures reproduced from arXiv: 2603.03447 by Haodong Li, Hao Liao, Jianxun Lian, Qi Ran, Tao Jin, Wang Lin, Weicai Yan, Xing Xie, Yuhong Dai.

Figure 1
Figure 1. Figure 1: Overview of Proact-VL. The top section shows Proact-VL collaborating with other commentators for real-time commentary, while the bottom section highlights its proactive player guidance capability. Abstract Proactive and real-time interactive experiences are essential for human-like AI companions, yet face three key challenges: (1) achieving low-latency inference under continuous streaming inputs, (2) auton… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the Live Gaming Dataset. The inner, middle, and outer rings represent the three data categories, 12 specific game titles, and their corresponding genres, respectively. Following the categorization of game genres, we selected representative popular games from each category to ensure broad coverage of the gaming landscape. We curated a diverse collection of high-traction titles spanning multiple … view at source ↗
Figure 3
Figure 3. Figure 3: Overview of data pipeline. clarity and professional instructional quality while preserv￾ing temporal accuracy and semantic fidelity. 3.2.3. PERSONA ENRICHMENT To enhance contextual awareness and support coherent role-playing across diverse gaming scenarios, we extract structured commentator and guide personas from processed narrative data. Specifically, DeepSeek-V3.2-Exp analyzes gameplay transcripts to sy… view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of the Proact-VL. At each second, Proact-VL consumes multi-source tokens (video, query, and context) and decides whether to speak by feeding the FLAG hidden state into a response head to obtain a score, then thresholding with τ . If triggered, it appends the assistant prefix and generates a short clip-level text; otherwise, it appends the prefix with a Silence token to output silence [PITH_FU… view at source ↗
Figure 5
Figure 5. Figure 5: Score curve visualization. Green: labeled response; Red: labeled silence; Dashed line: threshold; Above-threshold scores: model triggers responses. follows the label trend—response segments score slightly higher—suggesting the model can detect response-worthy moments from video alone when textual interference is ab￾sent. The middle threshold 0.5 yields the most practical pattern: an initial silent period (… view at source ↗
Figure 6
Figure 6. Figure 6: Illustration of ChatML template. To support real-time, proactive streaming commentary, we extend the Qwen-style ChatML with a lightweight, structured input format that explicitly separates (i) environment history, (ii) the current video chunk, and (iii) the user query. Our design enables a two-stage inference pipeline—decide-then-generate—where the model first decides whether to respond, and only then prod… view at source ↗
Figure 7
Figure 7. Figure 7: Game-Wise Analysis. We further break down performance by game to examine whether the gains of Proact-VL are consistent across diverse gameplay styles and visual contexts [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Interface for Pairwise Model Comparison. Proactive, and Real-Time models. The experimental data covers three scenarios: single-person commentary, multi-person commentary, and game instruction. For each scenario, we selected a typical game and extracted 10 video clips, resulting in a total of 30 test samples. Evaluation Platform and Stimuli Generation. We constructed a specialized evaluation platform to con… view at source ↗
Figure 9
Figure 9. Figure 9: Threshold ablation on CC and F1 across SOLO, Co-Commentary, and Guidance. Increasing the response threshold consistently reduces F1 in all three settings, indicating fewer triggered responses and degraded coverage. In contrast, CC favors more conservative triggering: Co-Commentary CC improves monotonically with higher thresholds (peaking at 0.9), SOLO CC peaks around 0.6, and Guidance achieves its best CC … view at source ↗
Figure 10
Figure 10. Figure 10: Window-size (context window) hyperparameter analysis on CC and F1 across SOLO, Co-Commentary, and Guidance. Increasing the context window generally improves CC up to a moderate-large range, while F1 remains relatively stable. For SOLO, CC rises from 50.58 at 2048 to a peak around 24576 (55.08), with only minor F1 variation (∼ 55–57). For Co-Commentary, CC steadily increases with larger windows and saturat… view at source ↗
Figure 11
Figure 11. Figure 11: Solo Commentary Scenario Case 1. Time 609s 615s 629s 635s So if you did everything, you'd be like, wait a second, there's nothing here. And that‘s because you’re not following along with the guide, and that‘s what I’m trying to teach you guys to do here. [Dataset Name]: Elden Ring [Filename]: walkthrough_game_59.mp4 [history]: “Hmm, am I going higher? Let's take it up to thirty, I guess. So we're going to… view at source ↗
Figure 12
Figure 12. Figure 12: Solo Commentary Scenario Case 2. Solo Commentary Scenario Cases [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Co-Commentary Scenario Case. Co-Commentary Scenario Case [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Guidance Scenario Case. Guidance Scenario Case. As illustrated in the timeline of [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Failure Case 1 for LoL. Oh, no! Oh, no! Oh, no! T=1560s T=1562s T=1564s [PITH_FULL_IMAGE:figures/full_fig_p027_15.png] view at source ↗
Figure 16
Figure 16. Figure 16 [PITH_FULL_IMAGE:figures/full_fig_p027_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Score distributions of human ratings, LiveU scores, and FinalQ scores across 100 sampled instances [PITH_FULL_IMAGE:figures/full_fig_p028_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Harnessing Streaming Video in the Wild

    cs.CV 2026-06 unverdicted novelty 6.0

    Presents Streaming-Train-248K dataset, Streaming Harness system, and Streaming-Eval benchmark to enable VLMs for proactive, memory-equipped streaming video understanding.

  2. Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding

    cs.CV 2026-06 unverdicted novelty 6.0

    LyraV uses FDTC and SToP for per-frame incremental decoding to reach 98.29% video synchrony at 3.89 FPS while preserving general understanding.

  3. Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse

    cs.CV 2026-05 unverdicted novelty 5.0

    The paper organizes research on generalist game AI into Dataset, Model, Harness, and Benchmark pillars and charts a five-level progression from single-game mastery to agents that create and live inside game multiverses.

  4. Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse

    cs.CV 2026-05 unverdicted novelty 3.0

    This work traces four eras of generalist game players across dataset, model, harness, and benchmark pillars and charts a five-level roadmap ending in agents that create and evolve within game multiverses.

Reference graph

Works this paper leans on

20 extracted references · 2 linked inside Pith · cited by 3 Pith papers

  1. [1]

    **Focus on stylistic feature**, not just content. Identify elements such as: - Tone (e.g., passionate, calm, analytical, humorous, dramatic, fast-paced, conversational) - V ocabulary (e.g., frequent use of game jargon, metaphors, emotional exclamations, team/player references) - Rhythm & Pacing (e.g., rapid-fire commentary during fights, slower descriptiv...

  2. [2]

    **Ignore transcription errors**

  3. [3]

    **Summarize in concise descriptive sentences or bullet points**

  4. [4]

    Tone": {Tone},

    **Word limit**: The whole style profile must be within **200** words. ## Output Format - Provide a structured style profile in JSON format. Example: { "Tone": {Tone}, "Vocabulary": {Vocabulary}, "Rhythm & Pacing": {Rhythm & Pacing}, "Overall Style Summary": {Overall Style Summary} } 30 Proact-VL: A Proactive VideoLLM for Real-Time AI Companions N.2. User ...

  5. [5]

    Keep **timestamps** aligned with their corresponding steps (use the original action_begin_time and ac- tion_end_time)

  6. [6]

    action_begin_time

    The output must be in **JSON format**, with each step represented as an object containing: * "action_begin_time" * "action_end_time" * "refined_description" — your improved, guidance-style version of the original action description. — #### Example Input: { "player_question": "How do I create a new world and set it to Survival mode?", "assistant_guidance":...

  7. [7]

    **Refine and rewrite the atomic action list** into a smoother, more human-readable sequence of steps

  8. [8]

    **Preserve key actions** that align with the assistant’s guidance

  9. [9]

    **Remove unnecessary or irrelevant actions**, such as indecisive steps, menu hovering, or redundant toggling, unless they meaningfully illustrate the learning process

  10. [10]

    Now click ’Create New World’

    Use **natural, instructive tone**, as if narrating the process to the player (e.g., *"Now click ’Create New World’..."*, *"Type in your desired name..."*)

  11. [14]

    Label Timeline Text (time-aligned reference timeline; includes all on-clip information such as [USER] questions, [SPEAKER*] commentary, and the current commentator’s lines; use mainly as a reference for rhythm and salient windows)

  12. [15]

    one huge dump

    Prediction Timeline Text (the current commentator’s output to be evaluated) Speaker tags (apply to all inputs): - [ASSISTANT] = the current commentator’s lines (may appear in Context as prior commentary; and appears in Label/Prediction as on-clip current commentator) - [USER] = user questions (reference/context only; may appear in Label Timeline) - [SPEAK...

  13. [16]

    Context Text (previous commentary or background context; may also contain prior lines from the same commen- tator)

  14. [17]

    Label Final Text (raw ASR final text from the original video; reference only)

  15. [18]

    - Any tagged lines in Context / Label Final are reference/context signals only and must NOT be scored

    Prediction Text (the final model output to be evaluated) Speaker definitions (apply to Context, Label Final, and Prediction): - [ASSISTANT] = current commentator’s lines (may appear in Context as prior commentary) - [USER] = user questions (reference/context only; may appear in Context and/or Label Final) - [SPEAKER*] = other commentators (reference/conte...

  16. [19]

    Scenario alignment: in Multi-commentator mode, check whether [ASSISTANT] complements [SPEAKER*] logically; in Guidance mode, check whether the [USER] query is actually addressed/resolved

  17. [20]

    Directional compatibility: use Label Final as a reference for major events; do NOT require word-by-word match; penalize **direct contradictions** (major event reversal, incompatible outcomes, or clearly wrong situation claims)

  18. [21]

    Scoring rules: - Output integer scores in [1..10] (no 0)

    Fidelity (lightweight): expressive commentary is allowed, but avoid unjustified concrete specifics (e.g., names, numbers, causes, outcomes) that would mislead; major invented events that conflict with Label should be penalized. Scoring rules: - Output integer scores in [1..10] (no 0). - Score bands: 10 excellent; 7–9 good; 4–6 mixed; 2–3 very poor but som...

  19. [336]

    fill-with-silence

    URL https://aclanthology.org/2025. findings-emnlp.336/. Wang, Y ., Meng, X., Wang, Y ., Zhang, H., and Zhao, D. Proactivevideoqa: A comprehensive benchmark evaluat- ing proactive interactions in video large language models. arXiv preprint arXiv:2507.09313, 2025b. Wang, Y ., Liu, S., Wang, D., Xu, N., Guanglu, W., Zhang, H., and Zhao, D. MMDuet2: Enhancing...

    Pith/arXiv arXiv 2025

  20. [8856]

    Li, K., Wang, Y ., He, Y ., Li, Y ., Wang, Y ., Liu, Y ., Wang, Z., Xu, J., Chen, G., Luo, P., et al

    URL https://openreview.net/forum? id=zKv8qULV6n. Li, K., Wang, Y ., He, Y ., Li, Y ., Wang, Y ., Liu, Y ., Wang, Z., Xu, J., Chen, G., Luo, P., et al. Mvbench: A compre- hensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22195–22206, 2024. Liao, Z., Ouyang, Y ., Lee,...

    Pith/arXiv arXiv 2024