{"id":"c5410249-7fc3-489e-9505-13e81b151a18","arxiv_id":"2606.02522","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Moment-Video benchmark shows top video MLLM achieves only 39.6% accuracy on momentary visual event tasks, with most open-source models below 25%.","lead":"The paper introduces Moment-Video, a benchmark of 1,000 human-verified video-QA pairs testing video MLLMs on brief, sampling-sensitive visual events across four task types. A smart generalist might read it to see why current AI video systems often fail on short decisive moments that matter in real applications like monitoring or robotics.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's weakest assumption matches the paper's explicit design claim. Given the human-verification step and the sampling diagnostic already performed, the assumption does not appear to be the point of failure for the headline result. No other load-bearing gap (e.g., in metric definition, model selection, or statistical reporting) is visible from the provided text.","tokens_in":1804,"tokens_out":266,"duration_ms":18083,"concrete_test":"Sample 100 questions from the released benchmark; for each, obtain model answers on (a) the full video, (b) a single randomly chosen frame, and (c) text-only input; compute the fraction where (b) or (c) matches the full-video answer within 5% of overall accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Moment-Video questions being verifiably sampling-sensitive and non-recoverable from language priors or static frames. The paper states that all 1,000 pairs were human-verified to meet this criterion across four task types and 25 subcategories, and reports that denser sampling improves but does not close the gap. No internal inconsistency appears in the reported construction or evaluation of 33 models.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces Moment-Video, a benchmark of 1,000 human-verified video-QA pairs across 7 domains and 25 subcategories, designed to diagnose video MLLMs' handling of momentary visual events (localized actions or state changes lasting only a few frames) via four task types: Temporal Occurrence, Temporal Counting, Action Description, and Temporal Reasoning. Each pair is claimed to require models to notice transient evidence rather than rely on persistent objects, global context, or language priors. Evaluation of 33 models shows Seed-2.0-Pro at 39.6% overall accuracy (most open-source models <25%), with diagnostics indicating denser sampling helps but does not close the gap and longer videos increase localization challenges; the central conclusion is that current video MLLMs lack temporally faithful representations.","tokens_in":1846,"tokens_out":617,"duration_ms":13604,"significance":"If the sampling-sensitivity and non-recoverability claims hold, the benchmark provides a targeted diagnostic for a previously underexplored failure mode in video MLLMs, with potential to drive improvements in frame sampling, visual token compression, and temporal aggregation. The scale, human verification, and multi-task coverage strengthen its utility as an evaluation tool beyond existing long-form video benchmarks.","major_comments":[{"comment":"Benchmark construction (Section 3): the claim that all 1,000 pairs are verifiably sampling-sensitive and non-recoverable from language priors or static frames rests on human verification, but the manuscript provides no details on verification protocol, inter-annotator agreement, or concrete tests (e.g., model performance on single-frame or text-only ablations) used to confirm this property for each subcategory.","section":"Section 3"},{"comment":"Evaluation setup (Section 4): the reported accuracies and diagnostic findings on frame sampling density lack specification of the exact sampling rates, token budgets, and prompting templates applied uniformly across the 33 models, making it impossible to isolate whether failures stem from temporal fidelity or from implementation choices.","section":"Section 4"},{"comment":"Error analysis (Section 5): the diagnostic claim that longer videos introduce stronger temporal-localization challenges is not supported by per-video-length breakdowns or statistical tests; aggregate accuracy alone does not establish this as load-bearing for the central temporal-fidelity conclusion.","section":"Section 5"}],"minor_comments":[{"comment":"Table 1 or equivalent: clarify the exact distribution of the 1,000 pairs across the 25 subcategories and four task types to allow readers to assess balance.","section":"Table 1"},{"comment":"Figure 2 or equivalent: the visualization of model performance gaps would benefit from error bars or per-task breakdowns to improve interpretability.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments, which help clarify key aspects of the benchmark and evaluation. We address each major comment below and will revise the manuscript to incorporate the suggested details.","responses":[{"response":"We agree that the current manuscript lacks sufficient detail on the verification process. In the revised version, we will add a new subsection in Section 3 that describes the full human verification protocol, including annotator instructions, number of annotators per item, inter-annotator agreement (Cohen's kappa), and the decision criteria for sampling sensitivity. We will also report single-frame and text-only ablation results across subcategories to empirically confirm that each QA pair requires transient visual evidence.","revision_made":"yes","referee_comment":"[Section 3] Benchmark construction (Section 3): the claim that all 1,000 pairs are verifiably sampling-sensitive and non-recoverable from language priors or static frames rests on human verification, but the manuscript provides no details on verification protocol, inter-annotator agreement, or concrete tests (e.g., model performance on single-frame or text-only ablations) used to confirm this property for each subcategory."},{"response":"We concur that precise implementation details are required for reproducibility and to isolate temporal effects. The revision will specify the exact sampling rates (e.g., uniform sampling at fixed FPS values), per-model visual token budgets, and the standardized prompting templates used for all 33 models. These additions will be placed in Section 4 and the appendix.","revision_made":"yes","referee_comment":"[Section 4] Evaluation setup (Section 4): the reported accuracies and diagnostic findings on frame sampling density lack specification of the exact sampling rates, token budgets, and prompting templates applied uniformly across the 33 models, making it impossible to isolate whether failures stem from temporal fidelity or from implementation choices."},{"response":"We acknowledge that the manuscript currently presents only aggregate observations without length-stratified breakdowns or statistical support. In the revision, we will add per-video-length accuracy tables (binned by duration), correlation coefficients between video length and accuracy, and appropriate statistical tests to substantiate the claim that longer videos increase localization difficulty.","revision_made":"yes","referee_comment":"[Section 5] Error analysis (Section 5): the diagnostic claim that longer videos introduce stronger temporal-localization challenges is not supported by per-video-length breakdowns or statistical tests; aggregate accuracy alone does not establish this as load-bearing for the central temporal-fidelity conclusion."}],"tokens_in":1578,"tokens_out":549,"duration_ms":32498,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing here is that the authors built Moment-Video, a set of 1,000 human-verified video-QA pairs that target short, localized events models often skip. They split it into four task types and 25 subcategories across seven domains, then ran 33 models. The best result is 39.6% from Seed-2.0-Pro; most open-source models sit below 25%. Denser sampling narrows the gap for some models but does not close it, and longer videos make the problem worse.\n\nWhat the paper does well is isolate a practical failure mode that general video benchmarks tend to blur. The focus on sampling-sensitive evidence, plus the scale of the evaluation, makes the performance numbers useful as a diagnostic. The task breakdown into occurrence, counting, description, and reasoning also gives readers concrete categories to think about.\n\nThe soft spot is the verification step. The claim that every pair requires the transient visual evidence and cannot be recovered from language priors or static frames depends on how carefully the humans checked that. The paper states they did this verification, but without more on the exact criteria or sample questions it is hard to judge how strict it was. That is the main place where additional detail would strengthen the work.\n\nThis is for researchers working on video MLLMs or temporal modeling who want a targeted test set. A reader running their own models on video understanding tasks would find the numbers and the sampling analysis directly relevant.\n\nI would send it to peer review. The evaluation covers enough models and the gap is consistent enough that referees should see the full paper.","headline":"Moment-Video gives a clear empirical signal on video MLLMs missing brief events, backed by broad model testing but resting on the strength of its human verification.","tokens_in":2365,"tokens_out":404,"would_cite":true,"duration_ms":25334,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Current video MLLMs fail to capture brief but decisive visual events that determine many practical answers.","keywords":["video MLLMs","temporal fidelity","momentary visual events","benchmark","video question answering","multimodal models","temporal reasoning"],"falsifier":"A model that scores near ceiling on the benchmark while using only sparse frames sampled away from the key events would indicate the questions do not require momentary visual evidence.","tokens_in":2702,"feed_emoji":"🎥","tokens_out":545,"duration_ms":19741,"temperature":0.7,"pith_summary":"The paper introduces Moment-Video, a benchmark of 1000 video-QA pairs that tests whether models can notice, count, describe or reason about short-lived visual events lasting only a few frames. It claims these events are often skipped by sparse sampling or lost in token compression, and that language reasoning cannot recover them reliably. Evaluations across 33 models show the best result at 39.6 percent accuracy and most open-source models below 25 percent. A sympathetic reader would care because many real questions hinge on such transient evidence rather than persistent objects or overall scene context.","feed_headline":"Top video model scores 39.6% on brief events benchmark","feed_subtitle":"Moment-Video shows most MLLMs miss short visual changes that decide answers, with open-source models below 25%.","key_machinery":"The Moment-Video benchmark, which grounds each of its 1000 questions in a localized, visually observable, and sampling-sensitive event across four task types.","core_discovery":"Moment-Video demonstrates that video MLLMs still lack temporally faithful representations for capturing, preserving, and using brief but decisive visual evidence, with the strongest model reaching only 39.6 percent overall accuracy across tasks that require attention to localized, sampling-sensitive events.","pith_inferences":["Future model designs may need new methods for preserving short-duration signals beyond current sampling or compression approaches.","The benchmark could serve as a diagnostic tool for comparing temporal handling across different video architectures."],"forward_implications":["Denser frame sampling raises accuracy for some models but leaves a remaining performance gap.","Longer videos increase the difficulty of temporal localization.","Proprietary models outperform open-source ones but none reach reliable understanding of momentary events."],"fun_headline_variants":["Video MLLMs score 39.6% on momentary visual events","Top MLLM reaches 39.6% on brief video events","Brief visual events challenge MLLMs to 39.6% success","Video MLLMs struggle with transient evidence at 39.6%"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Each question truly requires attention to a transient visual event that cannot be answered from persistent objects, global context, or language priors.","fun_headline_variants_meta":{"raw":{"variants":["Video MLLMs score 39.6% on momentary visual events","Top MLLM reaches 39.6% on brief video events","Brief visual events challenge MLLMs to 39.6% success","Video MLLMs struggle with transient evidence at 39.6%"]},"model":"grok-4.3","cost_usd":0.00578,"raw_usage":{"total_tokens":2784,"prompt_tokens":729,"num_sources_used":0,"completion_tokens":76,"cost_in_usd_ticks":57799500,"prompt_tokens_details":{"text_tokens":729,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1979,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":729,"tokens_out":76,"duration_ms":14484,"temperature":1.0,"reasoning_tokens":1979,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T14:47:52.168334+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A model that scores near ceiling on the benchmark while using only sparse frames sampled away from the key events would indicate the questions do not require momentary visual evidence.","supporting_citations":[],"review_version":1}