{"id":"5858d221-f2fb-45b2-9270-c88c08a799c4","arxiv_id":"2504.17213","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MASR combines coarse-to-fine relevance selection, dilated temporal expansion, and confidence-driven self-reflection to improve agent-based video question answering, and reports strong benchmark results.","lead":"MASR is a video question answering system that repeatedly zooms in on the frames most relevant to a question, guided by the model's own confidence score. It reports a large gain on EgoSchema and smaller gains on NExT-QA, IntentQA, and Video-MME benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The C==3 confidence-gated stop (Algorithm 1) is never calibrated; if self-reported confidence is miscalibrated, MASR halts before MCRS/DTE can improve, so the claimed self-reflection gain is unsupported.","rationale":"I agree with the reader's weakest assumption: the confidence-gated stopping rule in Algorithm 1 is load-bearing and untested. The paper's own Figure 5 indicates that MASR continues to improve with additional self-reflection rounds, so the decision to stop at C==3 can materially change reported accuracy. Without a calibration curve or an ablation of the threshold, the adaptive-stopping story is not supported. The proposed A/B test against a fixed three-round policy would directly show whether the confidence signal is necessary. I also note the Table I 'CLARF' and Figure 7 'MACF' inconsistencies as missing support for attributing the reported numbers to MASR, but I treat that as an editing/attribution issue rather than the primary technical concern. Because the method is plausible and the central weakness is an unvalidated design choice rather than an internal contradiction, the existing CONDITIONAL verdict remains appropriate; no verdict change is needed.","tokens_in":13303,"tokens_out":7961,"duration_ms":78626,"concrete_test":"Re-run MASR on the EgoSchema 500-question subset with one change: replace the `C == 3` break with 'always run exactly 3 self-reflection rounds' for every query, keeping all other parameters fixed. Compare the accuracy of this fixed-round variant to the published 73.4. If fixed-round accuracy is not lower (or is higher), the C==3 stopping rule is either unnecessary or miscalibrated, and the paper's claim that confidence-driven adaptive stopping contributes to the gain fails; if fixed-round accuracy is clearly lower, the concern is resolved and a calibration table (P(correct | C=1,2,3) per round) should be reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that adaptive confidence-driven self-reflection improves video QA—rests on Algorithm 1's stopping rule (Section III-D, line 9: `if C == 3 then break`). The paper provides no evidence that the LLM's self-reported integer confidence C is calibrated: no distribution of C across rounds, no P(correct | C), no threshold-sensitivity analysis, and no comparison against fixed-round stopping. If the reflector assigns C=3 too early on wrong answers, the loop terminates before MCRS and DTE can supply the missing context; if it assigns C<3 on correct answers, it wastes rounds but may still land on the same answer. Figure 5 shows MASR accuracy increases with more self-reflective rounds, which makes early stopping even more consequential: a miscalibrated high-confidence read could cut off exactly the iterations that produce the +5.2 EgoSchema gain. Therefore the claim that the attention-focusing loop is responsible for the improvement is not established. Separately, Table I labels the MASR row `CLARF (Ours)` and Figure 7 says `MACF`, so the manuscript currently does not explicitly tie the reported numbers to the MASR architecture described in Section III.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MASR, a training-free agent-based video question-answering framework. The pipeline samples and clusters video frames, then iteratively applies multimodal coarse-to-fine relevance sensing (MCRS) and dilated temporal expansion (DTE) to focus on query-relevant frames; a single LLM generates the answer, assigns a confidence score C, and decides whether to continue the loop until C==3 or a round cap N. Experiments on EgoSchema, NExT-QA, IntentQA, and the Video-MME long split report state-of-the-art accuracies, with the largest claimed gain being 5.2 points on EgoSchema.","tokens_in":13530,"tokens_out":6840,"duration_ms":61804,"significance":"If the reported numbers hold, the paper shows that a plug-and-play, training-free agent can improve video QA on several benchmarks without fine-tuning the underlying LLM. The architecture is clearly described, the ablation suite covers each module, and the decision to use a single LLM as responder, evaluator, and coarse selector is an appealing efficiency choice. The EgoSchema improvement is large enough that the mechanism deserves scrutiny. However, the evidence for the confidence-gated stopping mechanism, the small-margin SOTA claims, and the hyperparameter sensitivity is currently incomplete; the paper also does not release code or full prompt details, and several table and figure labels are inconsistent.","major_comments":[{"comment":"The stopping rule `if C == 3 then break` is never validated. The paper provides no calibration curve, no conditional probability P(correct | C), no threshold-sensitivity analysis, and no comparison against a fixed-round stopping baseline. Since Section IV-C and Figure 5 report that accuracy increases with repeated self-reflective rounds, a miscalibrated confidence read could terminate the loop before MCRS and DTE supply the missing context. The claim that adaptive, confidence-driven self-reflection is responsible for the reported accuracy gains is therefore not established without a calibration analysis.","section":"Section III-D, Algorithm 1 line 9"},{"comment":"All results appear to be single runs, and the NExT-QA and IntentQA margins over the prior best results are 0.2 and 0.3 percentage points, respectively. No error bars, confidence intervals, significance tests, or run-to-run variability are reported. On evaluation sets of roughly 4,969 and 2,134 items, these margins are within typical sampling noise; the SOTA claims need statistical support or a statement of variance.","section":"Section IV-C, Table I"},{"comment":"The DTE hyperparameters (wn, s, r, w) are adjusted per dataset in Section IV-B, and Kv, wn, and r are selected from the EgoSchema ablations in Tables VI, VII, and VIII. Because the same EgoSchema benchmark is used for the headline +5.2 improvement claim, the reported gain may partly reflect test-set selection rather than a property of the architecture. The authors should use a validation split for hyperparameter selection or report sensitivity of all four datasets to these parameters.","section":"Section IV-B and Section IV-E, Tables VI-VIII"}],"minor_comments":[{"comment":"The (+x.x) annotations are not tied to a named baseline; for example, the EgoSchema +5.2 is against LVNet at 68.2 rather than the adjacent VideoTree row at 66.2. Please state the reference baseline in the table or caption.","section":"Section IV-C, Table I"},{"comment":"The caption describes a 'demonstration' but the surrounding text describes a quantitative comparison of accuracy across self-reflective rounds; the caption should match the content.","section":"Section IV-C, Figure 5"},{"comment":"Please specify the exact clustering algorithm and number of clusters N, and provide the full text of the captioning prompt, the answer prompt, and the confidence-score prompt, since these are essential for reproducibility in an agent-based method.","section":"Section IV-B"},{"comment":"For EVA-CLIP-8B-plus, increasing the input resolution from 224 to 448 decreases accuracy from 73.4 to 71.4; adding a sentence of explanation would prevent confusion.","section":"Table III"},{"comment":"There are several typos and grammatical issues, including 'Mutilmodal', 'feet the LLMs', and 'This types'; the paper needs a careful copyedit.","section":"Section II and throughout"},{"comment":"The method is called MASR in Section III, but the result row in Table I is labeled 'CLARF (Ours)' and Figure 7 refers to 'MACF'. Please use one consistent name for the proposed method.","section":"Section IV-C and Figure 7"},{"comment":"The abstract says 'a remarkable 5% performance gain' while the table reports +5.2 points; please use consistent wording (percentage points versus percent) throughout.","section":"Abstract and Section IV-C"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a video understanding venue and the core idea is worth considering, but the empirical support for the confidence-gated stopping rule and for the small-margin SOTA claims needs to be strengthened before acceptance. The requested analyses, such as confidence calibration and error bars, are feasible within a revision, so I am not recommending rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: MASR reports a 5-point jump on EgoSchema and packages three known ideas into one agent loop. The large gain is worth taking seriously; the evidence that the self-reflection loop itself is responsible is not yet there.\n\nWhat's actually new: the specific combination of coarse-to-fine relevance sensing (MCRS), dilated temporal expansion (DTE), and a single-LLM confidence loop. MCRS improves on DrVideo's text-only relevance by adding visual token matching; DTE adds temporal context without exploding the context window. The ablation in Table II shows the full system beats no-self-reflection by 8.1 points, and Tables VII-VIII show a non-monotonic response to expansion, which is sensible. The single-LLM reflector is a clean design choice versus VCA's extra reward model. Credit where due: this is a coherent, well-motivated system and the EgoSchema result is not obviously cherry-picked.\n\nSoft spots, in order. First, the confidence gate. Algorithm 1 stops when C==3. The paper never shows C is calibrated: no P(correct|C), no threshold sweep, no comparison to running a fixed number of rounds. The ablation 'w/o self-reflection' is one-round versus many, so it tests the loop as a whole, not the adaptive stopping. If C is miscalibrated, the loop may stop early exactly when more rounds help. That is a load-bearing assumption and it is unmeasured. Second, test-set tuning. Kv is picked as the value with the best EgoSchema accuracy in Table VI; DTE parameters and sampling rates are per dataset. That inflates the SOTA claim. The 0.2 and 0.3 margins on NExT-QA and IntentQA are within noise, and there are no error bars anywhere. Third, presentation sloppiness: Table I calls the model CLARF, Figure 7 says MACF, and the abstract says Next-QA. Easy to fix, but it needs fixing.\n\nThe math is fine: DTE is a straightforward adaptation of dilated convolution to frame selection, and the pipeline is described clearly enough to reimplement once the prompt details are released. Citations to the agent-based literature are appropriate and self-citation is not a problem here.\n\nWho this is for: anyone building LLM agents for long-form video QA will get useful design ideas, especially the visual-token fine-focusing step. Treat the EgoSchema number as an upper bound pending code release. My recommendation: send it to peer review. The method is plausible, the main result is large enough to matter, and a referee round can force the confidence calibration analysis and code release that the paper currently lacks.","headline":"MASR's 5-point EgoSchema gain is worth a referee's time, but the confidence-gated stopping rule is uncalibrated and the small SOTA margins rest on test-set tuning.","tokens_in":14106,"tokens_out":2756,"would_cite":false,"duration_ms":26181,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MASR claims that a training-free self-reflective attention-focusing loop lifts video question answering accuracy by up to 5.2 points across four benchmarks.","keywords":["agent-based video understanding","self-reflective reasoning","attention focusing","video question answering","dilated temporal expansion","confidence feedback","multimodal relevance sensing","training-free video agent"],"falsifier":"On a labeled subset, run MASR and record the confidence score and round at which it stops, then compare accuracy for items that stop at C = 3 against items forced to stop at C = 2 or after one round; if forced-early answers are not measurably worse, the confidence-based stopping rule is not the cause of the reported gains.","tokens_in":13062,"feed_emoji":"🎬","tokens_out":5431,"duration_ms":47407,"temperature":0.7,"pith_summary":"The paper is trying to establish that an agent-based video understanding system can improve answer accuracy by repeatedly focusing attention on query-relevant frames rather than consuming an entire video at once. MASR clusters sampled frames, coarsely selects candidate clips through LLM reasoning, fine-selects individual frames by comparing visual tokens with the query text, expands each focused frame around its temporal neighborhood, and only then answers; if the LLM's self-assessed confidence is low, the loop repeats with a sharper selection. On the EgoSchema benchmark the reported accuracy is 73.4%, a 5.2-point gain over the previous best listed baseline, with smaller gains on NExT-QA and IntentQA and 57.1% on Video-MME's long split. These results matter because they suggest that training-free agents can match or beat fine-tuned video models by better allocating attention. If correct, the framework also inherits improvements in its underlying LLMs automatically.","feed_headline":"Self-reflective video agent beats prior best by 5 points","feed_subtitle":"MASR refocuses on query-relevant frames until the LLM is confident, and reports gains across four benchmarks without fine-tuning.","key_machinery":"The load-bearing mechanism is the MCRS-DTE-self-reflection loop. MCRS is a two-stage relevance sensor: an LLM coarsely names query-relevant clips from accumulated context, then a visual encoder embeds frames and counts how many visual tokens are highly similar to the query's text token to fine-select the most relevant frames. DTE treats each focused frame as a temporal anchor, concatenating neighboring frames at fixed dilation intervals to widen the receptive field, analogous to dilated convolution in 1D. The self-reflection stops only when the same LLM that answers the question reports a confidence score of C = 3; below that, the loop re-selects clips and frames with the new context. The argument requires this confidence signal to be meaningful.","core_discovery":"MASR's central claim is that the main bottleneck in video QA is not model capacity but attention allocation: a model needs to know where to look before it can answer. The paper argues that a multimodal coarse-to-fine relevance-sensing mechanism, which combines LLM-based clip selection with visual-token-to-query similarity matching, followed by dilated temporal expansion around the chosen frames, produces a more complete and more relevant context than uniform sampling or single-pass captioning. The self-reflection loop then monitors the LLM's confidence score and re-enters the focusing process whenever the score is below the stopping threshold. With this loop, MASR reports 73.4% on EgoSchema, 73.1% on IntentQA, 75.8% on NExT-QA, and 57.1% on the Video-MME long split, all without supervised fine-tuning of the backbone model.","pith_inferences":["A testable extension: vary the confidence threshold and the round cap; if accuracy is insensitive to them, the gain comes from the focusing mechanism, whereas if it spikes only at C = 3, the stopping rule is doing more work than the paper measures.","The paper does not calibrate confidence against correctness; I infer that a miscalibrated self-report could either halt the loop too early on hard questions or force extra rounds on easy ones, so the reported margins may depend on backbone-specific calibration.","Because DTE borrows the dilation idea from convolutional networks, a natural next step the paper leaves implicit is adaptive per-clip dilation rates chosen by the coarse selector rather than fixed hyperparameters.","The headline EgoSchema gain combines the MCRS/DTE loop with a specific captioner; unless the captioner is held fixed across baselines, part of the margin may be captioning quality rather than attention focusing."],"forward_implications":["If the reported numbers hold, training-free agent pipelines can outperform fine-tuned video-specific MLLMs on medium-length QA without any in-domain training.","A single LLM can simultaneously answer, evaluate its own confidence, and select relevant clips; no separate reward model is required for accurate attention focusing.","The ablation predicts that better visual encoders and stronger reasoning LLMs directly translate into higher video QA accuracy for the same framework.","Because the stopping rule caps self-reflection rounds, computation cost is bounded, but higher dilation windows or frame intervals degrade accuracy, so temporal expansion must be tuned rather than maximized.","Performance on very long videos, including the Video-MME long split, also improves over agent-based baselines, suggesting the same focusing loop scales to longer inputs."],"supporting_citations":[{"why":"Supplies the 180-second egocentric video QA benchmark used for the headline accuracy gain and for the ablation experiments.","marker":"[64]"},{"why":"Supplies the medium-length video QA validation set used for the zero-shot NExT-QA comparison.","marker":"[65]"},{"why":"Supplies the intent-reasoning test set used for the zero-shot IntentQA comparison.","marker":"[66]"},{"why":"Supplies the long-video benchmark whose long split evaluates MASR on ultra-long footage.","marker":"[67]"},{"why":"Supplies the single LLM that generates responses, evaluates confidence, and selects relevant clips in the main experiments.","marker":"[7]"},{"why":"Supplies the captioning model used for semantic extraction on EgoSchema and Video-MME.","marker":"[15]"},{"why":"Supplies the visual encoder whose fine-focusing configuration reports the best accuracy in the ablation.","marker":"[69]"},{"why":"Supplies the adaptive tree-based agent baseline reported at 66.2 on EgoSchema, the comparison behind the 5-point gain claim.","marker":"[59]"}],"fun_headline_variants":["MASR: refocus until confident, video QA jumps 5%","Self-reflective attention: 5% better on video understanding","Video agent beats prior best by 5% with self-focusing","MASR: Where you look beats how big your model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the LLM's self-reported confidence score is reliably correlated with answer correctness, yet the paper presents no calibration data for that score.","fun_headline_variants_meta":{"raw":{"variants":["MASR: refocus until confident, video QA jumps 5%","Self-reflective attention: 5% better on video understanding","Video agent beats prior best by 5% with self-focusing","MASR: Where you look beats how big your model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2750,"prompt_tokens":995,"completion_tokens":1755,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":1681}},"tokens_in":611,"tokens_out":1755,"duration_ms":13410,"temperature":1.0,"reasoning_tokens":1681,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:46:17.887864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a labeled subset, run MASR and record the confidence score and round at which it stops, then compare accuracy for items that stop at C = 3 against items forced to stop at C = 2 or after one round; if forced-early answers are not measurably worse, the confidence-based stopping rule is not the cause of the reported gains.","supporting_citations":[{"cited_title":"IntentQA: Context-aware Video Intent Reasoning,","cited_arxiv_id":null,"evidence_quote":"Supplies the intent-reasoning test set used for the zero-shot IntentQA comparison."}],"review_version":1}