{"id":"20b807c9-8238-4736-8a0a-8a1a6b236a18","arxiv_id":"2608.08612","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"REVEAL replaces relevance-based stopping with rubric-guided evidence-sufficiency verification, improving long-video QA accuracy without fine-tuning.","lead":"REVEAL is a training-free agent framework for long-video question answering that adds an explicit evidence-sufficiency verification step to the retrieval loop. It claims state-of-the-art accuracy on five long-video QA benchmarks by using automatically constructed rubrics to detect missing clues and guide targeted re-retrieval.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported +7.8/+10.9 ablation gain conflates rubric-guided sufficiency verification with rubric-based candidate reranking (§B.3), so the paper's central mechanistic claim is not yet isolated.","rationale":"REVEAL's central claim has two parts: (i) the full system is SOTA; (ii) the SOTA gain comes from explicit evidence-sufficiency verification rather than relevance-based stopping. Part (i) is plausibly supported by controlled comparisons against baselines with the same backbone, though Video-MME-L hyperparameter tuning and single runs remain caveats. The most load-bearing weakness is in part (ii). The implementation admits evidence via rubric-reranking before the verifier sees it, and Table 2's final step adds both the verifier and that reranking. This is an internal-validity confound, not an external transfer assumption: even if rubrics were perfectly transferable, the experiment would not tell us whether the +7.8/+10.9 comes from deciding when to stop and what to repair or simply from ranking better candidates in round 1. The paper's Figure 4a also turns off the rubric wholesale, so it cannot separate the two. The reader's transferability worry is reasonable but partly mitigated by consistent cross-benchmark gains; the reranking confound is not mitigated anywhere in the paper. A conditional acceptance is therefore still appropriate, but the condition should explicitly require (a) the separating ablation described in concrete_test, and (b) release of code and the exact rubric pool, so the reranking and verification prompts can be inspected. If the separating ablation shows that reranking accounts for the gain, the paper's contribution should be reframed.","tokens_in":17783,"tokens_out":7162,"duration_ms":75436,"concrete_test":"Run a 2x2 ablation on LVBench and Video-MME-L: (A) Ours with rubric-reranking enabled but verifier disabled (answer after a fixed 3 rounds, no sufficiency threshold, no missing-evidence repair); (B) Ours with verifier enabled but candidate admission ranked by BGE-M3 relevance only (no rubric-reranking); compare against Ours (79.1/65.9) and M+OnMem (71.3/55.0). If (A) recovers most of the gap, the paper should be reframed as rubric-guided retrieval, not sufficiency verification. If (B) retains most of the gain, the mechanistic claim is supported. Report both conditions with the same K=3, tau=0.7, theta=0.8 settings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline contribution is that explicitly verifying evidence sufficiency, rather than relevance-based stopping, recovers decisive clues. The primary evidence is Table 2's final step: adding the 'Rubric Suff.' component to M+OnMem improves Video-MME-L from 71.3 to 79.1 (+7.8) and LVBench from 55.0 to 65.9 (+10.9). But the 'Ours' pipeline uses the rubric in two distinct places. Besides the verifier's sufficiency score (Eq. 5) and its missing-evidence repair loop, §B.3 states that, in every retrieval round, 'the candidates are merged across queries and rubric-reranked to 8 evidence pieces before being admitted to the cache.' M+OnMem has neither mechanism, so the ablation does not isolate verification. The large jump could come substantially from better first-round candidate ranking by the rubric, without any sufficiency judgment or targeted repair. The Figure 4a analysis ('disable verification') likewise switches off the whole rubric component, so it cannot separate reranking from stopping/repair. The case study in Fig. 5 demonstrates one successful repair but is anecdotal. Consequently, the central claim that sufficiency verification, rather than rubric-improved relevance ranking, drives the gains is not yet supported by the reported experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"REVEAL is a training-free agentic framework for long-video question answering. It builds an offline hierarchical memory (L1 segments, L2 event timelines, L3 global narratives) from visually coherent, variable-length segments, and combines it with a question-conditioned online memory that tracks retrieved evidence, search history, and the verifier's missing-evidence analysis. The core novelty is an automatically constructed rubric library, induced from contrastive pairs of decisive and misleading evidence, that an inference-time verifier uses to decide whether accumulated evidence is sufficient (Eq. 5) and, if not, to produce concrete diagnoses that drive targeted re-retrieval. The paper reports state-of-the-art accuracy on five long-video QA benchmarks with a single open backbone (Qwen3.5-27B), and a component ablation in Table 2 attributes the largest gains to rubric-guided sufficiency verification.","tokens_in":18028,"tokens_out":6035,"duration_ms":68680,"significance":"If the central claim holds, REVEAL is a valuable contribution: it demonstrates that an interpretable, automatically constructed evidence-sufficiency criterion can be inserted into the inference loop of a long-video QA agent without any gradient-based training, and it provides extensive same-backbone comparisons with existing RAG and memory-augmented methods. The paper also ships useful analyses: the evidence-volume curve in Figure 4a shows that verification improves accuracy while using far fewer segments, the calibration analysis in Figure 4b uses oracle target-time annotations, and the per-category breakdown in Figure 4c localizes the gains to evidence-sensitive reasoning types. The detailed implementation notes (supplementary Sec. B) make the system largely reproducible. However, the central mechanistic claim that explicit sufficiency verification, rather than better rubric-based relevance ranking, drives the gains is not yet isolated by the reported experiments, and the main benchmark results are partly affected by hyperparameter tuning on the same dataset.","major_comments":[{"comment":"The implementation described in §B.3 admits evidence via 'the candidates are merged across queries and rubric-reranked to 8 evidence pieces before being admitted to the cache.' The 'Rubric Suff.' row in Table 2 therefore adds both the verifier/repair loop and rubric-based candidate reranking relative to M+OnMem, while the 'disable verification' analysis in Figure 4a also removes the whole rubric component rather than only the sufficiency judgment. The reported +7.8/+10.9 gain thus conflates two distinct mechanisms, and the paper's headline claim that explicit evidence-sufficiency verification, rather than improved relevance ranking, drives the gains is not yet supported. Please add an ablation that includes rubric reranking without the verifier/repair loop, and ideally one that keeps verification while using relevance-only admission, so that the two mechanisms can be separated.","section":"§B.3, Table 2"},{"comment":"Section B.5 states that the hyperparameters K, τ, and θ were tuned on Video-MME-L during development, and Table 1 reports Video-MME-L as one of the five main benchmarks. The Video-MME-L accuracy of 79.1 is therefore not fully held-out, and the same applies to the Video-MME-L row in Table 2. This weakens the headline claim of consistent state-of-the-art performance across all five benchmarks. Please either tune on a separate development set and report fixed-config results on all benchmarks, or explicitly qualify Video-MME-L as a development-tuned result in the main text.","section":"§B.5, Table 1"},{"comment":"The rubric library is constructed from contrastive pairs filtered by the answer model A's own correctness (Eq. 3), and the same backbone Qwen3.5-27B later serves as verifier and answerer. The sufficiency criteria therefore partially encode the decision boundary of the exact model they are meant to audit; if these criteria capture model-specific blind spots or training-distribution regularities rather than generic evidence-sufficiency requirements, the verifier's verdicts may be miscalibrated at test time. The 'one rubric pool across benchmarks' analysis in §4.4 does not fully settle this, because all five evaluation benchmarks share the same construction and evaluation protocol. A concrete transfer test would be to build the rubric library with a different teacher model, or on one benchmark family only (e.g., third-person), and then evaluate the same backbone on the remaining benchmarks; such an experiment would substantially strengthen the transferability claim.","section":"§3.2, Eq. (3)"}],"minor_comments":[{"comment":"The phrase 'without any extra training' is stronger than what is actually done: the rubric library is produced by an offline, LLM-based refinement loop over training pairs described in §B.4. Suggest rewording to 'without gradient-based training' or 'without fine-tuning' to avoid overclaiming.","section":"Abstract, §B.4"},{"comment":"The paper reports that each result comes from a single evaluation run, and some differences in Table 1 are small (e.g., 79.1 vs. 77.0 for Vgent on Video-MME-L). Reporting variance across multiple runs, or at least stating this as a limitation, would make the consistent-improvement claim more robust.","section":"§4.1, Table 1"},{"comment":"The per-category analysis includes a category ('Spatial Perception') with only three questions, so the claim that no category regresses rests on a very small sample for that category. Consider aggregating small categories or explicitly noting the low sample size.","section":"Figure 4c"},{"comment":"Section B.2 notes that L1 segments are approximately 10 seconds long on average, which is the same scale as the fixed 10s chunks criticized in §1. The intended distinction is variable-length, event-boundary-aware grouping rather than a different segment scale; the paper should state this explicitly to avoid an apparent inconsistency.","section":"§B.2, §1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is valid and load-bearing: the +7.8/+10.9 ablation gain conflates rubric-guided sufficiency verification with rubric-based candidate reranking, so the central mechanistic claim is not yet isolated. The hyperparameter tuning on Video-MME-L and the self-referential rubric construction are also real issues, though fixable. The paper is otherwise well written, the comparisons are extensive and controlled, and the supplementary details are unusually complete; I would be willing to see a revision with the requested ablations and a transfer experiment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The key thing to know: REVEAL is the first to use rubric-based criteria as an inference-time control for long-video QA, and the results across five benchmarks are strong. But the paper's central claim—that explicit sufficiency verification, not better candidate ranking, drives the gains—is not actually isolated by the experiments.\n\nWhat is new and what it does well: the rubric-in-the-loop idea is a real departure from prior rubric uses in evaluation and reward modeling. The adaptive event segmentation and offline-online memory separation are also sensible contributions, and the automatic rubric construction from contrastive evidence pairs is clever. The evaluation is broad: five benchmarks, controlled backbone comparisons, efficiency reporting, and a calibration analysis using Ego-R1's oracle annotations. The evidence-volume curve in Fig 4a is a nice idea—showing that more retrieved segments eventually hurt accuracy, while REVEAL is both more accurate and more compact.\n\nThe soft spots are real. The stress-test note is right: in B.3, the full pipeline's retrieval step includes \"rubric-reranked to 8 evidence pieces\" before cache admission. M+OnMem has no rubric at all, so the +7.8/+10.9 jump attributed to \"Rubric Suff.\" includes both the reranking and the verifier/repair loop. The paper does not separate them, and Fig 4a's \"disable verification\" variant also switches off the whole rubric component. The mechanism—sufficiency-based stopping and targeted repair—is consistent with the data but not proven. The case study is one anecdote. The hyperparameters were tuned on Video-MME-L, which is also a main benchmark; results are single runs without error bars; LVBench uses a subset; and no code or data are released. The circularity in rubric construction (the same model filters contrastive pairs and later serves as verifier) is a limitation, but held-out evaluation keeps the numbers from being forced by construction.\n\nWho this is for: anyone working on long-video understanding, agentic retrieval, or evidence-based reasoning. It deserves a serious referee. The fix is straightforward: add an ablation that includes rubric reranking but disables the verifier's stopping/repair, or a relevance-reranking control. Given the novelty and breadth, I would not desk reject this.\n\nRecommendation: accept for peer review. It needs heavy revision, but the core idea is worth the community's time.","headline":"A genuinely new inference-time mechanism with impressive benchmark numbers, but the main ablation does not isolate sufficiency verification from rubric-based reranking.","tokens_in":18608,"tokens_out":2534,"would_cite":true,"duration_ms":28036,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Long-video QA errors trace to stopping retrieval at relevance; an explicit sufficiency check recovers the decisive evidence.","keywords":["long-video question answering","evidence sufficiency","rubric-guided agent","retrieval-augmented reasoning","video memory","contrastive rubric construction","agentic retrieval","multimodal large language model"],"falsifier":"On a held-out set of long-video questions with manually annotated decisive-evidence intervals, compare the verifier's missing-evidence analysis with the actual content of those intervals; if questions whose answer flips from wrong to right after one targeted repair are not the ones the verifier flagged as missing that clue, the claimed sufficiency signal is not what drives the gain.","tokens_in":17533,"feed_emoji":"🎬","tokens_out":10619,"duration_ms":102065,"temperature":0.7,"pith_summary":"Long-video question answering, the paper argues, is limited less by how much relevant context a retrieval loop can find than by when it decides to stop: relevance-based stopping accepts topically related segments even when the decisive temporal, causal, or fine-grained detail is absent. REVEAL replaces that implicit stopping rule with an explicit evidence-sufficiency check. A fixed library of weighted criteria, built automatically from contrastive pairs of decisive and misleading evidence, is selected per question; the verifier scores the accumulated evidence against those criteria and, if the score falls below a threshold, produces a concrete missing-evidence analysis that drives a targeted repair query. On five long-video QA benchmarks, the training-free system reports state-of-the-art accuracy, and the single largest ablation gain comes from adding the sufficiency check. The paper's point is that “enough evidence” is a property that can be verified independently of relevance, and that making it explicit is what retrieves the decisive clues.","feed_headline":"Sufficiency checks beat relevance-only retrieval in long-video QA","feed_subtitle":"A training-free agent that names the missing clue and re-retrieves it beats fixed chunks on five benchmarks.","key_machinery":"The load-bearing mechanism is the rubric-guided retrieve–verify–answer loop. A rubric library is a fixed set of weighted evidence-sufficiency criteria, each with explicit score boundaries (0 missing, 0.5 partial, 1 sufficient), induced offline from contrastive pairs of evidence sets separated by whether they flip the answer model from wrong to right. At inference, the verifier selects the subset of criteria pertinent to the question, computes a weighted aggregate score, and either accepts the evidence or emits a missing-evidence analysis; the planner turns that analysis into targeted retrieval over a hierarchical offline memory of variable-length, visually coherent segments, with event-timeline and global-narrative summaries, while an online memory tracks admitted evidence, visited segments, query history, and per-round retrieval quality. This separation of “how much evidence is enough” (the scalar score) from “what is still missing” (the analysis) is what makes the repair retrieval directed rather than redundant.","core_discovery":"On its own terms, REVEAL's central discovery is that an automatically built rubric of evidence-sufficiency criteria, applied inside the retrieval loop, turns the retrieval-stopping decision into a calibrated, inspectable judgment. The system's verifier selects the question-relevant criteria from a shared rubric pool, assigns each a score of 0, 0.5, or 1 for the accumulated evidence, and stops only when the weighted confidence passes a threshold; on failure it returns a missing-evidence analysis rather than a generic “keep searching” signal. The planner then uses that analysis together with query history and retrieval-quality flags to issue targeted queries, or falls back to uniform timeline sampling when directed search is exhausted. After at most three rounds, the answerer receives a compact evidence cache. The authors report that this verification step—not larger memory, not more retrieved segments—produces the decisive jump in accuracy, and that the same fixed rubric pool transfers across egocentric and third-person benchmarks without per-dataset tuning.","pith_inferences":["A natural extension the paper does not test: because the contrastive-pair filter leans on the answer model's own correctness, swapping in a stronger or weaker teacher should shift the sharpness of every rubric boundary, so verifier calibration becomes a measurable proxy for teacher quality.","The verifier's missing-evidence analysis is already an explanation of what the video fails to establish; surfacing it to users as the justification for an answer, or for an admission of under-support, is an interpretability feature the paper mentions only in passing.","The same contrastive sufficiency logic should transfer to image QA and document QA, where topically relevant passages also omit decisive details; testing REVEAL's loop outside video would reveal whether “evidence sufficiency” is a general agentic-retrieval control or a video-specific fix.","The stopping threshold is treated as a tuned constant; varying it as a user-facing confidence knob would let the same system trade retrieval cost against answer reliability, something the efficiency analysis suggests is feasible."],"forward_implications":["If sufficiency, not relevance, is the right stopping criterion, retrieval systems for long videos should report an evidence-sufficiency score alongside each answer, so users can see when a response is under-supported.","Because a single rubric pool transfers across both egocentric and third-person benchmarks, evidence-sufficiency criteria appear to be reusable reasoning requirements rather than dataset-specific heuristics.","The finding that accuracy peaks with about a quarter of the evidence volume, and declines when more relevant-but-redundant segments are added, implies that context budgets should be set by sufficiency rather than by maximum context length.","The approach is training-free and uses one backbone model for planning, verification, and answering, suggesting that the same control loop can be dropped into existing retrieval-augmented pipelines without fine-tuning."],"supporting_citations":[{"why":"Supplies the Video-MME-L benchmark used for the main accuracy comparison and for hyperparameter development.","marker":"Fu et al. 2024"},{"why":"Supplies the LVBench benchmark whose long-range temporal structure drives the ablation that isolates the verification gain.","marker":"Wang et al. 2024b"},{"why":"Supplies the Ego-R1 Bench with target-time annotations used in the verifier-calibration analysis.","marker":"Tian et al. 2025"},{"why":"Supplies the LongVideoBench validation split used to evaluate referring and temporal reasoning.","marker":"Wu et al. 2024"},{"why":"Provides the BGE-M3 text embeddings that encode and retrieve the multi-level memory narratives.","marker":"Chen et al. 2024b"},{"why":"Provides the SigLIP visual embeddings used for similarity-based segmentation and the L1 visual retrieval channel.","marker":"Zhai et al. 2023"},{"why":"Serves as a representative retrieval-augmented baseline that relies on relevance-based stopping and must be outperformed.","marker":"Luo et al. 2025"},{"why":"Serves as a representative memory-augmented baseline for isolating the contribution of sufficiency verification over memory alone.","marker":"Long et al. 2025"}],"fun_headline_variants":["Rubric-guided verifier beats relevance-only retrieval in long-video QA","Training-free rubric checks boost long-video QA","Rubric-guided agent finds missing evidence in long-video QA","Evidence sufficiency verifier outdoes relevance-only retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the rubric library, built from training questions and filtered by a single model's judgments of what makes evidence sufficient, keeps its meaning on unseen videos, questions, and answer models without retuning.","fun_headline_variants_meta":{"raw":{"variants":["Rubric-guided verifier beats relevance-only retrieval in long-video QA","Training-free rubric checks boost long-video QA","Rubric-guided agent finds missing evidence in long-video QA","Evidence sufficiency verifier outdoes relevance-only retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001006,"raw_usage":{"total_tokens":4289,"prompt_tokens":1018,"completion_tokens":3271,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":3203}},"tokens_in":634,"tokens_out":3271,"duration_ms":25611,"temperature":1.0,"reasoning_tokens":3203,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:28:59.584151+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of long-video questions with manually annotated decisive-evidence intervals, compare the verifier's missing-evidence analysis with the actual content of those intervals; if questions whose answer flips from wrong to right after one targeted repair are not the ones the verifier flagged as missing that clue, the claimed sufficiency signal is not what drives the gain.","supporting_citations":[],"review_version":1}