{"id":"9556ff2c-ed98-4b7b-8531-8f6b1f7ebd80","arxiv_id":"2608.07663","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A simple multi-key caption retrieval system with on-demand temporal expansion beats complex graph and hierarchical memory methods on EgoLifeQA, LVBench, and Video-MME (Long).","lead":"MERIT is a retrieval-augmented video QA system that stores short captions with four different search keys per clip and, at question time, widens the search around matched clips. On three long-video benchmarks it outperforms more complex graph and hierarchical memory systems while using far fewer tokens during memory construction.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'state-of-the-art across three benchmarks' claim is under-supported for Video-MME(Long): the 77.7 vs. 76.6 margin is within sampling noise, no error bars are reported, and TopN/delta are selected on the test set.","rationale":"Read in good faith, MERIT is a coherent, well-ablated pipeline. The multi-key decomposition, neighbor filtering, round-distribution analysis, and efficiency measurements are genuine independent support; the failure-case discussion in Section E.2 is honest and identifies a real limitation. The reader's conditional verdict is reasonable. My stress-test picks a different emphasis: the headline claim is an empirical SOTA claim, and the weakest empirical link is the 1.1-point Video-MME(Long) margin with no significance evidence. Caption coverage is a real ceiling, but the paper's own Target Time Oracle (Table 3) shows that even a perfect selector leaves headroom, and the captioning pipeline is at least adequate to reach high scores under an oracle selector; it does not by itself make the benchmark comparison false. Statistical fragility and test-set hyperparameter selection are more directly load-bearing for 'state-of-the-art across three benchmarks.' The fix is cheap: report paired significance and validation-based tuning. This does not require rejecting the paper; it should remain conditional pending those analyses.","tokens_in":27435,"tokens_out":10863,"duration_ms":111712,"concrete_test":"Obtain or release per-question predictions for MERIT and WorldMM on the 900 Video-MME(Long) questions; run McNemar's test on the paired binary outcomes and bootstrap a 95% confidence interval for the accuracy difference. If the interval includes zero, revise the abstract to claim SOTA on EgoLifeQA and LVBench only, or treat Video-MME(Long) as tied. Additionally, split EgoLifeQA into validation and test sets, and re-select TopN (Table 7) and delta (Table 12) on the validation split before reporting test accuracy; if the test-set-tuned margins shrink materially, the SOTA claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim in the abstract rests on Table 2. On Video-MME(Long), MERIT (GPT-5) scores 77.7 against WorldMM (GPT-5) at 76.6 over 900 questions. A paired accuracy difference of 1.1 points has an approximate standard error of about 1.4 points under simple binomial sampling, so the observed margin is not clearly distinguishable from chance. No confidence intervals, bootstrap estimates, per-question predictions, or multiple-run statistics are provided for any table, so the third benchmark does not currently support 'state-of-the-art across three benchmarks.' The EgoLifeQA (+5.6) and LVBench (+9.9) margins are larger and more likely real, but Tables 7 and 12 select TopN and the neighbor window delta on the EgoLifeQA test set, introducing tuning optimism that could inflate those margins as well. The caption-coverage ceiling identified by the reader, arising from the Eq. 3 memory representation in Section 3.2, is a genuine limitation, but it is not the first thing that would falsify the headline claim; a statistically fragile 1.1-point SOTA on one of the three benchmarks is more directly load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MERIT, a retrieval-based framework for ultra-long video QA. The method partitions a video into 30-second clips, generates a dense caption and four textual keys (event, dialogue, object, summary) per clip, retrieves top-N clips by maximum cosine similarity between the query embedding and the stored keys, expands each retrieved clip's temporal neighborhood by a radius Delta, applies a query-aware filter to the concatenated neighbor captions, and runs a multi-turn agentic loop before final QA on frames from the retrieved clips. The central claim is that this simple, query-agnostic memory construction with inference-time temporal expansion achieves state-of-the-art results on EgoLifeQA, LVBench, and Video-MME(Long), outperforming hierarchical and graph-based memory systems.","tokens_in":27663,"tokens_out":10006,"duration_ms":93335,"significance":"If the reported results hold, MERIT is a valuable counterpoint to the trend of increasingly complex hierarchical and graph memory systems: it shows that query-agnostic preprocessing can be kept lightweight while deferring semantic composition to inference time. The paper has clear strengths: the method is conceptually simple, the ablations in Tables 3, 4, 11, and 12 show that both multi-key retrieval and neighbor filtering contribute, and Table 6 documents substantial memory-construction efficiency gains over WorldMM. The EgoLifeQA (+5.6 points) and LVBench (+9.9 points) gains are large and, if confirmed, would be strong empirical support for the authors' hypothesis. However, the 'across three benchmarks' SOTA claim currently rests on a statistically fragile 1.1-point margin on Video-MME(Long), the headline hyperparameters are selected on the test set, and some reproducibility details (complete prompts, baseline re-evaluation, uncertainty quantification) are missing. These issues are fixable and do not invalidate the core idea, but they need to be addressed before the paper's strongest claims can be accepted.","major_comments":[{"comment":"The claim of state-of-the-art performance on Video-MME(Long) is not statistically supported. In Table 2, MERIT (GPT-5) scores 77.7 versus WorldMM (GPT-5) at 76.6 on 900 questions, a 1.1-point margin that could easily arise from sampling noise; no confidence intervals or paired significance tests (e.g., McNemar) are reported. The comparison is further weakened because the baseline numbers in Table 2 are taken from [22,33] rather than re-evaluated under the same protocol; indeed, the paper's own reproduction of WorldMM on EgoLifeQA (Table 5) gives 63.2, whereas Table 1 reports 65.6 for the same system. Please provide uncertainty quantification and a controlled re-evaluation of all baselines on all three benchmarks, or temper the 'across three benchmarks' claim in the abstract.","section":"§4.3, Tables 2 and 5"},{"comment":"The default settings TopN=10 and Delta=2 are selected by sweeping on the EgoLifeQA test set (Table 7 sweeps TopN; Table 12 sweeps the neighbor-filtering radius), and the same settings are then used for the headline results in Tables 1 and 2. This test-set tuning introduces optimism that is not accounted for in the reported +5.6 and +9.9 gains. The paper should either use a held-out validation set, a nested selection protocol, or a sensitivity analysis showing that the qualitative conclusions are stable across a range of TopN and Delta values.","section":"Tables 7 and 12"},{"comment":"The memory representation in Eq. (3) stores, for each 30-second clip, a dense caption d_i and derived keys K_i; retrieval and final QA can only access visual frames from the few retrieved clips. Consequently, any information omitted by the GPT-5-mini captions—small object attributes, subtle actions, brief spoken exchanges—is unrecoverable. This caption-coverage ceiling is a fundamental scope limitation and should be stated in the main text; the failure case in Supp. E.2 (frequency/counting queries) is one manifestation and is currently only discussed in the supplement.","section":"§3.2, Eq. (3), and Supp. E.2"},{"comment":"The paper states in Section A that 'all specific LLM prompts used in this pipeline are provided in Section F,' but Section F contains only the multi-key extraction and neighbor-filtering prompts. The prompts for retrieval-query formulation, evidence-sufficiency assessment, and final QA generation—central to the agentic loop—are missing, and the manuscript does not describe a code/data release. Please make the complete prompt set and decoding settings available so that the reported numbers can be reproduced.","section":"Supp. A and F"}],"minor_comments":[{"comment":"Figure 2 contains garbled labels (e.g., 'ultra-ong', 'ltra-ong', 'Captionin', 'Similarit') that should be corrected.","section":"Fig. 2"},{"comment":"Reference [10] lists arXiv:2410.057792(3), which appears to contain a typo in the arXiv identifier.","section":"References"},{"comment":"The sentence 'the gap expands from +3.4% on Video-MME(L) to an impressive +11.4% on LVBench' compares MERIT to GPT-5 with uniform sampling, not to the previous memory-based SOTA; the wording should distinguish these comparisons to avoid implying a 3.4-point gain over the prior SOTA.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a clear and simple message, but the headline SOTA claim on Video-MME(Long) is fragile (1.1-point margin, no uncertainty quantification, baselines not re-evaluated under the same protocol), and the test-set hyperparameter selection is a real concern. The authors should be given a chance to fix these issues with paired significance tests, controlled baseline comparisons, and a validation-based tuning protocol. Please also encourage them to release code and complete prompts, as the current manuscript omits several prompts that are essential for reproducing the agentic pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MERIT is the rare 'simple beats complex' result that mostly delivers. The core idea: instead of building hierarchical or graph memory query-agnostically, index each 30-second clip with four caption-derived keys (event, dialogue, object, summary), retrieve by max-similarity late interaction, then expand around retrieved clips and filter with the solver at inference time. That combination is new relative to EgoRAG, WorldMM, and caption-RAG baselines, and it makes a clean point: defer semantic composition to query time and spend preprocessing budget on high-recall keys.\n\nThe paper does several things well. The ablations on EgoLifeQA isolate both components; the key-combination table shows the summary key is the best single key and all four together win; the neighbor filtering radius study shows a plateau; the retrieval hit rate and round-distribution comparisons against reproduced WorldMM and EgoRAG are informative; and the token counts demonstrate an 8.4x input-token reduction in memory construction versus WorldMM. The EgoLifeQA (+5.6) and LVBench (+9.9) margins are large and consistent across the reported settings.\n\nThe soft spots are real but concentrated. The abstract's 'state-of-the-art across three benchmarks' leans on Video-MME Long, where MERIT (GPT-5) scores 77.7 versus WorldMM's 76.6 over 900 questions. That 1.1-point gap is within ordinary sampling noise (standard error around 1.4 points), and no error bars, bootstrap intervals, or per-question results are reported anywhere. The hyperparameters TopN=10 and delta=2 are selected on the EgoLifeQA test set (Tables 7 and 12) and then used for the headline numbers, so some tuning optimism is baked in. The caption-coverage ceiling is a genuine limitation but it is not the first thing that would falsify the main claim; the fragile Video-MME margin and test-set selection are more directly load-bearing. Lack of code and data also makes the reproduced baselines hard to audit, though the paper does describe the reproduction setup and reports hit rates.\n\nWho gets value: researchers working on long-video memory architectures, especially those comparing hierarchical or graph-based preprocessing against lightweight indexing. The paper deserves a serious referee. It should be reviewed with requested revisions: report variance or paired statistics, fix hyperparameters on a validation split, release code and data, and soften the Video-MME SOTA language. The central intuition is worth engaging with even if the current evidence is not airtight.","headline":"Simple multi-key retrieval with inference-time expansion is a genuinely good idea and mostly backs its SOTA claims, but the Video-MME margin is thin and test-set-tuned hyperparameters blunt the headline.","tokens_in":28231,"tokens_out":1539,"would_cite":true,"duration_ms":16161,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For ultra-long videos, a simple multi-key caption memory can beat pre-built graph memories on three QA benchmarks.","keywords":["ultra-long video understanding","video question answering","retrieval-augmented generation","multi-key memory","episodic memory","neighbor filtering","multi-modal large language models","caption-based retrieval"],"falsifier":"Take a query whose answer depends on a detail absent from the caption of the correct 30-second clip—for example, the color of a small object visible only in frames—and force-retrieve exactly that clip; if the solver still cannot answer, the caption-completeness assumption fails and no amount of better retrieval can fix it.","tokens_in":27218,"feed_emoji":"🎬","tokens_out":6093,"duration_ms":54011,"temperature":0.7,"pith_summary":"The paper argues that the standard recipe for ultra-long video QA—spend heavy computation before seeing the question to build hierarchical or graph memories—is backwards. Instead, it proposes storing each 30-second clip as a dense caption plus four complementary retrieval keys (event, dialogue, object, summary), and deferring all semantic composition to query time. At inference, matching the query against these keys picks anchor clips, and a neighbor-filtering step expands the temporal context around those anchors so the solver can reason over local evidence. The paper reports that this simpler pipeline sets new state-of-the-art accuracy on EgoLifeQA, LVBench, and Video-MME(Long), while using many fewer tokens than graph memory construction. The broad claim a sympathetic reader takes away: high-recall retrieval plus a strong reasoning model beats query-agnostic preprocessing.","feed_headline":"Simple key matching beats graph memory on day-long video QA","feed_subtitle":"Four keys per clip, plus query-time neighbor expansion, tops three long-video QA benchmarks.","key_machinery":"The load-bearing object is the episodic multi-key memory entry: a key-value record $(v_i, d_i, K_i)$ in which each clip carries exactly four lightweight textual keys. The matching mechanism is maximum-similarity retrieval over these keys, plus a symmetric temporal window $W_i = \\{t : i - \\Delta \\le t \\le i + \\Delta\\}$ around each retrieved clip, with the solver acting as a query-conditioned filter on the concatenated neighbor captions. This replaces pre-computed hierarchies or graphs with a flat store and on-demand local expansion.","core_discovery":"The central claim is that an episodic multi-key memory is sufficient for ultra-long video understanding, provided retrieval is high-recall and semantic composition happens at inference. Memory is built query-agnostically as $M = \\{(v_i, d_i, K_i)\\}$, where $v_i$ is a 30-second clip, $d_i$ its dense caption, and $K_i$ a set of four keys—event/action, dialogue/mention, object-state, and summary—that let different query intents match the same clip. Retrieval scores each clip by the maximum cosine similarity between the query and any of its keys, which boosts recall without hierarchical indexing. For each retrieved anchor, neighbor filtering concatenates captions from a temporal window of radius $\\Delta = 2$ (roughly $\\pm 1$ minute) and asks the solver to distill only query-relevant information. The paper reports state-of-the-art QA accuracy across three benchmarks with GPT-5 as solver—71.2% on EgoLifeQA, 71.8% on LVBench, and 77.7% on Video-MME(Long)—and argues that these results support deferring high-level relation composition to inference rather than pre-modeling it during memory construction.","pith_inferences":["If the caption-omission assumption is the bottleneck, MERIT is a natural substrate for a hybrid that re-captions only the retrieved clips with a stronger vision model, rather than re-captioning the whole video per query.","The 'intelligence allocation' thesis generalizes beyond video RAG: any retrieval system with a strong reasoner may do better to preserve raw evidence and defer abstraction until the query is known, a claim that is testable in document and multimodal RAG settings.","The fixed radius $\\Delta = 2$ is a place to adapt: since hit rate keeps rising to $\\Delta = 4$ while accuracy plateaus, a query- or confidence-dependent temporal window could trade tokens for accuracy more efficiently."],"forward_implications":["Memory construction becomes a single pass: each clip is captioned once and keyed once, cutting input tokens by roughly 8.4 times and output tokens by 8.6 times relative to a multi-granularity graph memory baseline.","Fine-grained question types benefit most from retrieval quality: on EgoLifeQA, EntityLog and EventRecall improve sharply, while relation-heavy categories gain from temporal expansion.","Better solvers can absorb more retrieved clips: with GPT-5, raising TopN from 5 to 10 improves accuracy, whereas an open 8B solver degrades, so the optimal retrieval budget depends on solver capacity.","Most queries are answered early: 68.2% of EgoLifeQA questions finish in one retrieval round, averaging 1.72 rounds versus 2.45 for a graph-memory baseline, so simpler retrieval reduces inference steps as well as build cost."],"supporting_citations":[{"why":"Supplies the EgoLifeQA benchmark and the hierarchical-memory baseline whose accuracy MERIT must exceed.","marker":"[31]"},{"why":"Supplies the graph-memory baseline with multi-granularity episodic and semantic memories that MERIT compares against and whose token cost it measures.","marker":"[33]"},{"why":"Supplies the LVBench hour-long benchmark, including clue-duration annotations that drive the short/medium/long analysis.","marker":"[27]"},{"why":"Supplies the Video-MME benchmark and its certificate-length analysis for the Long subset used in evaluation.","marker":"[8]"},{"why":"Supplies the captioning and solving model that generates dense captions, multi-keys, and final answers.","marker":"[20]"},{"why":"Supplies the embedding model used as the sentence embedder for multi-key retrieval.","marker":"[37]"},{"why":"Supplies the late-interaction maximum-similarity retrieval idea used in the multi-key matching formulation.","marker":"[13]"}],"fun_headline_variants":["Key matching beats graph memory on day-long video QA","Simple keys, not graphs, set new long-video QA record","Four keys per clip plus query-time expansion tops long-video QA","MERIT: multi-key retrieval for practical day-long video understanding","Defer complex memory, retrieve with keys: new SOTA in video QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything downstream depends on the once-generated dense caption for each 30-second clip containing every visual and spoken detail a future question might ask about, because neither retrieval nor final reasoning can recover what the captioner omitted.","fun_headline_variants_meta":{"raw":{"variants":["Key matching beats graph memory on day-long video QA","Simple keys, not graphs, set new long-video QA record","Four keys per clip plus query-time expansion tops long-video QA","MERIT: multi-key retrieval for practical day-long video understanding","Defer complex memory, retrieve with keys: new SOTA in video QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000347,"raw_usage":{"total_tokens":1935,"prompt_tokens":1014,"completion_tokens":921,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":832}},"tokens_in":630,"tokens_out":921,"duration_ms":8150,"temperature":1.0,"reasoning_tokens":832,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:25:15.717069+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a query whose answer depends on a detail absent from the caption of the correct 30-second clip—for example, the color of a small object visible only in frames—and force-retrieve exactly that clip; if the solver still cannot answer, the caption-completeness assumption fails and no amount of better retrieval can fix it.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/CVF International Conference on Computer Vision","cited_arxiv_id":null,"evidence_quote":"Supplies the LVBench hour-long benchmark, including clue-duration annotations that drive the short/medium/long analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the captioning and solving model that generates dense captions, multi-keys, and final answers."}],"review_version":1}