{"id":"bf182ecb-42be-4388-9bed-5dec8ecc2be7","arxiv_id":"2506.17629","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CLiViS uses iterative LLM-planned, VLM-executed perception with a dynamic cognitive map to improve embodied visual question answering on OpenEQA, EgoTempo, and EgoSchema.","lead":"CLiViS is a new no-training pipeline that combines a language model planner with a vision-language model to answer questions about first-person videos. It builds an evolving 'cognitive map' of the scene and asks the vision model targeted follow-up questions until it can answer.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper does not establish that sub-instruction-guided VLM calls add genuinely new visual evidence; reported gains may stem from iterative LLM reasoning over already-rich captions.","rationale":"The reader's weakest_assumption identifies two intertwined concerns: (i) subtask-guided VLM perception may not recover details missing from coarse captions, and (ii) the Qwen2.5-Max judge may bias open-ended scores. I isolate (i) as the single most load-bearing concern because it attacks the core novelty of the framework: if the VLM's update rounds do not add genuinely new visual facts, then the 'linguistic-visual synergy' and 'dynamic cognitive map' are not the active ingredients, and the method reduces to an LLM prompting itself over a fixed set of captions. The paper's Table 3 ablation 'w/o multi-round interaction' shows a large drop, but that ablation removes both the extra LLM reasoning steps and the extra VLM calls, so it cannot attribute the gain to perception. The qualitative example in Fig. 4 actually illustrates the risk: the entity named in the update ('corn') was already mentioned in the initial caption as 'cornstalks'. A control experiment that feeds the initial captions back in place of VLM responses would settle whether the VLM is providing new evidence. If the control matches CLiViS, the central mechanism claim is unsupported; if the control is substantially worse, the concern is resolved. Because the reader's CONDITIONAL verdict already allows for this uncertainty, I keep the verdict UNCHANGED, but the proposed control should be a required condition for acceptance.","tokens_in":16415,"tokens_out":11877,"duration_ms":108546,"concrete_test":"On EgoTempo (or a 100-question subset), add two controls: (A) Replace every update-round VLM response with the corresponding segment's initial caption text, keeping all other components identical; if accuracy stays near 23.0, the update adds no measurable visual information. (B) An LLM-only iterative baseline: the same Qwen2.5-Max is given the initial captions and prompted to ask and answer its own sub-questions over multiple rounds, with no further VLM calls; if it reaches CLiViS's accuracy, the VLM loop is not the source of gains. Also annotate a sample of 50 update rounds for whether the VLM response contains at least one factual detail (entity, attribute, relation, or event) absent from the initial caption of the same segment; report the fraction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CLiViS's central premise (Sec. 3.2 stage 2, Eq. 7) is that LLM-generated sub-instructions elicit new visual evidence from the VLM, updating the cognitive map beyond what the initial captions contain. This premise is not tested. The initial cognitive map is built from per-30s VLM captions that already list entities, attributes, actions, and relations (Sec. 3.2 stage 1). In the paper's own running example (Fig. 4), the first caption already contains 'a few cornstalks' inside the refrigerator; the later 'new' finding that 'the object to the left of the hawthorn juice is corn' is a spatial relation re-describing an already-seen entity, not a novel visual discovery. More generally, the VLM is asked to answer targeted questions about segments it has already described, so the marginal information may be small. The only relevant ablation, 'w/o multi-round interaction' (Table 3), drops 10.5 points, but this compares a full iterative LLM+VLM loop against a single LLM+VLM pass; it cannot separate 'more LLM reasoning' from 'new VLM evidence'. A control that feeds the initial captions back to the LLM for multiple rounds, without additional VLM calls, is absent. If that control matches CLiViS, then the 'linguistic-visual synergy' is not the load-bearing ingredient; the observed gains could be produced by an LLM alone doing iterative self-questioning over rich captions. This directly affects the central claim that the dynamic cognitive map and VLM orchestration provide the benefit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"CLiViS is a training-free framework for embodied visual reasoning (episodic-memory EQA) over long egocentric video. It pairs an LLM planner (Qwen2.5-Max by default) with a VLM executor. In stage (1), the video is segmented into 30-second clips; the VLM produces coarse per-clip captions, and the LLM parses them into a two-part 'cognitive map' (a temporal navigation graph and an entity relation graph) together with an instruction-filtered evidence memory. In stage (2), the LLM iteratively issues focused sub-instructions for specific segments, the VLM answers them, and the LLM updates the map and memory (Eqs. 7-10). In stage (3), when the LLM decides the evidence is sufficient or the round limit is reached, it generates the final answer. Experiments on OpenEQA, EgoSchema, and EgoTempo claim consistent state-of-the-art results (55.4%, 69.4%, and 23.0%; average 49.4%), model-agnostic gains over three 7B-8B VLM backbones, component ablations on EgoTempo, and a latency-accuracy analysis on EgoSchema. The supplement adds per-category EgoTempo results, comparisons with GPT-4.1 and Gemini-2.5-flash, and qualitative reasoning traces.","tokens_in":16771,"tokens_out":17242,"duration_ms":140637,"significance":"The paper has real strengths worth crediting: code is released; evaluation uses standard metrics on three external benchmarks; the ablations in Tables 2 and 3 are informative; the latency analysis addresses a practical concern; and the fully traceable reasoning chain (sub-instructions, VLM responses, map updates) is a genuine interpretability asset. If the reported numbers hold, the practical contribution is meaningful: a modular, training-free recipe that consistently lifts several open 7B-8B VLMs, especially on EgoTempo and EgoSchema, and that can also wrap stronger proprietary backbones (Table 6). The significance is, however, bounded by two problems: the headline 'state-of-the-art' claim is contradicted by the paper's own Table 6 (GPT-4.1 alone scores 34.2% on EgoTempo versus 23.0-23.4% for the open-backbone CLiViS variants), and the ablations do not isolate the claimed mechanism of subtask-guided VLM eliciting new visual evidence from the alternative explanation of multi-round LLM reasoning over the initial captions.","major_comments":[{"comment":"The central 'state-of-the-art' claim is internally inconsistent. Sec. 4.3 asserts state-of-the-art performance with 23.0% on EgoTempo, but Table 6 in the supplement reports that GPT-4.1 alone achieves 34.2% on the same benchmark, which is at least 10.8 points above the best open-backbone CLiViS variant (23.4% for CLiViS with VideoLLaMA3). Likewise, the headline OpenEQA number of 55.4% is for CLiViS with InternVL3, yet the VideoLLaMA3 end-to-end baseline in Table 1 already scores 57.1%; the best CLiViS OpenEQA result is 57.3% (CLiViS with VideoLLaMA3). The authors should either scope the claim precisely (for example, state-of-the-art among open-source 7B-8B VLMs under a fixed LLM) and report the best-performing variant per benchmark, or reconcile the main text with Table 6, which contradicts the unqualified claim in the abstract.","section":"Sec. 4.3, Table 1, Table 6"},{"comment":"The core premise that LLM-generated sub-instructions elicit genuinely new visual evidence from the VLM is not isolated by the ablations. Table 3's 'w/o multi-round interaction' ablation (-10.5) conflates two factors: the number of LLM reasoning iterations and the presence of additional VLM perception calls. A control that feeds the initial captions and cognitive map back to the LLM for the same number of rounds without any further VLM calls is missing; if that control matches the full model, the claimed 'linguistic-visual synergy' of Eq. (7) is not the load-bearing ingredient and the gains would be attributable to iterative LLM self-questioning over rich captions. The Fig. 4 running example illustrates the risk: the initial caption already states that the refrigerator contains 'a few cornstalks', and the later 'discovery' that corn lies to the left of the hawthorn juice re-describes an already-seen entity. I request this control, and also an explanation of why single-round CLiViS (12.5) performs worse than the plain VLM baseline (17.0).","section":"Sec. 3.2 (Eq. 7), Table 3"},{"comment":"There is a generator-evaluator confound in the open-ended benchmarks. Qwen2.5-Max is used both as the LLM inside CLiViS that produces the final answers and as the scorer that rates all responses on the 5-point Likert scale. Because CLiViS's answers are generated in Qwen2.5-Max's own style, the judge may systematically favor CLiViS (and the Socratic baselines, which share the same LLM) over end-to-end-VLM outputs. Please add a control: re-score a random subset (for example, 100-200 items per benchmark) with an independent judge such as GPT-4o or human annotators and report agreement and the resulting accuracy; at minimum, discuss this limitation explicitly in Sec. 4.1.","section":"Sec. 4.1"}],"minor_comments":[{"comment":"In the EgoTempo columns, the Qwen2-VL row reports 10.3 for <30s and 10.5 for >=30s but 9.6 for 'All'; since the overall accuracy must be a weighted average of the two groups, it cannot lie outside [10.3, 10.5], so this row is internally inconsistent and should be corrected.","section":"Table 1, Qwen2-VL row"},{"comment":"The VideoTree row shows a latency of 7160.0 s with the accuracy entry missing (it should be 60.0 per Table 1), and the text saying 'Although VideoTree is faster' contradicts the printed numbers (7160 s versus 195 s for CLiViS); this is presumably a decimal typo (71.6 s) but must be fixed.","section":"Table 4, Sec. 4.6"},{"comment":"The reported 'overall average accuracy of 49.4%' differs from 49.3%, the average of 55.4, 23.0, and 69.4, and also from the value given in Table 2 for the same CLiViS (InternVL3) row.","section":"Sec. 4.3"},{"comment":"The notation M = [sum over T_i in LLM(I,M) of VLM(V, T_i)] is nonstandard and ambiguous because the LLM's output is used as the index set of the summation; please rewrite the reformulation as an explicit composition of functions.","section":"Eq. (4)"},{"comment":"Several numeric entries run together (for example, the Qwen2.5-VL row shows '12.026.016.0'), and the reported category averages for Qwen2-VL (9.6), CLiViS/Qwen2.5-VL (19.6), and CLiViS/InternVL3 (23.0) do not match the unweighted means of the ten category values (10.4, 19.2, and 22.8), so please either report weighted averages with category sizes or clarify how the overall averages are computed.","section":"Table 5"},{"comment":"The time range '00:00:00-00:0:30' in Fig. 4 contains a typo, and the caption of Fig. 7 spells 'Sub-Instrction' instead of 'Sub-Instruction'.","section":"Fig. 4 and Fig. 7"}],"recommendation":"major_revision","confidential_remarks":"The material contradiction between the main-text state-of-the-art claim and supplementary Table 6 should be resolved before acceptance; even if all experiments are reproducible, the paper's framing needs to be re-scoped. The missing control (iterative LLM reasoning over static captions without additional VLM calls) is inexpensive to run given the released code and would settle the mechanism question, so I recommend making it a condition of revision. The evaluator confound (Qwen2.5-Max as both generator and judge) is a common but under-discussed practice in open-ended evaluation; a re-scoring check with an independent judge on a subset would substantially strengthen the open-ended numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a solid, engineering-first paper that packages a training-free LLM-planner/VLM-executor loop with a dynamic cognitive map (navigation graph + relation graph) and evidence memory. It works across three egocentric QA benchmarks and three different 7B-8B VLM backbones, with ablations that mostly make sense. The code is out. If you build video-QA agents from off-the-shelf models, this is a useful recipe.\n\nThe genuinely new bit is the structured, evolving cognitive map that the LLM reads and writes over, plus the evidence buffer that tracks rationales. The ablations show each piece contributes something, and the 12.4-point drop when swapping the LLM for a VLM as reasoner is a clean result: the LLM is doing real planning, not just echoing captions.\n\nNow the soft spots, in proportion. First, \"state-of-the-art\" in the main text is overclaimed. The appendix (Table 6) shows GPT-4.1 alone scores 34.2% on EgoTempo versus 23.0% for the best CLiViS variant in Table 1; CLiViS with GPT-4.1 as the LLM does beat GPT-4.1 (36.8 vs 34.2), so the claim is defensible if you add the qualifier \"among open-source, training-free methods with 7B-8B VLMs.\" But the main text doesn't say that. The reader's summary said Table 6 contradicts the central claim; it doesn't, once you notice CLiViS w/ GPT-4.1 in that same table. Still, the framing matters.\n\nSecond, there's an internal inconsistency in Table 1: Qwen2-VL shows EgoTempo <30s = 10.3, >=30s = 10.5, but All = 9.6. That's arithmetically impossible and needs fixing.\n\nThird, the stress-test concern I largely buy: the paper never isolates whether the performance gain comes from new visual evidence elicited by sub-instructions, or just from iterative LLM reasoning over already-rich captions. The \"w/o multi-round interaction\" ablation drops 10.5 points, but that compares a full loop against a single pass; it conflates more LLM steps with more VLM perception. A control that feeds the initial captions back to the LLM for several rounds, with no additional VLM calls, would settle it. I checked their running example: the first caption already mentions cornstalks inside the refrigerator; the sub-instruction then elicits the spatial relation \"left of hawthorn juice,\" which is genuinely new relational evidence. So the mechanism is plausible, but the paper should demonstrate rather than assume it. This is the main weakness and it is fixable.\n\nMinor: no error bars or multiple-run variance, given reported temperatures of 0.5 and 0.3. The absolute EgoTempo numbers are all low (best 23%), but that's the benchmark's fault, not the method's.\n\nBottom line: worth a serious referee. I'd send it to review with a request for the caption-only multi-round control, a corrected Table 1, and a qualified SOTA sentence. The framework is useful, the writing is clear, and the missing pieces are addressable. For a reading group, I'd say maybe; it's a good example of structured reasoning for video QA, but the core claim needs one more experiment before I'd cite it as evidence for the synergy hypothesis.","headline":"Useful training-free video-QA pipeline with a genuinely structured cognitive map, but the 'SOTA' claim needs qualifiers and the paper never isolates whether the gains come from new VLM evidence or just iterative LLM reasoning.","tokens_in":17280,"tokens_out":3245,"would_cite":false,"duration_ms":31877,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a training-free LLM–VLM loop with a dynamic cognitive map reaches state-of-the-art embodied visual reasoning, reporting 49.4% average accuracy across three benchmarks.","keywords":["embodied visual reasoning","egocentric video question answering","LLM-VLM collaboration","cognitive map","training-free framework","temporal reasoning","scene graph","open-ended QA"],"falsifier":"Re-run the EgoTempo and OpenEQA evaluations with human raters or an independent judge model (e.g., GPT-4.1) instead of Qwen2.5-Max, and add a dense-caption control in place of the VLM's coarse captions; if the 23.0% and 55.4% scores fall to or below the end-to-end baselines under either change, the claimed synergy gain is a scoring artifact or a caption-impoverishment effect rather than a reasoning improvement.","tokens_in":16240,"feed_emoji":"🧠","tokens_out":5254,"duration_ms":45249,"temperature":0.7,"pith_summary":"CLiViS is a training-free framework for embodied visual reasoning that claims to resolve the trade-off between perception and reasoning by pairing an LLM planner with a VLM perceiver. The LLM decomposes a question into targeted visual subtasks, the VLM executes them on egocentric video clips, and a dynamic cognitive map accumulates the results. On OpenEQA, EgoTempo, and EgoSchema the framework reports 55.4%, 23.0%, and 69.4% accuracy, an average of 49.4%, ahead of Socratic caption-based pipelines, end-to-end VLMs, and existing video-reasoning agents. The authors argue the margin grows with video length, making the approach most relevant for long-horizon embodied tasks.","feed_headline":"Training-free LLM+VLM loop tops egocentric video QA","feed_subtitle":"CLiViS scores 55.4% on OpenEQA and 69.4% on EgoSchema, with its largest gains on long videos.","key_machinery":"The dynamic Cognitive Map is a graph-based representation of the embodied scene that evolves during inference. It has two components: a navigation graph whose nodes are fixed-length video segments with entities and captions, and a relation graph whose nodes are entities and actions and whose edges are spatial, agent-object, or functional relations. The map is initialized by VLM-generated segmented scene descriptions and refined in each round by new VLM evidence under LLM-generated sub-instructions, with conflict resolution favoring newer observations; an evidence memory stores time-stamped rationales. The map is what bridges low-level perception (VLM) and high-level reasoning (LLM), and ablations show that removing it, or collapsing the iterative loop, costs roughly 10 points on EgoTempo.","core_discovery":"The central claim is that embodied visual reasoning should be reformulated as an iterative LLM–VLM synergy rather than a single captioning or end-to-end step: $R = \\mathrm{LLM}(M, I \\mid M = \\bigcup_{T_i \\in \\mathrm{LLM}(I,M)} \\mathrm{VLM}(V, T_i))$, where $M$ is a structured cognitive map initialized from coarse per-segment descriptions and then updated round by round with VLM responses to LLM-generated sub-instructions. The paper presents evidence that this division of labor — LLM as high-level planner, VLM as focused open-vocabulary perceiver — outperforms both static-caption LLM reasoning and monolithic VLM answering, with the largest improvements on videos longer than 30 seconds and on reasoning-intensive question categories such as spatial relations and action sequences. The authors attribute the gains to the map's ability to localize evidence in time (navigation graph) and space (relation graph), plus an evidence memory that accumulates instruction-relevant rationales.","pith_inferences":["If the gains come from iterative verification rather than the specific graph structure, a similar cognitive-map loop could transfer to other long-form video tasks such as surveillance review or instructional video QA, where open-vocabulary detail and temporal localization matter.","The paper's assumption that subtask-guided VLMs recover caption-missing details could be stress-tested by replacing the VLM's coarse captions with human-written dense descriptions; if CLiViS still beats caption-plus-LLM on those, the benefit is architectural rather than an artifact of poor captions.","Because open-ended answers are judged by the same model family that generates them, the absolute scores may be optimistic; re-running with an independent judge model or human raters would clarify whether the 49.4% average reflects task skill or judge bias.","The authors observe that reasoning rounds increase with video duration and correlate with accuracy; an explicit round-allocation policy that scales with clip count could turn this correlation into a controllable latency-accuracy trade-off."],"forward_implications":["CLiViS's training-free design means the gains come at inference time, so any improved VLM backbone can be plugged in and still shows average accuracy gains (e.g., +3.6 points over the InternVL3 baseline).","The performance gap over baselines widens with video duration, implying the framework is suited to long-horizon egocentric tasks such as robotics and autonomous driving.","Replacing the LLM with a VLM for planning drops EgoTempo accuracy by 12.4 points, indicating a strong text-only reasoner is necessary for complex embodied questions.","Multi-round interaction is essential: collapsing the loop to a single round drops accuracy by 10.5 points, so the hypothesis-verification cycle, not the map alone, carries much of the benefit."],"supporting_citations":[{"why":"Supplies the OpenEQA benchmark whose 1,079 QA pairs are the primary open-ended testbed.","marker":"[23]"},{"why":"Provides the EgoSchema validation set of 500 long-video multiple-choice questions.","marker":"[24]"},{"why":"Defines the EgoTempo temporal-reasoning benchmark with 500 QA cases across 10 categories.","marker":"[29]"},{"why":"Qwen2.5-VL serves as a VLM backbone and as a Socratic-baseline captioner.","marker":"[3]"},{"why":"InternVL3 is the VLM backbone that yields CLiViS's best results.","marker":"[44]"},{"why":"VideoLLaMA3 provides an additional VLM backbone and a strong end-to-end baseline.","marker":"[42]"},{"why":"Qwen2.5-Max is the LLM planner and the judge used for open-ended responses.","marker":"[40]"},{"why":"DeepSeek-V3 serves as the alternative LLM in Socratic-based comparison pipelines.","marker":"[19]"},{"why":"VideoAgent is a training-free memory-augmented video-reasoning baseline compared on accuracy and latency.","marker":"[8]"},{"why":"VideoTree is a training-free frame-selection baseline compared on long-video QA.","marker":"[38]"}],"fun_headline_variants":["Iterative LLM-VLM map beats caption-only video QA","Dynamic cognitive map boosts LLM+VLM for long video","CLiViS: mapping vision-language synergy for embodied QA","LLM+VLM loop with map leads video reasoning benchmarks","Training-free VLM+LLM synergy excels on long clips"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's reported gains rest on the assumption that the LLM's sub-instructions make the VLM extract genuinely new, previously missing visual evidence, and that scoring open-ended answers with the same model family that generates them (Qwen2.5-Max) does not inflate the gains.","fun_headline_variants_meta":{"raw":{"variants":["Iterative LLM-VLM map beats caption-only video QA","Dynamic cognitive map boosts LLM+VLM for long video","CLiViS: mapping vision-language synergy for embodied QA","LLM+VLM loop with map leads video reasoning benchmarks","Training-free VLM+LLM synergy excels on long clips"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001186,"raw_usage":{"total_tokens":4923,"prompt_tokens":998,"completion_tokens":3925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":3841}},"tokens_in":614,"tokens_out":3925,"duration_ms":30050,"temperature":1.0,"reasoning_tokens":3841,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:04:35.933670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the EgoTempo and OpenEQA evaluations with human raters or an independent judge model (e.g., GPT-4.1) instead of Qwen2.5-Max, and add a dense-caption control in place of the VLM's coarse captions; if the 23.0% and 55.4% scores fall to or below the end-to-end baselines under either change, the claimed synergy gain is a scoring artifact or a caption-impoverishment effect rather than a reasoning improvement.","supporting_citations":[{"cited_title":"Openeqa: Embodied question answering in the era of foun- dation models","cited_arxiv_id":null,"evidence_quote":"Supplies the OpenEQA benchmark whose 1,079 QA pairs are the primary open-ended testbed."},{"cited_title":"Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural In- formation Processing Systems, 36:46212–46244, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the EgoSchema validation set of 500 long-video multiple-choice questions."},{"cited_title":"Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos","cited_arxiv_id":null,"evidence_quote":"Defines the EgoTempo temporal-reasoning benchmark with 500 QA cases across 10 categories."},{"cited_title":"Videoagent: A memory-augmented mul- timodal agent for video understanding","cited_arxiv_id":null,"evidence_quote":"VideoAgent is a training-free memory-augmented video-reasoning baseline compared on accuracy and latency."},{"cited_title":"Videotree: Adaptive tree-based video representation for llm reasoning on long videos","cited_arxiv_id":null,"evidence_quote":"VideoTree is a training-free frame-selection baseline compared on long-video QA."}],"review_version":2}