Pith. sign in

REVIEW 2 major objections 5 minor 15 references

A three-layer online memory with explicit causal event links, walked by a self-correcting agent, lifts multi-hop answers over hour-long video streams without keeping the full video.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 09:30 UTC pith:RPTN2IFB

load-bearing objection Solid systems paper: hierarchical memory with causal edges plus a harnessed agent actually moves online long-video QA, with honest ablations and a residual bottleneck the authors already name. the 2 major comments →

arxiv 2607.02588 v1 pith:RPTN2IFB submitted 2026-07-01 cs.CV cs.AIcs.CL

Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning

Yixin Ji , Fanghua Ye , Juntao Li , Bo Zhao , Zexuan Qiu , Zhaopeng Tu , Liefeng Bo , Min Zhang This is my paper
classification cs.CV cs.AIcs.CL
keywords long-form video understandingonline memoryhierarchical memoryagentic reasoningevent graphentity graphmulti-hop reasoningmultimodal large language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Models that handle short clips still fail when video arrives as a continuous stream and full history cannot be stored. Prior online systems either keep compressed visual features that lack structure, or higher-level notes ordered only by time, so multi-hop story logic must be rebuilt by the language model at every question. HOMER instead stores raw keyframes, character-centered facts, and event segments joined by explicit temporal and causal edges, then lets an agent locate the scene, drill into details, and verify each step under a harness that corrects failures and accumulates reusable skills. The design produces large accuracy gains on long-video benchmarks that hold when the reasoning backbone is swapped, showing the improvement comes from the memory structure itself rather than any single model.

Core claim

HOMER establishes that online long-video understanding improves when memory separates narrative coverage from perceptual detail and treats temporal-causal relations between events as first-class objects, and when the reasoner can detect insufficient retrieval and recover. With this hierarchical memory and harnessed multi-round agent, the method outperforms the previous best agent baseline by 5.5, 10.8, and 4.4 points on M3-Bench-robot, M3-Bench-web, and Video-MME-Long, and the same structural lift appears across three different language-model backbones.

What carries the argument

HOMER’s hierarchical online memory—Perceptual Buffer of keyframes, Entity Graph of character-centric episodic and semantic memories, and Event Graph of segments linked by temporal and causal edges—navigated by an agent that decomposes questions, retrieves at progressive granularity, and is supervised by a verify-and-correct harness with cross-question skill evolution.

Load-bearing premise

The system assumes an LLM can incrementally write accurate event segments and temporal-causal edges from streaming clip notes; if those summaries or edges are wrong or missing, multi-hop retrieval has nothing reliable to walk.

What would settle it

Replace the learned Event Graph with human-annotated segments and causal edges on the same videos and re-run the identical agent and harness; if accuracy does not rise substantially, structured event memory is not the main driver. If most remaining errors still fall in “memory absent or coarse” even with oracle graphs, the claim that construction—not the agent loop—is the bottleneck fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Explicit event-level temporal and causal edges move multi-hop narrative chaining out of the language model’s ad-hoc reconstruction at query time.
  • The same memory structure lifts both multimodal and text-only backbones, so the gain is structural rather than model-specific.
  • Failure attribution shows most remaining errors arise in what is written into memory, directing further work at denser online episodic construction.
  • Batch-level self-evolution produces a compact skill library that improves later questions without retraining any model weights.
  • Online agents can obtain offline-style multi-hop localization while only ever consulting a bounded, incrementally built memory.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because memory construction dominates residual errors, better online event segmenters and causal edge writers may yield larger gains than further agent-loop tuning.
  • The inform–constrain–verify–correct harness is portable to other streaming modalities that require multi-hop evidence under a hard context budget.
  • Character-centric entity graphs plus causal event graphs form a lightweight narrative schema that could transfer to long-horizon robot or world-model memory.
  • Skill promotion from trajectories without weight updates offers a practical path for agents that keep improving on a fixed video corpus after deployment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. HOMER is an online long-video understanding framework that maintains a three-layer hierarchical memory (Perceptual Buffer of quality-scored keyframes, EntityGraph of character-centric episodic/semantic memories, and EventGraph of narrative segments linked by explicit temporal and causal edges) and answers queries via an agentic multi-round retrieval loop supervised by a verify-and-correct harness plus a self-evolution skill library. On M3-Bench-robot, M3-Bench-web, and Video-MME-Long it reports +5.5, +10.8, and +4.4 points over the strongest prior agent baseline (M3-Agent-Gemini3, which shares the same memorization pipeline and Gemini-3 backbone but lacks EventGraph and harness). Ablations (Table 2) isolate EventGraph contributions; backbone swaps (Table 3) show gains on three LLMs including text-only models; failure attribution (§5.4) attributes most residual errors to memory construction rather than the agent loop.

Significance. If the reported gains hold under independent reimplementation, HOMER supplies a concrete, model-agnostic structural recipe for bringing multi-hop narrative reasoning into the online (streaming, bounded-memory) regime that prior online methods largely left to the LLM. The combination of typed temporal-causal edges, multilevel retrieval modes, harness engineering, and self-evolution is a useful engineering contribution for embodied and agentic video systems. Strengths that should be credited: transparent failure analysis, controlled memory-layer ablations, backbone-agnostic lifts (including open-source text-only models matching a closed multimodal baseline), and public code. The residual bottleneck the authors themselves quantify (memory construction) is a clear, falsifiable direction for follow-up rather than a hidden flaw.

major comments (2)
  1. §4.1 / Table 1: The strongest baseline comparison (M3-Agent-Gemini3) shares the memorization model and Gemini-3 backbone, which cleanly isolates EventGraph + harness. However, EventGraph construction itself is performed by Gemini-3-Flash-Preview (§3.3, §4.1). The paper should report at least one controlled run in which EventGraph construction uses a weaker or open model (or a non-LLM heuristic) so that the structural claim is not confounded by the same strong closed model writing both the graph and the answers. Without that, the model-agnostic claim is only partially supported for the reasoning agent (Table 3), not for the full pipeline.
  2. §5.4 and Figure 3: >75–80% of residual errors are attributed to Memory (absent) or Memory (coarse). This is a load-bearing limitation for the central claim that hierarchical memory enables reliable multi-hop reasoning. The manuscript should quantify EventGraph edge/segment fidelity (e.g., precision/recall of causal edges against a small human-annotated subset, or agreement under different update cadences) rather than only reporting downstream QA accuracy. Otherwise it remains unclear how much of the +Event Graph lift in Table 2 is genuine causal structure versus better coarse indexing.
minor comments (5)
  1. Table 2 vs. Table 1: Memory ablation is on a 600-question subset of M3-Bench-robot while main results use the full split; please state the subset sampling procedure and, if feasible, report the same incremental stack on the full robot split for direct comparability.
  2. §3.1 Eq. (1)–(2): τ_scene, w, n=3, and δ_min are free parameters; a short sensitivity note (or default values used for all experiments) would improve reproducibility.
  3. §5.2 / Table 3: Deepseek-v4-Flash and Qwen3-Coder-480B are text-only and never see keyframes; the text should explicitly note that KEYFRAME mode is disabled for those runs so readers do not over-interpret the Perceptual Buffer contribution under those backbones.
  4. Figure 1 and Algorithm 1 are clear; the harness case study (Table 4) is valuable but dense—consider moving the full multi-round transcript to the appendix and keeping a shorter schematic in the main text.
  5. Related work: concurrent PyraVid and MM-Mem are cited; a one-sentence contrast on agentic planning vs. automatic graph expansion would help position the contribution more sharply.

Circularity Check

0 steps flagged

No significant circularity: empirical system gains on public benchmarks with controlled ablations, not a derivation that reduces to its inputs by construction.

full rationale

HOMER is an empirical online video-understanding system paper. Its central claims are comparative accuracy lifts (+5.5 / +10.8 / +4.4) over the prior best agent (M3-Agent-Gemini3, sharing the same memorization pipeline and Gemini-3 backbone but lacking EventGraph + harness) on the public M3-Bench and Video-MME-Long benchmarks, plus consistent lifts across three LLM backbones (Table 1, Table 3). These are measured by external evaluation (LLM-as-judge or rule-based), not predicted from fitted parameters. Memory layers are constructed incrementally by off-the-shelf or previously published models (InsightFace, Qwen3-Omni, M3-Agent-Memorization, Gemini-3-Flash) and then ablated under controlled conditions (Table 2 isolates EntityGraph → +event nodes → +temporal-causal edges → +Perceptual Buffer). Self-evolution mines reusable skills from trajectories on the same distribution but validates them by downstream accuracy and automatically bans skills that degrade performance; this is ordinary online policy improvement, not a definitional loop that forces the reported gains. There are no self-definitional equations, no fitted quantity renamed as a prediction, no uniqueness theorem imported from overlapping authors, and no ansatz smuggled via self-citation that the central claim reduces to. Residual errors are openly attributed to memory construction quality (§5.4), which is shared with the baseline and does not circularly manufacture the comparative advantage. The paper is therefore self-contained against external benchmarks; circularity score is 0.

Axiom & Free-Parameter Ledger

6 free parameters · 3 axioms · 3 invented entities

The central empirical claim rests on a collection of engineering choices (thresholds, budgets, model choices) and on the cognitive-science-inspired premise that multi-scale memory with explicit causal links is the right inductive bias for online long-video reasoning. No free parameters are fitted to the evaluation metrics themselves; the listed free parameters are design knobs chosen a priori or by modest validation.

free parameters (6)
  • shot-boundary threshold τ_scene
    Fixed pixel-difference threshold used for hard-cut detection; value not reported as tuned on the test set but still a free design choice.
  • quality-score weight w and keyframe count n=3
    Controls sharpness-entropy trade-off and number of retained frames per 30 s clip.
  • EventGraph update cadence (every 10 clips) and max 40 clip_ids per segment
    Sliding-window patch size and segment-size cap that govern graph granularity.
  • agent budgets (≤5 subtasks, ≤3 attempts each, round budget 10, ≤5 keyframes)
    Hard resource limits that truncate search; chosen by authors.
  • self-evolution thresholds (batch 64, promote after ≥8 occurrences, route if similarity >0.45)
    Skill-library promotion and routing cut-offs that affect the reported self-evolution gains.
  • temperature 0.8 for Gemini-3-Flash reasoning agent
    Sampling temperature used at inference.
axioms (3)
  • domain assumption Human memory operates at perceptual, entity-centric episodic/semantic, and event levels linked by temporal and causal relations (Tulving, Zacks, Hahamy et al.).
    Explicitly invoked in §3 to justify the three-layer design.
  • ad hoc to paper Incremental LLM patching of an event graph from clip-level memories yields coherent long-range causal structure without future context.
    Core construction assumption of §3.3; never independently validated against human event annotations.
  • ad hoc to paper A verify-and-correct harness plus skill library can recover from retrieval failures and accumulate reusable policies without changing the underlying memory graphs.
    Stated as the second design principle and implemented in §3.4.
invented entities (3)
  • EventGraph with typed temporal-causal edges (before/overlap/enables/causes/...) no independent evidence
    purpose: Provide first-class multi-hop narrative structure for online retrieval.
    New graph schema relative to prior entity-only or proximity-based memories; independent evidence is only the downstream QA gains.
  • Harness engineering suite (Inform/Constrain/Verify/Correct + self-evolution skill library) no independent evidence
    purpose: Make multi-round agentic retrieval reliable and improvable across questions.
    Collection of guardrails and meta-learning rules introduced by the paper; success measured only on the same benchmarks.
  • Task ledger L with subtask status, evidence binding, and incremental replan no independent evidence
    purpose: Maintain persistent reasoning state across retrieval rounds.
    Internal bookkeeping structure required by the agent loop.

pith-pipeline@v1.1.0-grok45 · 23911 in / 2960 out tokens · 39952 ms · 2026-07-12T09:30:21.282978+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning." pith.science (2026). https://pith.science/paper/RPTN2IFB

@misc{pith2026260702588,
  author       = {Pith},
  title        = {Pith review of: Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RPTN2IFB}},
  note         = {Machine review of arXiv:2607.02588}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal large language models excel on short clips but struggle on hour-long videos in an online setting, where frames are processed incrementally under limited memory. Existing online methods either retain compact visual representations that lack semantic structure, or build higher-level memory stores organized around temporal proximity rather than explicit causal links, leaving multi-hop narrative reasoning to be reconstructed by the LLM at every query. We bridge this gap with \textsc{Homer}, a Hierarchical Online Memory Exploration and Reasoning framework. \textsc{Homer}'s memory mirrors the multi-scale structure of long videos, ranging from raw perception, to recurring entities, to events connected by explicit temporal and causal relations. Its agentic reasoner then explores this memory the way humans do, locating the relevant scene, looking up details, and composing the answer through multi-round memory retrieval, with a harness that verifies and corrects each step. \textsc{Homer} outperforms the previous best agent method by $+5.5$, $+10.8$, and $+4.4$ points on M3-Bench-robot, M3-Bench-web, and Video-MME-Long, and consistently lifts three various LLM backbones, indicating a model-agnostic structural capability for grounded retrieval over long videos.

Figures

Figures reproduced from arXiv: 2607.02588 by Bo Zhao, Fanghua Ye, Juntao Li, Liefeng Bo, Min Zhang, Yixin Ji, Zexuan Qiu, Zhaopeng Tu.

Figure 1
Figure 1. Figure 1: Overview of HOMER. Each incoming clip is processed online into a Perceptual Buffer of key frames, an Entity Graph of character-centric episodic and semantic memories, and an Event Graph of narrative segments linked by temporal and causal edges. At question time, an agentic reasoner decomposes the query and performs multi-turn retrievals at progressively finer granularity to produce the final answer. the on… view at source ↗
Figure 2
Figure 2. Figure 2: Self-evolution behaviour of HOMER on M3-Bench-robot, processed in batches of 64 questions. (a) Cumulative accuracy with and without self-evolution; (b) per-skill usage and downstream success rate, where green skills remain active and grey ones are automatically banned once their success rate falls below the running baseline. Memory (absent) Memory (coarse) Event localization Answer synthesis 0 10 20 30 40 … view at source ↗
Figure 3
Figure 3. Figure 3: Failure attribution by LLM-as-judge on M3- [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The full prompt for incremental Event Graph construction. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Task planning prompts [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Retrieval and synthesis prompts [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 4 linked inside Pith

  1. [1]

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Huichi Zhou, Qihui Zhang, Zhigang He, Yilin Bai, Chujie Gao, Liuyi Chen, et al

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631. Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Huichi Zhou, Qihui Zhang, Zhigang He, Yilin Bai, Chujie Gao, Liuyi Chen, et al. 2025a. Gui-world: A video benchmark and dataset for multimodal gui-oriented understanding. InIn- ternational Conference on Learning Represen- tations, volume 2025, page...

    Pith/arXiv arXiv 2025

  2. [2]

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al

    Cl-bench life: Can language models learn from real-life context?arXiv preprint arXiv:2604.27043. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2025. Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of th...

    Pith/arXiv arXiv 2025

  3. [3]

    InThe Fourteenth International Con- ference on Learning Representations

    Seeing, listening, remembering, and rea- soning: A multimodal agent with long-term memory. InThe Fourteenth International Con- ference on Learning Representations. Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai

  4. [4]

    thinking with long videos

    Drvideo: Document retrieval based long video understanding. InProceedings of the Computer Vision and Pattern Recognition Con- ference, pages 18936–18946. Jonas Pai, Liam Achenbach, Victoriano Mon- tesinos, Benedek Forrai, Oier Mees, and Elvis Nava. 2025. mimic-video: Video-action mod- els for generalizable robot control beyond vlas. arXiv preprint arXiv:2...

    Pith/arXiv arXiv 2025

  5. [5]

    Tatiana Zemskova, Solomon Andryushenko, Ilya Obrubov, Viktoriia Khoruzhaia, Ekate- rina Eroshenko, Ekaterina Derevyanka, and Dmitry Yudin

    Event perception: a mind-brain perspec- tive.Psychological bulletin, 133(2):273. Tatiana Zemskova, Solomon Andryushenko, Ilya Obrubov, Viktoriia Khoruzhaia, Ekate- rina Eroshenko, Ekaterina Derevyanka, and Dmitry Yudin. 2026. Focusgraph: Graph- structured frame selection for embodied long video question answering.arXiv preprint arXiv:2603.04349. Xiangyu Z...

    arXiv 2026

  6. [6]

    Heqing Zou, Tianze Luo, Guiyang Xie, Fengmao Lv, Guangcong Wang, Junyang Chen, Zhuochen Wang, Hansheng Zhang, Huaijian Zhang, et al

    Focus: Efficient keyframe selection for long video understanding.arXiv preprint arXiv:2510.27280. Heqing Zou, Tianze Luo, Guiyang Xie, Fengmao Lv, Guangcong Wang, Junyang Chen, Zhuochen Wang, Hansheng Zhang, Huaijian Zhang, et al

    arXiv

  7. [7]

    The last search added limited new information. Consider whether you have enough to answer

    From seconds to hours: Reviewing mul- timodal large language models on comprehen- sive long video understanding.arXiv preprint arXiv:2409.18938. Algorithm 1Agentic Reasoning over Hierarchical Memory Input:questionq, answer choicesA, hierarchical memoryM, round budgetR Output:final answery Prepare memory stateM q for the current video. Route relevant skill...

    Pith/arXiv arXiv

  8. [8]

    Each sub-question must be ATOMIC — ask ONE fact, one entity, one relation, one event

  9. [9]

    Use stable idst1, t2, t3, ...in generation order

  10. [10]

    If a later sub-question depends on the answer of an earlier one, declare this viadepends_on

  11. [11]

    Which/What

    Sub-questions should collectively be sufficient to answer the ORIGINAL question — no gaps, no overlap. QUESTION-TYPE-SPECIFIC RULES A) YES/NO or TRUE/FALSE: Decompose into 2–3 subtasks (supporting evidence, contradicting evidence, optional temporal scope). B) “Which/What” MCQ with multiple options: One subtask per option that needs verification, or subtas...

  12. [12]

    Do NOT re-add or rewrite them

    Preserve already-resolved subtasks verbatim. Do NOT re-add or rewrite them

  13. [13]

    If the plan looks fine and we are just querying badly, prefer empty edits

    Only modify the plan when evidence has revealed the original assumption is wrong or unreachable. If the plan looks fine and we are just querying badly, prefer empty edits

  14. [14]

    Prefer minimal edits: abandon a sub-branch, merge redundant siblings, or rewrite one subtask’s text

  15. [15]

    update_subtasks

    Any new subtask id MUST NOT collide with existing ones. Uset{N+1}style. 5.update_subtasks[ *].new_questionshould be DECLARATIVE (matchable to an event summary). OUTPUT SCHEMA {"update_subtasks": [{"id": "t2", "new_question": "...", "reason": "..."}], "abandon_subtasks": [{"id": "t3", "reason": "..."}], "new_subtasks": [{"id": "t5", "question": "...", "dep...