Agentic Very Long Video Understanding

Aniket Rege; Arka Sadhu; Hyo Jin Kim; Kejie Li; Ramya Korlakai Vinayak; Yong Jae Lee; Yuliang Li; Yuning Chai

REVIEW 3 major objections 5 minor 6 cited by

The paper seeks to establish that a planning agent with a time-stamped entity scene graph, queried through SQL alongside visual and audio search, enables compositional multi-hop reasoning over week-long egocentric video—beating uniform-samp

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 08:02 UTC pith:4FYAHYDJ

load-bearing objection A genuinely useful agentic framework for week-scale egocentric video, with strong EgoLifeQA gains and honest ablations; the main caveat is that one central ablation conflates the entity graph with the extra fused captions, though the graph is still very likely doing real work. the 3 major comments →

arxiv 2601.18157 v3 pith:4FYAHYDJ submitted 2026-01-26 cs.CV cs.LG

Agentic Very Long Video Understanding

Aniket Rege , Arka Sadhu , Yuliang Li , Kejie Li , Ramya Korlakai Vinayak , Yuning Chai , Yong Jae Lee , Hyo Jin Kim This is my paper

classification cs.CV cs.LG

keywords very long video understandingentity scene graphegocentric videoagentic retrievalmulti-hop reasoningtemporal localizationEgoLifeQAlong-context video question answering

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that very long video understanding, especially week-long egocentric footage from always-on wearable cameras, requires an explicit time-aware entity graph rather than just larger context windows or better frame retrieval. It proposes EGAgent, an agent that decomposes each query into sub-tasks, searches three sources—visual frames, transcripts, and the entity graph—and accumulates evidence before producing an answer. The entity graph is built by an LLM from fused visual captions and speaker-labeled transcripts, with nodes for people, places, and objects and edges for four relation types, each annotated with start and end times. On EgoLifeQA, EGAgent reaches 57.5% multiple-choice accuracy, up from 36.9% for the previous best EgoButler and 46.8% for uniform-sampling Gemini 2.5 Pro, with the largest gains on RelationMap and TaskMaster categories that require multi-hop relational reasoning. A sympathetic reader would care because this is the operating regime of future personal assistants: persistent, cross-day memory over continuous video from a first-person perspective.

Core claim

The paper's central claim is that compositional, multi-hop questions about week-long video—"Who was with us the last time we took the car?" or "Who usually speaks least in group discussions?"—cannot be answered by flat retrieval over frames and captions, because entity identities and relationships dissolve across days. EGAgent therefore extracts an entity scene graph from fused visual captions and diarized transcripts, with nodes typed as person, object, or location, and edges constrained to four relation types (talks-to, interacts-with, mentions, uses), each edge carrying a start/end time and the supporting snippet. At inference, a planning agent generates sub-tasks, issues strict-to-relaxe

What carries the argument

The load-bearing object is the time-annotated entity scene graph G=(V,E). Nodes are entities typed as person, object, or location; edges are relations drawn from a fixed four-type schema (talks-to, interacts-with, mentions, uses), each stored as a tuple (source, target, relation, start time, end time, supporting text snippet) in a SQLite table. The graph is built incrementally by an LLM extractor over 30-second fused captions and transcripts, and it preserves identity and temporal ordering across days—information that flat retrieval loses. The second mechanism is the planner's strict-to-relaxed SQL query strategy, which broadens time windows, entity names, and relation types in stages, so no

Load-bearing premise

The whole system leans on the LLM that builds the entity graph: if it misses, mislabels, or mis-times relationships between people, places, and objects, the multi-hop reasoning that drives the EgoLifeQA gains has nothing to reason over—and the authors audit only 100 of 13,968 extracted relationships (94% accurate) while depending on manually diarized transcripts.

What would settle it

Run EGAgent on EgoLifeQA with the entity-graph tool disabled but visual and transcript search intact; if accuracy stays near 57.5% instead of falling to roughly the 48.6% reported for the no-graph variant, the graph is not doing the causal work. Alternatively, replace EgoLife's manual diarization with an off-the-shelf diarizer and observe whether RelationMap and TaskMaster scores drop sharply, or enlarge the manual audit of graph edges to all 13,968 and see whether the 94% accuracy rate survives.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Entity graph reasoning lifts the multi-hop relational categories by roughly 20 percentage points over prior state of the art and by 17–21 points over a strong uniform-sampling baseline, suggesting structured representations matter more than raw context-window size.
Combined visual, audio, and graph search achieves 0.884 recall at a 10-second temporal window on EgoLifeQA, showing that the agent's overall temporal localization is strong, not just its retrieval of content.
The entity graph adds only about 12.8% inference overhead while occupying about 2 MB of storage for 52 hours of video, making it a lightweight persistent index for always-on assistants.
On Video-MME (Long), EGAgent matches AdaVideoRAG with over 10x fewer frames, indicating that graph-based retrieval is efficient even in the 30–60-minute regime.
Oracle experiments with perfect temporal localization saturate at 68.7% accuracy with GPT-4.1 and 57.9% with transcript-only search, implying that retrieval is not the only bottleneck—multimodal reasoning over retrieved evidence also limits performance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the entity graph is intended as persistent memory for wearable assistants, one testable extension is to log query patterns over graph edges and measure whether repeated personal questions can be answered without re-encoding raw video frames.
The paper's fixed four-relation schema trades granularity for robustness; an editorially suggested stress test is to re-tune the prompt schema for other domains (offices, kitchens, hospitals) and measure whether a finer relation taxonomy improves or fragments retrieval recall.
The reliance on manually diarized transcripts suggests a concrete external test: substitute off-the-shelf speaker diarization and watch whether RelationMap and TaskMaster scores collapse—if they do, the bottleneck shifts from reasoning to perception.
Because each graph edge carries a supporting text snippet, the system could be extended to provide evidence citations for its answers, enabling users to verify longitudinal claims by jumping directly to the relevant moment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely useful agentic framework for week-scale egocentric video, with strong EgoLifeQA gains and honest ablations; the main caveat is that one central ablation conflates the entity graph with the extra fused captions, though the graph is still very likely doing real work.

read the letter

You should know about this paper if you care about very-long egocentric video QA. EGAgent builds a temporally-annotated entity scene graph, exposes it to a planning agent through SQL search with a strict-to-relaxed relaxation strategy, and combines that with visual and transcript tools. On EgoLifeQA it gets 57.5% with a Gemini 2.5 Pro backbone, a 20.6-point jump over the prior SOTA, and the per-category gains are largest exactly where the paper says they should be: RelationMap and TaskMaster. That is a real, practical result.

What is genuinely new: the combination of time-stamped edges on an entity graph, SQL-based agentic querying, and a relaxation hierarchy that recovers from noisy extraction. The paper also does more homework than most: oracle search experiments, retrieval recall at multiple temporal windows, per-tool ablations, latency/memory tables, and public code. The limitations section explicitly flags dependence on upstream perception and manual diarization, which is honest.

Soft spots, in proportion. First, all numbers come from a single run, with no variance estimates; the paper says this in Appendix E, and for agentic systems it is understandable, but it still limits confidence in the 2-3 point margins. Second, the entity graph audit covers only 100 of 13,968 relations, so the 94% accuracy claim is a weak reed. Third — the point that matters most — Table 5's comparison of EGAgent (F+T) vs (EG+F+T) adds two things at once: the graph search tool and the fused captions used to build it. The RelationMap jump from 40.0 to 53.6 is therefore not cleanly attributable to graph structure alone. That said, Table 3 shows that switching the graph-extraction input from transcripts to fused captions only buys about 1.5 points for GPT-4.1, while the RelationMap gain in Table 5 is about 13.6 points. So the graph itself is almost certainly carrying the load, but a control condition (F+T+C without the graph) would settle it. The relation schema being tuned on EgoLife-like data is a lesser concern; the authors hand-wave that it is prompt-defined and domain-agnostic, which is plausible but untested.

Bottom line: this is a solid, well-engineered systems paper with a useful new architecture and evidence that is mostly on point. It deserves a serious referee. I would ask the authors to add a clean control, report variance or at least multiple runs, and expand the graph audit sample. If those land, this is a clear accept.

Referee Report

3 major / 5 minor

Summary. The paper proposes EGAgent, an agentic framework for question answering over very long (week-scale) egocentric video. The system first builds a temporally annotated entity scene graph from fused visual captions and audio transcripts, then uses a planning agent to decompose a query into subtasks and to call three retrieval tools: visual search over frame embeddings, audio transcript search, and SQL-style entity graph search. Retrieved evidence is distilled by an analyzer into a working memory that a VQA agent uses to produce the final answer. On EgoLifeQA, EGAgent with a Gemini 2.5 Pro backbone achieves 57.5% MCQ accuracy, a 20.6-point gain over EgoButler (Gemini 1.5 Pro) and 10.7 points over uniform-sampling Gemini 2.5 Pro; on Video-MME (Long) it reports 74.1%, behind native Gemini 2.5 Pro (82.0%) but ahead of other agentic baselines. The paper attributes the EgoLifeQA gains primarily to entity graph reasoning, especially on RelationMap and TaskMaster categories.

Significance. If the central claim is established, the paper makes a useful empirical contribution: it demonstrates that a structured, temporally localized entity graph can support multi-hop, cross-modal reasoning over week-long egocentric video, and it provides an extensive set of ablations, oracle upper bounds, recall analysis, and latency/memory measurements. The public code release, clear method description, and detailed appendices are strengths. However, the headline attribution of the gains to entity graph structure is currently under-supported because the graph is confounded with the addition of caption-derived content, and the reported numbers come from a single run without variance estimates. These issues are fixable with additional control experiments and uncertainty reporting, so the work is promising but not yet ready for acceptance.

major comments (3)

[Section 4.3 / Table 5 / Appendix E] The central claim that entity graph reasoning drives the RelationMap and TaskMaster gains is confounded. Table 5 compares EGAgent GPT-4.1 (F+T) with EGAgent GPT-4.1 (EG+F+T): in the latter condition the graph is extracted from fused captions and transcripts (C+T), so the comparison adds both the entity-graph search tool and the caption-derived content C. The same conflation appears in the headline Table 1 comparison (Gemini 2.5 Pro uniform F,T vs EGAgent Gemini EG+F+T). There is no control condition (F+T+C without the graph) that isolates the graph structure from the additional fused-caption information. Please add such a control — e.g., let the agent access the fused captions via a caption search tool or include them in the transcript tool — to demonstrate that the gains come from the entity graph itself rather than from the extra information content.
[Appendix E (Implementation Details)] All accuracies are reported from a single run, as acknowledged in Appendix E. For a benchmark paper whose main result is a 20.6-point SOTA gap over a strong baseline, point estimates with no variance are not sufficient, especially because LLM-based agent pipelines can have run-to-run variability despite temperature-0 settings. Please report at least 3 runs for the main conditions in Tables 1 and 5 (or provide bootstrap confidence intervals over the 500 EgoLifeQA questions), and state the random seed or API version used. This is important for assessing whether the reported gains are statistically reliable.
[Appendix D.5 / Section 6] The entity graph quality audit covers only 100 of the 13,968 extracted relationships (94% accuracy). Since the paper explicitly acknowledges that graph construction depends on upstream perception and language models, a 100-edge sample gives a wide confidence interval and does not characterize failures across relation types or temporal intervals. Please either enlarge the audit, report a confidence interval, or provide a breakdown of error types (e.g., wrong entity, wrong relation, wrong timestamp). This is secondary to the confound in the ablation, but it matters for the claimed robustness of the graph-based mechanism.

minor comments (5)

[Tables 1 and 5] The column labeled 'A verage' appears to be an overall accuracy that is not the simple mean of the five category accuracies. For example, EGAgent GPT-4.1 (F+T) reports 48.6 while the five categories listed average to about 50.8; EGAgent GPT-4.1 (EG+F+T) reports 50.7 while the categories average to about 53.8. Please clarify whether 'average' is a weighted overall accuracy, and if so, report per-category question counts. This will let readers verify the table.
[Section 3.2, Equation (3)] The relation types are written with hyphens in Equation (3) (talks-to, interacts-with) but as uppercase underscores elsewhere (TALKS_TO, INTERACTS_WITH). Please make the notation consistent.
[Table 2 caption] The caption says the current state-of-the-art is bolded and the second-highest is underlined, but the formatting is not visible in the text. If the camera-ready version uses actual bolding/underlining, this is fine; otherwise please state the ranking explicitly (e.g., Gemini 2.5 Pro is first, EGAgent Gemini 2.5 Pro is second).
[Figure 3 caption] There is an apparent stray annotation in the caption ('D: se other diagram to reflect that this is a set of doc') that looks like an editing note rather than intended text. Please remove or correct it.
[Appendix D.1, final paragraph] The text says 'when we add the entity graph search tool T oolaud', but T oolaud is the audio transcript tool; the sentence should refer to the entity graph tool T ooleg. Please fix the notation.

Circularity Check

0 steps flagged

No circularity: benchmark numbers are external and entity-graph construction is not fitted to test answers.

full rationale

The paper contains no derivation that reduces to its own inputs. Entity graph construction (Eqs. 1-5) applies an LLM extractor F to documents D (audio transcripts plus scene captions) and stores temporally annotated relationship tuples; no parameter is fit to EgoLifeQA or Video-MME answer labels. The reported accuracies (e.g., Table 1: EGAgent Gemini 2.5 Pro 57.5% vs uniform-sampling Gemini 2.5 Pro 46.8%) are external benchmark results, not quantities implied by the method's definitions. The closest issue is an ablation confound: in Tables 1 and 5, the EG+F+T condition also adds fused captions C compared to the F+T control, so the incremental gain attributed to the entity graph is not cleanly isolated from the added caption content. This is an experimental design weakness, not circularity: the graph is still a precomputed representation of the same external video/audio inputs, and no predicted accuracy equals an input by construction. The paper also flags its dependence on upstream perception/language models (Section 6) and manually audits only 100/13,968 graph edges (Appendix D.5), which are evidence-quality limitations. There are no load-bearing self-citations; related graph-based methods and baselines are independent prior work.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No numerical free parameters are fitted to the test benchmarks, but the system has hand-chosen hyperparameters (k=50, 30s captions, max 5 subtasks) and a dataset-derived relation schema. The only invented abstraction is the entity-scene-graph representation, which is a data structure rather than a physical entity.

free parameters (3)

k_nearest_visual_frames = 50
Number of frames returned by visual search and later analyzed; a fixed context/token budget chosen by hand, not derived from data.
visual_caption_interval = 30 seconds
GPT-4.1 generates captions for each 30-second snippet; this granularity affects graph extraction and temporal annotation but is not fitted.
max_subtasks = 5
The planner prompt caps sub-tasks at five, which impacts retrieval coverage and latency; a hand-chosen budget.

axioms (4)

domain assumption LLM-based extractor F (Eq. 1) produces sufficiently accurate and complete temporal relations from fused captions/transcripts.
The entity graph is the core new data structure; errors propagate to all downstream reasoning. Only 100 of 13,968 relations were manually audited (94%).
domain assumption EgoLifeQA and Video-MME (Long) MCQ accuracy is a valid measure of very long video understanding.
All claims are benchmark numbers; no independent task validation is provided.
domain assumption Manual diarized transcripts for EgoLife are available and accurate.
The authors note that using off-the-shelf diarization would likely degrade performance (Section 6).
ad hoc to paper Four relation types R={TALKS_TO, INTERACTS_WITH, MENTIONS, USES} are sufficient for cross-domain adaptation.
The schema was chosen because free-form relations on EgoLife collapsed into these buckets; the authors claim domain-agnostic but provide no transfer test.

pith-pipeline@v1.3.0-alltime-deepseek · 28867 in / 11546 out tokens · 121821 ms · 2026-08-03T08:02:57.793811+00:00 · methodology

0 comments

read the original abstract

The advent of always-on personal AI assistants, enabled by all-day wearable devices such as smart glasses, demands a new level of contextual understanding, one that goes beyond short, isolated events to encompass the continuous, longitudinal stream of egocentric video. Achieving this vision requires advances in long-horizon video understanding, where systems must interpret and recall visual and audio information spanning days or even weeks. Existing methods, including large language models and retrieval-augmented generation, are constrained by limited context windows and lack the ability to perform compositional, multi-hop reasoning over very long video streams. In this work, we address these challenges through EGAgent, an enhanced agentic framework centered on entity scene graphs, which represent people, places, objects, and their relationships over time. Our system equips a planning agent with tools for structured search and reasoning over these graphs, as well as hybrid visual and audio search capabilities, enabling detailed, cross-modal, and temporally coherent reasoning. Experiments on the EgoLifeQA and Video-MME (Long) datasets show that our method achieves state-of-the-art performance on EgoLifeQA (57.5%) and competitive performance on Video-MME (Long) (74.1%) for complex longitudinal video understanding tasks. Code is available at https://github.com/facebookresearch/egagent.

discussion (0)

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Grounding
cs.CV 2026-05 unverdicted novelty 7.0

ReTool-Video uses a 134-tool meta-augmented library and recursive grounding to translate abstract video intents into fine-grained multimodal operations, outperforming baselines on MVBench, MLVU, and Video-MME.
ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory
cs.AI 2026-07 conditional novelty 6.0

A robotic agent operating system with source-grounded graph memory and split-wise self-evolution improves long-horizon embodied task success and memory QA scores over baseline controllers.
Rethinking RAG in Long Videos: What to Retrieve and How to Use It?
cs.AI 2026-06 unverdicted novelty 6.0

Introduces V-RAGBench benchmark and CARVE method that selects per-chunk retrieval configurations via parallel retrievers and adaptive reranking, outperforming eight VideoRAG baselines.
Personal Visual Context Learning in Large Multimodal Models
cs.CV 2026-05 unverdicted novelty 6.0

Introduces Personal VCL formalization and benchmark revealing LMM context gaps, plus an Agentic Context Bank baseline that boosts personalized visual reasoning.
ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory
cs.AI 2026-07 conditional novelty 5.5

A hierarchical robotic Agent OS with source-grounded multi-modal graph memory and split-gated self-evolution improves long-horizon embodied execution and memory QA over single-controller and prior memory baselines.
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
cs.CV 2026-06 unverdicted novelty 4.0

This is a survey that frames video MLLM research via a human-view formulation of perceptual representations, memory states, reasoning traces, and predictions, then reviews methods, datasets, benchmarks, and open problems.

Reference graph

Works this paper leans on

22 extracted references · cited by 5 Pith papers

[1]

a list of relationships: each relationship has relationship_id, source_id node, target_id node, and relationship type
[2]

a caption containing dialogue: the caption contains a timestamp (start_t → end_t) and visual information from the scene as well as information on spoken dialogue
[3]

For every single provided relationship, find all transcripts and captions that support it

a list of transcripts [t1, t2, ...], each containing a ‘timestamp’ (start_t → end_t) and ‘text’ containing spoken dialogue. For every single provided relationship, find all transcripts and captions that support it. Relationships : {relationships} Captions: {captions} Transcripts: {transcripts} Rules: - First, try to use only timestamps already present in ...
[4]

Visual Caption - a detailed description of what is visible in the video
[5]

Your task is to fuse these into a single, coherent, and natural paragraph that integrates both visual and spoken information

Diarized Transcript - spoken dialogue transcribed with timestamps and speaker names. Your task is to fuse these into a single, coherent, and natural paragraph that integrates both visual and spoken information. Follow these rules carefully: * Focus on relevant spoken content (who says what and its meaning) and highlight visual content (location, people, a...
[6]

Visual Caption: {caption_text}
[7]

Diarized Transcript: {transcript_text} Produce one fused caption that naturally combines both.””” 23 System Prompt for Entity Graph Extraction: “““ Knowledge Graph Instructions
[8]

Try to capture as much information from the text as possible without sacrificing accuracy

Overview You are a top-tier algorithm designed for extracting information in structured formats to build a knowledge graph. Try to capture as much information from the text as possible without sacrificing accuracy. Do not add any information that is not explicitly mentioned in the text. - Nodes represent entities and concepts. - The aim is to achieve simp...
[9]

Ensure you use basic or elementary types for node labels

Labeling Nodes - Consistency: Ensure you use available types for node labels. Ensure you use basic or elementary types for node labels. - For example, when you identify an entity representing a person, always label it as ’person’. Avoid using more specific terms like ’mathematician’ or ’scientist’. - Node IDs: Never utilize integers as node IDs. Node IDs ...
[10]

Coreference Resolution - Maintain Entity Consistency: When extracting entities, it’s vital to ensure consistency. If an entity, such as John Doe, is mentioned multiple times in the text but is referred to by different names or pronouns (e.g., Joe, he), always use the most complete identifier for that entity throughout the knowledge graph. In this example,...
[11]

Person”, “Location

Strict Compliance Adhere to the rules strictly. Non-compliance will result in termination. """ User Prompt for Entity Graph Extraction: “““ Based on the following example, extract entities and relations from the provided text. Use the following entity types, don’t use other entity that is not defined below: ENTITY TYPES: {allowed_nodes} Use the following ...
[12]

Your goal is to find relevant rows describing relationships between entities
[13]

You must construct SQL queries progressively, starting with the strictest filter and relaxing step by step if no results are found
[14]

The order of relaxation is: (a) Strict: exact day, exact timestamp (start_t ≥x and end_t ≤y), exact source_id, exact target_id, exact rel_type

Each stage should keep only the necessary filters. The order of relaxation is: (a) Strict: exact day, exact timestamp (start_t ≥x and end_t ≤y), exact source_id, exact target_id, exact rel_type. (b) Relax time: same day, exact source_id/target_id, same rel_type. (c) Relax day: all days, exact source_id/target_id, same rel_type. Day has to be≤to the query ...
[15]

Always return your reasoning, and the SQL for each step, in a structured format
[16]

Do not hallucinate entity names; use = or LIKE matching only to suggest similar candidates
[17]

Do not use SELECT transcript or any other specific element of the schema

Always use SELECT * FROM entity_graph_table WHERE ... Do not use SELECT transcript or any other specific element of the schema
[18]

Do not search the transcript unless you have exhausted other options
[19]

reasoning

Keep relaxing until the last SQL query has ONLY target_id (and optionally transcript). ””” 25 Entity Graph Search User Prompt: “““ User question: {question} asked at {query_time}, and relevant context gathered thus far: ‘{working_- memory}’. Your job is to create a SQL query to answer this specific goal given by an expert planner: ‘{current_task}’. Return...
[20]

You may search any day between the start_t and end_t of that day
[21]

Select a set of 1 to 3 concise text queries e.g., [’q1’] or [’q1’, ’q2’, ’q3’] for each day you would like to search, and optionally when during that day to search (between start_t and end_t)
[22]

me” or “I

If you only need one text query, only use one text query. Only use additional text queries if they are semantically distinct from one another.””” 26 Audio Transcript Search System Prompt: “““You are a helpful assistant who analyzes how retrieved transcripts are relevant to answer a multiple-choice question about a long video. You are given a single step o...

[1] [1]

a list of relationships: each relationship has relationship_id, source_id node, target_id node, and relationship type

[2] [2]

a caption containing dialogue: the caption contains a timestamp (start_t → end_t) and visual information from the scene as well as information on spoken dialogue

[3] [3]

For every single provided relationship, find all transcripts and captions that support it

a list of transcripts [t1, t2, ...], each containing a ‘timestamp’ (start_t → end_t) and ‘text’ containing spoken dialogue. For every single provided relationship, find all transcripts and captions that support it. Relationships : {relationships} Captions: {captions} Transcripts: {transcripts} Rules: - First, try to use only timestamps already present in ...

[4] [4]

Visual Caption - a detailed description of what is visible in the video

[5] [5]

Your task is to fuse these into a single, coherent, and natural paragraph that integrates both visual and spoken information

Diarized Transcript - spoken dialogue transcribed with timestamps and speaker names. Your task is to fuse these into a single, coherent, and natural paragraph that integrates both visual and spoken information. Follow these rules carefully: * Focus on relevant spoken content (who says what and its meaning) and highlight visual content (location, people, a...

[6] [6]

Visual Caption: {caption_text}

[7] [7]

Diarized Transcript: {transcript_text} Produce one fused caption that naturally combines both.””” 23 System Prompt for Entity Graph Extraction: “““ Knowledge Graph Instructions

[8] [8]

Try to capture as much information from the text as possible without sacrificing accuracy

Overview You are a top-tier algorithm designed for extracting information in structured formats to build a knowledge graph. Try to capture as much information from the text as possible without sacrificing accuracy. Do not add any information that is not explicitly mentioned in the text. - Nodes represent entities and concepts. - The aim is to achieve simp...

[9] [9]

Ensure you use basic or elementary types for node labels

Labeling Nodes - Consistency: Ensure you use available types for node labels. Ensure you use basic or elementary types for node labels. - For example, when you identify an entity representing a person, always label it as ’person’. Avoid using more specific terms like ’mathematician’ or ’scientist’. - Node IDs: Never utilize integers as node IDs. Node IDs ...

[10] [10]

Coreference Resolution - Maintain Entity Consistency: When extracting entities, it’s vital to ensure consistency. If an entity, such as John Doe, is mentioned multiple times in the text but is referred to by different names or pronouns (e.g., Joe, he), always use the most complete identifier for that entity throughout the knowledge graph. In this example,...

[11] [11]

Person”, “Location

Strict Compliance Adhere to the rules strictly. Non-compliance will result in termination. """ User Prompt for Entity Graph Extraction: “““ Based on the following example, extract entities and relations from the provided text. Use the following entity types, don’t use other entity that is not defined below: ENTITY TYPES: {allowed_nodes} Use the following ...

[12] [12]

Your goal is to find relevant rows describing relationships between entities

[13] [13]

You must construct SQL queries progressively, starting with the strictest filter and relaxing step by step if no results are found

[14] [14]

The order of relaxation is: (a) Strict: exact day, exact timestamp (start_t ≥x and end_t ≤y), exact source_id, exact target_id, exact rel_type

Each stage should keep only the necessary filters. The order of relaxation is: (a) Strict: exact day, exact timestamp (start_t ≥x and end_t ≤y), exact source_id, exact target_id, exact rel_type. (b) Relax time: same day, exact source_id/target_id, same rel_type. (c) Relax day: all days, exact source_id/target_id, same rel_type. Day has to be≤to the query ...

[15] [15]

Always return your reasoning, and the SQL for each step, in a structured format

[16] [16]

Do not hallucinate entity names; use = or LIKE matching only to suggest similar candidates

[17] [17]

Do not use SELECT transcript or any other specific element of the schema

Always use SELECT * FROM entity_graph_table WHERE ... Do not use SELECT transcript or any other specific element of the schema

[18] [18]

Do not search the transcript unless you have exhausted other options

[19] [19]

reasoning

Keep relaxing until the last SQL query has ONLY target_id (and optionally transcript). ””” 25 Entity Graph Search User Prompt: “““ User question: {question} asked at {query_time}, and relevant context gathered thus far: ‘{working_- memory}’. Your job is to create a SQL query to answer this specific goal given by an expert planner: ‘{current_task}’. Return...

[20] [20]

You may search any day between the start_t and end_t of that day

[21] [21]

Select a set of 1 to 3 concise text queries e.g., [’q1’] or [’q1’, ’q2’, ’q3’] for each day you would like to search, and optionally when during that day to search (between start_t and end_t)

[22] [22]

me” or “I

If you only need one text query, only use one text query. Only use additional text queries if they are semantically distinct from one another.””” 26 Audio Transcript Search System Prompt: “““You are a helpful assistant who analyzes how retrieved transcripts are relevant to answer a multiple-choice question about a long video. You are given a single step o...