Pith. sign in

REVIEW 4 major objections 5 minor 40 references

GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenes

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GESTO claims that a robot can answer retrospective questions about human activities by building its own two-level activity hierarchy over a persistent 4D scene graph, so that it matches methods supplied with human-annotated event and…

desk verdict Genuinely new representation idea with solid ablations, but the evaluation is under-specified and the headline numbers are within the noise floor—worth peer review, not worth believing yet. read the letter →

arxiv 2608.10886 v1 pith:RYDDS26R submitted 2026-08-11 cs.CV cs.RO

Ermanno Bartoli , Buwei He , Dennis Rotondi , Sebastian Koch , Federico Tombari , Kai O. Arras , Patric Jensfelt , Yixi Cai
show 1 more author
Iolanda Leite
This is my paper · ORCID
classification cs.CVcs.RO
keywords 4Dscenegraphsactivityhierarchyhuman-objectinteractionspatio-temporalmemorygroundedreasoningrobotquestionansweringdynamicscenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GESTO claims that a robot can keep a useful memory of what people did and with which objects, without any human-supplied event boundaries or object correspondences, by constructing the activity structure itself from an RGB-D observation stream. The proposed representation couples a persistent 4D scene graph, a record of objects and places over time, with a two-level activity hierarchy in which timestamped atomic human-object interactions are grouped under goal-driven events such as 'prepares coffee,' and both levels are grounded to persistent scene entities. On the text, binary, and time query categories of the benchmark, GESTO reports scores of 0.71, 0.75, and 0.70, close to a method given ground-truth event and object grounding and well above the same reasoning framework when those inputs are removed, with 0.73 and 0.75 on new Space2Event and Event2Space query sets. If these results hold, a grounded activity hierarchy built automatically would give robots a practical memory for answering retrospective questions about human behavior in dynamic scenes.

What carries the argument

The central object is the coupled pair $\mathcal{G}^+ = (\mathcal{G}_S, \mathcal{G}_A, L)$: the spatial layer $\mathcal{G}_S$ organizes persistent object and place nodes as in existing 4D scene graphs, while the activity layer $\mathcal{G}_A$ organizes timestamped atomic human-object interactions $I$ under goal-driven events $E$, and the grounding links $L$ attach each interaction to the object node it acts on. This linkage is what lets one query move from an object to the events it participated in, or from an event to the objects and places that support it. The construction pipeline carries the argument: atomic interactions are extracted from short clips by a vision-language model, grounded by mean intersection-over-union between interaction boxes and tracked object boxes plus text-embedding similarity between labels and object descriptions, grouped into events by an LLM, and then refined through one alternating pass $L^{(1)} = R_L(L^{(0)}; E^{(0)})$, $E^{(1)} = R_E(E^{(0)}; L^{(1)})$ in which event context infers links for unlinked interactions and corrected links re-check event membership. A relation-aware tool-calling agent exposes these structures through event, interaction, object, place, and time-interval tools.

What would settle it

A decisive check would be re-scoring GESTO and the fully automatic EGG baseline with an independent LLM judge plus a human-rated sample: if the text gap from 0.71 versus 0.33 collapses or reverses, the advantage is an artifact of grading, not of the activity hierarchy.

Watch

Extended reading notes

Core claim

GESTO's central discovery is that the missing ingredient for activity-centric robot memory is not more perception or larger language models, but a structured coupling between persistent spatial records and a two-level activity graph. The representation $\mathcal{G}^+ = (\mathcal{G}_S, \mathcal{G}_A, L)$ pairs a spatio-temporal scene graph $\mathcal{G}_S$ of objects and places with an activity graph $\mathcal{G}_A$ in which atomic interactions $I$ are grouped into goal-driven events $E$, connected by grounding links $L$ that attach each interaction to the persistent object it acts on. The pipeline constructs everything automatically: a vision-language model reads short clips to extract timestamped interactions, geometric and semantic matching grounds them to tracked objects, an LLM groups them into coherent events, and one alternating refinement pass uses event context to infer links for ungrounded interactions and re-evaluate event membership. The result, as the paper reports, is performance on text, binary, and time queries that approaches a method provided with externally annotated events and object associations, and ablation evidence that the event level itself, not just the grounded interactions, is what carries temporal and activity-space reasoning.

Load-bearing premise

The load-bearing premise is that the evaluation protocol—an LLM-based semantic score for text answers plus a two-minute tolerance for time answers—is a fair, stable measure of reasoning ability, because if that judge favors GESTO's verbose evidence-rich responses or is noisy at the current query counts, the reported margins over the fully automatic baseline could shrink or disappear.

Editorial extensions

If this is right

  • A robot equipped with GESTO could answer queries such as 'Which mug was used to drink coffee this morning?' from its own recordings, with no human annotation of event boundaries or object correspondences.
  • Removing the event hierarchy drops time accuracy from 0.70 to 0.40 and Space2Event from 0.73 to 0.53, which indicates that the goal-level organization, not the stored interactions alone, is what makes temporal and activity-space queries answerable.
  • Context-aware grounding refinement recovers interactions that initially lack a geometric match; without it, roughly 30% of interactions stay ungrounded and time accuracy falls from 0.70 to 0.59.
  • Because the pipeline treats the 4D scene graph as a swappable backbone and needs only a prompt change for egocentric video, the same memory construction can in principle be reused across robot-mounted and wearable camera configurations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single alternating refinement pass is a deliberate stopping point; iterating link and event refinement to a fixed point could plausibly improve consistency, so the reported gains may be a lower bound on what the hierarchy can support.
  • The paper notes the vision-language model is weaker at localizing the handled object than at recognizing the action, an appearance-based limitation; adding affordance or hand-contact cues to the grounding score is a concrete way to test whether the time-accuracy gap to ground-truth-grounded EGG can be closed.
  • Since memory grows with session length, detecting and consolidating recurrent activities into routines would be the natural next step for long-term deployments, a capability the single-session benchmark cannot assess.
  • The two-minute tolerance on time answers means 'when' questions are coarse; re-scoring with a stricter tolerance would test whether the hierarchy's temporal precision is real or just enough to pass a generous window.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. GESTO couples a persistent 4D scene graph with a two-level activity hierarchy of atomic human–object interactions and goal-driven events, both grounded to persistent scene entities. The pipeline extracts interactions from RGB-D clips with a vision-language model, groups them into events via an LLM, refines uncertain object links using event context, and exposes the memory through a relation-aware tool-calling agent. The paper evaluates on the text, binary, and time QA categories from the EGG benchmark plus 40 new Space2Event/Event2Space queries, reporting that GESTO achieves 0.71/0.75/0.70 on the standard categories, 0.73/0.75 on the new ones, and that ablations show complementary benefits of video fragmentation, grounding refinement, and the event hierarchy.

Significance. If reliable, the central claim is that an automatically constructed, grounded two-level activity hierarchy is sufficient to bring a robot memory close to methods that receive externally provided event and object grounding — a meaningful step beyond object-centric 4D scene graphs. The paper's strengths include a clear system design, an explicit problem formulation (G+ = (G_S, G_A, L)), an ablation that isolates the event hierarchy and grounding refinement, and honest reporting of benchmark limitations such as the exclusion of node queries and known object-description inconsistencies. The promised release of prompts, queries, and code would support reproducibility. However, the headline comparisons currently rest on small, unreleased query sets and an underspecified LLM judge, so the strength of the contribution cannot yet be assessed.

major comments (4)
  1. [V-A(b), Table I] The central comparison rests on aggregate scores over small per-category query counts with no error bars, per-query scores, or significance tests. Because node queries are excluded from the 80-query benchmark, the text, binary, and time categories likely contain only about 15–30 queries each; one answer changes a score by roughly 0.03–0.07, which covers the 0.01 text advantage over EGG(auto-caption) and part of the 0.08 time gap. Please report category sizes, per-query results, and confidence intervals, and temper the comparative claims accordingly.
  2. [V-A(d), IV-D] The text metric is defined only as an LLM-based semantic score between 0 and 1 without specifying the judge model, prompt, temperature, or aggregation rule. Since GESTO returns answers together with supporting evidence (Section IV-D), an LLM judge that rewards verbosity or evidence citation could systematically inflate GESTO's text scores relative to flat baselines. Please specify the judge and scoring prompt, validate against human ratings on a subset, and check for judge bias toward evidence-rich answers.
  3. [V-A(c)] The 40 extended queries were authored by the authors before running any method, but they are not released, and the ground-truth answers are scored with the same unspecified LLM semantic metric. These queries are used to support the cross-domain reasoning claims (0.73 Space2Event, 0.75 Event2Space). Without releasing the queries and answers and providing human-rated or otherwise validated scores, these numbers cannot be independently reproduced or fairly compared. Please include the query set and human agreement.
  4. [V-B, Table I] The EGG (w/o GT input) baseline is not the released EGG system; it is EGG's tool-calling agent exposed to the same automatically constructed scene and activity information used by GESTO. The phrase including its predicted activity information is ambiguous: if it includes GESTO's event structure and refined links, the baseline is not independent; if it includes only raw predicted interactions, the comparison conflates representation structure with tool-calling differences. Please specify precisely which inputs each baseline receives and how the agent was adapted, or run the original EGG with automatic event segmentation.
minor comments (5)
  1. [V-A(b)] Binary are evaluated using F1 scores should read Binary answers are evaluated using F1 scores.
  2. [III] The definition of an atomic interaction uses o_action in the tuple but later text refers to the interaction object label of o action; please standardize the notation.
  3. [V-A(a)] The free parameters, including the 15-second clip length, tau_mIoU = 0.3, tau_sem = 0.7, and one refinement round, are fixed without sensitivity analysis; a small sweep would strengthen the claim that the components transfer across settings.
  4. [V-D] The statement that GESTO transfers across sensor configurations without retraining is based on a single qualitative example; please soften the wording or add a small quantitative egocentric evaluation.
  5. [Availability statement] The availability statement says code, prompts, and queries will be open-sourced upon acceptance; for a reproducibility-focused evaluation, releasing these artifacts at submission time or in a supplementary package would allow reviewers to verify the numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GESTO's activity hierarchy is inferred from observations and evaluated against an external benchmark; the LLM-judge and self-authored extended queries raise evaluation-validity questions, not definitional circularity.

full rationale

The paper's central derivation constructs the representation G+ = (GS, GA, L) from an RGB-D observation stream: atomic interactions are extracted by a vision-language model, grounded to persistent objects via geometric and semantic matching, grouped into goal-driven events, and refined in one alternating pass (Sec. IV). No benchmark answer is used as an input to this construction: event summaries and grounding links are inferred from sensor data and scene-graph nodes, not from the text, binary, or time queries used for evaluation. The comparison with EGG is also not forced by construction: EGG(auto-caption) is reported with externally provided event and object grounding, while the fully automatic EGG setting is exposed to the same automatically constructed scene and activity estimates as GESTO, so the comparison isolates representation and querying structure rather than reusing GESTO's outputs as the baseline's inputs. The extended Space2Event and Event2Space queries were authored before running any evaluated method, and although they are scored with the same LLM-based semantic metric, this is a measurement choice rather than a definitional equivalence between the metric and GESTO's output. Concerns about LLM-judge noise, small per-category query counts, or possible judge preference for evidence-rich answers are legitimate evaluation-validity risks, but they do not demonstrate that any reported score reduces by construction to a fitted parameter or to a self-citation. The only self-citations ([27], [28]) appear in related-work context and are not load-bearing for the main claim. Therefore no circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on off-the-shelf perception models, hand-set thresholds, and an imposed partition of interactions into events. No new physical entities are introduced; the goal-driven event is the main invented abstraction, validated only indirectly through the question-answering benchmark. These are free choices that the reported scores depend on, and the paper does not report sensitivity to them.

free parameters (4)
  • clip_length = 15 seconds
    The RGB-D stream is split into 15-second clips for VLM interaction extraction. No sensitivity analysis is reported, and the ablation shows performance drops when clips exceed the visual-token budget, so results depend on this choice.
  • tau_mIoU = 0.3
    Geometric agreement threshold for grounding interactions to tracked objects. Chosen by hand; no ablation across values is reported.
  • tau_sem = 0.7
    Semantic similarity threshold for object grounding, with borderline candidates checked by an LLM judge. Chosen by hand; no sensitivity analysis is reported.
  • refinement_rounds = 1
    The link and event refinement alternation is performed exactly once. The paper says iterating to a fixed point is left for future work, so the benefit of refinement may depend on this early stopping choice.
assumptions (4)
  • domain assumption The DAAAM/Hydra 4D scene graph backbone supplies persistent object nodes with tracked bounding boxes and semantic appearance descriptions of sufficient quality for grounding.
    Section IV-A says GESTO uses DAAAM as the backbone and grounds interactions by matching to tracked boxes and descriptions, so tracking or description errors propagate into the grounding links.
  • domain assumption Cosmos-Reason2 reliably extracts atomic interactions and localizes manipulated objects within 15-second clips.
    Section IV-A and the ablation in Section V-C show the VLM degrades on longer clips, and Section VI admits object localization is unreliable under occlusion or with visually similar objects, so the whole hierarchy depends on this model's quality.
  • ad hoc to paper The LLM grouping of interactions into events can recover a complete, time-ordered partition at a consistent goal granularity.
    Section IV-B instructs the LLM that blocks of interactions form a complete partition and that events are broader than atomic actions but not as sweeping as general kitchen work; this is an imposed modeling constraint, not a derived fact.
  • domain assumption The LLM-based semantic score for text and extended answers is a valid measure of correctness and is not biased toward GESTO's answer style.
    Section V-A specifies that text and extended answers are scored with an LLM-based semantic score; if the judge rewards longer or evidence-rich answers, the head-to-head comparison could be inflated.
invented entities (1)
  • Goal-driven event (epsilon = (I_epsilon, s_epsilon))
    purpose: Groups a coherent set of atomic human-object interactions under a semantic summary so that queries can traverse from objects and places to activities and back.
    Events are created by LLM grouping from the video stream and are not validated against external annotations of event boundaries or summaries; the QA benchmark provides only indirect evidence of their utility.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenes." pith.science (2026). https://pith.science/paper/RYDDS26R

@misc{pith2026260810886,
  author       = {Pith},
  title        = {Pith review of: GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYDDS26R}},
  note         = {Machine review of arXiv:2608.10886}
}
read the original abstract

Robots operating in human environments need memories that capture not only what objects exist and where, but also how people use them over time and how individual interactions compose into goal-directed activities. Existing 4D scene graphs preserve object and place histories but omit activity structure, whereas activity representations are either not grounded in persistent 3D scenes or rely on externally provided event boundaries and object associations. We present GESTO (Grounded Event and Spatio-Temporal memOry), a spatio-temporal memory that couples a persistent 4D scene graph with a two-level hierarchy of atomic human--object interactions and goal-driven events. From an RGB-D observation stream, GESTO automatically extracts timestamped interactions, grounds them to persistent scene entities, groups them into events, and uses event context to refine uncertain object associations. A relation-aware tool-calling agent queries the resulting memory for activity-centric spatio-temporal reasoning. We evaluate GESTO on the reproducible text, binary, and time categories of an existing benchmark, together with 40 new Space2Event and Event2Space queries. GESTO achieves scores of 0.71, 0.75, and 0.70 on the standard categories, approaching a method supplied with ground-truth event and object grounding, while substantially outperforming the same reasoning framework when these inputs are removed. It further achieves 0.73 and 0.75 on Space2Event and Event2Space queries. Ablations show that hierarchical event structure and context-aware grounding refinement provide complementary benefits, supporting activity-grounded hierarchical memory for retrospective reasoning in dynamic human environments.

Figures

Figures reproduced from arXiv: 2608.10886 by the authors.

Figure 1
Figure 1. GESTO answers retrospective queries about human behavior by [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of GESTO. (Left) The online stage segments the RGB-D stream into short clips and extracts atomic human–object interactions, matching detections to tracked objects in GS. Then, event grouping and context-aware grounding refinement organize atomic interactions into goal-driven events and resolve unmatched object links using activity context. (Right) The resulting G + jointly represents the spatio-temporal sce… view at source ↗
Figure 3
Figure 3. The coupled hierarchies of G +. The activity graph GA (left plane) organizes atomic interactions under goal-driven events, and the spatio￾temporal graph GS (right plane) organizes persistent objects under places. The grounding links L connect them at their finest granularity, linking atomic interactions to object nodes, so that events inherit spatial context and objects become queryable through the activities they p… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative demonstration of GESTO on egocentric datasets. Given [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 31 canonical work pages

  1. [1]

    Describe anything anywhere at any moment,

    N. Gorlo, L. Schmid, and L. Carlone, “Describe anything anywhere at any moment,”arXiv preprint arXiv:2512.00565, 2025

    arXiv 2025

  2. [2]

    Towards spatio-temporal world scene graph generation from monocular videos,

    R. Peddi, Saurabh, S. Shanmugam, L. Pallapothula, Y . Xiang, P. Singla, and V . Gogate, “Towards spatio-temporal world scene graph generation from monocular videos,”arXiv 2603.13185, 2026

    work page arXiv 2026

  3. [3]

    Worth re- membering: Surprise-gated robot episodic memory,

    N. Gorlo, D. K. Wise, A. Speranzon, and L. Carlone, “Worth re- membering: Surprise-gated robot episodic memory,”arXiv preprint arXiv:2606.03787, 2026

    arXiv 2026

  4. [4]

    A V A: A video dataset of spatio-temporally localized atomic visual actions,

    C. Gu, C. Sun, D. A. Ross, C. V ondrick, C. Pantofaru, Y . Li, S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar, C. Schmid, and J. Malik, “A V A: A video dataset of spatio-temporally localized atomic visual actions,” inProceedings of the IEEE CVPR, 2018

    work page 2018

  5. [5]

    Action genome: Ac- tions as compositions of spatio-temporal scene graphs,

    J. Ji, R. Krishna, L. Fei-Fei, and J. C. Niebles, “Action genome: Ac- tions as compositions of spatio-temporal scene graphs,” inProceedings of the IEEE/CVF CVPR, 2020, pp. 10 236–10 247

    work page 2020

  6. [6]

    Moma: Multi-object multi-actor activity parsing,

    Z. Luo, W. Xie, S. Kapoor, Y . Liang, M. Cooper, J. C. Niebles, E. Adeli, and F.-F. Li, “Moma: Multi-object multi-actor activity parsing,” inAdvances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 17 939–17 955

    work page 2021

  7. [7]

    Ego4d goal-step: Toward hierarchical understanding of procedu- ral activities,

    Y . Song, E. Byrne, T. Nagarajan, H. Wang, M. Martin, and L. Torre- sani, “Ego4d goal-step: Toward hierarchical understanding of procedu- ral activities,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023

    work page 2023

  8. [8]

    Event-grounding graph: Uni- fied spatio-temporal scene graph from robotic observations,

    P. Nguyen, F. Verdoja, and V . Kyrki, “Event-grounding graph: Uni- fied spatio-temporal scene graph from robotic observations,”IEEE Robotics and Automation Letters, 2026

    work page 2026

Show all 40 references
  1. [9]

    3d scene graph: A structure for unified semantics, 3D space, and camera,

    I. Armeni, Z.-Y . He, A. Zamir, J. Gwak, J. Malik, M. Fischer, and S. Sneuavarese, “3d scene graph: A structure for unified semantics, 3D space, and camera,” in2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2019

    2019

  2. [10]

    Hier- archical Open-V ocabulary 3D Scene Graphs for Language-Grounded Robot Navigation,

    A. Werby, C. Huang, M. B ¨uchner, A. Valada, and W. Burgard, “Hier- archical Open-V ocabulary 3D Scene Graphs for Language-Grounded Robot Navigation,” inProceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024

    2024

  3. [11]

    Foundations of spatial perception for robotics: Hierarchical representations and real-time systems,

    N. Hughes, Y . Chang, S. Hu, R. Talak, R. Abdulhai, J. Strader, and L. Carlone, “Foundations of spatial perception for robotics: Hierarchical representations and real-time systems,”IJRR, 2024

    2024

  4. [12]

    Keysg: Hierar- chical keyframe-based 3d scene graphs,

    A. Werby, D. Rotondi, F. Scaparro, and K. O. Arras, “Keysg: Hierar- chical keyframe-based 3d scene graphs,” inProceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2026

    2026

  5. [13]

    3d scene graphs: Open challenges and future directions,

    D. Rotondi, F. Argenziano, S. Koch, N. Hughes, M. B ¨uchner, J. Wald, L. Rosenberger Schmid, D. Nardi, A. Valada, L. Paull, F. Tombari, L. Carlone, and K. O. Arras, “3d scene graphs: Open challenges and future directions,”arXiv 2606.19383, 2026

    2026 arXiv

  6. [14]

    Hydra: A real-time spatial perception system for 3d scene graph construction and optimization,

    N. Hughes, Y . Chang, and L. Carlone, “Hydra: A real-time spatial perception system for 3d scene graph construction and optimization,” arXiv preprint arXiv:2201.13360, 2022

    2022 arXiv

  7. [15]

    Conceptgraphs: Open-vocabulary 3D scene graphs for perception and planning,

    Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, C. Gan, C. M. de Melo, J. B. Tenenbaum, A. Torralba, F. Shkurti, and L. Paull, “Conceptgraphs: Open-vocabulary 3D scene graphs for perception and planning,” i...

    2024

  8. [16]

    Hi- erarchical open-vocabulary 3D scene graphs for language-grounded robot navigation,

    A. Werby, C. Huang, M. B ¨uchner, A. Valada, and W. Burgard, “Hi- erarchical open-vocabulary 3D scene graphs for language-grounded robot navigation,” inRobotics: Science and Systems XX. Robotics: Science and Systems Foundation, 2024

    2024

  9. [17]

    Clio: Real-time task-driven open-set 3D scene graphs,

    D. Maggio, Y . Chang, N. Hughes, M. Trang, D. Griffith, C. Dougherty, E. Cristofalo, L. Schmid, and L. Carlone, “Clio: Real-time task-driven open-set 3D scene graphs,”IEEE Robotics and Automation Letters, vol. 9, no. 10, pp. 8921–8928, 2024

    2024

  10. [18]

    Open-vocabulary functional 3D scene graphs for real- world indoor spaces,

    C. Zhang, A. Delitzas, F. Wang, R. Zhang, X. Ji, M. Pollefeys, and F. Engelmann, “Open-vocabulary functional 3D scene graphs for real- world indoor spaces,” in2025 IEEE/CVF CVPR. IEEE, 2025

    2025

  11. [19]

    Fungraph: Functionality aware 3D scene graphs for language-prompted scene interaction,

    D. Rotondi, F. Scaparro, H. Blum, and K. O. Arras, “Fungraph: Functionality aware 3D scene graphs for language-prompted scene interaction,” in2025 IEEE/RSJ IROS. IEEE, 2025

    2025

  12. [20]

    Ashita: Automatic scene-grounded hierarchical task analysis,

    Y . Chang, L. Fermoselle, D. Ta, B. Bucher, L. Carlone, and J. Wang, “Ashita: Automatic scene-grounded hierarchical task analysis,” in2025 IEEE/CVF CVPR. IEEE, 2025

    2025

  13. [21]

    3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans,

    A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone, “3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans,” inRobotics: Science and Systems XVI. Robotics: Science and Systems Foundation, 2020

    2020

  14. [22]

    Kimera: From SLAM to spatial perception with 3D dynamic scene graphs,

    A. Rosinol, A. Violette, M. Abate, N. Hughes, Y . Chang, J. Shi, A. Gupta, and L. Carlone, “Kimera: From SLAM to spatial perception with 3D dynamic scene graphs,”The International Journal of Robotics Research, vol. 40, no. 12-14, pp. 1510–1546, 2021

    2021

  15. [23]

    Khronos: A unified approach for spatio-temporal metric-semantic SLAM in dynamic envi- ronments,

    L. Schmid, M. Abate, Y . Chang, and L. Carlone, “Khronos: A unified approach for spatio-temporal metric-semantic SLAM in dynamic envi- ronments,” inRobotics: Science and Systems XX. Robotics: Science and Systems Foundation, 2024

    2024

  16. [24]

    Aion: Towards hierarchical 4D scene graphs with temporal flow dynamics,

    I. Catalano, E. Montijano, J. Civera, J. A. Placed, and J. Pena-Queralta, “Aion: Towards hierarchical 4D scene graphs with temporal flow dynamics,” in2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026, forthcoming

    2026

  17. [25]

    Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation,

    A. Anwar, J. Welsh, J. Biswas, S. Pouya, and Y . Chang, “Remembr: Building and reasoning over long-horizon spatio-temporal memory for robot navigation,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 2838–2845

    2025

  18. [26]

    Embodied- RAG: General non-parametric embodied memory for retrieval and generation,

    Q. Xie, S. Y . Min, P. Ji, Y . Yang, T. Zhang, K. Xu, A. Bajaj, R. Salakhutdinov, M. Johnson-Roberson, and Y . Bisk, “Embodied- RAG: General non-parametric embodied memory for retrieval and generation,”arXiv preprint arXiv:2409.18313, 2024

    2024 arXiv

  19. [27]

    Long-term planning around humans in domestic environments with 3d scene graphs,

    E. Bartoli, D. Rotondi, K. O. Arras, and I. Leite, “Long-term planning around humans in domestic environments with 3d scene graphs,”arXiv 2503.09173, 2025

    2025 arXiv

  20. [28]

    Social 3d scene graphs: Modeling human actions and relations for interactive service robots,

    E. Bartoli, D. Rotondi, B. He, P. Jensfelt, K. O. Arras, and I. Leite, “Social 3d scene graphs: Modeling human actions and relations for interactive service robots,”IROS, 2025

    2025

  21. [29]

    Event segmentation,

    J. M. Zacks and K. M. Swallow, “Event segmentation,”Current Directions in Psychological Science, vol. 16, no. 2, pp. 80–84, 2007

    2007

  22. [30]

    The language of actions: Re- covering the syntax and semantics of goal-directed human activities,

    H. Kuehne, A. Arslan, and T. Serre, “The language of actions: Re- covering the syntax and semantics of goal-directed human activities,” inProceedings of the IEEE CVPR, 2014, pp. 780–787

    2014

  23. [31]

    4D panoptic scene graph generation,

    J. Yang, J. Cen, W. Peng, S. Liu, F. Hong, X. Li, K. Zhou, Q. Chen, and Z. Liu, “4D panoptic scene graph generation,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  24. [32]

    Action scene graphs for long-form understanding of egocentric videos,

    I. Rodin, A. Furnari, K. Min, S. Tripathi, and G. M. Farinella, “Action scene graphs for long-form understanding of egocentric videos,” in Proceedings of the IEEE/CVF CVPR, 2024

    2024

  25. [33]

    Building a mind palace: Structuring environment-grounded semantic graphs for effective long video analysis with LLMs,

    Z. Huang, Y . Ji, X. Wang, N. Mehta, T. Xiao, D. Lee, S. Vanvalken- burgh, S. Zha, B. Lai, Y . Ren, L. Yu, N. Zhang, Y . J. Lee, and M. Liu, “Building a mind palace: Structuring environment-grounded semantic graphs for effective long video analysis with LLMs,” inProceedings of...

    2025

  26. [34]

    Ego4d: Around the world in 3,000 hours of egocentric video,

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu,et al., “Ego4d: Around the world in 3,000 hours of egocentric video,” inProceedings of the IEEE/CVF CVPR, 2022, pp. 18 995–19 012

    2022

  27. [35]

    The epic-kitchens dataset: Collection, challenges and baselines,

    D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price,et al., “The epic-kitchens dataset: Collection, challenges and baselines,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 11, pp...

    2020

  28. [36]

    Egolive: A large-scale egocentric dataset from real-world human tasks,

    Y . Li, X. Wei, J. Luo, Y . Xiao, Y . Bai, G. Zhou, T. Zou, C. Gui, J. Wen, H. Zhang,et al., “Egolive: A large-scale egocentric dataset from real-world human tasks,”arXiv preprint arXiv:2604.23570, 2026

    2026 arXiv

  29. [37]

    Cosmos-Reason2-8B: Physical ai common sense and embodied reasoning model,

    NVIDIA, “Cosmos-Reason2-8B: Physical ai common sense and embodied reasoning model,” https://huggingface.co/nvidia/ Cosmos-Reason2-8B, Mar. 2026, hugging Face model card, updated March 10, 2026; accessed June 10, 2026

    2026

  30. [38]

    Fast segment anything,

    X. Zhao, W. Ding, Y . An, Y . Du, T. Yu, M. Li, M. Tang, and J. Wang, “Fast segment anything,”arXiv preprint arXiv:2306.12156, 2023

    2023 arXiv

  31. [39]

    Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models,

    J. Ni, G. H. Abrego, N. Constant, J. Ma, K. Hall, D. Cer, and Y . Yang, “Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models,” inFindings of the association for computational linguistics: ACL 2022, 2022, pp. 1864–1874

    2022

  32. [40]

    Hoi4d: A 4d egocentric dataset for category- level human-object interaction,

    Y . Liu, Y . Liu, C. Jiang, K. Lyu, W. Wan, H. Shen, B. Liang, Z. Fu, H. Wang, and L. Yi, “Hoi4d: A 4d egocentric dataset for category- level human-object interaction,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.