Pith. sign in

REVIEW 5 major objections 4 minor 60 references

Designing Memory-Augmented AR Agents for Spatiotemporal Reasoning in Personalized Task Assistance

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that AR assistants need a persistent memory of the user's own experiences, and proposes a four-module framework that stores past procedures as scene graph sequences and aligns live context against them to give personalized

desk verdict A clear, honest position paper for memory-augmented AR agents; the four-module architecture is new as a combination, but the hypothesis is not yet backed by evidence and the load-bearing scene-graph alignment step is asserted rather than demonstrated. read the letter →

arxiv 2508.08774 v1 pith:CNGUMCNA submitted 2025-08-12 cs.AI cs.CL

Dongwook Choi , Taeyoon Kwon , Dongil Yang , Hyojun Kim , Jinyoung Yeo This is my paper · ORCID
classification cs.AIcs.CL
keywords augmentedrealitymemory-augmentedagentsscenegraphsspatiotemporalreasoningpersonalizedtaskassistancemultimodallargelanguagemodelsegocentricvideoproceduralmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that augmented-reality assistants fail at complex, personalized multi-step tasks because they cannot retain and reuse a user's own past experiences. It proposes a four-module architecture—perception, memory, spatiotemporal reasoning, and actuation—built around storing recorded egocentric activities as sequences of scene graphs. In a later recall phase, the agent aligns a live scene graph against the stored sequence to infer the user's task, track their progress, and suggest the next step based on how the user previously did the task. If the architecture works as claimed, AR assistance could shift from generic instructions to guidance that reproduces a user's personal cooking, organizing, health, or laboratory workflows. The paper is a design proposal: it supplies the framework, an implementation roadmap, and an evaluation plan rather than a tested system.

What carries the argument

The dynamic scene graph $G_t = (\mathcal{V}_t, \mathcal{E}_t)$, a structured snapshot in which nodes denote objects, user hands, actions, and UI elements with multimodal features, and directed edges denote observed physical interactions, attentional cues, or planned guidance relations. A temporal sequence of such graphs forms an episode memory, and the same graph format is used for both stored memories and live perception, making it the shared substrate that lets the Spatiotemporal Reasoning Module align past and present context and plan guidance.

What would settle it

Run a controlled repeat-task study: users perform a personalized multi-step task twice (e.g., cooking their own recipe) with AR glasses; build scene-graph memories from the first pass, then have the system predict each next step during the second pass. If predicted next steps do not beat a fixed text-instruction baseline, or if accuracy collapses when the user is interrupted or the camera moves, the central claim is contradicted.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that the reason current AR agents fall short on personalized tasks is not perception or language grounding but the absence of persistent, spatiotemporal memory, and that this memory can be supplied by converting recorded egocentric experiences into sequences of scene graphs. The framework's four modules—Perception, Memory, Spatiotemporal Reasoning, and Actuator—are organized around that claim. The load-bearing operation happens in the Spatiotemporal Reasoning Module: it aligns the incoming scene graph with the stored episode to infer task intent, identify the current stage, and generate an action plan, which the Actuator renders as AR overlays o

Load-bearing premise

The whole proposal rests on whether a user's past actions and surroundings can be captured, from egocentric video, as a faithful sequence of structured scene snapshots, and whether the live view can be matched against that stored sequence accurately enough to infer the task, the current step, and the right next step.

Editorial extensions

If this is right

  • If the framework is sound, an AR agent can reproduce a user's own prior procedure—such as a personal recipe—rather than giving generic instructions.
  • The two-phase recording and recall design turns ordinary egocentric recordings into reusable procedural memory, so personalized assistance does not require the user to describe their workflow in words.
  • Scene graph memory makes retrieval spatial and procedural: the system can match by object layout and action sequence, not only by verbal query.
  • The planned user study comparing memory-augmented guidance with fixed on-screen text would give a direct test of whether personalization improves task completion rate, completion time, workload, and satisfaction.
  • The same framework generalizes beyond cooking to household organizing, personalized health routines, and repeatable laboratory experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same recording-then-alignment pattern could be applied to egocentric assembly or repair tasks, and a proof-of-concept on existing long-form egocentric video would test the core assumption before any head-mounted display deployment.
  • The authors leave implicit that if scene graphs are a workable memory substrate, the stored procedure could be distilled into a natural-language workflow; retrieval and alignment might then be done by text reasoning, which is easier to evaluate and debug.
  • A longer-horizon consequence the paper does not develop: memory value compounds with use, so the same architecture becomes more personalized over time, making memory capacity, forgetting, and user privacy first-order design questions rather than add-ons.
  • A stricter evaluation than the proposed text-only baseline would be to compare the full memory agent against the same agent with memory access removed, isolating whether the memory module itself drives any gains apart from the AR visualization or language model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes a conceptual framework for memory-augmented AR agents that provide personalized task assistance by recalling and reasoning over a user's long-term interaction history. It identifies a limitation of current AR agents (lack of persistent, spatiotemporal user memory), and proposes four interconnected modules: Perception, Memory, Spatiotemporal Reasoning, and Actuator. Scene graphs are chosen as the unified representation for both stored episodic memories and live egocentric observations. The paper also provides a two-phase recording/recall scenario, an implementation roadmap using Unity and GPT-4o-realtime, a planned user-study evaluation, and a target cooking-recall application with additional use cases. No implementation, prototype, or empirical evaluation is reported; Section 5 is explicitly a future plan.

Significance. If the proposed framework were realized, it would address a genuine gap: current AR assistants generally support immediate tasks but do not leverage user-specific long-term workflows. The modular decomposition and the use of scene graphs as a shared representation are sensible and provide a concrete starting point for building such systems. The roadmap and target application give the proposal actionable specificity. However, the paper's central claim that the framework 'supports adaptive guidance grounded in user-specific workflows' (Section 7) is not established by any evidence; it remains an existence conjecture. The main strength is the conceptual synthesis, not a validated system. The paper is clearly written and the use cases are well chosen, but the load-bearing reasoning and alignment steps need significantly more specification or empirical grounding before the central claim can be accepted.

major comments (5)
  1. [§1, §7] The paper's title, abstract, and conclusion assert that the framework 'can provide personalized task assistance' and 'supports adaptive guidance grounded in user-specific workflows.' No implementation or user study is presented; Section 5 is only a plan. This is a causal/existence claim that is not supported by the manuscript's evidence. The authors should either provide at least a proof-of-concept implementation/feasibility study, or explicitly reframe the contribution as a proposed framework and hypothesis with the conclusion appropriately qualified.
  2. [§4.5.2, §5.1] The load-bearing step of the framework is the alignment between a streamed, noisy current scene graph and a stored scene-graph sequence. This component is asserted in §4.5.2 ('interprets short-term behavioral patterns... and aligns them with expected procedural steps') and in §5.1 ('aligns the current scene graph with recalled memory entities'), but no algorithm, graph-similarity metric, or handling of skipped/reordered/repeated steps is specified. Similarly, the claimed distinction of off-task behavior is not operationalized. Without this specification, the framework's core mechanism is untestable and the conclusion in §7 is not justified. Please provide a concrete alignment/robustness design or evidence of feasibility.
  3. [§4.3] The Perception Module cites [21,29,45] as recent advances in MLLMs that 'generate scene graph structures directly from complex, multimodal inputs.' These papers (Action Genome, video relation detection with spatio-temporal global context, and target-adaptive context aggregation) are not MLLM-based; they are supervised scene-graph-generation methods using convolutional/transformer architectures. This misattribution weakens the stated support for the perception pipeline. Please correct the citations and accurately describe the state of the art in MLLM-based scene graph generation, or qualify the feasibility claim accordingly.
  4. [§5.2] The proposed evaluation plan measures task completion rate, completion time, NASA-TLX, and user satisfaction, but it does not measure the framework's internal validity: memory construction fidelity, scene-graph alignment accuracy, step-detection precision/recall, or robustness to off-task actions. Without such metrics, even a successful between-group user study cannot attribute benefits to the memory-augmented architecture rather than to the novelty of the system or the content of instructions. Please add component-level evaluation metrics (e.g., graph edit distance, alignment error, misstep detection rate) to the plan.
  5. [§3, §4.4] The memory construction phase assumes that a user's past procedure can be faithfully captured as a sequence of scene graphs via offline processing of egocentric video and sensor data. This assumption is load-bearing because the entire Recall phase depends on the constructed memory. The paper offers no evidence that current scene-graph-generation methods achieve the required fidelity on noisy, long-form egocentric recordings, nor does it discuss failure modes (e.g., missing objects, misdetected relations, temporal segmentation errors). Please either provide a feasibility analysis or explicitly state this as an open research challenge that the framework does not solve.
minor comments (4)
  1. [§4.2.2] The notation for the dynamic scene graph appears garbled (e.g., 'Gt �Vt�Et�'). Please ensure proper typesetting and define the node/edge notation more clearly.
  2. [§5.1] The phrase 'we set the base simulation engine with Unity' is awkward; consider 'we use Unity as the base simulation engine.'
  3. [Figure 2] The text under the example recipe title appears as corrupted/non-rendered characters ('����� ������� ����'). Please verify the figure's font embedding and character encoding.
  4. [Author list] The author affiliations and names contain spacing artifacts (e.g., 'Y onsei University', 'T aeyoon Kwon'). Please correct these typographical issues.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an untested conceptual framework, not a derivation whose outputs are built into its inputs.

full rationale

The paper proposes a four-module architecture for memory-augmented AR agents and provides an implementation roadmap, but it reports no experiments and contains no equations, fitted parameters, or quantitative predictions. The central assertion that 'memory-augmented AR agents are essential for providing personalized assistance' is a design motivation, not an empirical result, and the paper's concrete contribution is the modular framework itself. The scene-graph representation is adopted partly on the basis of same-lab prior work [53] on LLM scene-graph reasoning, and the value of user-specific memory is supported by same-lab work [26]; both are empirical/benchmark studies rather than unverified assertions, so they do not constitute load-bearing circularity. The framework's conclusion that the system 'supports adaptive guidance grounded in user-specific workflows' is a restatement of the described module functions, not a derived prediction that reduces to its inputs by construction. The main weakness is evidential: the critical scene-graph alignment step in Sec. 4.5.2 is asserted without an algorithm, similarity metric, or error-recovery mechanism. That is a correctness and feasibility risk, not a circularity. No circular step can be exhibited with a specific reduction, so the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The framework rests entirely on domain assumptions about scene graph representational fidelity and MLLM reasoning reliability. There are no fitted parameters or invented entities; the 'memory' is a store of scene graph sequences, not a new physical or mathematical construct.

assumptions (4)
  • domain assumption Scene graphs are a sufficient unified representation for encoding both stored user memories and current observations such that spatiotemporal reasoning can operate on them.
    Section 4.2 asserts this without direct evidence; it is the backbone of the framework.
  • domain assumption MLLMs are capable of generating accurate scene graphs from egocentric multimodal input (video, gaze, hand pose, audio).
    Section 4.3 relies on citations [21,29,45] for scene graph generation, but the specific integration with AR glass sensor streams is not demonstrated.
  • domain assumption LLMs can reason over scene graphs to infer intent, track progress, and plan actions reliably enough for real-time guidance.
    Section 4.5 and Section 5.1 rely on [53] (same authors) and the GPT-4o backbone; this capacity is a precondition for the entire reasoning module.
  • domain assumption Offline processing of egocentric recordings yields structured memories that retain the fidelity needed to reproduce a personal workflow weeks later.
    Section 3 describes the conversion pipeline but provides no analysis of information loss or retrieval accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Designing Memory-Augmented AR Agents for Spatiotemporal Reasoning in Personalized Task Assistance." pith.science (2026). https://pith.science/paper/CNGUMCNA

@misc{pith2026250808774,
  author       = {Pith},
  title        = {Pith review of: Designing Memory-Augmented AR Agents for Spatiotemporal Reasoning in Personalized Task Assistance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CNGUMCNA}},
  note         = {Machine review of arXiv:2508.08774}
}
read the original abstract

Augmented Reality (AR) systems are increasingly integrating foundation models, such as Multimodal Large Language Models (MLLMs), to provide more context-aware and adaptive user experiences. This integration has led to the development of AR agents to support intelligent, goal-directed interactions in real-world environments. While current AR agents effectively support immediate tasks, they struggle with complex multi-step scenarios that require understanding and leveraging user's long-term experiences and preferences. This limitation stems from their inability to capture, retain, and reason over historical user interactions in spatiotemporal contexts. To address these challenges, we propose a conceptual framework for memory-augmented AR agents that can provide personalized task assistance by learning from and adapting to user-specific experiences over time. Our framework consists of four interconnected modules: (1) Perception Module for multimodal sensor processing, (2) Memory Module for persistent spatiotemporal experience storage, (3) Spatiotemporal Reasoning Module for synthesizing past and present contexts, and (4) Actuator Module for effective AR communication. We further present an implementation roadmap, a future evaluation strategy, a potential target application and use cases to demonstrate the practical applicability of our framework across diverse domains. We aim for this work to motivate future research toward developing more intelligent AR systems that can effectively bridge user's interaction history with adaptive, context-aware task assistance.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 43 canonical work pages

  1. [53]

    LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study

    Dongil Yang, Minjin Kim, Sunghwan Kim, Beong-woo Kwak, Minjun Park, Jinseok Hong, Woontack Woo, and Jiny- oung Yeo. Llm meets scene graph: Can large language mod- els understand and generate scene graphs? a benchmark and empirical study. arXiv preprint arXiv:2505.19510, 2025. 2, 3

    work page Pith review arXiv 2025

  2. [26]

    Embodied agents meet personalization: Exploring memory utilization for personalized assistance

    Taeyoon Kwon, Dongwook Choi, Sunghwan Kim, Hyojun Kim, Seungjun Moon, Beong-woo Kwak, Kuan-Hao Huang, and Jinyoung Yeo. Embodied agents meet personalization: Exploring memory utilization for personalized assistance. arXiv preprint arXiv:2505.16348, 2025. 1, 2

    arXiv 2025

  3. [1]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Cheb- otar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. 1

    arXiv 2022

  4. [2]

    A survey of augmented reality

    Ronald T Azuma. A survey of augmented reality. Presence: teleoperators & virtual environments, 6(4):355–385, 1997. 1

    work page 1997

  5. [3]

    Large language models empowered personalized web agents

    Hongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu, Xiaoyu Shen, Wenjie Li, and Tat-Seng Chua. Large language models empowered personalized web agents. In Proceedings of the ACM on Web Conference 2025, pages 198–215, 2025. 2

    work page 2025

  6. [4]

    Argus: Visualization of ai-assisted task guidance in ar

    Sonia Castelo, Joao Rulff, Erin McGowan, Bea Steers, Guande Wu, Shaoyu Chen, Iran Roman, Roque Lopez, Ethan Brewer, Chen Zhao, et al. Argus: Visualization of ai-assisted task guidance in ar. IEEE Transactions on Visualization and Computer Graphics, 30(1):1313–1323, 2023. 1, 2

    work page 2023

  7. [5]

    Web agents with world models: Learning and leveraging environment dynamics in web navi- gation

    Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navi- gation. arXiv preprint arXiv:2410.13232, 2024. 1

    arXiv 2024

  8. [6]

    Web-shepherd: Advancing prms for reinforcing web agents

    Hyungjoo Chae, Sunghwan Kim, Junhee Cho, Seungone Kim, Seungjun Moon, Gyeom Hwangbo, Dongha Lim, Min- jin Kim, Yeonjun Hwang, Minju Gwak, et al. Web-shepherd: Advancing prms for reinforcing web agents. arXiv preprint arXiv:2505.15277, 2025. 1

    arXiv 2025

Show all 60 references
  1. [7]

    Spatial-temporal transformer for dynamic scene graph generation

    Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosen- hahn, and Michael Ying Yang. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 16372–16382, 2021. 2

    2021

  2. [8]

    Towards context- aware adaptation in extended reality: A design space for xr interfaces and an adaptive placement strategy

    Shakiba Davari and Doug A Bowman. Towards context- aware adaptation in extended reality: A design space for xr interfaces and an adaptive placement strategy. arXiv preprint arXiv:2411.02607, 2024. 1

    2024 arXiv

  3. [9]

    Augmented object intelligence with xr-objects

    Mustafa Doga Dogan, Eric J Gonzalez, Karan Ahuja, Ruofei Du, Andrea Colac ¸o, Johnny Lee, Mar Gonzalez-Franco, and David Kim. Augmented object intelligence with xr-objects. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–15, 2...

    2024

  4. [10]

    The faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazar ´e, Maria Lomeli, Lucas Hosseini, and Herv ´e J´egou. The faiss library. arXiv preprint arXiv:2401.08281, 2024. 4

    2024 arXiv

  5. [11]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. 2023. 1

    2023

  6. [12]

    3d spatial multimodal knowledge ac- cumulation for scene graph prediction in point cloud

    Mingtao Feng, Haoran Hou, Liang Zhang, Zijie Wu, Yulan Guo, and Ajmal Mian. 3d spatial multimodal knowledge ac- cumulation for scene graph prediction in point cloud. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9182–9191, 2023. 2

    2023

  7. [13]

    Towards pervasive augmented reality: Context- awareness in augmented reality

    Jens Grubert, Tobias Langlotz, Stefanie Zollmann, and Holger Regenbrecht. Towards pervasive augmented reality: Context- awareness in augmented reality. IEEE transactions on visual- ization and computer graphics, 23(6):1706–1724, 2016. 1

    2016

  8. [14]

    Ar secretary agent: Real- time memory augmentation via llm-powered augmented re- ality glasses

    Rapha ¨el A El Haddad, Zeyu Wang, Yeonsu Shin, Ranyi Liu, Yuntao Wang, and Chun Yu. Ar secretary agent: Real- time memory augmentation via llm-powered augmented re- ality glasses. arXiv preprint arXiv:2505.11888, 2025. 1, 2

    2025 arXiv

  9. [15]

    Supporting student laboratory experiments with aug- mented reality experience

    Jona Hallmann, Carsten Stechert, and Syed Imad-Uddin Ahmed. Supporting student laboratory experiments with aug- mented reality experience. Proceedings of the Design Society, 3:3235–3244, 2023. 5

    2023

  10. [16]

    Development of nasa- tlx (task load index): Results of empirical and theoretical re- search

    Sandra G Hart and Lowell E Staveland. Development of nasa- tlx (task load index): Results of empirical and theoretical re- search. In Advances in psychology , volume 52, pages 139–

  11. [17]

    Web- voyager: Building an end-to-end web agent with large multi- modal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Web- voyager: Building an end-to-end web agent with large multi- modal models. arXiv preprint arXiv:2401.13919, 2024. 1

    2024 arXiv

  12. [18]

    Object detecting augmented reality system

    Mingwei Hu, Dongdong Weng, Feng Chen, and Yongtian Wang. Object detecting augmented reality system. In 2020 IEEE 20th International Conference on Communication Tech- nology (ICCT), pages 1432–1438. IEEE, 2020. 2

    2020

  13. [19]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 4

    2024 arXiv

  14. [20]

    Ac- ceptance of augmented reality for laboratory safety training: methodology and an evaluation study.Frontiers in Virtual Re- ality, 5:1322543, 2024

    Muhannad Ismael, Roderick McCall, Fintan McGee, Ilyasse Belkacem, Micka¨el Stefas, Joan Baixauli, and Didier Arl. Ac- ceptance of augmented reality for laboratory safety training: methodology and an evaluation study.Frontiers in Virtual Re- ality, 5:1322543, 2024. 5

    2024

  15. [21]

    Action genome: Actions as compositions of spatio- temporal scene graphs

    Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio- temporal scene graphs. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 10236–10247, 2020. 2, 3

    2020

  16. [22]

    Flowar: How different augmented real- ity visualizations of online fitness videos support flow for at- home yoga exercises

    Hye-Young Jo, Laurenz Seidel, Michel Pahud, Mike Sinclair, and Andrea Bianchi. Flowar: How different augmented real- ity visualizations of online fitness videos support flow for at- home yoga exercises. InProceedings of the 2023 CHI Confer- ence on Human Factors in Computing S...

    2023

  17. [23]

    Integrative object and pose to task detection for an augmented-reality-based human assistance system using neu- ral networks

    Linh K ¨astner, Leon Eversberg, Marina Mursa, and Jens Lam- brecht. Integrative object and pose to task detection for an augmented-reality-based human assistance system using neu- ral networks. In 2020 IEEE Eighth International Conference on Communications and Electronics (ICC...

    2020

  18. [24]

    Marker tracking and hmd calibration for a video-based augmented reality confer- encing system

    Hirokazu Kato and Mark Billinghurst. Marker tracking and hmd calibration for a video-based augmented reality confer- encing system. In Proceedings 2nd IEEE and ACM Inter- national Workshop on Augmented Reality (IWAR’99) , pages 85–94. IEEE, 1999. 2

    1999

  19. [25]

    Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships

    Sebastian Koch, Narunas Vaskevicius, Mirco Colosi, Pedro Hermosilla, and Timo Ropinski. Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set relationships. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Rec...

    2024

  20. [27]

    Satori: Towards proactive ar assistant with belief-desire-intention user modeling

    Chenyi Li, Guande Wu, Gromit Yeuk-Yin Chan, Dishita Gdi Turakhia, Sonia Castelo Quispe, Dong Li, Leslie Welch, Claudio Silva, and Jing Qian. Satori: Towards proactive ar assistant with belief-desire-intention user modeling. In Pro- ceedings of the 2025 CHI Conference on Human ...

    2025

  21. [28]

    Virtual marker technique to en- hance user interactions in a marker-based ar system

    Boyang Liu and Jiro Tanaka. Virtual marker technique to en- hance user interactions in a marker-based ar system. Applied Sciences, 11(10):4379, 2021. 2

    2021

  22. [29]

    Beyond short-term snippet: Video relation de- tection with spatio-temporal global context

    Chenchen Liu, Yang Jin, Kehan Xu, Guoqiang Gong, and Yadong Mu. Beyond short-term snippet: Video relation de- tection with spatio-temporal global context. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10840–10849, 2020. 2, 3

    2020

  23. [30]

    Sgformer: Semantic graph transformer for point cloud-based 3d scene graph generation

    Changsheng Lv, Mengshi Qi, Xia Li, Zhengyuan Yang, and Huadong Ma. Sgformer: Semantic graph transformer for point cloud-based 3d scene graph generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4035–4043, 2024. 2

    2024

  24. [31]

    Augmented reality based interactive cooking guide.Sensors, 22(21):8290,

    Isaias Majil, Mau-Tsuen Yang, and Sophia Yang. Augmented reality based interactive cooking guide.Sensors, 22(21):8290,

  25. [32]

    Augmented reality maintenance assistant using yolov5

    Ana Malta, Mateus Mendes, and Torres Farinha. Augmented reality maintenance assistant using yolov5. Applied Sciences, 11(11):4758, 2021. 2

    2021

  26. [33]

    A taxonomy of mixed re- ality visual displays

    Paul Milgram and Fumio Kishino. A taxonomy of mixed re- ality visual displays. IEICE TRANSACTIONS on Information and Systems, 77(12):1321–1329, 1994. 1

    1994

  27. [34]

    Text and code embeddings by contrastive pre-training

    Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005, 2022. 4

    2022 arXiv

  28. [35]

    Gorilla: Large language model connected with massive apis

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gon- zalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 126544–126565, 2024. 4

    2024

  29. [36]

    Autonomous workflow for multimodal fine-grained training assistants towards mixed re- ality

    Jiahuan Pei, Irene Viola, Haochen Huang, Junxiao Wang, Moonisa Ahsan, Fanghua Ye, Jiang Yiming, Yao Sai, Di Wang, Zhumin Chen, et al. Autonomous workflow for multimodal fine-grained training assistants towards mixed re- ality. arXiv preprint arXiv:2405.13034, 2024. 1, 2

    2024 arXiv

  30. [37]

    Toolllm: Facilitating large language models to mas- ter 16000+ real-world apis

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to mas- ter 16000+ real-world apis. arXiv preprint arXiv:2307.16789,

    arXiv

  31. [38]

    Sam 2: Segment any- thing in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment any- thing in images and videos. arXiv preprint arXiv:2408.00714,

    arXiv

  32. [39]

    Action scene graphs for long- form understanding of egocentric videos

    Ivan Rodin, Antonino Furnari, Kyle Min, Subarna Tripathi, and Giovanni Maria Farinella. Action scene graphs for long- form understanding of egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18622–18632, 2024. 4

    2024

  33. [40]

    Crossover: 3d scene cross-modal alignment

    Sayan Deb Sarkar, Ondrej Miksik, Marc Pollefeys, Daniel Barath, and Iro Armeni. Crossover: 3d scene cross-modal alignment. In Proceedings of the Computer Vision and Pat- tern Recognition Conference, pages 8985–8994, 2025. 2

    2025

  34. [41]

    Location-based augmented reality with pervasive smartphone sensors: Inside and beyond pokemon go! IEEE Access, 5:9619–9631, 2017

    Ryan Shea, Di Fu, Andy Sun, Chao Cai, Xiaoqiang Ma, Xi- aoyi Fan, Wei Gong, and Jiangchuan Liu. Location-based augmented reality with pervasive smartphone sensors: Inside and beyond pokemon go! IEEE Access, 5:9619–9631, 2017. 2

    2017

  35. [42]

    Caring-ai: Towards authoring context-aware augmented re- ality instruction through generative artificial intelligence

    Jingyu Shi, Rahul Jain, Seunggeun Chi, Hyungjun Doh, Hyung-gun Chi, Alexander J Quinn, and Karthik Ramani. Caring-ai: Towards authoring context-aware augmented re- ality instruction through generative artificial intelligence. In Proceedings of the 2025 CHI Conference on Human ...

    2025

  36. [43]

    Cognitive architectures for language agents

    Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive architectures for language agents. Transactions on Machine Learning Research , 2023. 2

    2023

  37. [44]

    Outdoors aug- mented reality on mobile phone using loxel-based visual fea- ture organization

    Gabriel Takacs, Vijay Chandrasekhar, Natasha Gelfand, Yin- gen Xiong, Wei-Chao Chen, Thanos Bismpigiannis, Radek Grzeszczuk, Kari Pulli, and Bernd Girod. Outdoors aug- mented reality on mobile phone using loxel-based visual fea- ture organization. In Proceedings of the 1st ACM...

    2008

  38. [45]

    Tar- get adaptive context aggregation for video scene graph gen- eration

    Yao Teng, Limin Wang, Zhifeng Li, and Gangshan Wu. Tar- get adaptive context aggregation for video scene graph gen- eration. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision , pages 13688–13697, 2021. 2, 3

    2021

  39. [46]

    Archef: An ios-based aug- mented reality cooking assistant powered by multimodal gem- ini llm

    Rithik Vir and Parsa Madinei. Archef: An ios-based aug- mented reality cooking assistant powered by multimodal gem- ini llm. arXiv preprint arXiv:2412.00627, 2024. 4

    2024 arXiv

  40. [47]

    Vector databases: What’s really new and what’s next?(vldb 2024 panel)

    Jianguo Wang, Eric Hanson, Guoliang Li, Yannis Papakon- stantinou, Harsha Simhadri, and Charles Xie. Vector databases: What’s really new and what’s next?(vldb 2024 panel). Proceedings of the VLDB Endowment, 17(12):4505– 4506, 2024. 4

    2024

  41. [48]

    Karma: Augmenting embodied ai agents with long-and-short term memory systems

    Zixuan Wang, Bo Yu, Junzhe Zhao, Wenhao Sun, Sai Hou, Shuai Liang, Xing Hu, Yinhe Han, and Yiming Gan. Karma: Augmenting embodied ai agents with long-and-short term memory systems. arXiv preprint arXiv:2409.14908, 2024. 2

    2024 arXiv

  42. [49]

    Agent workflow memory

    Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Gra- ham Neubig. Agent workflow memory. arXiv preprint arXiv:2409.07429, 2024. 2

    2024 arXiv

  43. [50]

    Artist: Automated text sim- plification for task guidance in augmented reality

    Guande Wu, Jing Qian, Sonia Castelo Quispe, Shaoyu Chen, Jo˜ao Rulff, and Claudio Silva. Artist: Automated text sim- plification for task guidance in augmented reality. In Pro- ceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–24, 2024. 1, 2

    2024

  44. [51]

    Augmented reality for maintenance tasks with chatgpt for automated text-to-action

    Fang Xu, Tri Nguyen, and Jing Du. Augmented reality for maintenance tasks with chatgpt for automated text-to-action. Journal of Construction Engineering and Management , 150 (4):04024015, 2024. 1, 2

    2024

  45. [52]

    Socialmind: Llm- based proactive ar social assistive system with human-like perception for in-situ live interactions

    Bufang Yang, Yunqi Guo, Lilin Xu, Zhenyu Yan, Hongkai Chen, Guoliang Xing, and Xiaofan Jiang. Socialmind: Llm- based proactive ar social assistive system with human-like perception for in-situ live interactions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiqu...

    2025

  46. [54]

    Authoring of a mixed reality assembly instructor for hierarchical structures

    J ¨urgen Zauner, Michael Haller, Alexander Brandl, and Werner Hartman. Authoring of a mixed reality assembly instructor for hierarchical structures. In The Second IEEE and ACM International Symposium on Mixed and Augmented Reality,

  47. [55]

    Sgedit: Bridging llm with text2image generative model for scene graph-based image editing

    Zhiyuan Zhang, DongDong Chen, and Jing Liao. Sgedit: Bridging llm with text2image generative model for scene graph-based image editing. arXiv preprint arXiv:2410.11815,

    arXiv

  48. [56]

    Editroom: Llm-parameterized graph dif- fusion for composable 3d room layout editing

    Kaizhi Zheng, Xiaotong Chen, Xuehai He, Jing Gu, Linjie Li, Zhengyuan Yang, Kevin Lin, Jianfeng Wang, Lijuan Wang, and Xin Eric Wang. Editroom: Llm-parameterized graph dif- fusion for composable 3d room layout editing. arXiv preprint arXiv:2410.12836, 2024. 4

    2024 arXiv

  49. [57]

    Openpsg: Open-set panoptic scene graph generation via large multimodal models

    Zijian Zhou, Zheng Zhu, Holger Caesar, and Miaojing Shi. Openpsg: Open-set panoptic scene graph generation via large multimodal models. In European Conference on Computer Vision, pages 199–215. Springer, 2024. 2

    2024

  50. [58]

    Retrieval-augmented embodied agents

    Yichen Zhu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Retrieval-augmented embodied agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17985–17995, 2024. 2

    2024

  51. [59]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 1

    2023

  52. [2003]

    IEEE, 2003

    Proceedings., pages 237–246. IEEE, 2003. 2

    2003

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.