pith. sign in

arxiv: 2606.05275 · v1 · pith:RYCLFXDFnew · submitted 2026-06-03 · 💻 cs.CV · cs.AI

Personal AI Agent for Camera Roll VQA

Pith reviewed 2026-06-28 06:17 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords personal AI agentcamera roll VQAvisual question answeringhierarchical memorylong-context visual reasoningpersonalized visual memorycamroll dataset
0
0 comments X

The pith

A hierarchical-memory agent outperforms baselines when answering questions over a user's personal photo collection.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines the task of visual question answering over a single user's camera roll spanning years and thousands of images. It releases the camroll dataset containing 2,500 manually written questions tied to 31,476 photos from 50 people. It then presents camroll-agent, which stores the photo stream in a hierarchy of memories and uses a small set of retrieval and reasoning tools to locate relevant images. Experiments show this design beats a range of long-context baselines on the new questions. The results indicate that handling consistent, user-specific visual streams differs from handling long text passages.

Core claim

The paper claims that camroll-agent, a conversational system equipped with hierarchical memory and a minimal tool set, enables effective navigation and question answering over large personalized visual memory streams, outperforming existing long-context methods and demonstrating that visual personal memory requires approaches distinct from standard textual long-context handling, especially for consistency, visual detail, and user-specific context.

What carries the argument

camroll-agent: a conversational AI agent that uses hierarchical memory plus a minimal set of tools to navigate and reason over a user's long-horizon personal photo stream.

If this is right

  • Personalized visual memory streams need specialized memory hierarchies rather than direct application of text-based long-context techniques.
  • The camroll dataset serves as a benchmark exposing gaps in current long-context agents when consistency and user-specific visual details are required.
  • With hierarchical memory and few tools, agents can support both factual recall and open-ended recommendations drawn from years of personal photos.
  • Handling visual personal memory at scale demands different engineering choices than handling textual memory alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same hierarchical-memory pattern could be tested on other personal data streams such as location history or saved documents.
  • Deploying such agents would require explicit controls on which parts of the camera roll are visible to the model at any time.
  • Re-running the evaluation on camera-roll data collected from additional demographic groups would test whether the reported gap holds beyond the current 50 users.

Load-bearing premise

The manually written questions in the camroll dataset reflect the kinds of queries people would actually pose to an assistant with access to their camera roll.

What would settle it

A controlled experiment in which a flat long-context baseline matches or exceeds camroll-agent accuracy on the full set of 2,500 camroll questions would falsify the claim that the hierarchical-memory design is required.

Figures

Figures reproduced from arXiv: 2606.05275 by Donghyun Kim, Krishna Kumar Singh, Thao Nguyen, Yong Jae Lee, Yuheng Li.

Figure 1
Figure 1. Figure 1: We study the VQA setting over the personal camera roll, where an AI assistant can search [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of camroll. Left: photos are captured across 25+ countries. Right: smartphone users (in-house subset) take substantially more images than digital camera users (YFCC-100M). In this paper, we take a step toward studying question answering over personal camera rolls. We construct a dataset, camroll, from real user camera roll with annotated personalized visual question answering, and highlight the un… view at source ↗
Figure 3
Figure 3. Figure 3: Hierarchical memory for personal camera rolls, organized from low-level visual pixels ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Tool-call distributions across turns and question types. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: ClaudeCode vs. camroll-agent tool call distributions. Do we need domain-specific agents? A gener￾alist coding agent can be repurposed for camera roll setting, but its tool inventory imposes a strong inductive bias toward filesystem traversal and byte￾level inspection. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
read the original abstract

We study the personal camera roll visual question answering setting. In this setting, a conversational AI assistant can access a user's personal camera roll and retrieve relevant photos to answer queries, ranging from simple factual questions (e.g., ``Name of the food I tried yesterday?'') to more open-ended ones (e.g., ``Recommend some dishes I have never eaten before''). Given the vast nature of the personal camera roll (i.e., multiple years, hundreds to thousands of photos), a successful AI assistant needs to understand a long-horizon, highly personalized visual content stream in order to navigate and locate the correct and/or relevant information. To support this, we collect and manually annotate questions that mimic real-world usage. The final dataset, camroll, contains 50 users, 31,476 images, and 2,500 QA pairs. We further design camroll-agent, a conversational AI agent equipped with hierarchical memory and a minimal set of tools for efficient navigation over large, personalized visual memory. Experimental results show that camroll-agent outperforms numerous baselines and methods for long-context understanding AI agents system. Together, the camroll dataset and camroll-agent highlight the gap in AI agents' long-context reasoning: personalized visual memory requires different approaches from standard long-context textual memory, especially when consistency, visual details, and user-specific context are present.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces the camroll dataset (50 users, 31,476 images, 2,500 manually annotated QA pairs) for personal camera-roll visual question answering and proposes camroll-agent, a conversational agent using hierarchical memory and a minimal tool set for navigating long-horizon personalized visual streams. It claims experimental results demonstrate that camroll-agent outperforms baselines for long-context understanding and that personalized visual memory requires distinct approaches from standard textual long-context methods, particularly regarding consistency, visual details, and user-specific context.

Significance. If the outperformance holds under a validated evaluation protocol and the dataset reflects organic usage, the work would usefully highlight limitations of existing long-context agents when applied to visual memory and provide a new benchmark for personalized visual agents. The dataset collection and hierarchical-memory design are concrete contributions that could stimulate follow-up on visual vs. textual memory distinctions.

major comments (2)
  1. [Dataset Construction] Dataset section: the central claim that results demonstrate a gap between personalized visual memory and textual long-context methods rests on the 2,500 QA pairs accurately mimicking real-world usage patterns (consistency, visual details, user-specific context). The manuscript supplies no supporting evidence such as inter-annotator agreement, comparison against logged user queries, or a user validation study, leaving the representativeness assumption unverified and load-bearing for the gap conclusion.
  2. [Experimental Results] Experimental Results section: the abstract asserts that camroll-agent 'outperforms numerous baselines' yet the provided description contains no quantitative metrics, baseline specifications, or evaluation protocol details. Without these, the outperformance claim cannot be assessed for statistical significance or fairness of comparison.
minor comments (2)
  1. [Methods] Clarify the exact composition of the hierarchical memory (e.g., how image embeddings are indexed and retrieved) and the minimal tool set in the methods section to allow reproducibility.
  2. [Discussion] Add a limitations paragraph discussing potential annotator bias in the QA pairs and the scope of the 50-user sample.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on dataset representativeness and experimental clarity. We address each major comment below with plans for revision where appropriate.

read point-by-point responses
  1. Referee: [Dataset Construction] Dataset section: the central claim that results demonstrate a gap between personalized visual memory and textual long-context methods rests on the 2,500 QA pairs accurately mimicking real-world usage patterns (consistency, visual details, user-specific context). The manuscript supplies no supporting evidence such as inter-annotator agreement, comparison against logged user queries, or a user validation study, leaving the representativeness assumption unverified and load-bearing for the gap conclusion.

    Authors: We agree that stronger evidence for representativeness would better support the central claim. The manuscript describes manual annotation to mimic real-world usage, but we will revise to add a dedicated subsection on the annotation protocol, including inter-annotator agreement metrics that were computed internally. We will also elaborate on how the QA pairs were designed around the 50 users' actual photo collections to capture consistency, visual details, and personalization. A comparison to logged user queries is not feasible due to the private nature of personal camera rolls, and a dedicated user validation study was outside the original scope; we will explicitly discuss these as limitations while arguing that the manual process by domain-aware annotators provides a reasonable proxy for organic patterns. revision: partial

  2. Referee: [Experimental Results] Experimental Results section: the abstract asserts that camroll-agent 'outperforms numerous baselines' yet the provided description contains no quantitative metrics, baseline specifications, or evaluation protocol details. Without these, the outperformance claim cannot be assessed for statistical significance or fairness of comparison.

    Authors: The full experimental results section does contain quantitative metrics (accuracy on the 2,500 QA pairs), baseline specifications (including long-context LLMs and agent variants), and the evaluation protocol. However, we acknowledge these elements were not presented with sufficient prominence or detail for easy assessment. In the revised manuscript we will expand the section with explicit result tables, baseline descriptions, the precise evaluation protocol, and statistical significance analysis to enable full verification of the outperformance claims. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical evaluation on newly collected dataset

full rationale

The paper collects a new dataset (camroll) with manual annotations and introduces camroll-agent with hierarchical memory and tools, then reports empirical outperformance on 2,500 QA pairs against baselines. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the abstract or described content. The central claim rests on direct experimental comparison rather than any reduction to inputs by construction. This is a standard empirical setup with no load-bearing self-referential steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical derivations or parameter fittings described; assessment limited to abstract.

pith-pipeline@v0.9.1-grok · 5774 in / 899 out tokens · 30604 ms · 2026-06-28T06:17:56.381696+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    Survey: The states that phlush away the most memories, 2023

    Mixbook. Survey: The states that phlush away the most memories, 2023. Accessed: 2026-05-06

  2. [2]

    Hunters and gatherers of pictures: Why photography has become a human universal.Frontiers in Psychology, 12, 2021

    Lisa Kislinger and Kurt Kotrschal. Hunters and gatherers of pictures: Why photography has become a human universal.Frontiers in Psychology, 12, 2021

  3. [3]

    David Fernández-Pérez, Lucía Ros, and José M. Latorre. The role of the personal relevance of images in retrieving autobiographical memories for emotion regulation: A randomized controlled trial study.Current Psychology, 43:3523–3537, 2024

  4. [4]

    Mobile photography statistics, 2024

    PhotoAid. Mobile photography statistics, 2024

  5. [5]

    Digital hoarding: Why your phone has 10,000 photos you’ll never look at, 2025

    Affenstunde. Digital hoarding: Why your phone has 10,000 photos you’ll never look at, 2025. Accessed: 2026-04-26

  6. [6]

    https://support.apple.com/guide/iphone/ use-apple-intelligence-in-photos-iphf7de217f0/ios

    Use apple intelligence in photos on iphone. https://support.apple.com/guide/iphone/ use-apple-intelligence-in-photos-iphf7de217f0/ios. Apple Inc., Accessed: 2026-05-06

  7. [7]

    https://techcommunity.microsoft.com/blog/onedriveblog/ copilot--onedrive-intelligence-in-every-click-inspiration-in-every-memory/ 4458882

    Copilot + onedrive: Intelligence in every click, inspiration in every memory. https://techcommunity.microsoft.com/blog/onedriveblog/ copilot--onedrive-intelligence-in-every-click-inspiration-in-every-memory/ 4458882. Microsoft, Accessed: 2026-05-06

  8. [8]

    OpenAI, Accessed: 2026-05-06

    Chatgpt.https://chat.openai.com. OpenAI, Accessed: 2026-05-06

  9. [9]

    Google, Accessed: 2026-05-06

    Gemini.https://gemini.google.com. Google, Accessed: 2026-05-06. 10

  10. [10]

    Anthropic, Accessed: 2026-05-06

    Claude.https://claude.ai. Anthropic, Accessed: 2026-05-06

  11. [11]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. InACL, 2024

  12. [12]

    Gonzalez, Trevor Darrell, and David Chan

    Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, and David Chan. Visual haystacks: A vision-centric needle-in-a-haystack benchmark. InICLR, 2025

  13. [13]

    From RAG to memory: Non-parametric continual learning for large language models

    Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From RAG to memory: Non-parametric continual learning for large language models. InICML, 2025

  14. [14]

    Self-RAG: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. InICLR, 2024

  15. [15]

    Mem0: Building production-ready ai agents with scalable long-term memory

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. InarXiv, 2025

  16. [16]

    Memos: An operating system for memory-augmented generation (mag) in large language models

    Zhiyu Li, Shichao Song, Hanyu Wang, Simin Niu, Ding Chen, Jiawei Yang, Chenyang Xi, Huayi Lai, Jihao Zhao, Yezhaohui Wang, Junpeng Ren, Zehao Lin, Jiahao Huo, Tianyi Chen, Kai Chen, Kehang Li, Zhiqiang Yin, Qingchen Yu, Bo Tang, Hongkang Yang, Zhi-Qin John Xu, and Feiyu Xiong. Memos: An operating system for memory-augmented generation (mag) in large langu...

  17. [17]

    Simplemem: Efficient lifelong memory for llm agents

    Jiaqi Liu, Yaofeng Su, Peng Xia, Siwei Han, Zeyu Zheng, Cihang Xie, Mingyu Ding, and Huaxiu Yao. Simplemem: Efficient lifelong memory for llm agents. InarXiv, 2026

  18. [18]

    Lightmem: Lightweight and efficient memory-augmented generation

    Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, and Ningyu Zhang. Lightmem: Lightweight and efficient memory-augmented generation. InICLR, 2026

  19. [19]

    Evaluating very long-term conversational memory of llm agents.arxiv, 2024

    Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of llm agents.arxiv, 2024

  20. [20]

    Taylor, and Dan Roth

    Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J. Taylor, and Dan Roth. Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale. InarXiv, 2025

  21. [21]

    Perltqa: A personal long-term memory dataset for memory classification, retrieval, and fusion in question answering

    Yiming Du, Hongru Wang, Zhengyi Zhao, Bin Liang, Baojun Wang, Wanjun Zhong, Zezhong Wang, and Kam-Fai Wong. Perltqa: A personal long-term memory dataset for memory classification, retrieval, and fusion in question answering. InProceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10), 2024

  22. [22]

    Deepimagesearch: Benchmarking multimodal agents for context-aware image retrieval in visual histories

    Chenlong Deng, Mengjie Deng, Junjie Wu, Dun Zeng, Teng Wang, Qingsong Xie, Jiadeng Huang, Shengjie Ma, Changwang Zhang, Zhaoxiang Wang, Jun Wang, Yutao Zhu, and Zhicheng Dou. Deepimagesearch: Benchmarking multimodal agents for context-aware image retrieval in visual histories. InarXiv, 2026

  23. [23]

    Photobench: Beyond visual matching towards personalized intent-driven photo retrieval

    Tianyi Xu, Rong Shan, Junjie Wu, Jiadeng Huang, Teng Wang, Jiachen Zhu, Wenteng Chen, Minxin Tu, Quantao Dou, Zhaoxiang Wang, Changwang Zhang, Weinan Zhang, Jun Wang, and Jianghao Lin. Photobench: Beyond visual matching towards personalized intent-driven photo retrieval. InarXiv, 2026

  24. [24]

    Ask photos: Search your memories in google photos

    Google. Ask photos: Search your memories in google photos. https://blog.google/ products-and-platforms/products/photos/ask-button-ask-photos-tips/ , 2024. Accessed: 2026-05-04

  25. [25]

    Personal intelligence in gemini app with nano banana

    Google. Personal intelligence in gemini app with nano banana. https://blog.google/ innovation-and-ai/products/gemini-app/personal-intelligence-nano-banana/ , 2026. Accessed: 2026-05-04

  26. [26]

    Introducing muse spark: Msl’s first model, purpose-built to prioritize people

    Meta AI. Introducing muse spark: Msl’s first model, purpose-built to prioritize people. https://ai. meta.com/blog/introducing-muse-spark-msl/, 2026. Accessed: 2026-05-04

  27. [27]

    Cottrell

    Yufei Wang, Zhe Lin, Xiaohui Shen, Radomˇir Mˇech, Gavin Miller, and Garrison W. Cottrell. Recognizing and curating photo albums via event-specific image importance. InBMVC, 2017

  28. [28]

    Realfill: Reference-driven generation for authentic image completion.arXiv preprint arXiv:2309.16668, 2023

    Luming Tang, Nataniel Ruiz, Chu Qinghao, Yuanzhen Li, Aleksander Holynski, David E Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, and Michael Rubinstein. Realfill: Reference-driven generation for authentic image completion.arXiv preprint arXiv:2309.16668, 2023

  29. [29]

    Myvlm: Personal- izing vlms for user-specific queries, 2024

    Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. Myvlm: Personal- izing vlms for user-specific queries, 2024. 11

  30. [30]

    Yo’LLaV A: Your personalized language and vision assistant

    Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. Yo’LLaV A: Your personalized language and vision assistant. InNeurIPS, 2024

  31. [31]

    Yo’chameleon: Personalized vision and language generation.CVPR, 2025

    Thao Nguyen, Krishna Kumar Singh, Jing Shi, Trung Bui, Yong Jae Lee, and Yuheng Li. Yo’chameleon: Personalized vision and language generation.CVPR, 2025

  32. [32]

    Personavlm: Long-term personal- ized multimodal llms

    Chang Nie, Chaoyou Fu, Yifan Zhang, Haihua Yang, and Caifeng Shan. Personavlm: Long-term personal- ized multimodal llms. InCVPR, 2026

  33. [33]

    Patel, and Yuheng Li

    Viet Nguyen, Thao Nguyen, Vishal M. Patel, and Yuheng Li. Personal visual memory from explicit and implicit evidence, 2026

  34. [34]

    Context length alone hurts llm performance despite perfect retrieval

    Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, and Hao Peng. Context length alone hurts llm performance despite perfect retrieval. InEMNLP, 2025

  35. [35]

    Memorybank: Enhancing large language models with long-term memory

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. InarXiv, 2023

  36. [36]

    Re- Act: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- Act: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

  37. [37]

    Claude Code: Agentic coding

    Anthropic. Claude Code: Agentic coding. https://www.anthropic.com/product/claude-code,

  38. [38]

    Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li

    Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: the new data in multimedia research. 2016

  39. [39]

    According to me: Long-term personalized referential memory qa

    Jingbiao Mei, Jinghong Chen, Guangyu Yang, Xinyu Hou, Margaret Li, and Bill Byrne. According to me: Long-term personalized referential memory qa. InarXiv, 2026

  40. [40]

    Episodic memory: From mind to brain.Annual review of psychology, 2002

    Endel Tulving. Episodic memory: From mind to brain.Annual review of psychology, 2002

  41. [41]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InICCV, 2015

  42. [42]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

  43. [43]

    M3-embedding: Multi- linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation

    Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multi- linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In ACL 2024, 2024

  44. [44]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

  45. [45]

    GPT-4o System Card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. A Appendix A.1 Broader Impacts This work studies long-horizon reasoning over personal camera rolls, a setting with potential appli- cations in personalize...

  46. [46]

    Choose exactly one operation for the event table: - ADD: create a new event row - UPDATE: update the latest event row - NO OP: do not modify the event table

  47. [47]

    a man",

    If you choose ADD or UPDATE, return a full event row with fields: - event name - description - date - images Rules: - The image caption must always be present. - Use first-person wording when natural. - The image caption must describe image 2 only, not the profile/reference image. - Be careful about person identity; the person in the first image is the us...