Pith. sign in

REVIEW 4 major objections 6 minor 79 references

ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Modeling the user's working memory, rather than letting an LLM judge context directly, determines when a smart-glasses assistant should interrupt, and this timing engine—not the message text—is what made assistance feel more valuable and…

desk verdict A genuinely new WM-based timing mechanism for proactive wearables, with a plausible but under-powered user study; the engagement claim needs a count-adjusted analysis before it carries the load. read the letter →

arxiv 2507.21378 v1 pith:FTHFIXFB submitted 2025-07-28 cs.HC cs.AI

Kevin Pu , Ting Zhang , Naveen Sendhilnathan , Sebastian Freitag , Raj Sodhi , Tanya Jonker This is my paper
classification cs.HCcs.AI
keywords proactiveassistanceworkingmemorymodelingsmartglassesinterruptioncosttimingpredictionegocentricperceptionhuman-AIinteractionuserstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ProMemAssist claims that the right moment to give proactive assistance on a wearable device can be found by modeling the user's working memory from egocentric camera and microphone signals. The system turns what the user sees and hears into memory items with recency, relevance, and importance, binds them into episodic chunks, and uses a utility function that weighs the value of a candidate message against the predicted cost of displacing or interfering with current mental content. In a within-subject study of 12 participants doing four tabletop organization tasks, the WM-informed timing delivered fewer messages than an LLM baseline that saw the same observations, yet drew positive engagement on 24.6% of delivered messages versus 9.34%, and significantly reduced self-reported frustration. The paper's claim is that the timing mechanism, not the message content, drives these differences, since both conditions used the same LLM to generate assistance content.

What carries the argument

The load-bearing mechanism is the working memory model itself: a perception memory of seven item slots for visuospatial and phonological content, plus an episodic buffer of four chunks. Each item carries recency (linear decay with a 30-second horizon), relevance (average CLIP cosine similarity to episode summaries), and importance (LLM-assigned score), which together decide displacement and binding. The timing predictor treats every candidate assistance message as a would-be new memory item and computes a utility that subtracts predicted displacement and modality-matched semantic interference from weighted importance and relevance. Thresholds then route messages to immediate delivery, a deferred queue that is re-evaluated as the model updates, or discard.

What would settle it

Give the same 12 or more participants the same four tasks but replace ProMemAssist's WM utility gate with a scheduler that delivers the same messages at randomly chosen moments matched for frequency; if positive engagement and frustration ratings do not differ from the WM-timed condition, then the WM model contributes nothing beyond message selectivity. A complementary check is to compare the model's predicted utility at delivery time against an independent receptivity signal such as self-reported interruption annoyance; if high-utility deliveries are not rated less annoying than low-utility ones, the proxy fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a real-time computational model of working memory—built from visual object detections, speech transcriptions, semantic embeddings, and LLM-assigned importance scores—can be used to decide when a wearable assistant should speak up, and that this timing improves user experience over having an LLM judge timing directly from the same observations. Concretely, ProMemAssist maintains a seven-item perception store and a four-chunk episodic buffer; candidate assistance is scored by $\text{Utility} = (0.6\,I + 0.4\,R) - (C_D + C_I)$, with immediate delivery above 0.75, deferral and re-evaluation between 0 and 0.75, and discard at or below zero. The study found lower frustration (mean 2.32 vs 3.14, $p = 0.043$) and a higher proportion of positively engaged responses (24.6% vs 9.34%) while relevance, helpfulness, and task-completion time stayed comparable, which the authors take as evidence that WM-informed selectivity, not message content, produced the improvement.

Load-bearing premise

The system assumes that the memory items it builds from detected objects, transcribed speech, semantic similarity, and LLM-scored importance truly mirror the user's actual working-memory load and receptivity; if these proxies diverge from real cognitive availability, the utility scores stop measuring interruption cost and the reported engagement differences could come from message sparsity or experimenter expectations rather than cognitive alignment.

Editorial extensions

If this is right

  • WM-informed timing can improve proactive assistance without changing what the assistant says, since assistance content was held constant across conditions.
  • Selective delivery raises the ratio of positively received messages, suggesting that fewer, better-timed interventions can outperform more frequent ones.
  • Deferring borderline messages and re-scoring them as the user's mental state changes offers a second chance for low-cost delivery.
  • Because WM state is estimated from camera and audio signals only, the approach transfers to off-the-shelf wearable hardware rather than requiring physiological sensors.
  • The observed lower frustration, with no significant change in task time or other workload dimensions, indicates the main benefit is attentional and emotional rather than purely productivity-related.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same utility signal could be repurposed to filter content, not just timing, by using predicted interference cost to reword or drop messages that would collide with active memory.
  • Beyond the paper: an ablation that delivers ProMemAssist's chosen messages at random or at fixed intervals—holding frequency and content fixed—would test whether the WM model's specific timing, rather than mere message sparsity, causes the engagement gain.
  • Beyond the paper: calibrating recency horizon, capacity, and threshold weights per user could improve the model; the paper's fixed defaults (30 seconds, 7 items, 4 chunks, threshold 0.75) are starting points, not measured optima.
  • Beyond the paper: urgent or safety-critical messages would need to bypass the WM utility gate, since the paper's non-urgent tasks cannot evaluate how cost-benefit trade-offs shift when missing a moment is dangerous.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. ProMemAssist is a smart-glasses system that constructs a real-time computational working-memory (WM) model from egocentric visual and audio signals, then uses a utility-based timing predictor to decide when to deliver LLM-generated proactive assistance. The system encodes perceived objects and speech as memory items with recency, relevance, and importance scores, binds them into episodic chunks, and predicts displacement and interference costs to compute a delivery utility. The authors report a 12-participant within-subject study comparing ProMemAssist against an LLM-only baseline that uses the same observations to generate and time assistance. They report that ProMemAssist delivered fewer messages (130 vs 332), achieved a higher positive-engagement rate (24.6% vs 9.34%), and produced a statistically significant reduction in self-reported frustration (Q10, p=0.043), with qualitative feedback supporting cognitively aligned timing. The paper claims that WM-informed timing, not assistance content, drives these improvements.

Significance. If the central claim were established, the paper would make a genuine contribution to proactive assistance: a theoretically grounded, implementable model of working memory as a timing signal, evaluated in real physical tasks against a reasonable baseline. Strengths include the use of system logs to document generation, deferral, and delivery decisions; a within-subject design with counterbalanced conditions; validation of logged engagements against video; and a candid limitations section that acknowledges content-timing coupling and the need for future ablations. The paper is transparent about the proxy nature of its WM model. However, the current analysis does not separate the effect of WM modeling from the effect of delivering fewer messages, and the primary engagement measure is logged by an experimenter who was not blinded to condition. The result is promising as a proof-of-concept but does not yet support the strong causal wording of the abstract and Section 6.1.

major comments (4)
  1. [§6.1 (System logs and engagement comparison)] The headline engagement comparison (32/130 = 24.6% for ProMemAssist vs 31/332 = 9.34% for baseline) is reported without a significance test and confounds timing policy with delivery volume. Because the baseline delivered more than twice as many messages, the observed difference could be explained by a simple volume effect: any policy that delivers fewer, more selective messages would likely raise the positive-engagement proportion, regardless of whether the selection uses a cognitive model. The authors should report a per-participant mixed-effects logistic regression with delivery count (or rate) as a covariate, or a count-matched subsample comparison, and provide a significance test for the proportional difference. Without such an analysis, the abstract's claim that WM-informed timing caused higher engagement is not supported.
  2. [§5.5 (Data Analysis)] Positive engagements were manually logged by the experimenter in real time using keypresses on the companion laptop, with no reported blinding to condition. Because the experimenter had to know which system was running to operate the session, and because deciding whether a visible or verbal response counts as a positive engagement is a subjective judgment, the engagement difference could be influenced by experimenter expectation. Video validation checks only whether the logged events match the video, not whether the logging threshold differed across conditions. The authors should report a blinded coding protocol (e.g., independent coders viewing video with condition labels removed) and inter-rater reliability, or explicitly acknowledge this limitation and temper the §6.1 conclusion.
  3. [§4.3–§4.5 (WM model parameters and attribution)] The computational WM model contains at least ten hand-set parameters (recency time constant T = 30 s, displacement weights alpha = 0.3, beta = 0.4, gamma = 0.3, binding weight lambda = 0.6, binding threshold theta = 0.5, utility weights W_I = 0.6 and W_R = 0.4, utility delivery threshold 0.75, duplicate-object similarity threshold 0.95, and the capacities 7 and 4), all justified as 'based on initial testing' without a sensitivity or ablation analysis. The user study compares the complete ProMemAssist system to the LLM baseline, so the observed effects cannot be attributed specifically to WM modeling, to the particular parameter values, or to the utility threshold that happens to produce fewer deliveries. The authors note in §7.3 that ablations are future work; for the present claims, at least a parameter sensitivity analysis or an ablation of the interference and displacement terms is needed before 'WM-informed filtering' can be identified as the causal mechanism.
  4. [§6.3 (Frustration and workload results)] With 12 participants and multiple correlated Likert dimensions, a single significant p = 0.043 on Q10 is weak evidence, and the adjacent measures—perceived interruptiveness (p = 0.158) and well-timed ratings—are not significant. The paper does not report a multiple-comparison correction or a pre-registered primary outcome. Since the frustration difference is the only significant survey effect and is used to support the WM-timing benefit, the authors should either report adjusted p-values (e.g., Benjamini–Hochberg) or reframe the survey findings as exploratory rather than confirmatory.
minor comments (6)
  1. [Throughout] The manuscript contains numerous typographical and grammatical errors, including 'environements' (§2.1), 'Necklack' (Reference [2]), 'curent' (§4.1), 'exisiting' (§5.2), 'serailized' (§4.3), and 'recall coverage' (§6.3). A careful proofreading pass is needed.
  2. [Figure 5] Figure 5 is referenced as showing overall system behavior, but the figure is not described in the text and appears to lack axis labels and legends that would let a reader understand what the bars represent. Add a clear textual description and make the figure self-contained.
  3. [§5.5 / Figure 6] The description of Q8 states '1 is Perfect and 7 is Failure', but the box-and-whisker plot description says anchors range from 1 (Strongly disagree, Very low) to 7 (Strongly agree, Very high). Verify that the anchor direction is consistent between the text and the figure for all adapted NASA-TLX items.
  4. [§5.1 (Participants)] Participants are described as research engineers or scientists familiar with wearable and mixed-reality devices, recruited from an internal pool. The potential impact of this specialist, technically sophisticated sample on generalizability is not discussed in the Limitations; a brief comment would be appropriate.
  5. [§7.2 (Limitations)] The limitations section notes that the system 'does not distinguish well between first-person and third-person perspectives' and can encode information from the experimenter's actions. Since the WM model is meant to represent the user's working memory, the paper should explain why this contamination does not differentially bias the two conditions, or report how often such third-person encodings occurred.
  6. [§4.5.2 (Interference cost)] The formula C_I = C'_I / |WM_same_modality| is undefined when |WM_same_modality| = 0. The paper should state the assumed handling for that case (e.g., C_I = 0), since an empty same-modality set is plausible during some task phases.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the WM-timing model is empirically evaluated and not defined by its outcome, though delivery-rate confounds and LLM-scored importance are validity caveats.

full rationale

The paper's derivation chain is: multimodal observations are encoded into WM items and episodes; recency, relevance, and importance are computed from CLIP embeddings and an LLM prompt; displacement and interference costs are defined by cosine similarity and capacity constants; a utility function combines these to make a delivery decision; and user engagement is measured independently by experimenter-coded reactions and surveys. At no point is the outcome variable (positive engagement, frustration, or qualitative preference) inserted into the model definitions, nor is any equation such that the predicted outcome equals the input by construction. The baseline LLM receives the same perceptual inputs and task prompts but lacks the WM state, so the comparison is not definitionally forced. The parameters alpha = 0.3, beta = 0.4, gamma = 0.3 and the 0.75 utility threshold are described as coming from initial testing, but the paper does not present these as predictions fitted to the engagement DV, so this is a tuning-transparency issue rather than a circular reduction. Two non-circular caveats should be noted: (1) Section 6.1 compares 130 delivered messages for ProMemAssist against 332 for the baseline, so the higher positive-engagement proportion could plausibly reflect delivery volume or experimenter non-blinding rather than WM modeling; (2) Section 4.6 lets the same LLM assign the importance scores that enter the utility value term, meaning the timing filter is partly self-scored by the message generator. Both are threats to evidential strength, not examples of the derivation reducing to its inputs, and the paper itself acknowledges the timing/content coupling in Section 7.2. The self-citations present, such as [24] and [55], are hardware descriptions or related-work references and are not load-bearing. The central claim is therefore not circular.

Assumptions & free parameters 9 free parameters · 6 assumptions · 1 invented entities

The timing decisions and the evaluation rest on a stack of hand-set parameters and strong mapping assumptions (perception equals working memory, CLIP similarity equals semantic interference, LLM score equals importance). The user study tests the assembled system, so the contribution of the WM model per se is not isolated from these choices.

free parameters (9)
  • Recency time constant T = 30 seconds
    Chosen as upper bound of 15-30 s WM retention from Peterson & Peterson (1959); used in Recency = 1 - t/T (Section 4.4.1).
  • Displacement composite weights alpha, beta, gamma = 0.3, 0.4, 0.3
    Set 'based on initial testing' (Section 4.4.2); no ablation or sensitivity analysis.
  • Binding weight lambda = 0.6
    Chosen to prioritize episode summary coherence (Section 4.4.3).
  • Binding similarity threshold theta = 0.5
    Default threshold for chunk binding (Section 4.4.3).
  • Utility weights W_I, W_R = 0.6, 0.4
    Tunable weights in utility function (Section 4.5); no derivation or fitting procedure reported.
  • Delivery utility threshold = 0.75
    Chosen 'from testing' (Section 4.5.3); affects which messages are delivered, deferred, or discarded.
  • Duplicate-object CLIP similarity threshold = 0.95
    Threshold to avoid re-encoding the same object (Section 4.4.2); no reported calibration.
  • Perception memory capacity = 7 items
    Based on Miller (1956) [48]; used to force displacement.
  • Episodic buffer capacity = 4 chunks
    Based on Cowan (2010) [22]; used to force chunk displacement.
assumptions (6)
  • domain assumption Working memory has the tripartite structure of Baddeley (visuospatial, phonological, episodic), and the episodic buffer can be derived from perception memory.
    Invoked in Section 4.3 to justify memory item types and the episodic buffer design.
  • domain assumption The user's working memory content is observable from egocentric visual objects and transcribed speech around the user.
    Section 4.2 encodes detected objects and speech as memory items; a key idealization for the whole approach.
  • domain assumption Recency decays linearly with a 30-second time constant, and CLIP cosine similarity measures semantic relevance and interference.
    Section 4.4.1 defines Recency = 1 - t/T with T=30 s; Sections 4.4.2 and 4.5.2 use CLIP similarity for relevance, displacement, and interference.
  • domain assumption LLM-generated importance scores from 0 to 1 are valid proxies for the cognitive value of memory items and assistance messages.
    Sections 4.4.1, 4.5.1 and Appendix A.1.5; the utility function and displacement depend on these scores.
  • ad hoc to paper The cost of an interruption is well modeled by the displacement of the lowest-score memory item plus the mean semantic dissimilarity of same-modality items.
    Section 4.5.2 introduces C_D and C_I with these operational definitions; no independent validation is provided.
  • ad hoc to paper The utility threshold 0.75 and weights (W_I=0.6, W_R=0.4, alpha=0.3, beta=0.4, gamma=0.3) are appropriate for the value/cost tradeoff.
    Sections 4.4.2 and 4.5.3 report these are set 'from testing' or 'by default'; no formal derivation or sensitivity analysis.
invented entities (1)
  • Computational working memory state (MemoryItem list and episodic buffer chunks)
    purpose: Latent representation of the user's current mental contents used to compute assistance timing utility.
    Introduced in Sections 4.3 to 4.5 as a proxy for the user's real working memory; no direct behavioral or neural validation is provided; only indirect evidence via user study outcomes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices." pith.science (2026). https://pith.science/paper/FTHFIXFB

@misc{pith2026250721378,
  author       = {Pith},
  title        = {Pith review of: ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FTHFIXFB}},
  note         = {Machine review of arXiv:2507.21378}
}
read the original abstract

Wearable AI systems aim to provide timely assistance in daily life, but existing approaches often rely on user initiation or predefined task knowledge, neglecting users' current mental states. We introduce ProMemAssist, a smart glasses system that models a user's working memory (WM) in real-time using multi-modal sensor signals. Grounded in cognitive theories of WM, our system represents perceived information as memory items and episodes with encoding mechanisms, such as displacement and interference. This WM model informs a timing predictor that balances the value of assistance with the cost of interruption. In a user study with 12 participants completing cognitively demanding tasks, ProMemAssist delivered more selective assistance and received higher engagement compared to an LLM baseline system. Qualitative feedback highlights the benefits of WM modeling for nuanced, context-sensitive support, offering design implications for more attentive and user-aware proactive agents.

Figures

Figures reproduced from arXiv: 2507.21378 by the authors.

Figure 1
Figure 1. ProMemAssist Workflow. ○A: Multimodal sensor input (visual and auditory) is captured from smart glasses. ○B : Perception memory encodes visual-spatial (e.g., “forks, bottle, banana”) and phonological (e.g., “There’s gonna be four of us for dinner”) signals, which are summarized into an episodic buffer describing the current task context. ○C : The assistance generator uses the working memory state to produce candidat… view at source ↗
Figure 2
Figure 2. Working Memory Displacement. As the user interacts with objects in the environment, ○A the system captures egocentric visual input as the user interacts with objects in the environment. ○B A new memory item (e.g., a banana) is encoded with a corresponding feature embedding. If the perception memory is already at capacity, the system calculates the recency, relevance, and importance scores for existing items. ○C The … view at source ↗
Figure 3
Figure 3. Working Memory Interference. Interference is com￾puted for each memory item in the same modality (e.g., visual or auditory) using cosine similarity. ○A A new proactive assistance message (e.g., “You might need more utensils for all guests”) is eval￾uated for delivery. ○B ProMemAssist identifies memory items in the perception memory that share the same modality—in this case, phonological. ○C For each overlapping memo… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Task Settings. Four task scenarios were used in the user study: Setting up a dining table, Organizing an office desk, Packing for a work trip, and Arranging a living room table. Each task has a fixed starting position of objects (task start), and participants are asked…
Figure 5
Figure 5. Figure 5: Overall system behavior from ProMemAssist (left) and the Baseline (right). [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Likert-scale responses comparing ProMemAssist and baseline conditions. Box-and-whisker plots for each question show participant ratings across the two conditions. The red dotted lines indicate mean values. Anchors ranged from 1 (Strongly disagree, Very low) to 7 (Stron…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 49 canonical work pages

  1. [1]

    The Cambridge Handbook of the Learning Sciences (2 ed.)

    2014. The Cambridge Handbook of the Learning Sciences (2 ed.). Cambridge University Press

    work page 2014

  2. [2]

    Friend: AI Necklack

    2025. Friend: AI Necklack. https://www.friend.com/wearable/index.html Re- trieved April, 2025

    work page 2025

  3. [3]

    Humane Ai Pin | See the World, Not Your Screen

    2025. Humane Ai Pin | See the World, Not Your Screen. | Humane. https: //humane.com/ Retrieved April, 2025

    work page 2025

  4. [4]

    Ray-Ban Meta Glasses

    2025. Ray-Ban Meta Glasses. https://www.ray-ban.com/rayban-meta-ai-glasses Retrieved April, 2025. ProMemAssist: Working Memory Modeling for Timely Proactive Assistance UIST ’25, September 28-October 1, 2025, Busan, Republic of Korea

    work page 2025

  5. [5]

    Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human- AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Asso...

    arXiv 2019

  6. [6]

    Riku Arakawa, Jill Fain Lehman, and Mayank Goel. 2024. PrISM-Q&A: Step- Aware Voice Assistant on a Smartwatch Enabled by Multimodal Procedure Track- ing and Large Language Models. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8, 4 (Nov. 2024), 180:1–180:26. https://doi.org/10.1145/3699759

    doi:10.1145/3699759 2024

  7. [7]

    Riku Arakawa, Hiromu Yakura, and Mayank Goel. 2024. PrISM-Observer: In- tervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24) . Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3654777.3676350

    arXiv 2024

  8. [8]

    Alan Baddeley. 2012. Working Memory: Theories, Models, and Controversies. Annual Review of Psychology 63, 1 (Jan. 2012), 1–29. https://doi.org/10.1146/ annurev-psych-120710-100422

    work page 2012

Show all 79 references
  1. [9]

    Seyed Ali Bahrainian and Fabio Crestani. 2017. Towards the Next Generation of Personal Assistants: Systems that Know When You Forget. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR ’17). Association for Computing Machinery, N...

    2017

  2. [10]

    Seyed Ali Bahrainian and Fabio Crestani. 2018. Augmentation of Human Memory: Anticipating Topics that Continue in the Next Meeting. InProceedings of the 2018 Conference on Human Information Interaction & Retrieval (New Brunswick, NJ, USA) (CHIIR ’18). Association for Computing...

    2018

  3. [11]

    Seyed Ali Bahrainian, Fattane Zarrinkalam, Ida Mele, and Fabio Crestani. 2019. Predicting the Topic of Your Next Query for Just-In-Time IR. In Advances in Information Retrieval, Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra (Eds.)....

    2019 doi

  4. [12]

    Brian P Bailey, Joseph A Konstan, and John V Carlis. 2000. Measuring the ef- fects of interruptions on task performance in the user interface. In Smc 2000 conference proceedings. 2000 ieee international conference on systems, man and cy- bernetics. ’cybernetics evolving to sys...

    2000

  5. [13]

    Jimmy Baraglia, Maya Cakmak, Yukie Nagai, Rajesh Rao, and Minoru Asada

  6. [14]

    Marc G Berman, John Jonides, and Richard L Lewis. 2009. In search of decay in verbal short-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition 35, 2 (2009), 317

    2009

  7. [15]

    Elise Bonnail, Wen-Jie Tseng, Mark Mcgill, Eric Lecolinet, Samuel Huron, and Jan Gugenheimer. 2023. Memory Manipulations in Extended Reality. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (April 2023), 1–20. https://doi.org/10.1145/3544548.358098...

    2023

  8. [16]

    Michael D. Byrne. 1996. A computational theory of working memory. In Confer- ence Companion on Human Factors in Computing Systems (CHI ’96) . Association for Computing Machinery, New York, NY, USA, 31–32. https://doi.org/10.1145/ 257089.257117

    1996

  9. [17]

    Runze Cai, Nuwan Janaka, Hyeongcheol Kim, Yang Chen, Shengdong Zhao, Yun Huang, and David Hsu. 2025. AiGet: Transforming Everyday Moments into Hidden Knowledge Discovery with AI Assistance on Smart Glasses. InProceedings of the 2025 CHI Conference on Human Factors in Computing...

    2025

  10. [18]

    Narae Cha, Auk Kim, Cheul Young Park, Soowon Kang, Mingyu Park, Jae-Gil Lee, Sangsu Lee, and Uichin Lee. 2020. Hello There! Is Now a Good Time to Talk? Opportune Moments for Proactive Interactions with Smart Speakers. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3 ...

    2020 doi

  11. [19]

    Samantha W. T. Chan, Shardul Sapkota, Rebecca Mathews, Haimo Zhang, and Suranga Nanayakkara. 2020. Prompto: Investigating Receptivity to Prompts Based on Cognitive Load from Memory Training Conversational Agent. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4 (2020), 1...

    2020

  12. [20]

    Kuan-Wen Chen, Yung-Ju Chang, and Liwei Chan. 2022. Predicting Opportune Moments to Deliver Notifications in Virtual Reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, ...

    2022

  13. [21]

    Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David Sontag, and Ameet Talwalkar. 2024. Need Help? Designing Proactive AI Assistants for Pro- gramming. (2024). https://doi.org/10.48550/ARXIV.2410.04596 Publisher: arXiv Version Number: 1

    2024 doi arXiv

  14. [22]

    Nelson Cowan. 2010. The Magical Mystery Four: How is Working Memory Capacity Limited, and Why? Current directions in psychological science 19, 1 (Feb. 2010), 51–57. https://doi.org/10.1177/0963721409359277

    2010 doi

  15. [23]

    Mary Czerwinski, Edward Cutrell, and Eric Horvitz. 2000. Instant messaging: Effects of relevance and timing. In People and computers XIV: Proceedings of HCI , Vol. 2. 71–76

    2000

  16. [24]

    Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Mered- ith, Cheng Peng, Chris Sweeney, Cole Wilson, Dan Barnes, Daniel DeTone, David Caruso, Derek Valleroy, Dinesh Ginjupalli, Dunc...

    2023 arXiv

  17. [25]

    Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psy- chology. Vol. 52. Elsevier, 139–183

    1988

  18. [26]

    Morgan Harvey, Marc Langheinrich, and Geoff Ward. 2016. Remembering through lifelogging: A survey of human memory augmentation.Pervasive and Mo- bile Computing 27 (April 2016), 14–26. https://doi.org/10.1016/j.pmcj.2015.12.002

    2016 doi

  19. [27]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. https://doi.org/10.1145...

    1999

  20. [28]

    Edward Cutrell Mary Czerwinski Eric Horvitz. 2001. Notification, disruption, and memory: Effects of messaging interruptions on memory and performance. In Human-Computer Interaction: INTERACT, Vol. 1. 263

    2001

  21. [29]

    Jiaxiong Hu, Jingya Guo, Ningjing Tang, Xiaojuan Ma, Yuan Yao, Changyuan Yang, and Yingqing Xu. 2024. Designing the Conversational Agent: Asking Follow-up Questions for Information Elicitation. Proceedings of the ACM on Human-Computer Interaction 8, CSCW1 (April 2024), 1–30. h...

    2024

  22. [30]

    Gaoping Huang, Xun Qian, Tianyi Wang, Fagun Patel, Maitreya Sreeram, Yuanzhi Cao, Karthik Ramani, and Alexander J. Quinn. 2021. AdapTutAR: An Adaptive Tutoring System for Machine Tasks in Augmented Reality. InProceedings of the 2021 CHI Conference on Human Factors in Computing...

    2021

  23. [31]

    Glenn Jocher and Jing Qiu. 2024. Ultralytics YOLO11 . https://github.com/ ultralytics/ultralytics

    2024

  24. [32]

    John E. Laird. 2001. It knows what you’re going to do: adding anticipation to a Quakebot. In Proceedings of the fifth international conference on Autonomous agents. ACM, Montreal Quebec Canada, 385–392. https://doi.org/10.1145/375735.376343

    2001

  25. [33]

    M. Lamming. 1994. The Design of a Human Memory Prosthesis. Comput. J. 37, 3 (March 1994), 153–163. https://doi.org/10.1093/comjnl/37.3.153

    1994 doi

  26. [34]

    Smeaton, Noel E

    Hyowon Lee, Alan F. Smeaton, Noel E. O’Connor, Gareth Jones, Michael Blighe, Daragh Byrne, Aiden Doherty, and Cathal Gurrin. 2008. Constructing a SenseCam visual diary as a media process. Multimedia Systems 14, 6 (Dec. 2008), 341–349. https://doi.org/10.1007/s00530-008-0129-x

    2008 doi

  27. [35]

    Lewis and Oğuz Osman Erdinç

    James R. Lewis and Oğuz Osman Erdinç. 2017. User experience rating scales with 7, 11, or 101 points: does it matter? Journal of Usability Studies archive 12 (2017), 73–91. https://api.semanticscholar.org/CorpusID:27640663

    2017

  28. [36]

    Turakhia, Sonia Castelo Quispe, Dong Li, Leslie Welch, Claudio Silva, and Jing Qian

    Chenyi Li, Guande Wu, Gromit Yeuk-Yin Chan, Dishita G. Turakhia, Sonia Castelo Quispe, Dong Li, Leslie Welch, Claudio Silva, and Jing Qian. 2024. Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling. https: //doi.org/10.48550/arXiv.2410.16668 arXiv...

    2024 doi arXiv

  29. [37]

    Jiahao Nick Li, Yan Xu, Tovi Grossman, Stephanie Santosa, and Michelle Li. 2024. OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMs. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24) . A...

    2024

  30. [38]

    Jiahao Nick Li, Zhuohao Jerry Zhang, and Jiaju Ma. 2024. OmniQuery: Contex- tually Augmenting Captured Multimodal Memory to Enable Personal Question Answering. https://doi.org/10.48550/arXiv.2409.08250 arXiv:2409.08250

    2024 doi arXiv

  31. [39]

    Hong, and Jeffrey Nichols

    Tianshi Li, Julia Katherine Haines, Miguel Flores Ruiz De Eguino, Jason I. Hong, and Jeffrey Nichols. 2023. Alert Now or Never: Understanding and Predicting UIST ’25, September 28-October 1, 2025, Busan, Republic of Korea Pu et al. Notification Preferences of Smartphone Users....

    2023 doi

  32. [40]

    Tianjian Liu, Hongzheng Zhao, Yuheng Liu, Xingbo Wang, and Zhenhui Peng

  33. [41]

    Steven J Luck and Edward K Vogel. 2013. Visual working memory capacity: from psychophysics and neurobiology to individual differences. Trends in cognitive sciences 17, 8 (2013), 391–400

    2013

  34. [42]

    Zhanat Makhataeva, Tolegen Akhmetov, and Huseyin Atakan Varol. 2023. Augmented-Reality-Based Human Memory Enhancement Using Artificial Intelli- gence. IEEE Transactions on Human-Machine Systems 53, 6 (Dec. 2023), 1048–1060. https://doi.org/10.1109/THMS.2023.3307397 Conference ...

    2023

  35. [43]

    Bruno Massoni Sguerra, Amine Benamara, Samuel Benveniste, and Pierre Jou- velot. 2018. Adapting Human-Computer Interfaces to Working Memory Limita- tions Using MATCHS. In2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 1309–1314. https://doi.org/10.11...

    2018

  36. [44]

    An Unscented Hound for Working Memory

    Bruno Massoni Sguerra and Pierre Jouvelot. 2019. "An Unscented Hound for Working Memory" and the Cognitive Adaptation of User Interfaces. InProceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization (Larnaca, Cyprus) (UMAP ’19). Association for Com...

    2019

  37. [45]

    Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2011. Ambient help. In Proceedings of the SIGCHI Conference on Human Factors in Comput- ing Systems (<conf-loc>, <city>Vancouver</city>, <state>BC</state>, <coun- try>Canada</country>, </conf-loc>) (CHI ’11). Association f...

    2011 doi

  38. [46]

    Tom Mercer and Denis McKeown. 2014. Decay uncovered in nonverbal short- term memory. Psychonomic bulletin & review 21 (2014), 128–135

    2014

  39. [47]

    Mihale-Wilson, Adrian Hawlitschek, Florian Giger, Florian Müller, Oliver Hinz, and Max Mühlhäuser

    Christian Meurisch, Cristina A. Mihale-Wilson, Adrian Hawlitschek, Florian Giger, Florian Müller, Oliver Hinz, and Max Mühlhäuser. 2020. Exploring User Expectations of Proactive AI Systems. Proc. ACM Interact. Mob. Wearable Ubiqui- tous Technol. 4, 4 (Dec. 2020), 146:1–146:22....

    2020 doi

  40. [48]

    George A. Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review 63, 2 (March 1956), 81–97. https://doi.org/10.1037/h0043158

    1956 doi

  41. [49]

    Klaus Oberauer. 2009. Interference between storage and processing in working memory: Feature overwriting, not similarity-based competition. Memory & cognition 37 (2009), 346–357

    2009

  42. [50]

    O’Reilly and Michael J

    Randall C. O’Reilly and Michael J. Frank. 2006. Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Gan- glia. Neural Computation 18, 2 (Feb. 2006), 283–328. https://doi.org/10.1162/ 089976606775093909

    2006

  43. [51]

    Bernstein

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simu- lacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’2...

    2023

  44. [52]

    Zhenhui Peng, Yunhwan Kwon, Jiaan Lu, Ziming Wu, and Xiaojuan Ma. 2019. Design and Evaluation of Service Robot’s Proactivity in Decision-Making Support Process. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (May 2019), 1–13. https://doi.org/10.11...

    2019

  45. [53]

    Peterson and Margaret Jean Peterson

    Lloyd R. Peterson and Margaret Jean Peterson. 1959. Short-term retention of individual verbal items. Journal of experimental psychology 58 (1959), 193–8. https://api.semanticscholar.org/CorpusID:40600538

    1959

  46. [54]

    Preston and Andrew M

    Carolyn C. Preston and Andrew M. Colman. 2000. Optimal number of re- sponse categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta psychologica 104 1 (2000), 1–15. https: //api.semanticscholar.org/CorpusID:14372956

    2000

  47. [55]

    Kevin Pu, Daniel Lazaro, Ian Arawjo, Haijun Xia, Ziang Xiao, Tovi Grossman, and Yan Chen. 2025. Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support. ArXiv abs/2502.18658 (2025). https://api.semanticscholar.org/Corpus...

    2025 arXiv

  48. [56]

    Mashfiqui Rabbi, Min Hane Aung, Mi Zhang, and Tanzeem Choudhury. 2015. MyBehavior: automatic personalized health feedback from user behaviors and preferences using smartphones. InProceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (...

    2015

  49. [57]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.000...

    2021 arXiv

  50. [58]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2212.04356 [eess.AS] https://arxiv.org/abs/2212.04356

    2022 arXiv

  51. [59]

    Judith Spencer Reitman. 1971. Mechanisms of forgetting in short-term memory. Cognitive Psychology 2, 2 (1971), 185–195

    1971

  52. [60]

    Judith S Reitman. 1974. Without surreptitious rehearsal, information in short- term memory decay. Journal of verbal learning and verbal behavior 13, 4 (1974), 365–377

    1974

  53. [61]

    Brad Rhodes and Thad Starner. 1996. Remembrance Agent: A Continuously Run- ning Automated Information Retrieval System. https://www.semanticscholar. org/paper/Remembrance-Agent%3A-A-Continuously-Running-Automated- Rhodes-Starner/24d7e19ea677dc052df91e7a226187a37ca0c3c2

    1996

  54. [62]

    Mahbubur Rahman, Rummana Bari, Syed Monowar Hossain, and Santosh Kumar

    Hillol Sarker, Moushumi Sharmin, Amin Ahsan Ali, Md. Mahbubur Rahman, Rummana Bari, Syed Monowar Hossain, and Santosh Kumar. 2014. Assessing the availability of users to engage in just-in-time intervention in the natural environment. Proceedings of the 2014 ACM International J...

    2014

  55. [63]

    Junxiao Shen, John Dudley, and Per Ola Kristensson. 2024. Encode-Store-Retrieve: Augmenting Human Memory through Language-Encoded Egocentric Perception. https://doi.org/10.48550/arXiv.2308.05822 arXiv:2308.05822

    2024 doi arXiv

  56. [64]

    John Sweller. 2011. CHAPTER TWO - Cognitive Load Theory. Psychology of Learning and Motivation, Vol. 55. Academic Press, 37–76. https://doi.org/10. 1016/B978-0-12-387691-1.00002-8

    2011

  57. [65]

    Mirko Thalmann, Alessandra S Souza, and Klaus Oberauer. 2019. How does chunking help working memory? Journal of Experimental Psychology: Learning, Memory, and Cognition 45, 1 (2019), 37

    2019

  58. [66]

    Quang-Linh Tran, Binh Nguyen, Gareth J. F. Jones, and Cathal Gurrin. 2024. MemoriQA: A Question-Answering Lifelog Dataset. InProceedings of the 1st ACM Workshop on AI-Powered Q&A Systems for Multimedia (AIQAM ’24) . Association for Computing Machinery, New York, NY, USA, 7–12....

    2024

  59. [67]

    Iskander Umarov and Maxim Mozgovoy. 2012. Believable and Effective AI Agents in Virtual Worlds: Current State and Future Perspectives. International Journal of Gaming and Computer-Mediated Simulations 4, 2 (April 2012), 37–59. https: //doi.org/10.4018/jgcms.2012040103

    2012 doi

  60. [68]

    Marlieke TR van Kesteren, Paul Rignanese, Pierre G Gianferrara, Lydia Krabben- dam, and Martijn Meeter. 2020. Congruency and reactivation aid memory inte- gration through reinstatement of prior knowledge. Scientific Reports 10, 1 (2020), 4776

    2020

  61. [69]

    Marlieke TR Van Kesteren, Dirk J Ruiter, Guillén Fernández, and Richard N Henson. 2012. How schema and novelty augment memory formation. Trends in neurosciences 35, 4 (2012), 211–219

    2012

  62. [70]

    Geoffrey F Woodman and Steven J Luck. 2010. Why is information displaced from visual working memory during visual search? Visual Cognition 18, 2 (2010), 275–295

    2010

  63. [71]

    Meng-Hsin Wu, Su-Fang Yeh, XiJing Chang, and Yung-Ju Chang. 2021. Exploring Users’ Preferences for Chatbot’s Guidance Type and Timing. In Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing (CSCW ’21 Companion) . Association...

    2021

  64. [72]

    Erik Wästlund. 2007. Experimental Studies of Human-Computer Interaction : Working memory and mental workload in complex cognition. https:// www.semanticscholar.org/paper/Experimental-Studies-of-Human-Computer- Interaction-W%C3%A4stlund/3ec3d53b895d67e7f5cf66ac89bda273edec3c9d

    2007

  65. [73]

    Lam, and James A

    Jackie Junrui Yang, Leping Qiu, Emmanuel Angel Corona-Moreno, Louisa Shi, Hung Bui, Monica S. Lam, and James A. Landay. 2024. AMMA: Adaptive Multi- modal Assistants Through Automated State Tracking and User Model-Directed Guidance Planning. 2024 IEEE Conference Virtual Reality...

    2024

  66. [74]

    dining

    Wazeer Deen Zulfikar, Samantha Chan, and Pattie Maes. 2024. Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory Augmentation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24) . Association for Computi...

    2024

  67. [77]

    Example : The user is holding a hot cup of coffee near a child

    If the memory is task - relevant and requires immediate attention , it 's very important and should have a score close to 1. Example : The user is holding a hot cup of coffee near a child . Importance score : 0.9

  68. [78]

    Example : The user hears a notification from their phone while working on a project

    If the memory is task - irrelevant , it should have a low importance score close to 0. Example : The user hears a notification from their phone while working on a project . Importance score : 0.1

  69. [79]

    per ce pt io n_m em or y

    If the memory is indirectly related to the task , it should have a moderate to high importance score . Example : The user is packing for a trip , and is reminded of the weather . Importance score : 0.65 Only output the JSON object : { " per ce pt io n_m em or y ": [ importance...

    2025

  70. [2016]

    2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (March 2016), 67–74

    Initiative in robot assistance during collaborative task execution. 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (March 2016), 67–74. https://doi.org/10.1109/HRI.2016.7451735 Conference Name: 2016 11th ACM/IEEE International Conference on Human-...

    2016

  71. [2024]

    In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24)

    ComPeer: A Generative Conversational Agent for Proactive Peer Support. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24) . Association for Computing Machinery, New York, NY, USA, 1–22. https://doi.org/10.1145/3654777.3676430

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.