pith. sign in

arxiv: 2606.17767 · v1 · pith:JT5YTSM5new · submitted 2026-06-16 · 💻 cs.HC · cs.AI

Talking to Your Data: Exploring Embodied Conversation as an Interface for Personal Health Reflection

Nikola Kovacevic , Bastien Husler , Di Zhuang , Rafael Wampfler , Barbara Solenthaler This is my paper

Pith reviewed 2026-06-26 23:18 UTC · model grok-4.3

classification 💻 cs.HC cs.AI
keywords embodied conversational agentspersonal health datawearable datadata sensemakingdashboard interfacesvirtual agentshealth reflection
0
0 comments X

The pith

Embodied conversation with a virtual agent leads to higher perceived understanding and more specific health actions than viewing dashboard charts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tests an alternative to dashboard charts for personal health data by letting users talk with an embodied virtual character about their wearable metrics. The system uses two agents: one extracts statistics and trends from the data, while the other voices those findings in dialogue without offering medical advice. A within-subject study with five participants adopting personas showed that the conversational format produced stronger feelings of understanding, more concrete action ideas, and a move from passive looking to active interpretation. The work matters because wearables generate continuous data that most users find hard to turn into meaningful insights or decisions.

Core claim

The paper claims that embodied conversational reflection on personal health data, using a dual-agent system in which an Observer extracts descriptive statistics and temporal trends while a Presenter communicates findings through spoken statistics with a Unity-based character, produces higher perceived understanding, more specific generated actions, and a cognitive shift from passive viewing to active sensemaking compared with traditional dashboard exploration, as measured in a within-subject simulated-self study with N=5 participants adopting personas from the LifeSnaps dataset.

What carries the argument

Dual-agent embodied conversational system with an Observer agent for statistics and trends and a Presenter agent for objective spoken narrative, implemented in Unity to isolate the effect of dialogue modality.

If this is right

  • Users generate more specific health-related actions after conversational reflection than after dashboard exploration.
  • The conversational format produces a measurable cognitive shift toward active sensemaking rather than passive viewing.
  • The dual-agent design isolates interaction modality effects by deliberately avoiding clinical advice.
  • The approach supplies a reusable design pattern for generating objective health data narratives.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Integrating the system directly with live wearable feeds rather than pre-processed personas could reveal whether personal ownership strengthens the observed effects.
  • Users with lower data literacy might benefit more from the conversational format, suggesting a possible equity angle for future tests.
  • Long-term behavior change could be tracked by measuring whether the specific actions generated in dialogue are actually followed.

Load-bearing premise

A within-subject study with five participants using simulated personas from a dataset can validly measure real differences in how people interpret their own personal health data.

What would settle it

A follow-up study with participants reflecting on their own real wearable data that finds no difference in perceived understanding or action specificity between the embodied conversational interface and the dashboard would falsify the central claim.

Figures

Figures reproduced from arXiv: 2606.17767 by Barbara Solenthaler, Bastien Husler, Di Zhuang, Nikola Kovacevic, Rafael Wampfler.

Figure 1
Figure 1. Figure 1: The Dual-Agent Architecture and Unity Integration. The system employs a decoupled design to ensure data-driven narrative reflection. The Observer Agent utilizes rule-based feature extraction to preprocess longitudinal health traces into a canonical Insight JSON containing both raw metrics and natural-language insights. The Presenter Agent consumes this JSON to ground its responses, functioning as a "narrat… view at source ↗
Figure 2
Figure 2. Figure 2: Screenshot of the User Interface. The left screen shows the health data charts alongside persona information and health goals specific to the user study. The right screen shows the embodied conversational agent during a conversation. and "spoken statistics" (e.g., using weekdays like "last Wednesday") rather than raw dates to maintain a natural conversational flow. 2.4. User Interaction Users interact with… view at source ↗
Figure 3
Figure 3. Figure 3: Quantitative evaluation of Dashboard vs. Conversational Agent. While the Agent reduces effort and increases actionability, users default to the Dashboard for initial data exploration. 4.6. Design Implications and Limitations Based on these findings, we propose that the separation of the Observer (data analysis) and Presenter (dialogue) remains essential also for health interface safety. Grounding the inter… view at source ↗
read the original abstract

Personal health data from wearables are typically presented through dashboards of charts and summary statistics, requiring users to actively interpret patterns and implications. We explore an alternative interaction paradigm: engaging with personal health data through an embodied conversational agent that facilitates objective data reflection in dialogue with the user. We present a system that combines lightweight preprocessing of wearable data with a Unity-based embodied character. Internally, the system follows a dual-agent design in which an Observer agent extracts descriptive statistics and temporal trends, and a Presenter agent communicates these findings through "spoken statistics," intentionally refraining from clinical advice to isolate the impact of the interaction modality. We evaluate this approach through a simulated-self user study (N=5) using a within-subject design. Participants adopted health personas and goals derived from the LifeSnaps dataset to compare traditional dashboard exploration with embodied conversational reflection. Our evaluation focuses on perceived understanding, the specificity of generated actions, and the cognitive shift from passive viewing to active sensemaking. The paper contributes a functional prototype, a design pattern for objective health data narrative generation, and early empirical insights into how embodiment affects the interpretation of personal health metrics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript describes a Unity-based embodied conversational agent system for personal health data reflection from wearables. It uses a dual-agent architecture (Observer for extracting descriptive statistics and temporal trends; Presenter for communicating via 'spoken statistics' without clinical advice) and evaluates the approach in a within-subject study (N=5) where participants adopted LifeSnaps-derived personas to compare embodied conversation against traditional dashboard exploration. The evaluation targets perceived understanding, specificity of generated actions, and cognitive shift from passive viewing to active sensemaking. Contributions include a functional prototype, a design pattern for objective health data narrative generation, and early empirical insights.

Significance. If substantiated, the work offers a novel interaction paradigm for personal informatics that could reduce the interpretive burden of dashboard-based health data. The prototype and narrative generation pattern represent concrete, reusable contributions to HCI. The small-scale simulated study, however, provides only preliminary evidence whose generalizability is constrained by sample size and role-play design.

major comments (2)
  1. [Abstract] Abstract / Evaluation description: The central claims of higher perceived understanding, more specific actions, and a cognitive shift rest on a within-subject comparison with N=5 simulated-self participants. No statistical results, effect sizes, power analysis, or explicit controls for demand characteristics and persona-adoption fidelity are supplied, leaving the comparative outcomes unsupported.
  2. [Evaluation] Evaluation section: The simulated-self design using LifeSnaps personas directly threatens validity for claims about real personal data sensemaking, because participants lack personal stakes; this confound is load-bearing for the three measured outcomes yet receives no qualitative safeguards or discussion.
minor comments (1)
  1. [Abstract] Abstract: The evaluation focus is stated but no summary of actual findings (even directional) is provided; adding a brief results clause would improve completeness.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. The evaluation is exploratory and small-scale by design, and we agree that the manuscript should more explicitly frame the results as preliminary insights rather than supported comparative outcomes. We address each major comment below and will revise the abstract, evaluation section, and limitations discussion accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract / Evaluation description: The central claims of higher perceived understanding, more specific actions, and a cognitive shift rest on a within-subject comparison with N=5 simulated-self participants. No statistical results, effect sizes, power analysis, or explicit controls for demand characteristics and persona-adoption fidelity are supplied, leaving the comparative outcomes unsupported.

    Authors: We agree that N=5 precludes meaningful statistical analysis, effect sizes, or power calculations, and the manuscript does not report any such results. The evaluation is presented as a within-subject simulated-self study yielding qualitative observations rather than statistically supported claims. To address the concern, we will revise the abstract to remove any phrasing that could imply comparative support and will add explicit language in the evaluation section stating that outcomes are exploratory and not statistically validated. We will also note the absence of controls for demand characteristics and persona fidelity as a limitation. revision: yes

  2. Referee: [Evaluation] Evaluation section: The simulated-self design using LifeSnaps personas directly threatens validity for claims about real personal data sensemaking, because participants lack personal stakes; this confound is load-bearing for the three measured outcomes yet receives no qualitative safeguards or discussion.

    Authors: The simulated-self approach was chosen to permit a controlled within-subject comparison while sidestepping privacy and ethical issues with real personal health data. We acknowledge that the absence of personal stakes is a core limitation that weakens claims about authentic sensemaking and that the paper currently provides insufficient discussion of this issue. We will expand the evaluation and limitations sections to discuss this confound explicitly, its potential effects on perceived understanding, action specificity, and cognitive shift, and the lack of additional qualitative safeguards beyond the described persona briefing. Future work with actual users will be highlighted as necessary. revision: partial

Circularity Check

0 steps flagged

No significant circularity; empirical system + study paper

full rationale

The paper contains no equations, derivations, fitted parameters, or mathematical claims. It describes a prototype system (dual-agent Observer/Presenter design) and reports results from a within-subject user study (N=5 simulated-self personas). The evaluation outcomes (perceived understanding, action specificity, cognitive shift) are measured directly from participant responses rather than derived from any self-citation chain, ansatz, or input data by construction. No load-bearing uniqueness theorems or renamed empirical patterns appear. The derivation chain is therefore self-contained and non-circular.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No mathematical model, free parameters, or invented physical entities; the work rests on standard HCI evaluation practices and the LifeSnaps dataset.

pith-pipeline@v0.9.1-grok · 5741 in / 1002 out tokens · 38392 ms · 2026-06-26T23:18:07.810837+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 4 canonical work pages

  1. [1]

    Demiris, S

    G. Demiris, S. J. Iribarren, K. Sward, S. Lee, R. Yang, Consumer health informatics: Past, present, and future, Yearbook of Medical Informatics 25 (2016) S42–S47

    2016

  2. [2]

    Demiris, Y

    G. Demiris, Y. Choi, K. J. Chun, Personal health informatics: New tools and roles for health care, International Journal of Medical Informatics 166 (2022) 104832

    2022

  3. [3]

    A. M. Lai, J. L. Warner, S. Luhanga, H.-C. Kum, C. Condit, Present and future trends in consumer health informatics and patient-generated health data, Yearbook of Medical Informatics 26 (2017) 38–43

    2017

  4. [4]

    M. R. Turchioe, D. A. Asch, A. B. Troxel, H. J. Sweitzer, R. R. Townsend, R. M. Merchant, A systematic review of patient-facing visualizations of personal health data, Applied Clinical Informatics 10 (2019) 47–63

    2019

  5. [5]

    L. K. Chan, et al., The shape of mobile health: A systematic review of health visualization on mobile devices, Journal of the American Medical Informatics Association 31 (2024) 275–289

    2024

  6. [6]

    I. Li, A. K. Dey, J. Forlizzi, A stage-based model of personal informatics systems, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2010, pp. 557–566

    2010

  7. [7]

    Consolvo, P

    S. Consolvo, P. Klasnja, D. W. McDonald, D. Avrahami, J. Froehlich, L. LeGrand, R. Libby, K. Mosher, J. A. Landay, Activity sensing in the wild: A field trial of ubifit garden, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2008, pp. 1797–1806

    2008

  8. [8]

    D. A. Epstein, A. Ping, J. Fogarty, S. A. Munson, Beyond abandonment to next steps: Understanding and designing for life after personal informatics tool use, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2015, pp. 1105–1114

    2015

  9. [9]

    Y. Kim, X. Xu, D. McDuff, C. Breazeal, H. W. Park, Health-llm: Large language models for health prediction via wearable sensor data, in: Proceedings of the fifth Conference on Health, Inference, and Learning, volume 248 ofProceedings of Machine Learning Research, PMLR, 2024, pp. 522–539

    2024

  10. [10]

    Cosentino, A

    J. Cosentino, A. Belyaeva, X. Liu, N. A. Furlotte, Z. Yang, C. Lee, E. Schenck, Y. Patel, J. Cui, L. D. Schneider, R. Bryant, R. G. Gomes, A. Jiang, R. Lee, Y. Liu, J. Perez, J. K. Rogers, C. Speed, S. Tailor, M. Walker, J. Yu, T. Althoff, C. Heneghan, J. Hernandez, M. Malhotra, L. Stern, Y. Matias, G. S. Corrado, S. Patel, S. Shetty, J. Zhan, S. Prabhaka...

    work page doi:10.48550/arxiv.2406.06474

  11. [11]

    X. Wang, J. Griffith, D. A. Adler, J. Castillo, T. Choudhury, F. Wang, Exploring personalized health support through data-driven, theory-guided llms: A case study in sleep health, in: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, ACM, 2025, pp. 1–15. doi:10.1145/3706598.3713852

    work page doi:10.1145/3706598.3713852 2025

  12. [12]

    C. M. Fang, V. Danry, N. Whitmore, A. Bao, A. Hutchison, C. Pierce, P. Maes, Physiollm: Supporting personalized health insights with wearables and large language models, in: 2024 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), IEEE, 2024, pp. 1–8. doi: 10. 1109/bhi62660.2024.10913781

    arXiv 2024

  13. [13]

    Cassell, J

    J. Cassell, J. Sullivan, S. Prevost, E. Churchill, More than just a pretty face: Conversational protocols and the role of embodiment in hci, Communications of the ACM 43 (2000) 70–84

    2000

  14. [14]

    Paiva, I

    A. Paiva, I. Leite, H. Boukricha, I. Wachsmuth, Empathy in virtual agents and robots: A survey, in: ACM Transactions on Interactive Intelligent Systems (TiiS), volume 7, 2017, pp. 1–40

    2017

  15. [15]

    Provoost, H

    S. Provoost, H. M. Lau, J. Ruwaard, H. Riper, Embodied conversational agents in clinical psychology: A scoping review, Journal of Medical Internet Research 19 (2017) e151

    2017

  16. [16]

    ter Stal, L

    S. ter Stal, L. L. Kramer, M. Tabak, H. op den Akker, H. Hermens, Design features of embodied conversational agents in ehealth applications: Scoping review, Journal of Medical Internet Research 22 (2020) e18839

    2020

  17. [17]

    T. W. Bickmore, D. Schulman, C. L. Sidner, A reusable framework for health counseling dialogue systems based on a behavioral medicine ontology, Journal of Biomedical Informatics 44 (2011) 183–197. doi:https://doi.org/10.1016/j.jbi.2010.12.006

    work page doi:10.1016/j.jbi.2010.12.006 2011

  18. [18]

    C. Nass, J. Steuer, E. R. Tauber, Computers are social actors, in: Proceedings of the SIGCHI conference on Human factors in computing systems, 1994, pp. 72–78

    1994

  19. [19]

    E. K. Choe, B. Lee, S. Munson, W. Pratt, J. A. Kientz, Understanding quantified-selfers’ practices in collecting and exploring personal data, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), ACM, 2014, pp. 1143–1152

    2014

  20. [20]

    Pirolli, S

    P. Pirolli, S. Card, The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis, in: Proceedings of the International Conference on Intelligence Analysis, volume 5, 2005, pp. 2–4

    2005

  21. [21]

    H. Lee, S. J. Joo, C. Kim, J. Jang, D. Kim, K.-W. On, M. Seo, How well do large language models truly ground?, in: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Association for Computational Linguistics, 2024, pp. 2437–2465. doi: 10.18...

    work page doi:10.18653/v1/ 2024

  22. [22]

    Yfantidou, C

    S. Yfantidou, C. Karagianni, S. Efstathiou, A. Vakali, J. Palotti, D. P. Giakatos, T. Marchioro, A. Kazlouski, E. Ferrari, v. Girdzijauskas, Lifesnaps, a 4-month multi-modal dataset captur- ing unobtrusive snapshots of our lives in the wild, Scientific Data 9 (2022). doi: 10.1038/ s41597-022-01764-x

    2022