Pith. sign in

REVIEW 4 major objections 6 minor 25 references

"We need to avail ourselves of GenAI to enhance knowledge distribution": Empowering Older Adults through GenAI Literacy

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Litti, a chatbot built to teach generative AI to older adults, produces a positive learning experience and a positive but statistically non-significant shift in self-reported AI literacy; trust and safety perceptions stay flat.

desk verdict Useful qualitative pilot on GenAI literacy chatbots for older adults, but the quantitative trend is circular and internally contradictory. read the letter →

arxiv 2506.06225 v1 pith:GWQ7RR2K submitted 2025-06-06 cs.HC cs.AI

Eunhye Grace Ko , Shaini Nanayakkara , Earl W. Huff Jr This is my paper
classification cs.HCcs.AI
keywords generativeAIliteracyolderadultseducationalchatbotsafetyandethicstrustinmixedmethodsstudyassistedliving
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a purpose-built generative AI chatbot can teach adults aged 75 and older what generative AI is, how to use it, and how to recognize its risks. Twelve residents of a senior assisted-living center spent about an hour with Litti, a chatbot whose guided tasks cover general search, personal assistance, health questions, emotional support, safety, and ethics. Pre- and post-surveys built from an existing AI-literacy questionnaire showed a positive but not statistically significant trend: average self-reported literacy rose 0.55 points on a 5-point scale. Interviews showed participants enjoyed the experience and wanted to learn more, yet trust and safety concerns remained largely unchanged. If the pattern holds, chat-based instruction is a workable entry point for GenAI literacy, but it is not, by itself, a trust-building or safety intervention.

What carries the argument

The machinery is Litti, a chatbot whose large-language-model responses are shaped by a prompt that fixes its persona as an empathetic AI literacy instructor and sequences six tasks: explaining generative AI, general search, personal assistance, health information seeking, emotional support, and safety and ethics. Around this sits an adapted pre/post questionnaire drawn from a published multi-subscale AI-literacy instrument; the authors prioritized the sub-scales for knowing about AI, detecting AI, and AI ethics, and built the chatbot's tasks from those same sub-scales. The prompt and the aligned questionnaire are what carry the argument: every quantitative claim is a comparison of the same instrument before and after one hour with the bot.

What would settle it

Run a larger pre-registered study with a no-intervention control group and a non-interactive video condition using the same adapted questionnaire; if the chatbot condition does not outperform the video on post-test gain, or if the no-intervention group shows the same gain, the claim that the chatbot drives literacy improvement is falsified. A simpler check is to administer the questionnaire twice a week apart with no intervention; a large practice effect would undermine the measured trend.

Watch

Extended reading notes

Core claim

The paper's central claim is that an interactive chatbot is a viable format for introducing generative AI to older adults, and that a single guided session can move self-reported AI literacy in the positive direction. On the paper's own evidence, the claim is supported only as a trend: 12 participants aged 75 to 90, with a mean age of 81.1, showed an average increase of 0.55 points (SD 0.72) on a 5-point AI-literacy scale after interacting with Litti, with sub-scale increases of 0.72 for knowledge, 0.47 for ethics, and 0.37 for AI detection, none reaching statistical significance. Qualitative focus-group data show broad enthusiasm for the learning experience and a strong desire for more education, alongside persistent skepticism about privacy, scams, and the trustworthiness of AI outputs. The paper therefore establishes, provisionally, that this demographic will engage with chatbot-facilitated GenAI education; it does not establish that such education measurably improves literacy, trust, or safety.

Load-bearing premise

The load-bearing premise is that the adapted AI-literacy questionnaire measures real change in adults aged 75 and older, so that the 0.55-point gain reflects learning rather than practice on questions similar to the tasks.

Editorial extensions

If this is right

  • If the positive trend is real, a single one-hour chat session is enough to nudge self-reported AI literacy upward in this age group, with the largest gains appearing in participants who started with the lowest scores.
  • Chatbot-based instruction can be run in an assisted-living setting with laptops and about one hour of supervision, making it a practical low-cost format to deploy more widely.
  • Because trust and safety perceptions did not improve, AI literacy curricula for older adults need separate, content-rich safety modules and explicit trust-building activities rather than relying on hands-on use alone.
  • A formal between-subjects trial comparing Litti against a website or video covering the same material would be the natural next test of whether interactivity, rather than content exposure, drives the effect.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The observed effect size, mean gain 0.55 points with SD 0.72, is large enough that a replication with a few dozen participants might reach statistical significance; a pre-registered power calculation based on these numbers would settle whether the trend is worth pursuing.
  • The inverse relation between pre-test score and gain, if it survives replication, suggests chatbot literacy education functions mainly as a leveling tool for novices; the paper only reports this pattern descriptively.
  • A natural extension is to test Litti against a non-interactive version with identical content; if gains match, the chatbot's interactivity is not the active ingredient and simpler static tutorials would suffice.
  • Trust may be harder to move than knowledge because the bot's own polished, human-like output can sharpen concerns about deception; one participant's reaction in the focus group illustrates this, and a future measure of trust after safety-specific modules could test it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper reports a mixed-methods study of Litti, a GenAI literacy chatbot, with 12 older adults (mean age 81.1) in a single one-hour session. Participants completed pre/post surveys adapted from the MAILS framework, interacted with the chatbot through five to six tasks, and joined focus group interviews. The paper claims a positive qualitative learning experience and a non-significant positive trend in AI literacy (an average increase of 0.55 points on a 5-point scale, SD=0.72), while also reporting that the intervention did not significantly change trust or safety perceptions. The contribution is framed as an exploratory case study with design recommendations for AI literacy education for older adults.

Significance. The qualitative portion of the study is a useful exploratory contribution: it documents varied prior familiarity with GenAI among adults over 75, highlights trust and safety as persistent concerns, and offers concrete, participant-grounded design suggestions such as larger fonts, multimodal input, and familiar task scenarios. The detailed description of the Litti intervention (built on Claude and FlowXO) supports replication, and the explicit limitations paragraph is a strength. However, the quantitative evidence is not load-bearing as reported. The abstract's 'not statistically significant' statement is internally contradicted by the reported mean and standard deviation, the outcome measure is aligned with the intervention content (a teaching-to-the-test risk), and the adapted MAILS instrument is not validated for this population. The paper's significance therefore rests primarily on the qualitative findings and on the suggestion of a viable chatbot format for future, better-controlled studies, rather than on the claimed quantitative trend.

major comments (4)
  1. [Abstract and Section 5.2] The printed quantitative results are internally inconsistent. With n=12 and a reported average increase of 0.55 points (SD=0.72) for the pre/post difference, a paired t-test yields t ≈ 2.65 with df=11 and a two-tailed p ≈ 0.023, which is significant at the conventional .05 level. The abstract and Section 5.2 state that the results were 'not statistically significant' and frame the change as a 'positive trend,' but they neither report this test nor explain which test was actually used. The authors must reanalyze and re-report these numbers, or provide a clear justification for a different test that yields non-significance; as it stands, the conclusion and the data contradict each other.
  2. [Section 3 and Section 4.3] The outcome measure is not independent of the intervention content. Section 3 states that Litti's tasks were 'designed around' the MAILS sub-scales 'Know & AI,' 'Detect AI,' and 'AI Ethics,' while Section 4.3 states that the pre/post questionnaire was 'developed adopting' the same MAILS framework. The measured gains are therefore plausibly attributable to practice effects, item familiarity, or teaching-to-the-test rather than to generalizable AI literacy learning. The paper should explicitly discuss this alignment as a threat to construct validity; the current text does not acknowledge this circularity. A transfer measure, a control condition, or a delayed post-test would partly mitigate the concern, but none is present.
  3. [Section 5.2] The claim that the 0.55-point overall increase is 'meaningful' is unsupported by any inferential statistics. The section reports only means and standard deviations for the overall score and the three sub-scales, with no test statistics, confidence intervals, or effect sizes; the only explicitly reported statistical results are the non-significant correlations. To support any quantitative claim, the authors should report the paired t-test (or an appropriate nonparametric alternative) result, the associated effect size with a confidence interval, and an interpretation of that magnitude relative to the instrument's scale and measurement properties.
  4. [Section 4.1 and Section 4.3] The sample and instrument limitations are more severe than the Discussion acknowledges. With n=12, a single one-hour session, and 10 of 12 participants holding graduate degrees, the quantitative results have extremely limited generalizability. In addition, the MAILS framework was originally developed and validated on other populations, and no psychometric evidence is provided for its use with adults aged 75 and over; without measurement invariance, test-retest reliability, or known floor/ceiling behavior, the pre/post change scores may partly reflect measurement artifact. These issues should be moved from 'future work' into the primary limitations of the quantitative analysis.
minor comments (6)
  1. [Section 3 and Section 4.2] The number of tasks is inconsistent: Section 3 says 'four tasks' but then describes five (general search, personal assistance, emotional companion, AI safety, AI ethics), while Section 4.2 enumerates six steps. The authors should align these counts.
  2. [Section 6] The sentence beginning 'As enAI tools become...' contains a typo; it should read 'As GenAI tools...'.
  3. [Section 5.1.3] The chatbot is referred to as 'Littie' in one participant quote; this should be 'Litti'.
  4. [Section 6] The citation 'Shandilya and Fan [?]' has a placeholder question mark instead of a reference number; it should be numbered consistently with the bibliography.
  5. [Section 2.2] Reference '[48]' is cited in the sentence about age-related cognitive and visual impairments, but the reference list only goes up to [24]; the intended citation is likely one of the existing AI-safety references, such as [16] or [24].
  6. [Appendix C] Figure 2's caption is identical to Figure 1's caption ('AI literacy increase by individual'), which could confuse readers; it should be made more descriptive or referenced explicitly in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No by-construction circularity: the shared MAILS framework is a construct-validity concern, not a derivation that reduces to its inputs.

full rationale

The paper's central quantitative claim is a pre/post change in self-reported AI literacy scores, and its qualitative findings are thematic descriptions of interviews. There is no fitted parameter, no equation, and no result derived from an assumption that already contains the conclusion. The only candidate for circularity is the shared MAILS framework: Section 3 states that Litti's tasks were 'designed around' the sub-scales 'Know & AI,' 'Detect AI,' and 'AI Ethics,' and Section 4.3 says the questionnaire was 'developed adopting' the same MAILS framework. This raises a legitimate construct-validity and teaching-to-the-test concern, but the paper does not claim that the trend is statistically significant, the survey items are not shown to be identical to the training tasks, and no numerical result is forced by construction. The cited MAILS scale is external, not a self-citation, so no self-citation chain is load-bearing. The statistical inconsistency noted by a skeptical reader (a paired t-test on the reported values would be significant) is a reporting and correctness issue, not a circularity issue. Accordingly, no circular step is exhibited, and the paper is best assessed on validity and statistical-reporting grounds rather than circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The central quantitative claim rests on an unvalidated self-report instrument, a brief single-session exposure, and no control group; the thematic analysis relies on unreported reliability. The only invented entity is the chatbot itself, which is an intervention artifact with an externally viewable prompt.

assumptions (4)
  • domain assumption The MAILS-based questionnaire is a valid measure of AI literacy for adults aged 75+.
    Section 4.3 adopts MAILS without psychometric validation for this age group; all pre/post quantitative claims rely on it.
  • domain assumption Self-reported pre/post survey scores reflect genuine AI literacy changes.
    Section 4.2 procedure has no control group, no behavioral tasks, and no checks for social desirability or practice effects.
  • domain assumption A single one-hour session with six chatbot tasks can produce measurable changes in AI literacy.
    Section 4.2 states the study took place on a single day; the pre/post design implicitly assumes a detectable short-term effect.
  • domain assumption Inductive thematic analysis without inter-rater reliability or a shared codebook yields reliable themes.
    Section 5.1 reports only 'inductive coding' and thematic analysis; no reliability metrics are given.
invented entities (1)
  • Litti, the AI literacy education chatbot independent evidence
    purpose: Intervention delivering six GenAI literacy tasks to older adults through a Claude-based FlowXO chatbot.
    The paper links to the live prototype and the full prompt (Section 3), so the artifact is externally checkable, but its educational effectiveness is only assessed in this 12-person study.

how reviews work

0 comments
Cite this review

Pith. "Pith review of "We need to avail ourselves of GenAI to enhance knowledge distribution": Empowering Older Adults through GenAI Literacy." pith.science (2026). https://pith.science/paper/GWQ7RR2K

@misc{pith2026250606225,
  author       = {Pith},
  title        = {Pith review of: "We need to avail ourselves of GenAI to enhance knowledge distribution": Empowering Older Adults through GenAI Literacy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWQ7RR2K}},
  note         = {Machine review of arXiv:2506.06225}
}
read the original abstract

As generative AI (GenAI) becomes increasingly widespread, it is crucial to equip users, particularly vulnerable populations such as older adults (65 and older), with the knowledge to understand its benefits and potential risks. Older adults often exhibit greater reservations about adopting emerging technologies and require tailored literacy support. Using a mixed methods approach, this study examines strategies for delivering GenAI literacy to older adults through a chatbot named Litti, evaluating its impact on their AI literacy (knowledge, safety, and ethical use). The quantitative data indicated a trend toward improved AI literacy, though the results were not statistically significant. However, qualitative interviews revealed diverse levels of familiarity with generative AI and a strong desire to learn more. Findings also show that while Litti provided a positive learning experience, it did not significantly enhance participants' trust or sense of safety regarding GenAI. This exploratory case study highlights the challenges and opportunities in designing AI literacy education for the rapidly growing older adult population.

Figures

Figures reproduced from arXiv: 2506.06225 by the authors.

Figure 1
Figure 1. AI literacy increase by individual more common to older adults, such as health information seek￾ing. Providing scenarios and tasks that are familiar with the target audience will improve the learning process, enabling them to com￾prehend the knowledge faster. Also, chatbots, when incorporated with ethical AI principles, can encourage critical engagement with AI technologies, raising awareness of bias, fairness, and … view at source ↗
Figure 2
Figure 2. AI literacy increase by individual [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 15 canonical work pages

  1. [1]

    Abeer Alessa and Hend Al-Khalifa. 2023. Towards Designing a ChatGPT Conver- sational Companion for Elderly People. InProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments. ACM, Corfu Greece, 667–674. doi:10.1145/3594806.3596572

    arXiv 2023

  2. [2]

    Keith Atkinson, Jaclyn Barnes, Judith Albee, Peter Anttila, Judith Haataja, Kanak Nanavati, Kelly Steelman, and Charles Wallace. 2016. Breaking Barriers to Digital Literacy: An Intergenerational Social-Cognitive Approach. InProceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’16). Association for Computing...

    arXiv 2016

  3. [3]

    Koch, Samantha Straka, Marc Erich Latoschik, and Carolin Wienrich

    Astrid Carolus, Martin J. Koch, Samantha Straka, Marc Erich Latoschik, and Carolin Wienrich. 2023. MAILS - Meta AI literacy scale: Development and testing of an AI literacy questionnaire based on well-founded competency models and psychological change- and meta-competencies.Computers in Human Behavior: Artificial Humans1, 2 (Aug. 2023), 100014. doi:10.101...

    arXiv 2023

  4. [4]

    Krish Chetty. 2023. AI Literacy for an Ageing Workforce: Leveraging the Experi- ence of Older Workers.OBM Geriatrics7, 3 (July 2023), 1–17. doi:10.21926/obm. geriatr.2303243

    work page doi:10.21926/obm 2023

  5. [5]

    Alexandra Crampton. 2011. Population aging and social work practice with older adults: Demographic and policy challenges.International Social Work54, 3 (May 2011), 313–329. doi:10.1177/0020872810396257

    work page doi:10.1177/0020872810396257 2011

  6. [6]

    Vítor Crista, Diogo Martinho, and Goreti Marreiros. 2025. A Multi-agent System Approach with Generative AI for Improved Elderly Daily Living. InProgress in Artificial Intelligence, Manuel Filipe Santos, José Machado, Paulo Novais, Paulo Cortez, and Pedro Miguel Moreira (Eds.). Vol. 14967. Springer Nature Switzerland, Cham, 128–140. doi:10.1007/978-3-031-7...

    work page doi:10.1007/978-3-031-73497-7_11 2025

  7. [7]

    Dhimolea, Regina Kaplan-Rakowski, and Lin Lin

    Tetyana K. Dhimolea, Regina Kaplan-Rakowski, and Lin Lin. 2022. Supporting Social and Emotional Well-Being with Artificial Intelligence. InBridging Human Intelligence and Artificial Intelligence, Mark V. Albert, Lin Lin, Michael J. Spector, and Lemoyne S. Dunn (Eds.). Springer International Publishing, Cham, 125–138. doi:10.1007/978-3-030-84729-6_8

    work page doi:10.1007/978-3-030-84729-6_8 2022

  8. [8]

    Alice Gomstyn and Alexandra Jonker. 2025. AI Literacy: Closing the Artificial Intelligence Skills Gap | IBM. https://www.ibm.com/think/insights/ai-literacy

    work page 2025

Show all 25 references
  1. [9]

    Yueqiao Jin, Kaixun Yang, Lixiang Yan, Vanessa Echeverria, Linxuan Zhao, Ri- ordan Alfredo, Mikaela Milesi, Jie Fan, Xinyu Li, Dragan Gašević, and Roberto Martinez-Maldonado. 2024. Chatting with a Learning Analytics Dashboard: The Role of Generative AI Literacy on Learner Inte...

    2024 doi arXiv

  2. [10]

    Yusuke Kajiwara and Kouhei Kawabata. 2024. AI literacy for ethical use of chatbot: Will students accept AI ethics?Computers and Education: Artificial Intelligence6 (June 2024), 100251. doi:10.1016/j.caeai.2024.100251

    2024

  3. [11]

    Amrat Kaur and Weiqin Chen. 2023. Exploring AI Literacy Among Older Adults. InAssistive Technology: Shaping a Sustainable and Inclusive World. IOS Press, 9–16. doi:10.3233/SHTI230589

    2023 doi

  4. [12]

    Matthias Carl Laupichler, Alexandra Aster, Jana Schirch, and Tobias Raupach

  5. [13]

    We need to avail ourselves of [GenAI] to enhance knowledge distribution

    Duri Long and Brian Magerko. 2020. What is AI Literacy? Competencies and Design Considerations. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–16. doi:10.1145/3313831.3376727 “...

    2020

  6. [14]

    Duri Long, Anthony Teachey, and Brian Magerko. 2022. Family Learning Talk in AI Literacy Learning Activities. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–20. doi:10.1145/349...

    2022

  7. [15]

    Xinran Lu, Yao Yao, and Yinzi Jin. 2022. Digital exclusion and functional depen- dence in older people: Findings from five longitudinal cohort studies.eClini- calMedicine54 (Dec. 2022), 101708. doi:10.1016/j.eclinm.2022.101708

    2022

  8. [16]

    Yisroel Mirsky, Ambra Demontis, Jaidip Kotak, Ram Shankar, Deng Gelei, Liu Yang, Xiangyu Zhang, Maura Pintor, Wenke Lee, Yuval Elovici, and Battista Biggio. 2023. The Threat of Offensive AI to Organizations.Computers & Security 124 (Jan. 2023), 103006. doi:10.1016/j.cose.2022.103006

    2023

  9. [17]

    Davy Tsz Kit Ng, Jac Ka Lok Leung, Maggie Jiahong Su, Iris Heung Yue Yim, Maggie Shen Qiao, and Samuel Kai Wah Chu. 2022. AI Literacy for All. InAI Literacy in K-16 Classrooms. Springer International Publishing, Cham, 21–29. doi:10.1007/978-3-031-18880-0_3

    2022 doi

  10. [18]

    Kim, Minsu Kim, Jaeuk Oh, Sang Hui Chu, and JiYeon Choi

    Sarah Soyeon Oh, Kyoung-A. Kim, Minsu Kim, Jaeuk Oh, Sang Hui Chu, and JiYeon Choi. 2021. Measurement of Digital Literacy Among Older Adults: Sys- tematic Review.Journal of Medical Internet Research23, 2 (Feb. 2021), e26145. doi:10.2196/26145

    2021 doi

  11. [19]

    Jennifer Ortman, Victoria Velkoff, and Howard Hogan. 2014. An Aging Nation: The Older Population in the United States. https://www.census.gov/library/ publications/2014/demo/p25-1140.html

    2014

  12. [20]

    Kit Pribble. 2024. Fostering Critical AI Literacy in the Russian Language Class- room.Russian Language Journal74, 1 (Dec. 2024). doi:10.70163/0036-0252.1399

    2024

  13. [21]

    Esha Shandilya and Mingming Fan. 2024. Understanding Older Adults’ Per- ceptions and Challenges in Using AI-enabled Everyday Technologies. InPro- ceedings of the Tenth International Symposium of Chinese CHI (Chinese CHI ’22). Association for Computing Machinery, New York, NY, ...

    2024

  14. [22]

    2020.World population ageing, 2019 highlights.Number 430 in [Population studies]

    United Nations. 2020.World population ageing, 2019 highlights.Number 430 in [Population studies]. United Nations, New York. OCLC: 1134853628

    2020

  15. [23]

    Honglian Xiang, Jia Zhou, and Bingjun Xie. 2023. AI tools for debunking on- line spam reviews? Trust of younger and older adults in AI detection criteria. Behaviour & Information Technology42, 5 (April 2023), 478–497. doi:10.1080/ 0144929X.2021.2024252

    2023

  16. [24]

    Baobao Zhang and Allan Dafoe. 2019. Artificial Intelligence: American Attitudes and Trends.SSRN Electronic Journal(2019). doi:10.2139/ssrn.3312874 Appendix A Participant Demographics Table 1: Participant demographics PID Age Gender Race Education AI Familiarity AI Usage Freque...

    2019 doi

  17. [2022]

    2022), 100101

    Artificial intelligence literacy in higher and adult education: A scoping literature review.Computers and Education: Artificial Intelligence3 (Jan. 2022), 100101. doi:10.1016/j.caeai.2022.100101

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.