pith. sign in

arxiv: 2605.00796 · v1 · submitted 2026-05-01 · 💻 cs.CR · cs.AI· cs.CL

When RAG Chatbots Expose Their Backend: An Anonymized Case Study of Privacy and Security Risks in Patient-Facing Medical AI

Alfredo Madrid-Garc\'ia , Miguel Rujas This is my paper

Pith reviewed 2026-05-09 18:48 UTC · model grok-4.3

classification 💻 cs.CR cs.AIcs.CL
keywords RAG chatbotmedical AIprivacy risksecurity vulnerabilitypatient databrowser inspectionretrieval-augmented generation
0
0 comments X

The pith

A patient-facing medical RAG chatbot exposed its backend configuration and recent patient conversations through ordinary browser inspection.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a case study of a publicly accessible medical chatbot using retrieval-augmented generation that was assessed for security and privacy issues. Inspection with standard browser tools revealed the system prompt, model settings, retrieval details, API structure, knowledge base content, and the one thousand most recent conversations that included health-related questions. This access required no login or advanced skills and contradicted the system's own privacy statements. Readers would care because these tools are meant to provide reliable health information yet can leak sensitive data and internal operations.

Core claim

The assessment demonstrates that in this deployment, browser inspection of network traffic and stored data exposed the full system prompt, model and embedding configuration, retrieval parameters, backend endpoints, API schema, document and chunk metadata, knowledge-base content, and the 1,000 most recent patient-chatbot conversations, which contained health queries.

What carries the argument

The client-side exposure of RAG system configuration and conversation history through visible network requests and browser storage.

Load-bearing premise

The examined chatbot deployment reflects risks common to other patient-facing medical RAG systems and did not have additional server-side protections that would block the observed browser access.

What would settle it

Performing the same browser inspection on the chatbot after security fixes and finding that the system prompt, configurations, and past conversations are no longer visible would indicate the exposure has been addressed.

Figures

Figures reproduced from arXiv: 2605.00796 by Alfredo Madrid-Garc\'ia, Miguel Rujas.

Figure 1
Figure 1. Figure 1: Two-stage workflow of the security assessment. view at source ↗
read the original abstract

Background: Patient-facing medical chatbots based on retrieval-augmented generation (RAG) are increasingly promoted to deliver accessible, grounded health information. AI-assisted development lowers the barrier to building them, but they still demand rigorous security, privacy, and governance controls. Objective: To report an anonymized, non-destructive security assessment of a publicly accessible patient-facing medical RAG chatbot and identify governance lessons for safe deployment of generative AI in health. Methods: We used a two-stage strategy. First, Claude Opus 4.6 supported exploratory prompt-based testing and structured vulnerability hypotheses. Second, candidate findings were manually verified using Chrome Developer Tools, inspecting browser-visible network traffic, payloads, API schemas, configuration objects, and stored interaction data. Results: The LLM-assisted phase identified a critical vulnerability: sensitive system and RAG configuration appeared exposed through client-server communication rather than restricted server-side. Manual verification confirmed that ordinary browser inspection allowed collection of the system prompt, model and embedding configuration, retrieval parameters, backend endpoints, API schema, document and chunk metadata, knowledge-base content, and the 1,000 most recent patient-chatbot conversations. The deployment also contradicted its privacy assurances: full conversation records, including health-related queries, were retrievable without authentication. Conclusions: Serious privacy and security failures in patient-facing RAG chatbots can be identified with standard browser tools, without specialist skills or authentication; independent review should be a prerequisite for deployment. Commercial LLMs accelerated this assessment, including under a false developer persona; assistance available to auditors is equally available to adversaries.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The manuscript reports an anonymized case study of privacy and security risks in a patient-facing medical RAG chatbot. Using LLM-assisted hypothesis generation followed by manual verification with Chrome Developer Tools on network traffic and stored data, the authors demonstrate that client-side inspection revealed the system prompt, model and embedding configuration, retrieval parameters, backend endpoints, API schema, document and chunk metadata, knowledge-base content, and the 1,000 most recent patient conversations, contradicting the system's privacy assurances.

Significance. The result, if it holds, is significant because it offers direct, reproducible empirical evidence of backend exposure in a medical AI system using only standard browser tools. This strengthens the case for rigorous security controls in health-related generative AI and explicitly credits the two-stage method that combines commercial LLM assistance with manual confirmation, making the findings verifiable and highlighting risks to both defenders and adversaries.

minor comments (2)
  1. [Abstract, Results] The specific number of 1,000 conversations is presented without detailing the mechanism by which this exact count was obtained from the browser inspection; adding this would improve reproducibility of the verification.
  2. [Conclusions] The statement that 'independent review should be a prerequisite for deployment' could be clarified to specify the scope, as the evidence is from one deployment.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive assessment of our manuscript and for recommending acceptance. We appreciate the recognition of the empirical value of the two-stage LLM-assisted and manually verified approach to identifying client-side exposures in a patient-facing medical RAG system.

Circularity Check

0 steps flagged

No significant circularity

full rationale

This is an empirical case study reporting direct observations from browser-based inspection of one live RAG deployment. The two-stage method (LLM-assisted hypothesis generation followed by manual Chrome DevTools verification of network payloads, API schemas, and stored data) is self-contained and externally falsifiable; no equations, fitted parameters, predictions, or self-citation chains are used to derive the listed exposures. The central claims reduce to verifiable facts about the inspected system rather than any internal definitional loop.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical security case study with no mathematical models, fitted parameters, or postulated new entities.

pith-pipeline@v0.9.0 · 5595 in / 1009 out tokens · 31213 ms · 2026-05-09T18:48:59.713882+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages

  1. [1]

    RAG in Health Care: A Novel Framework for Improving Communication and Decision-Making by Addressing LLM Limitations.NEJM AI2025;2

    Ng KKY , Matsuba I, Zhang PC. RAG in Health Care: A Novel Framework for Improving Communication and Decision-Making by Addressing LLM Limitations.NEJM AI2025;2. https://doi.org/10.1056/ AIra2400380

  2. [2]

    Retrieval augmented generation for large language models in healthcare: A systematic review.PLOS Digital Health2025;4:e0000877

    Amugongo LM, Mascheroni P, Brooks S, Doering S, Seidel J. Retrieval augmented generation for large language models in healthcare: A systematic review.PLOS Digital Health2025;4:e0000877. https://doi.org/10. 1371/journal.pdig.0000877

  3. [3]

    The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)

    Zeng S, Zhang J, He P, Liu Y , Xing Y , Xu H, et al. The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG). InFindings of the Association for Computational Linguistics ACL 2024, Stroudsburg, PA, USA: Association for Computational Linguistics; 2024, p. 4505–24. https://doi.org/10. 18653/v1/2024.findings-acl.267

    2024

  4. [4]

    Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directions, 2025

    Guan S, Kwok HC, Law NF, Stiglic G, Qin H, Hui V . Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directions, 2025

    2025

  5. [5]

    Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarks, 2026

    Mu Y , Hu H, Li F, Yuan Q, Wu J, Liu Z, et al. Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarks, 2026

    2026

  6. [6]

    Dellavalle NS, Ellis JR, Moore AA, Akerson M, Andazola M, Campbell EG, et al. What patients want from healthcare chatbots: insights from a mixed-methods study.Journal of the American Medical Informatics Association2025;32:1735–45.https://doi.org/10.1093/jamia/ocaf164

    work page doi:10.1093/jamia/ocaf164

  7. [7]

    Patients are disclosing sensitive information to AI tools—clinicians must adapt.BMJ2026;392:s124

    Blease C. Patients are disclosing sensitive information to AI tools—clinicians must adapt.BMJ2026;392:s124. https://doi.org/10.1136/bmj.s124

    work page doi:10.1136/bmj.s124

  8. [8]

    A Survey of Vibe Coding with Large Language Models, 2025

    Ge Y , Mei L, Duan Z, Li T, Zheng Y , Wang Y , et al. A Survey of Vibe Coding with Large Language Models, 2025

    2025

  9. [9]

    Is Vibe Coding Safe? Benchmarking Vulnerability of Agent- Generated Code in Real-World Tasks, 2026

    Zhao S, Wang D, Zhang K, Luo J, Li Z, Li L. Is Vibe Coding Safe? Benchmarking Vulnerability of Agent- Generated Code in Real-World Tasks, 2026

    2026

  10. [10]

    Teams of LLM Agents can Exploit Zero-Day Vulnerabilities

    Zhu Y , Kellermann A, Gupta A, Li P, Fang R, Bindu R, et al. Teams of LLM Agents can Exploit Zero-Day Vulnerabilities. V ol. 1. Long Papers; 2026

    2026

  11. [11]

    Our evaluation of Claude Mythos Preview’s cyber capabilities, 2026

    AI Security Institute. Our evaluation of Claude Mythos Preview’s cyber capabilities, 2026

    2026

  12. [12]

    Claude Mythos Preview, 2026

    Anthropic Frontier Red Team. Claude Mythos Preview, 2026

    2026

  13. [13]

    Testing and Evaluation of Health Care Applications of Large Language Models.JAMA2025;333:319

    Bedi S, Liu Y , Orr-Ewing L, Dash D, Koyejo S, Callahan A, et al. Testing and Evaluation of Health Care Applications of Large Language Models.JAMA2025;333:319. https://doi.org/10.1001/jama.2024. 21700

    work page doi:10.1001/jama.2024 2024

  14. [14]

    Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review.JMIR AI2025;4:e69006.https://doi.org/10.2196/69006

    Hua Y , Xia W, Bates D, Hartstein GL, Kim HT, Li M, et al. Standardizing and Scaffolding Health Care AI-Chatbot Evaluation: Systematic Review.JMIR AI2025;4:e69006.https://doi.org/10.2196/69006. 10 APREPRINT- MAY4, 2026

    work page doi:10.2196/69006 2026

  15. [15]

    Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models

    World Health Organization. Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models. Geneva: World Health Organization; 2025

    2025

  16. [16]

    Information security and confidentiality in health chatbots: A scoping review and development of a conceptual model.Digital Health2025;11

    Talebi Azadboni T, Solat F, Hematti H, Rahmani M. Information security and confidentiality in health chatbots: A scoping review and development of a conceptual model.Digital Health2025;11. https://doi.org/10.1177/ 20552076251406637. 11