Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries

Aoife Keane; Grace Thompson; Haruka Notsu; Jill Noorily; John Torous; Laura Ospina-Pinillos; Matthew Flathers; Phuong Anh Nguyen; Samantha Clark; Tommy Nguyen

REVIEW

Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries

Phuong Anh Nguyen, Jill Noorily, Matthew Flathers, Haruka Notsu, Laura Ospina-Pinillos, Tommy Nguyen, Samantha Clark, Aoife Keane

show 2 more authors

Grace Thompson, John Torous

Computers and Society · Submitted 31 Aug 2026 · 4 references

Pith's verdict: A careful, transparent audit of what three AI products cite for mental health queries, but the retry protocol undermines some of the headline claims about no-citation rates and the effect of asking for sources.

T0 · AI review

How this review was made: An AI referee (deepseek/deepseek-v4-flash) read the paper on September 9, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

T0 AI review An AI read the paper and answered a fixed list of questions.
T1 Audit Automatic checks that the references and files the paper relies on exist and match.
T2 Compute The paper's code or calculations re-run with the same results.
T3 Formal statement The claims written out in a formal math language a computer can read.
T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 2 major objections · 4 minor · Ask about this paper · The compiled record · Read the full review →

Challenge this review Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2609.00319 (version 1)

Pith IDZT5JWB6B, Pith's permanent record of this paper and its review

CitationAoife Keane, Grace Thompson, Haruka Notsu, Jill Noorily, John Torous, Laura Ospina-Pinillos, Matthew Flathers, Phuong Anh Nguyen, Samantha Clark, Tommy Nguyen. "Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries." arXiv:2609.00319 (2026). https://pith.science/paper/ZT5JWB6B

Cite this reviewBibTeX for Pith's review of this paper

Claim paper

Why Pith says this

Highlight any text to comment on it.

Editor's note

read the letter

This paper is a solid empirical contribution to the AI audit literature. It systematically records what ChatGPT, Perplexity, and Google AI Overview cite when asked 20 mental health questions in English and three in six other languages, building a validated source-type classifier and releasing the full corpus. The main findings – that citations concentrate on a narrow institutional core (top-10 domains account for 44% of English citations), that platforms lead with different source types, and that non-English queries get fewer and less locally appropriate citations – are well supported and useful.

What you need to know is that the results about no-citation rates and the effect of explicitly requesting sources are weaker than they appear. The annotator protocol in Appendix S3 says: if a query returns no sources, run it again up to five times until you get sources. So the reported 7.3% no-citation rate is the fraction of response blocks where all five attempts failed, not the per-query probability. The first-attempt failure rate is almost certainly higher. This matters most for the claim that appending “List your sources.” changes behavior only modestly: the protocol censors the very outcome where the request might matter (whether any sources appear at all). Given that the API audit in Appendix S1 found that adding the source request flipped gpt-5.4-mini’s retrieval rate from 19% to 99% in English, the product-level null is partly an artifact of retrying until sources appeared. The volume and composition shifts within responses that do contain citations may still be small, but the claim about “little benefit” is not established on the data as collected.

That said, the paper is transparent about the protocol, releases everything, and the core descriptive findings about domain concentration and source-type composition are robust to this issue. The cross-lingual results are valuable even if based on only three questions per language.

For whom? Researchers in AI accountability, health informatics, and information retrieval. It deserves a serious peer review, but the authors should be asked to report first-attempt data (or state they no longer have it) and to adjust their conclusions about the source-request manipulation accordingly.

I’d take this to reading group, but I’d cite it only after the revision addresses the retry bias.

Referee report

Significance

If the findings hold, this is a valuable and reusable audit of the consumer-facing surfaces where mental-health information seeking now occurs. Its strengths include an outcome measure defined on the product rather than on an inaccessible underlying model, a deterministic classifier with documented human agreement, a leave-one-annotator-out sensitivity analysis, a multilingual design covering resource tiers that are rarely audited together, and public release of the instrument and data. The API-side comparison in Appendix S1 is a useful attempt to separate wrapper-level from model-level retrieval behavior, and its own limitations are stated candidly. However, the retry protocol in Appendix S3 directly affects the interpretation of the no-citation rates, the citation-volume comparisons, and the “little benefit” conclusion for explicit source requests, all of which are central claims. The issue is fixable by reanalysis or targeted re-collection, but it must be resolved before these claims are published as stated.

Major objections (2)

[Appendix S3; Sections 3.1 and 3.5] The annotator protocol instructs: “If no sources, run the question again until you get sources (Cap at 5 times); record how many times you run it till you get sources.” The reported no-citation rates therefore describe completed blocks of up to five attempts, not a single user-issued query. Section 3.1 reports 83/1,140 (7.3%) responses with no citations, and Section 3.5 reports per-language rates such as 33% for Twi and 23% for Japanese under ChatGPT. If attempts were independent, a 33% block-level rate corresponds to roughly an 80% per-attempt no-source probability, so these numbers may substantially understate what a user who asks once experiences. Mean citation counts are also computed over final successful or cap-exhausted attempts, so they overstate typical single-query yield and are not directly comparable across prompt conditions. The authors should recover the recorded retry counts and report first-attempt no-citation rates and first-attempt citation-volume distributions, or re-run the affected conditions, and should not interpret the current block-level estimates as per-query probabilities.
[Section 3.4 and Discussion §4] The conclusion that appending “List your Sources.” “yields little benefit” is not established as stated, because the censoring produced by the capped retry protocol removes exactly the margin along which the instruction plausibly acts: whether any sources are returned at the first attempt. Appendix S1 shows a large model-level effect, with gpt-5.4-mini's English retrieval rate rising from 19.0% to 99.0% and mean English citations per response rising from 0.60 to 7.05 when sources were requested. The product-level null in Section 3.4 could be at least partly an artifact of rerunning no-source attempts in the question-only condition before recording a response. The authors should analyze the effect of the instruction on first-attempt no-citation rates and first-attempt volume, or collect new first-attempt product data, before claiming that explicitly requesting sources changes citation behavior only modestly.

Minor objections (4)

[Abstract, Section 3.1, Section 3.3, Section 5] Citation totals are not internally consistent: the Abstract states 15,942 recorded citations while the Conclusion states 15,929 classified citations, and Section 3.1 reports 10,972 English citations while Section 3.3 uses 10,959 English citations. These differences should be reconciled or explicitly explained.
[Appendix S2] The translated prompts are partly unreadable: the Japanese items appear as garbled or mojibake characters, and the three Nepali source-request variants are rendered identically as “लंखत गनुर्”. Because the translated instrument is part of the replication package, these items should be corrected and presented in clean Unicode text.
[Section 4.2] The connection between the observed lexical-neighbor anomalies (e.g., mayonnaise for Mayo Clinic, DSM-Firmenich for DSM) and adversarial typosquatting/combosquatting is appropriately labeled as a future risk, but the paragraph could be tightened by noting explicitly that the examples observed involve benign content and that no adversarial exploitation was found in this dataset.
[Sections 2.1 and 3.5] After the reanalysis requested above, the manuscript should define and consistently use the terms “single attempt,” “retried block,” and “response” so that the distinction intended by the protocol is explicit in every estimate and figure caption.

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

Consumer AI search products answering mental health questions concentrate their citations on a narrow core of institutional domains, differ sharply by platform, and route non-English queries to local resources far less reliably.

The reading

This paper tries to establish that when people ask consumer AI search products—ChatGPT, Perplexity, and Google's AI Overview—about mental health, the sources those products cite are neither neutral nor uniform. Auditing 1,140 responses and 15,942 citations across seven languages, it finds that the ten most-cited domains account for 43.6% of English citations, with government, commercial health, and academic sources each contributing roughly 22%. It also finds that the three platforms lead with different source types, that appending "List your Sources." barely changes citation behavior, and that non-English queries receive fewer citations routed to language-appropriate resources. The reason this matters is that these products now act as gatekeepers of mental-health information: where their citations concentrate, and where they fail, shapes what millions of users see and trust.

What carries the argument

The carrier of the argument is a nine-category organizational typology—government/public health, academic/journal, nonprofit health system, commercial health, nonprofit/advocacy, encyclopedia (wiki), news/media, social/video, and other—applied to every citation by an ordered, rule-based classifier that reads the registrable domain, the host, and the public suffix, and validated against human coding with Cohen's $κ$ of 0.86–0.89. This classifier does the work of turning the 15,942 recorded citations into comparable source-type distributions across platforms, question categories, and languages. For the multilingual claims, the load-bearing instrument is a pair of complementary routing measures: citations on a country-code top-level domain associated with the query language, and citations whose URL carries a native-language signal such as a locale path or percent-encoded script—the second catching translated pages like the Spanish section of MedlinePlus that the first measure alone would miss.

Extended reading notes

Core claim

The paper's central claim is that citation behavior in generative mental-health search is measurable, systematic, and skewed. For English queries, the products concentrate attention on a narrow institutional core—ten destinations, led by PubMed/NCBI and Mayo Clinic, account for 43.6% of citations, and government, commercial health, and academic sources each supply about 22%—so a handful of pages effectively define the information environment for sensitive health questions. The platforms are not interchangeable: ChatGPT leans toward government sources and Wikipedia, Perplexity toward academic sources, and Google AI Overview toward commercial health and social/video. Explicitly requesting sources moves the mix only slightly toward institutional sources and does not fix the concentration. For non-English queries the picture worsens: citation counts per response fall, no-citation responses rise sharply for some product–language pairs (a third of ChatGPT's Twi responses and 30% of Google AI Overview's Hindi responses carried no citations), and language-appropriate routing, measured by both country-code domains and native-language pages on international domains, drops steeply with language-resource tier and tracks the supply of authoritative in-language web content rather than any property of the language itself. A parallel audit of comparable vendor API models suggests that the consumer wrapper, not just the underlying model, shapes this surface.

Load-bearing premise

The cross-lingual results rest on only three translated questions per non-English language—covering diagnosis, crisis resources, and trustworthy sources—and the paper itself flags that these three must stand in for the full range of mental health queries in each language.

Editorial extensions

If this is right

A user's choice of product changes the evidence they see: the same mental-health question yields a government-led, academic-led, or commercial-led citation mix depending on whether the answer comes from ChatGPT, Perplexity, or Google AI Overview.
A user's strategy of appending "List your Sources." buys little—citation counts and source composition barely move—so source evaluation effectively stays with the platform even when the user asks for accountability.
For non-English users, especially in medium- and low-resource languages, answers arrive with fewer citations or none at all, and the sources offered are less often local, making the equity gap in mental-health information access partly a citation-routing gap.
Citation concentration concentrates risk: errors, gaps, or biases in a few heavily cited pages propagate to all users, and the observed lexical-neighbor retrieval errors (Mayo Clinic vs. mayonnaise, DSM vs. the fragrance firm DSM-Firmenich) mark an occupiable surface for typosquatting and combosquatting attacks.
The released typology, classifier, and annotated corpus make source composition a quantity that can be tracked over time, so a single-point snapshot can become a continuing audit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the consumer wrapper is what broadens citation diversity for at least one provider, product-level interventions—capping citation counts per answer, diversifying source types, or prioritizing local content—could change what users see without retraining models; the paper's API comparison supports but does not test such levers.
Localization appears to track the supply of authoritative in-language web content rather than retrieval ability, which implies that publishing and indexing more high-quality local health content would move the language-appropriateness metric more than better multilingual models; that is a testable prediction, not a result the paper establishes.
Citation concentration is plausibly self-reinforcing: systems retrieve what is already heavily indexed and linked, so the top domains would be expected to grow their share across repeated audits, a trend the released instruments could directly measure.
The lexical-neighbor failures suggest a concrete adversarial experiment: plant innocuous pages whose names sit adjacent to the most-cited clinical domains and test whether retrieval systems pull them into mental-health answers, quantifying an attack surface the paper identifies but cannot measure from a single snapshot.

Original abstract

Online health information seeking is shifting from keyword search, where users consider a ranked list of links, to conversational systems that compose a single answer and curate its citations. Source evaluation therefore passes from user to platform, yet what these systems surface is poorly characterized. We audited three free consumer products (ChatGPT, Perplexity, Google AI Overview) on twenty English mental health questions under two prompt conditions, with a subset of three also translated into six further languages of varying resource tiers. We recorded 15,942 citations across 1,140 responses and 1,713 unique domains, then classified every citation with a nine-category organizational typology applied by a deterministic classifier validated against human coding. Citations were heavily concentrated: the ten most-cited domains accounted for 43.6% of English citations, and government, commercial health, and academic sources were closely matched at roughly 22% each. Platforms differed little in typical citation volume but sharply in consistency and in the source types they favored. Explicitly requesting sources shifted composition only modestly. Non-English queries surfaced fewer citations and were routed to language-appropriate resources at significantly lower rates. We release the typology, classifier, and annotated corpus as reusable instruments for auditing generative health search.

The compiled record

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,905 input tokens.

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.49 · human participants 0.17 · controlled comparison 0.13 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.25 · mentions code or data availability 0.95.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

Online health information seeking is shifting from keyword search, where users consider a ranked list of links, to conversational systems that compose a single answer and curate its citations. Source evaluation therefore passes from user to platform, yet what these systems surface is poorly characterized. We audited three free consumer products (ChatGPT, Perplexity, Google AI Overview) on twenty English mental health questions under two prompt conditions, with a subset of three also translated into six further languages of varying resource tiers. We recorded 15,942 citations across 1,140 responses and 1,713 unique domains, then classified every citation with a nine-category organizational typology applied by a deterministic classifier validated against human coding. Citations were heavily concentrated: the ten most-cited domains accounted for 43.6% of English citations, and government, commercial health, and academic sources were closely matched at roughly 22% each. Platforms differed little in typical citation volume but sharply in consistency and in the source types they favored. Explicitly requesting sources shifted composition only modestly. Non-English queries surfaced fewer citations and were routed to language-appropriate resources at significantly lower rates. We release the typology, classifier, and annotated corpus as reusable instruments for auditing generative health search.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Online health information seeking is shifting from keyword search, where users consider a ranked list of links, to conversational systems that compose a single answer and curate its citations.	background 1.00	0.05	0.03	asserted 1.00
s1	Source evaluation therefore passes from user to platform, yet what these systems surface is poorly characterized.	gap 0.92	0.05	0.06	asserted 1.00
s2	We audited three free consumer products (ChatGPT, Perplexity, Google AI Overview) on twenty English mental health questions under two prompt conditions, with a subset of three also translated into six further languages of varying resource tiers.	method 0.95	0.97	0.03	asserted 0.94
s3	We recorded 15,942 citations across 1,140 responses and 1,713 unique domains, then classified every citation with a nine-category organizational typology applied by a deterministic classifier validated against human coding.	method 0.98	0.98	0.05	asserted 1.00
s4	Citations were heavily concentrated: the ten most-cited domains accounted for 43.6% of English citations, and government, commercial health, and academic sources were closely matched at roughly 22% each.	result 1.00	0.97	0.15	asserted 1.00
s5	Platforms differed little in typical citation volume but sharply in consistency and in the source types they favored.	result 1.00	0.95	0.22	asserted 1.00
s6	Explicitly requesting sources shifted composition only modestly.	result 1.00	0.94	0.05	asserted 1.00
s7	Non-English queries surfaced fewer citations and were routed to language-appropriate resources at significantly lower rates.	result 1.00	0.95	0.13	asserted 1.00
s8	We release the typology, classifier, and annotated corpus as reusable instruments for auditing generative health search.	contribution 0.95	0.93	0.06	asserted 0.99

What kind of paper the classifier reports this is: empirical study 0.88 · evidence from observation or experiment 0.98.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

Figures

**Figure 1.** Citation volume by platform and question type. Figure 1A shows the number of sources per response (English query only) for ChatGPT, Perplexity and Google AI Overview; ChatGPT had the highest mean but similar medians to the others. Figure 1B breaks down citation counts by question category, showing the mean number of sources per response. 5

**Figure 2.** A distribution chart of the top 58 domains cited across English-language queries, color-coded by source type (academic, government, commercial health, etc.), showing how citations concentrate heavily on a small handful of sites like PubMed/NCBI and Mayo Clinic. Platforms differed in source-type composition (Figure 3A). Each led with a different type: government/public for ChatGPT (26.4%), academic/journal for Perplexity (24.3%), and commercial health for Google AI Overview (25.7%). The sharpest contrasts were in the smaller categories: ChatGPT cited Wikipedia far more than the…

**Figure 3.** Source-type composition for English queries. Figure 3A shows the percentage of citation types by platform. Figure 3B shows the percentage of citation types by question category. 3.4 Effect of explicitly requesting sources Appending ”List your Sources.” had little effect on the overall number of citations surfaced: across all English responses, the mean rose only modestly from 16.6 in the question-only condition to 19.9 when sources were explicitly requested (Figure 4A). This shift was driven almost entirely by ChatGPT, which showed a more substantial change (mean 27.3 to 37.2…

**Figure 4.** Effect of explicitly asking for sources. Figure 4A shows the shift in citation volume between questiononly and “List your Sources” conditions across platforms. Figure 4B shows the resulting small shift in source-type composition when “List your Sources” is appended. 3.5 Cross-lingual citation behavior On the three questions administered in all seven languages, non-English queries surfaced fewer sources than English ones (mean 9.2, SD 8.7, n=540 vs 12.3, SD 11.6, n=90), ranging from 11.2 for Spanish to 7.1 for Twi (Figure 5A). The gap was driven by the products that fail rather than…

**Figure 5.** Cross-lingual results. Figure 5A compares citation volume across platforms and languages. Figure 5B shows the share of citations pointing to language-appropriate resources (ccTLD or native-language). 3.6 Sensitivity to individual annotators As a sensitivity analysis covering all languages, every reported estimate was recomputed seven times, each time excluding one annotator's contributions (leave-one-annotator-out jackknife). Across 82 estimates, the median largest absolute deviation from the full-sample value was 0.82 percentage points (90th percentile 3.25). Every platform-level…

Figures reproduced from arXiv:2609.00319 by the authors.

References

Works cited (4)

[1]

For each question a) Use incognito mode b) Start a new chat (clear chat history) in the AI platform. No account logged in c) Use the model's base/free version. d) Copy and paste one question into the chat

View on Pith
[2]

Review the response and copy and paste every source cited in the appropriate tab. a) Copy and paste citations in the tab chart; include both: i) In-text citations and the citations of the in-text pop-outs ii) References listed at the end of the response iii) When you prompt it to list sources, it may provide a list of sources in its response in addition t...

View on Pith
[3]

Clear the chat and repeat the process with the next question until all questions have been com- pleted

View on Pith
[4]

Repeat the entire process for the other AI platforms (ChatGPT, Perplexity, and Google), keeping each platform's responses in the correct tab. 25 Appendix S4. Sensitivity to individual annotators Citations were collected by seven annotators. Annotators were not assigned evenly across the study design, with some contributing only English responses, others o...

View on Pith

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

"If I Had to Buy Just ONE: Galaxy S26 Ultra": Auditing AI-Generated Product Recommendations
cs.CY 2026-09 conditional novelty 7.0 of 10

AI chatbots vary drastically in how they recommend products and cite sources, and their APIs cannot substitute for auditing the consumer-facing interfaces.
How Generative AI Redefines Personalization and Diversity in the Newsroom
cs.CY 2026-11 conditional novelty 6.0 of 10

Based on 16 interviews and two model-behavior checks, news diversity decisions increasingly happen inside journalists' personally configured AI tools and across multiple versions of one story, where standard editorial...

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.

[1] [1]

For each question a) Use incognito mode b) Start a new chat (clear chat history) in the AI platform. No account logged in c) Use the model's base/free version. d) Copy and paste one question into the chat

View on Pith

[2] [2]

Review the response and copy and paste every source cited in the appropriate tab. a) Copy and paste citations in the tab chart; include both: i) In-text citations and the citations of the in-text pop-outs ii) References listed at the end of the response iii) When you prompt it to list sources, it may provide a list of sources in its response in addition t...

View on Pith

[3] [3]

Clear the chat and repeat the process with the next question until all questions have been com- pleted

View on Pith

[4] [4]

Repeat the entire process for the other AI platforms (ChatGPT, Perplexity, and Google), keeping each platform's responses in the correct tab. 25 Appendix S4. Sensitivity to individual annotators Citations were collected by seven annotators. Annotators were not assigned evenly across the study design, with some contributing only English responses, others o...

View on Pith