{"id":"feb339aa-ade3-42f8-abf5-ee48df583f95","arxiv_id":"2609.00319","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"AI search products for mental health queries concentrate citations on a narrow set of institutional domains, vary by platform in source type, and show degraded performance for non-English languages.","lead":"This study examined which external sources three AI search products (ChatGPT, Perplexity, Google AI Overview) cite when answering mental health questions across seven languages. It found that citations concentrate on a narrow set of institutional sources, platforms differ in the types of sources they favor, and non-English queries receive fewer and less locally relevant citations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The annotator protocol required rerunning queries that returned no citations, so the reported no-citation rates and the 'listing sources changes little' conclusion describe capped retry blocks rather than a single user query; first-attempt data are needed.","rationale":"The reader's weakest assumption concerned the three-question multilingual base. That is a genuine and self-acknowledged limitation, but it is not the most load-bearing issue. A more consequential problem appears in the annotator instructions: queries that returned no citations were rerun until sources appeared, capped at five attempts. The manuscript's central descriptive statistics—no-citation rates, mean citation counts, and the effect of explicitly requesting sources—are all sensitive to this retry rule. If only successful attempts are recorded, the reported 14.5% ChatGPT no-citation rate is not the rate a user would observe on a single query; it is closer to the probability that five independent attempts all fail. Likewise, the conclusion that 'List your Sources.' yields little benefit cannot be sustained if the protocol systematically discards no-source outcomes, since the request may act primarily on whether sources are produced at all. The API audit in Appendix S1 strengthens this worry: for gpt-5.4-mini, explicitly requesting sources raised retrieval invocation from 19% to 99% in English. Because the released data may include retry counts, the concern is checkable rather than fatal; the paper should either report first-attempt analyses or explicitly redefine its no-citation and volume statistics as outcomes after up to five attempts. I therefore recommend conditional acceptance pending that reanalysis. The paper's other strengths—released code and data, validated classifier, jackknife sensitivity analysis, and candid limitations—are real and preserved.","tokens_in":21151,"tokens_out":5326,"duration_ms":59564,"concrete_test":"Inspect the released Hugging Face dataset and annotator records for the retry counts that Appendix S3 requires annotators to record. Recompute Section 3.1 no-citation rates, Section 3.5 per-language no-citation and mean-citation estimates, and the source-request effect using only first-attempt responses, and separately compute first-attempt no-citation rates by prompt condition. If first-attempt no-citation rates are materially higher than the reported 7.3% overall, or if the source-request condition lowers first-attempt no-citation rates by more than a few points, the corresponding conclusions must be revised. If retry counts are not present in the release, the reported no-citation and source-request estimates cannot be independently verified from the released data alone.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The protocol in Appendix S3 instructs annotators: 'If no sources, run the question again until you get sources (Cap at 5 times); record how many times you run it till you get sources.' Yet Section 3.1 reports 83/1,140 responses with no citations (7.3%) and Section 3.5 reports per-language no-citation rates such as 33% for Twi and 23% for Japanese under ChatGPT. If each logged response is the first successful attempt, or the fifth failed attempt, then these rates are not per-response probabilities: the observed no-citation rate would be the probability that all attempts in a block fail, not the probability that a user's single query returns no sources. Mean citation counts are likewise conditional on at least one successful retry. This is most damaging for the source-request finding: the paper claims that appending 'List your Sources.' changes behavior only modestly, but the protocol censors the very outcome where the request may matter—whether any sources are returned at all. Appendix S1 shows that gpt-5.4-mini's retrieval rate jumps from 19% to 99% in English when sources are requested, so the product-level null could be partly an artifact of rerunning no-source attempts. Without first-attempt data, the no-citation claims, the volume comparisons, and the 'little benefit' conclusion are not established as stated.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper reports a product-level audit of citations returned by three consumer generative-AI search products (ChatGPT, Perplexity, Google AI Overview) for twenty English mental-health questions under question-only and “List your Sources” prompt conditions, plus a three-question subset translated into six non-English languages spanning high-, medium-, and low-resource tiers. Using 1,140 responses, 15,942 citations, and a nine-category deterministic domain classifier validated against human coding (Cohen's κ ≈ 0.88), the authors find heavy citation concentration (top-ten domains account for 43.6% of English citations), platform differences in source-type composition, modest compositional changes under explicit source requests, and lower citation counts and lower language-appropriate routing for several non-English languages. The authors release the typology, classifier, code, and annotated corpus, and an appendix compares product behavior with developer-API model behavior.","tokens_in":21549,"tokens_out":8258,"duration_ms":87451,"significance":"If the findings hold, this is a valuable and reusable audit of the consumer-facing surfaces where mental-health information seeking now occurs. Its strengths include an outcome measure defined on the product rather than on an inaccessible underlying model, a deterministic classifier with documented human agreement, a leave-one-annotator-out sensitivity analysis, a multilingual design covering resource tiers that are rarely audited together, and public release of the instrument and data. The API-side comparison in Appendix S1 is a useful attempt to separate wrapper-level from model-level retrieval behavior, and its own limitations are stated candidly. However, the retry protocol in Appendix S3 directly affects the interpretation of the no-citation rates, the citation-volume comparisons, and the “little benefit” conclusion for explicit source requests, all of which are central claims. The issue is fixable by reanalysis or targeted re-collection, but it must be resolved before these claims are published as stated.","major_comments":[{"comment":"The annotator protocol instructs: “If no sources, run the question again until you get sources (Cap at 5 times); record how many times you run it till you get sources.” The reported no-citation rates therefore describe completed blocks of up to five attempts, not a single user-issued query. Section 3.1 reports 83/1,140 (7.3%) responses with no citations, and Section 3.5 reports per-language rates such as 33% for Twi and 23% for Japanese under ChatGPT. If attempts were independent, a 33% block-level rate corresponds to roughly an 80% per-attempt no-source probability, so these numbers may substantially understate what a user who asks once experiences. Mean citation counts are also computed over final successful or cap-exhausted attempts, so they overstate typical single-query yield and are not directly comparable across prompt conditions. The authors should recover the recorded retry counts and report first-attempt no-citation rates and first-attempt citation-volume distributions, or re-run the affected conditions, and should not interpret the current block-level estimates as per-query probabilities.","section":"Appendix S3; Sections 3.1 and 3.5"},{"comment":"The conclusion that appending “List your Sources.” “yields little benefit” is not established as stated, because the censoring produced by the capped retry protocol removes exactly the margin along which the instruction plausibly acts: whether any sources are returned at the first attempt. Appendix S1 shows a large model-level effect, with gpt-5.4-mini's English retrieval rate rising from 19.0% to 99.0% and mean English citations per response rising from 0.60 to 7.05 when sources were requested. The product-level null in Section 3.4 could be at least partly an artifact of rerunning no-source attempts in the question-only condition before recording a response. The authors should analyze the effect of the instruction on first-attempt no-citation rates and first-attempt volume, or collect new first-attempt product data, before claiming that explicitly requesting sources changes citation behavior only modestly.","section":"Section 3.4 and Discussion §4"}],"minor_comments":[{"comment":"Citation totals are not internally consistent: the Abstract states 15,942 recorded citations while the Conclusion states 15,929 classified citations, and Section 3.1 reports 10,972 English citations while Section 3.3 uses 10,959 English citations. These differences should be reconciled or explicitly explained.","section":"Abstract, Section 3.1, Section 3.3, Section 5"},{"comment":"The translated prompts are partly unreadable: the Japanese items appear as garbled or mojibake characters, and the three Nepali source-request variants are rendered identically as “लंखत गनुर्”. Because the translated instrument is part of the replication package, these items should be corrected and presented in clean Unicode text.","section":"Appendix S2"},{"comment":"The connection between the observed lexical-neighbor anomalies (e.g., mayonnaise for Mayo Clinic, DSM-Firmenich for DSM) and adversarial typosquatting/combosquatting is appropriately labeled as a future risk, but the paragraph could be tightened by noting explicitly that the examples observed involve benign content and that no adversarial exploitation was found in this dataset.","section":"Section 4.2"},{"comment":"After the reanalysis requested above, the manuscript should define and consistently use the terms “single attempt,” “retried block,” and “response” so that the distinction intended by the protocol is explicit in every estimate and figure caption.","section":"Sections 2.1 and 3.5"}],"recommendation":"major_revision","confidential_remarks":"The reader's acceptance is too optimistic about the retry protocol: the no-citation rates, volume comparisons, and the source-request null are load-bearing and are stated at the block level rather than the first-attempt level. The paper is otherwise well designed and the release is valuable; I would expect a revision that either recovers first-attempt statistics from the recorded retry counts or re-runs the affected conditions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper is a solid empirical contribution to the AI audit literature. It systematically records what ChatGPT, Perplexity, and Google AI Overview cite when asked 20 mental health questions in English and three in six other languages, building a validated source-type classifier and releasing the full corpus. The main findings – that citations concentrate on a narrow institutional core (top-10 domains account for 44% of English citations), that platforms lead with different source types, and that non-English queries get fewer and less locally appropriate citations – are well supported and useful.\n\nWhat you need to know is that the results about no-citation rates and the effect of explicitly requesting sources are weaker than they appear. The annotator protocol in Appendix S3 says: if a query returns no sources, run it again up to five times until you get sources. So the reported 7.3% no-citation rate is the fraction of response blocks where all five attempts failed, not the per-query probability. The first-attempt failure rate is almost certainly higher. This matters most for the claim that appending “List your sources.” changes behavior only modestly: the protocol censors the very outcome where the request might matter (whether any sources appear at all). Given that the API audit in Appendix S1 found that adding the source request flipped gpt-5.4-mini’s retrieval rate from 19% to 99% in English, the product-level null is partly an artifact of retrying until sources appeared. The volume and composition shifts within responses that do contain citations may still be small, but the claim about “little benefit” is not established on the data as collected.\n\nThat said, the paper is transparent about the protocol, releases everything, and the core descriptive findings about domain concentration and source-type composition are robust to this issue. The cross-lingual results are valuable even if based on only three questions per language.\n\nFor whom? Researchers in AI accountability, health informatics, and information retrieval. It deserves a serious peer review, but the authors should be asked to report first-attempt data (or state they no longer have it) and to adjust their conclusions about the source-request manipulation accordingly.\n\nI’d take this to reading group, but I’d cite it only after the revision addresses the retry bias.","headline":"A careful, transparent audit of what three AI products cite for mental health queries, but the retry protocol undermines some of the headline claims about no-citation rates and the effect of asking for sources.","tokens_in":21907,"tokens_out":1486,"would_cite":false,"duration_ms":15332,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Consumer AI search products answering mental health questions concentrate their citations on a narrow core of institutional domains, differ sharply by platform, and route non-English queries to local resources far less reliably.","keywords":["generative AI search","citation audit","mental health information seeking","source credibility","multilingual retrieval","algorithmic gatekeeping","health misinformation","online health information seeking"],"falsifier":"Run the identical protocol with a far larger question base—for example, fifty or more translated questions per language—or re-run the original twenty English questions at a later date: if the ten most-cited domains no longer account for roughly 40% or more of English citations, or if non-English citation volumes and language-appropriate routing match English levels, the concentration and cross-lingual claims would fail to generalize.","tokens_in":22795,"feed_emoji":"🔍","tokens_out":10034,"duration_ms":87914,"temperature":0.7,"pith_summary":"This paper tries to establish that when people ask consumer AI search products—ChatGPT, Perplexity, and Google's AI Overview—about mental health, the sources those products cite are neither neutral nor uniform. Auditing 1,140 responses and 15,942 citations across seven languages, it finds that the ten most-cited domains account for 43.6% of English citations, with government, commercial health, and academic sources each contributing roughly 22%. It also finds that the three platforms lead with different source types, that appending \"List your Sources.\" barely changes citation behavior, and that non-English queries receive fewer citations routed to language-appropriate resources. The reason this matters is that these products now act as gatekeepers of mental-health information: where their citations concentrate, and where they fail, shapes what millions of users see and trust.","feed_headline":"AI mental health answers lean on just 10 websites","feed_subtitle":"An audit of 15,942 citations shows three AI products funnel users to a narrow institutional core.","key_machinery":"The carrier of the argument is a nine-category organizational typology—government/public health, academic/journal, nonprofit health system, commercial health, nonprofit/advocacy, encyclopedia (wiki), news/media, social/video, and other—applied to every citation by an ordered, rule-based classifier that reads the registrable domain, the host, and the public suffix, and validated against human coding with Cohen's $κ$ of 0.86–0.89. This classifier does the work of turning the 15,942 recorded citations into comparable source-type distributions across platforms, question categories, and languages. For the multilingual claims, the load-bearing instrument is a pair of complementary routing measures: citations on a country-code top-level domain associated with the query language, and citations whose URL carries a native-language signal such as a locale path or percent-encoded script—the second catching translated pages like the Spanish section of MedlinePlus that the first measure alone would miss.","core_discovery":"The paper's central claim is that citation behavior in generative mental-health search is measurable, systematic, and skewed. For English queries, the products concentrate attention on a narrow institutional core—ten destinations, led by PubMed/NCBI and Mayo Clinic, account for 43.6% of citations, and government, commercial health, and academic sources each supply about 22%—so a handful of pages effectively define the information environment for sensitive health questions. The platforms are not interchangeable: ChatGPT leans toward government sources and Wikipedia, Perplexity toward academic sources, and Google AI Overview toward commercial health and social/video. Explicitly requesting sources moves the mix only slightly toward institutional sources and does not fix the concentration. For non-English queries the picture worsens: citation counts per response fall, no-citation responses rise sharply for some product–language pairs (a third of ChatGPT's Twi responses and 30% of Google AI Overview's Hindi responses carried no citations), and language-appropriate routing, measured by both country-code domains and native-language pages on international domains, drops steeply with language-resource tier and tracks the supply of authoritative in-language web content rather than any property of the language itself. A parallel audit of comparable vendor API models suggests that the consumer wrapper, not just the underlying model, shapes this surface.","pith_inferences":["If the consumer wrapper is what broadens citation diversity for at least one provider, product-level interventions—capping citation counts per answer, diversifying source types, or prioritizing local content—could change what users see without retraining models; the paper's API comparison supports but does not test such levers.","Localization appears to track the supply of authoritative in-language web content rather than retrieval ability, which implies that publishing and indexing more high-quality local health content would move the language-appropriateness metric more than better multilingual models; that is a testable prediction, not a result the paper establishes.","Citation concentration is plausibly self-reinforcing: systems retrieve what is already heavily indexed and linked, so the top domains would be expected to grow their share across repeated audits, a trend the released instruments could directly measure.","The lexical-neighbor failures suggest a concrete adversarial experiment: plant innocuous pages whose names sit adjacent to the most-cited clinical domains and test whether retrieval systems pull them into mental-health answers, quantifying an attack surface the paper identifies but cannot measure from a single snapshot."],"forward_implications":["A user's choice of product changes the evidence they see: the same mental-health question yields a government-led, academic-led, or commercial-led citation mix depending on whether the answer comes from ChatGPT, Perplexity, or Google AI Overview.","A user's strategy of appending \"List your Sources.\" buys little—citation counts and source composition barely move—so source evaluation effectively stays with the platform even when the user asks for accountability.","For non-English users, especially in medium- and low-resource languages, answers arrive with fewer citations or none at all, and the sources offered are less often local, making the equity gap in mental-health information access partly a citation-routing gap.","Citation concentration concentrates risk: errors, gaps, or biases in a few heavily cited pages propagate to all users, and the observed lexical-neighbor retrieval errors (Mayo Clinic vs. mayonnaise, DSM vs. the fragrance firm DSM-Firmenich) mark an occupiable surface for typosquatting and combosquatting attacks.","The released typology, classifier, and annotated corpus make source composition a quantity that can be tracked over time, so a single-point snapshot can become a continuing audit."],"supporting_citations":[{"why":"Supplies the framing that a consumer product is a model plus a retrieval and citation wrapper, and that retrieval is nondeterministic, motivating the product-level audit and the parallel API comparison.","marker":"Kirsten et al. (2026)"},{"why":"Prior audit establishing that generative search engines produce citations that fail to support their text, the verifiability problem this paper extends to mental-health queries.","marker":"Liu et al. (2023)"},{"why":"Establishes systematic cross-lingual performance inequalities in language technology, grounding the paper's multilingual routing analysis.","marker":"Blasi et al. (2022)"},{"why":"Provides the language resource-tier taxonomy used to assign the seven test languages to high-, medium-, and low-resource tiers.","marker":"Joshi et al. (2020)"},{"why":"Documentation of combosquatting and typosquatting abuse, the basis for the claim that lexical-neighbor retrieval errors mark an occupiable attack surface.","marker":"Kintis et al. (2017)"},{"why":"Public Suffix List used to standardize citations to registrable domains, the step the entire source-type classification depends on.","marker":"Mozilla Foundation (2026)"},{"why":"Credibility-heuristics framework supporting the claim that users trust composed answers and delegate source triage to platforms.","marker":"Metzger & Flanagin (2013)"}],"fun_headline_variants":["AI mental health citations cluster around 10 sites","Ten domains carry 43.6% of AI mental health citations","Chatbots funnel mental health queries to 10 domains","Three AI tools lean on the same narrow citation core","Non-English queries get fewer AI health citations"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The cross-lingual results rest on only three translated questions per non-English language—covering diagnosis, crisis resources, and trustworthy sources—and the paper itself flags that these three must stand in for the full range of mental health queries in each language.","fun_headline_variants_meta":{"raw":{"variants":["AI mental health citations cluster around 10 sites","Ten domains carry 43.6% of AI mental health citations","Chatbots funnel mental health queries to 10 domains","Three AI tools lean on the same narrow citation core","Non-English queries get fewer AI health citations"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"cost":0.00028042,"is_byok":false,"cost_details":{"upstream_inference_cost":0.00028042,"upstream_inference_prompt_cost":0.00016842,"upstream_inference_completions_cost":0.000112},"total_tokens":1603,"prompt_tokens":1203,"completion_tokens":400,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":338}},"tokens_in":1203,"tokens_out":400,"cost_source":"openrouter_usage","duration_ms":5497,"temperature":1.0,"reasoning_tokens":338,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-09T01:08:13.980098+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"falsifier":"Run the identical protocol with a far larger question base—for example, fifty or more translated questions per language—or re-run the original twenty English questions at a later date: if the ten most-cited domains no longer account for roughly 40% or more of English citations, or if non-English citation volumes and language-appropriate routing match English levels, the concentration and cross-lingual claims would fail to generalize.","supporting_citations":[],"review_version":1}