Pith. sign in

REVIEW 4 major objections 5 minor 51 references

Across three model families and four non-English languages, LLM health answers align more strongly with English Wikipedia than with source-language pages; placing a translated local excerpt in the prompt shifts factual alignment toward cult

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Health Q&A chatbots align their answers with English Wikipedia even for non-English prompts, and injecting a non-English Wikipedia excerpt at query time shifts answers toward local references.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The headline English-centric alignment finding is likely a scoring artifact (max over multiple English paragraphs vs a single non-English paragraph); the dataset is useful but the paper's central claim needs a matched re-analysis. the 4 major comments →

arxiv 2510.17476 v2 pith:ELVRHMGE submitted 2025-10-20 cs.CL

Zoom In Disparities in Healthcare LLM Q&A

Ipek Baris Schlicht , Burcu Sayin , Zhixue Zhao , Frederik M. Labont\'e , Cesare Barbera , Marco Viviani , Paolo Rosso , Lucie Flek This is my paper
classification cs.CL
keywords multilingual LLMshealthcare question answeringcross-lingual disparitiesWikipedia coveragefactual alignmentretrieval-augmented generationAlignScoreEnglish-centric bias
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a specific mechanism behind multilingual health-Q&A disparities: a large language model's answers are anchored to whatever the English side of its training data contains, not to the language the user speaks. The authors build a multilingual Wikipedia-derived benchmark, ask three open-weight LLMs health questions in English, German, Turkish, Chinese, and Italian, and score answers against both English and same-language Wikipedia evidence. They find the English reference wins in every language-model combination, and that the bias is steerable: putting a translated excerpt from the non-English Wikipedia page into the prompt raises alignment with that local source dramatically. The authors are careful to call this an alignment-with-Wikipedia result, not a clinical-accuracy result, and this matters because it suggests cheap inference-time interventions rather than retraining could make AI health information more culturally grounded.

Core claim

The central discovery is a stable cross-lingual asymmetry: when the same healthcare question is posed in five languages, the generated answers are consistently more similar, as measured by AlignScore on machine-translated text, to English Wikipedia evidence than to the same-topic page in the prompt's language. The asymmetry is visible across every model-language pair reported in the paper. The second, mechanism-level finding is that the bias can be redirected at inference time: if a short excerpt from the German, Turkish, Chinese, or Italian Wikipedia edition is included in the prompt, alignment with that non-English source jumps sharply, and the English-reference score drops below it. The s

What carries the argument

Two instruments carry the argument. First, MultiWikiHealthCare, a dataset of aligned cross-lingual facts mined from over 800 Wikipedia articles across five languages, with each fact paired to a bilingual evidence paragraph and a generated healthcare question. Second, AlignScore, an English-trained sentence-level consistency model used to measure how close each LLM answer is to a Wikipedia excerpt after all non-English text is machine-translated. The case study's active ingredient is the contextual excerpt: a single translated paragraph from the local Wikipedia edition placed in the prompt, which the paper shows is sufficient to flip the reference-alignment ordering; retrieval-augmented gener

Load-bearing premise

The comparison hinges on the assumption that translating all answers and Wikipedia evidence into English and scoring them with an English-trained consistency model preserves each language's factual ordering fairly; if translation or the scorer systematically favors text that looks like English, the 'English-centrism' is partly a measurement artifact rather than a property of the models.

What would settle it

Have bilingual physician raters judge a random sample of the same answers against original-language Wikipedia evidence without any translation, or use a newly trained multilingual consistency scorer. If native-language alignment is not lower than English alignment once the English-only scorer is removed, the central claim is an artifact of the evaluation language. A cheaper check: replace AlignScore with a round-trip-translation sanity test in which English evidence is translated into the target language and then back, to see how much of the gap is translation noise.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • For healthcare Q&A in German, Turkish, Chinese, and Italian, standard open-weight LLMs are more likely to reproduce the English Wikipedia version of a fact than the local Wikipedia version of the same fact, even when the question is asked in the local language.
  • The English-centrism is not fixed at training time: a single translated local excerpt at inference is enough to move factual alignment toward the source-language reference, suggesting a low-cost mitigation.
  • Retrieval-augmented generation is not automatically a fix; retrieving from an English-biased source reinforces rather than reduces the English alignment.
  • Because non-English Wikipedia editions are much thinner, low local alignment scores partly reflect reference coverage gaps, not purely model bias.
  • Deployments that route non-English users through English-centric knowledge should explicitly ground answers in local-language references if culturally specific practices and guidelines are at stake.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The measured magnitude of 'English alignment' probably overstates the model bias because scoring happens in English after machine translation; an untranslated multilingual scorer or native-speaker judgment could shrink the gap.
  • The same inference-time steering should generalize to any domain where Wikipedia editions differ by language, such as law, nutrition, or local public-health guidance; this can be tested by repeating the case study with, say, dosage guidelines or immunization schedules.
  • A stronger test of the mitigation claim would be to let the model retrieve from the non-English Wikipedia directly instead of using a pre-translated English excerpt; if that works, the practical path is a local-language retrieval pipeline, not manual excerpts.
  • The Chinese result hints that English-bias hurts most exactly where the local Wikipedia is thinnest; an equitable system might combine local excerpts with English-derived knowledge only when local sources are absent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MultiWikiHealthCare, a multilingual Wikipedia-derived healthcare Q&A dataset covering English, German, Turkish, Chinese, and Italian. It first analyzes cross-lingual disparities in Wikipedia coverage, fact overlap, and citation patterns; it then generates questions from source-language Wikipedia facts and evaluates answers from Llama 3.3-70B, Qwen3-Next-80B-A3B-Instruct, and Aya Expanse-32b using AlignScore against both source-language and English Wikipedia evidence. The reported headline finding is that LLM answers align more closely with English Wikipedia than with same-topic pages in the prompt language, and a case study reports that inserting non-English Wikipedia excerpts into the prompt shifts alignment toward the non-English reference, while BM25 RAG with PubMed moves alignment toward English references. The authors explicitly disclaim that the results reflect alignment with Wikipedia rather than clinical correctness.

Significance. The paper addresses a timely and important question for multilingual healthcare AI: whether LLM answers are implicitly anchored to English-language knowledge even when prompted in other languages. The dataset construction is substantial, three model families are compared, and the source of grounding is an external, independently maintained corpus rather than quantities fitted by the authors. The cross-lingual Wikipedia coverage statistics (Tables 6-7) are a useful standalone contribution. If the English-centric alignment result were robust, it would have practical implications for multilingual RAG and prompt design. However, two load-bearing measurement issues currently undermine the central claims: the asymmetric scoring of English versus source-language references in Table 8, and the partly tautological scoring of the Wiki-condition in Table 9. The significance is therefore conditional on a re-analysis with symmetric, non-tautological evaluation.

major comments (4)
  1. [§3.2, §4.2, Table 8] The English-reference AlignScores are computed with a max over multiple aligned English passages: 'For questions with multiple aligned English passages, we report the passage with the highest AlignScore' (§4.2). In contrast, the non-English reference is the single paragraph p_x from the source-language article (§3.2). Because English Wikipedia has significantly more paragraphs and facts (Tables 6-7), a max over a larger candidate set is biased upward relative to a single draw even if the underlying alignment distributions are identical. The small (roughly 1-6 point) and consistent English advantage in Table 8 is exactly what such a bias would produce. The paired direction across the 12 model×language cells does not control for this. Please re-score with a symmetric protocol: e.g., restrict to instances where n=1, or aggregate over all candidate English passages with the same statistic us
  2. [§4.3, Table 9] In the Wiki condition, the non-English Wikipedia excerpt is inserted into the prompt, and the answer is then scored with AlignScore against that same excerpt (translated into English). Conditioning on a document and then measuring similarity to that document makes high alignment partly tautological. This condition demonstrates prompt sensitivity, not that the model's factual knowledge is shifted toward culturally relevant or locally grounded knowledge. To support the stated conclusion, the evaluation should use independent held-out facts from the non-English edition, or at least show that the alignment increase survives when the inserted text is excluded from scoring. The current Table 9 therefore does not establish the abstract's corollary claim.
  3. [§3.1.4, §4.3] The Q&A dataset retains only bidirectionally matched facts: 'the intersection of the two directions for each language pair' (§3.1.4). This means every test question is derived from a fact that also exists in English Wikipedia. The paper's stated goal includes culturally specific knowledge, but the construction excludes facts unique to the non-English edition. The case study's claim about shifting alignment toward culturally relevant knowledge is consequently restricted to facts that already have English counterparts. Please either extend the case study to language-specific facts or explicitly limit the claim to overlapping facts.
  4. [§4.2, Limitations (i)] Because AlignScore is English-only, all non-English answers and evidence are translated into English before scoring. This is disclosed in the limitations, but the abstract and contribution statements call the measured quantity 'factuality alignment.' Translationese and an English-centric scorer provide a plausible alternative explanation for lower source-language-reference scores, particularly for Chinese where only 7.0% of facts align with English (Table 6). Given that the headline finding is about alignment with English Wikipedia versus the prompt language's Wikipedia, I ask for a robustness check (e.g., a multilingual consistency model or native-speaker fact-level judgments on a subset) or, at minimum, a consistent rewording of the abstract to match the cautious framing already used in §4.2 and the Limitations.
minor comments (5)
  1. [§4.2, after Table 8] The sentence 'In particular, factual alignment is higher when evaluated against source-language evidence than against evidence from the English Wikipedia pages' appears to contradict Table 8, where source-language reference scores are lower than English-reference scores in every query/reference pair. Please rephrase to match the table.
  2. [Table 6] The Facts column for English appears as '20,5468', which is likely a typo for '205,468'. Please correct the formatting.
  3. [Table 8] The table reports means only, with no standard deviations, confidence intervals, or significance tests. Given the small differences (1-6 points), please add uncertainty estimates or paired significance tests for the non-en vs en comparison.
  4. [Abstract, §4.2, Limitations] The abstract uses 'factuality alignment' while §4.2 describes AlignScore as a 'relative proxy' and Limitations (iv) disclaims clinical correctness. Please align the abstract wording with this more cautious framing.
  5. [Figures 2 and 7] The judge model is labeled 'ChatGPT-5' in Figure 2 and 'GPT-5' in Figure 7; use one consistent name.

Circularity Check

2 steps flagged

Headline English-alignment result is partly an artifact of scoring by construction: English scores are maximized over multiple aligned passages while non-English scores use a single paragraph; the Wiki-condition case study scores against the very excerpt it inserts.

specific steps
  1. self definitional [Section 3.2 (data instance definition) and Section 4.2 (AlignScore evaluation), Table 8]
    "Formally, let each data instance be represented as d= (f x, fen, px, pen), where f_x is a fact in Language X, f_en = {f^1_en, ..., f^n_en} (n≥1) is its aligned English fact(s), p_x is the paragraph in Language X containing f_x, and p_en is the aligned English paragraph(s). ... For questions with multiple aligned English passages, we report the passage with the highest AlignScore."

    The non-English alignment score is computed against a single source paragraph p_x, while the English alignment score is the maximum over a set of aligned English paragraphs p_en (n≥1). A maximum over more candidates is biased upward relative to a single draw even if the underlying distributions are identical. Tables 6 and 7 show English Wikipedia has far more paragraphs and aligned facts, so Table 8's 'answers align more closely with English pages' is partly produced by the scoring definition itself, not measured on a like-for-like comparison.

  2. self definitional [Section 4.3 (Case Study: Factuality Alignment), Table 9]
    "For the first method, we incorporate translated excerpts from non-English Wikipedia pages that are semantically aligned with the given question. ... LLMs incorporating excerpts from Wikipedia produce responses that align more closely with non-English references"

    In the 'Wiki' condition, the exact non-English Wikipedia passage is inserted into the prompt and the answer is then scored with AlignScore against that same non-English reference. Therefore the high non-en scores in Table 9 (about 77–83) largely reflect the model conditioning on, and often echoing, the supplied text; the measured 'shift toward culturally relevant knowledge' is partly tautological because the reference used for evaluation is the same text provided as input. The comparison to the Base condition has some content, but this intervention is not an independent test of the model's latent alignment.

full rationale

The paper is not circular in the usual fit/self-citation sense: the dataset is built from Wikipedia, the LLMs are external systems, and no parameter is fitted to force the headline table. However, the central claim 'responses align more with English Wikipedia, even when the prompts are non-English' rests on an internal asymmetry in the scoring construction. Section 3.2 defines p_x as a single non-English paragraph while p_en is a set, and Section 4.2 explicitly reports the maximum AlignScore over multiple aligned English passages. Since English Wikipedia has far more passages per topic (Tables 6–7), the English score is inflated by construction relative to a single non-English paragraph, so Table 8's English-reference advantage is partly a property of the metric rather than an independently established model behavior. Similarly, the case-study 'Wiki' condition inserts the non-English reference into the prompt and then scores the answer against that same reference, making the large alignment gains in Table 9 partly definitional. The paper does disclose translation and English-centric scoring limitations, and those are separate concerns; the circularity here is the comparison of a max-over-candidates English score with a single-candidate non-English score, and the use of the injected reference as the evaluation target. This warrants a partial-circularity score of 6 rather than a full 10, because the Wikipedia coverage analysis, the dataset, and the general observation of cross-lingual variation have independent content.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The ledger contains no fitted parameters in the forecasting sense: nothing is tuned to maximize the headline numbers. The entries are curation and configuration choices that define what 'factual alignment' means in this paper and therefore what the central claim depends on: which facts survive the alignment intersection, which classifier tags relevance, which judge accepts questions, and which RAG configuration injects context. The axioms make the reference-side assumptions explicit: Wikipedia is the proxy for pretraining knowledge and reference truth; translated AlignScore is treated as language-fair; InfoGap alignments are trusted at measured F1 0.638–0.841. The single 'invented' entity is the benchmark dataset itself, which is not yet released and so has no independent check outside this paper.

free parameters (4)
  • Fact retention rule: bidirectionally matched InfoGap facts only = Intersection of en↔X alignments (7.0–33.8% of facts retained per language)
    Drops one-directional matches; the retained fact set defines both the questions and the reference evidence, so this curation choice shapes every alignment score in the paper.
  • Relevance classifier per language = Fine-tuned transformers; F1-macro 68.5–88.4 (Table 4)
    Facts are discarded/tagged by classifiers trained on GPT-4o-mini (DeepSeek-R1 for Chinese) labels; for all languages except Chinese the cross-lingual model was selected on dev F1, changing which content enters the dataset.
  • Question-quality pre-filter (ChatGPT-5) = Human agreement 44–76%; filter applied
    Used to accept/reject generated questions despite low and uneven agreement with human labels; alters the evaluation set before any model is tested.
  • RAG configuration = BM25-Sparse, top-10 articles, discard entities with <50 PubMed articles
    Case-study only; retrieval configuration determines the context injected into the prompt and hence the alignment direction observed.
axioms (5)
  • domain assumption Wikipedia editions are a representative proxy for the health content LLMs acquire in pretraining and the reference truth for healthcare facts.
    Invoked in Sections 1 and 4 (Wikipedia is 'a pretraining corpus for many LLMs', citing Singhal et al. 2023); both the source-level analysis and the response-level references rest on it.
  • domain assumption AlignScore computed on English translations is a language-fair measure of factual alignment.
    Section 4.2 translates all non-English answers and evidence into English before scoring; the central en-vs-non-en comparison assumes translation does not bias the comparison. Acknowledged in Limitations (i) but not tested.
  • domain assumption InfoGap (GPT-4o-mini backbone) cross-lingual fact extraction and alignment is sufficiently accurate.
    Section 3.1.3: fact pairs are the unit of analysis, validated on only ~50 facts/language with F1-macro 0.638–0.841 vs. humans (Table 2); alignment errors propagate into the Q&A pairs and all downstream scores.
  • domain assumption Higher AlignScore against a Wikipedia reference implies higher factual alignment, not just lexical-semantic overlap.
    Section 4.2 calls the scores 'a relative proxy for factual alignment rather than a comprehensive quality metric'; Wikipedia excerpts are not independently fact-checked (Limitations (ii)).
  • standard math Standard ML/NLP machinery works as expected: pretrained encoders, fine-tuning, BM25 retrieval, paired t-tests, Spearman correlation.
    Background tooling used throughout Sections 3–4; no unusual mathematics is introduced.
invented entities (1)
  • MultiWikiHealthCare dataset no independent evidence
    purpose: Reference corpus of aligned cross-lingual health facts plus generated Q&A; defines both the questions and the evidence used for every alignment score in the paper.
    Not yet released ('will be released publicly after review'), so the central artifact has no external inspection, replication, or falsifiable handle outside this paper.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Zoom In Disparities in Healthcare LLM Q&A." pith.science (2026). https://pith.science/paper/ELVRHMGE

@misc{pith2026251017476,
  author       = {Pith},
  title        = {Pith review of: Zoom In Disparities in Healthcare LLM Q&A},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ELVRHMGE}},
  note         = {Machine review of arXiv:2510.17476}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Equitable access to reliable health information is vital when integrating AI into healthcare. Yet, information quality varies across languages, raising concerns about the reliability and consistency of multilingual Large Language Models (LLMs). We systematically examine cross-lingual disparities in pre-training source and factuality alignment in LLM answers for multilingual healthcare Q&A across English, German, Turkish, Chinese (Mandarin), and Italian. We (i) constructed Multilingual Wiki Health Care (MultiWikiHealthCare), a multilingual dataset from Wikipedia; (ii) analyzed cross-lingual healthcare coverage; (iii) assessed LLM response alignment with these references; and (iv) conducted a case study on factual alignment through the use of contextual information and Retrieval-Augmented Generation (RAG). Our findings reveal substantial cross-lingual disparities in both Wikipedia coverage and LLM factual alignment. Across LLMs, responses align more with English Wikipedia, even when the prompts are non-English. Providing contextual excerpts from non-English Wikipedia at inference time effectively shifts factual alignment toward culturally relevant knowledge. These results highlight practical pathways for building more equitable, multilingual AI systems for healthcare.

Figures

Figures reproduced from arXiv: 2510.17476 by Burcu Sayin, Cesare Barbera, Frederik M. Labont\'e, Ipek Baris Schlicht, Lucie Flek, Marco Viviani, Paolo Rosso, Zhixue Zhao.

Figure 1
Figure 1. Figure 1: Analyzing source- and response-level disparity and factuality alignment: (1) comparison of Turkish and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: MultiWikiHealthCare - Pipeline for Q&A construction 3.1 Construction of Aligned facts 3.1.1 Healthcare Topics To construct MultiWikiHealthCare, we first curated a list of trending and controversial health topics2 by using Google Trends,3 and Wikipedia’s list of controversial issues in science, biology, health,4 and related surveys (Schlicht et al., 2024). These topics formed the basis of content collection… view at source ↗
Figure 3
Figure 3. Figure 3: Word cloud of trending subtopics derived [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Prompt for finding Wikipedia pages work (Kwon et al., 2023), to (i) filter out entities not related to healthcare and (ii) link the remain￾ing entities to their corresponding Wikipedia pages. After removing duplicates, we retained 918 unique Wikipedia pages. Wikipedia page titles often differ across lan￾guages, especially when scripts differ or the title is a common noun rather than a proper name (e.g., En… view at source ↗
Figure 5
Figure 5. Figure 5: Prompt for selecting relevant facts After cross-lingual extraction and alignment, we retained only bidirectionally matched facts: the intersection of the two directions for each language 6 In (Samir et al., 2024), random guessing outperformed Natural Language Inference transformers. 4 [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompt for health-care question generation [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt for quality check with GPT-5 7 FacebookAI/xlm-roberta-base 5 [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 1 canonical work pages

  1. [1]

    Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner - Fushman. 2017. Overview of the medical question answering task at TREC 2017 liveqa. In TREC , volume 500-324 of NIST Special Publication . National Institute of Standards and Technology (NIST)

    2017

  2. [2]

    Goodwin, Sonya E

    Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis R. Goodwin, Sonya E. Shooshan, and Dina Demner - Fushman. 2019. Bridging the gap between consumers' medication questions and trusted answers. In MedInfo, volume 264 of Studies in Health Technology and Informatics, pages 25--29. IOS Press

    2019

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  4. [4]

    Vibhor Agarwal, Yiqiao Jin, Mohit Chandra, Munmun De Choudhury, Srijan Kumar, and Nishanth Sastry. 2024. Medhalu: Hallucinations in responses to healthcare queries by large language models. arXiv preprint arXiv:2409.19492

    arXiv 2024

  5. [5]

    John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, and 1 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261

    Pith/arXiv arXiv 2024

  6. [6]

    Myles Davaris, Stephen Barnett, Robert Abouassaly, Nathan Lawrentschuk, and 1 others. 2017. Thoracic surgery information on the internet: a multilingual quality assessment. Interactive Journal of Medical Research, 6(1):e6732

    2017

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171--4186, M...

    2019

  8. [9]

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. https://doi.org/10.18653/v1/2024.eacl-demo.16 RAGA s: Automated evaluation of retrieval augmented generation . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150--158, St. Julians, Malta. A...

    doi:10.18653/v1/2024.eacl-demo.16 2024

  9. [10]

    John M Ford. 2004. Content analysis: An introduction to its methodology. Personnel psychology, 57(4):1110

    2004

  10. [11]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The L lama 3 H erd of M odels. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  11. [12]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. https://arxiv.org/abs/2411.15594 A survey on llm-as-a-judge . arXiv, abs/2411.15594

    Pith/arXiv arXiv 2025

  12. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633--638

    2025

  13. [14]

    Ashim Gupta, Maitrey Mehta, Zhichao Xu, and Vivek Srikumar. 2025. Found in translation: Measuring multilingual llm consistency as simple as translate then evaluate. arXiv preprint arXiv:2505.21999

    Pith/arXiv arXiv 2025

  14. [15]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. GPT -4o S ystem C ard. arXiv preprint arXiv:2410.21276

    Pith/arXiv arXiv 2024

  15. [16]

    Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, and Srijan Kumar. 2024. Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries. In Proceedings of the ACM Web Conference 2024, page 2627–2638, New York, NY, USA. Association for Computing Machinery

    2024

  16. [17]

    Navreet Kaur, Monojit Choudhury, and Danish Pruthi. 2024. Evaluating large language models for health-related queries with presuppositions. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14308--14331, Bangkok, Thailand. Association for Computational Linguistics

    2024

  17. [18]

    Yubin Kim, Hyewon Jeong, Shan Chen, Shuyue Stella Li, Mingyu Lu, Kumail Alhamoud, Jimin Mun, Cristina Grau, Minseok Jung, Rodrigo Gameiro, and 1 others. 2025. Medical hallucinations in foundation models and their impact on healthcare. arXiv preprint arXiv:2503.05777

    arXiv 2025

  18. [19]

    Bevan Koopman and Guido Zuccon. 2023. Dr C hat GPT tell me what I want to hear: How different prompts impact health answer correctness. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15012--15022, Singapore. Association for Computational Linguistics

    2023

  19. [20]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  20. [21]

    Health information quality on the internet in gynecological oncology: a multilingual evaluation

    N Lawrentschuk, R Abouassaly, E Hewitt, A Mulcahy, DM Bolton, and T Jobling. Health information quality on the internet in gynecological oncology: a multilingual evaluation. Eur. J. Gynaecol. Oncol, 37(4):2016

    2016

  21. [22]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Sy...

    2020

  22. [23]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    Pith/arXiv arXiv 2019

  23. [24]

    Xing Han Lù. 2024. https://arxiv.org/abs/2407.03618 Bm25s: Orders of magnitude faster lexical search via eager sparse scoring

    Pith/arXiv arXiv 2024

  24. [25]

    Hellina Hailu Nigatu, Nuredin Ali Abdelkadir, Fiker Tewelde, Stevie Chancellor, and Daricia Wilkinson. 2025. I nto the V oid: U nderstanding O nline H ealth I nformation in L ow- W eb D ata L anguages. arXiv preprint arXiv:2509.20245

    arXiv 2025

  25. [26]

    OpenAI. 2025. Introducing gpt-5. https://openai.com/index/introducing-gpt-5/. Accessed 2025-09-29

    2025

  26. [27]

    Chan Young Park, Xinru Yan, Anjalie Field, and Yulia Tsvetkov. 2021. Multilingual C ontextual A ffective A nalysis of LGBT P eople P ortrayals in W ikipedia. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 479--490

    2021

  27. [28]

    Surangika Ranathunga and Nisansa de Silva. 2022. https://doi.org/10.18653/v1/2022.aacl-main.62 Some languages are more equal than others: Probing deeper into the linguistic disparity in the NLP world . In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conferen...

    doi:10.18653/v1/2022.aacl-main.62 2022

  28. [29]

    David Restrepo, Chenwei Wu, Zhengxu Tang, Zitao Shuai, Thao Nguyen Minh Phan, Jun-En Ding, Cong-Tinh Dao, Jack Gallifant, Robyn Gayle Dychiao, Jose Carlo Artiaga, André Hiroshi Bando, Carolina Pelegrini Barbosa Gracitelli, Vincenz Ferrer, Leo Anthony Celi, Danielle Bitterman, Michael G Morley, and Luis Filipe Nakayama. 2025. Multi-ophthalingua: A multilin...

    2025

  29. [30]

    Amanda Ross and Victor L Willson. 2017. Paired samples t-test. In Basic and advanced statistical tests: Writing results sections and creating tables and figures, pages 17--19. Springer

    2017

  30. [31]

    Farhan Samir, Chan Young Park, Anjalie Field, Vered Shwartz, and Yulia Tsvetkov. 2024. Locating information gaps and narrative inconsistencies across languages: A case study of LGBT people portrayals on W ikipedia. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6747--6762, Miami, Florida, USA. Association ...

    2024

  31. [32]

    Burcu Sayin, Pasquale Minervini, Jacopo Staiano, and Andrea Passerini. 2024. Can LLM s correct physicians, yet? investigating effective interaction methods in the medical domain. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 218--237, Mexico City, Mexico. Association for Computational Linguistics

    2024

  32. [33]

    Ipek Baris Schlicht, Eugenia Fernandez, Berta Chulvi, and Paolo Rosso. 2024. Automatic detection of health misinformation: a systematic review. Journal of Ambient Intelligence and Humanized Computing, 15(3):2009--2021

    2024

  33. [34]

    Ipek Baris Schlicht, Zhixue Zhao, Burcu Sayin, Lucie Flek, and Paolo Rosso. 2025. Do llms provide consistent answers to health-related questions across languages? In Advances in Information Retrieval (ECIR 2025), pages 314--322, Cham. Springer Nature Switzerland

    2025

  34. [35]

    Stefan Schweter. 2020. https://doi.org/10.5281/zenodo.3770924 BERT urk - BERT models for T urkish . https://doi.org/10.5281/zenodo.3770924

    doi:10.5281/zenodo.3770924 2020

  35. [36]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, and 13 others. 2023. Large language models encode cli...

    2023

  36. [37]

    Aaron A Tierney, Mary E Reed, Richard W Grant, Florence X Doo, Denise D Pay \'a n, and Vincent X Liu. 2025. H ealth E quity in the E ra of L arge L anguage? M odels. American Journal of Managed Care, 31(3)

    2025

  37. [38]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others. 2023. Llama 2: O pen F oundation and F ine- T uned C ha...

    Pith/arXiv arXiv 2023

  38. [39]

    Zining Wang, Yuxuan Zhang, Dongwook Yoon, Nicholas Vincent, Farhan Samir, and Vered Shwartz. 2025. Wikigap: Promoting epistemic equity by surfacing knowledge gaps between english wikipedia and other language editions. arXiv preprint arXiv:2505.24195

    arXiv 2025

  39. [40]

    u ller, S Bartelt, S Schulz, JS M \

    Christian Weissenberger, S Jonassen, J Beranek-Chiu, M Neumann, D M \"u ller, S Bartelt, S Schulz, JS M \"o nting, K Henne, G Gitsch, and 1 others. 2004. B reast cancer: patient information needs reflected in E nglish and G erman web sites. British Journal of Cancer, 91(8):1482--1487

    2004

  40. [41]

    Clark Wissler. 1905. The spearman correlation formula. Science, 22(558):309--311

    1905

  41. [42]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \' e mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transformers: State-of-the-art natural language processing....

    2020

  42. [43]

    Fangyuan Xu, Junyi Jessy Li, and Eunsol Choi. 2022. How do we answer complex questions: Discourse structure of long-form answers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3556--3572, Dublin, Ireland. Association for Computational Linguistics

    2022

  43. [44]

    Niraj Yagnik, Jay Jhaveri, Vivek Sharma, and Gabriel Pila. 2024. Medlm: Exploring language models for medical question answering systems. arXiv preprint arXiv:2401.11389

    Pith/arXiv arXiv 2024

  44. [45]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  45. [46]

    Tian Yang and Susana Valdez. 2025. https://doi.org/10.1075/dt.25015.yan H ow machine translation is used in healthcare . Digital Translation

    work page doi:10.1075/dt.25015.yan 2025

  46. [47]

    Haoran Yu, Chang Yu, Zihan Wang, Dongxian Zou, and Hao Qin. 2024. E nhancing H ealthcare through L arge L anguage M odels: A S tudy on M edical Q uestion A nswering. In 2024 IEEE 6th International Conference on Power, Intelligent Computing and Systems (ICPICS), pages 895--900. IEEE

    2024

  47. [48]

    Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. A lign S core: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328--11348, Toronto, Canada. Association for Computational Linguistics

    2023

  48. [49]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, and 1 others. 2025. Siren’s song in the ai ocean: A survey on hallucination in large language models. Computational Linguistics, pages 1--46

    2025

  49. [50]

    Ming Zhu, Aman Ahuja, Wei Wei, and Chandan K. Reddy. 2019. A hierarchical attention retrieval model for healthcare question answering. In The World Wide Web Conference, WWW '19, page 2472–2482, New York, NY, USA. Association for Computing Machinery

    2019

  50. [51]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  51. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.