HealthQA-BR: A System-Wide Benchmark Reveals Critical Knowledge Gaps in Large Language Models

Andrew Maranh\~ao Ventura D'addario

REVIEW 3 major objections 5 minor 1 cited by

HealthQA-BR: A System-Wide Benchmark Reveals Critical Knowledge Gaps in Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that aggregate accuracy scores on healthcare benchmarks hide a "spiky" knowledge profile, where a top model like GPT 4.1 can score 86.6% overall yet only 60.0% in neurosurgery and 68.4% in social work.

desk verdict HealthQA-BR is a genuinely useful new benchmark and the aggregate-score warning is right, but the specialty-level numbers are too fragile to support the current claims. read the letter →

arxiv 2506.21578 v1 pith:XOJ3ESFG submitted 2025-06-16 cs.CL cs.AI

Andrew Maranhão Ventura D'addario This is my paper · ORCID

classification cs.CLcs.AI

keywords largelanguagemodelshealthcarebenchmarkmedicalAIevaluationPortuguese-languageNLPBrazilianlicensingexamsalliedhealthprofessionsspecialty-levelaccuracyzero-shot

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a single top-line accuracy score creates a dangerous illusion of competence in medical AI evaluation. To test this, it introduces HealthQA-BR, a 5,632-question benchmark built from Brazil's national licensing and residency exams that spans medicine, nursing, dentistry, psychology, social work, and other allied health professions. Zero-shot evaluation of more than 20 large language models shows that top models meet or exceed overall passing thresholds, but specialty-level accuracy varies wildly: GPT 4.1 drops from 98.7% in ophthalmology to 60.0% in neurosurgery and 68.4% in social work. The paper claims this uneven, "spiky" profile is systemic across all evaluated models, so reporting only aggregate scores is insufficient for safety validation.

What carries the argument

The central instrument is HealthQA-BR, a 5,632-question multiple-choice dataset drawn from Brazilian licensing and residency examinations and tagged with more than 30 professional and subspecialty categories. The mechanism that carries the argument is the fine-grained metadata tagging, which lets the authors compute accuracy per specialty and profession rather than only overall; the "spiky profile" claim is exactly the variance among those per-category scores.

What would settle it

Compute exact binomial confidence intervals for each specialty-level accuracy from the reported question counts; if the intervals for top and bottom specialties overlap, the spiky-profile claim loses much of its evidentiary support. An independent re-tagging of the full dataset's specialty labels would likewise test whether the observed gaps are artifacts of metadata noise.

Watch

Extended reading notes

Core claim

The central discovery is that high overall performance masks a spiky, specialty-dependent knowledge profile: the same model that nears perfection in one area can barely reach the passing threshold in another, and this pattern is not unique to one model but appears across the entire cohort. The paper presents this as direct evidence that single-score evaluation of medical AI is inadequate and that granular, specialty- and profession-level auditing is a prerequisite for assessing safety and reliability.

Load-bearing premise

Specialty-level accuracy estimates are treated as reliable even though several categories contain very few questions, so the claimed gaps may partly reflect sampling noise.

Editorial extensions

If this is right

Even top-performing models cannot be certified as clinically safe on the basis of overall accuracy; specialty-level validation is argued to be a fundamental prerequisite.
Disciplines integral to public health, such as social work and collective medicine, show notably lower scores, suggesting a bias in training data toward specialized medicine at the expense of community-focused care.
Public release of the benchmark and evaluation suite lets other research groups audit additional models and run similar system-wide evaluations for Portuguese-speaking healthcare.
Developers can use the specialty-level breakdown to target remediation, such as fine-tuning or retrieval-augmented generation on the specific areas where a model is weakest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the spiky-profile finding generalizes, a natural extension is that safety validation for medical LLMs should require a per-specialty coverage map rather than a single pass/fail threshold; the paper implies this but does not formalize an evaluation standard.
Because specialty-level samples are small and no confidence intervals are reported, the headline gaps should be treated as provisional until replicated on larger or independently tagged question sets.
A direct test of the proposed training-data bias would be to compare model performance on public-health and primary-care items against specialized items while controlling for question difficulty or wording.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

HealthQA-BR is a real contribution: the first large Portuguese-language benchmark covering not just medicine but nursing, psychology, social work and other allied professions, built from 5,632 official licensing and residency questions. That fills a genuine gap, and the aggregate evaluation of 21 models is useful. The overall rankings look plausible, and the authors deserve credit for framing the work around interprofessional care rather than another physician-only MCQ set.

The central warning is also right: an 86.6% overall score can hide 60% in Neurosurgery. The qualitative point that knowledge is uneven is well taken. But the quantitative case is thinner than the paper suggests. Several specialty categories are tiny: 25 Neurosurgery, 14 Dermatology, 10 Anesthesiology, 8 Rheumatology. A one-answer change moves those percentages by 4 to 12 points, and the paper gives no confidence intervals. So the specific gap between 98.7% and 60.0% is real in direction but not in magnitude.

The bigger problem is the "systemic across all models" claim in Section 4.2. The paper only shows GPT 4.1's specialty breakdown in detail. There is no per-model table or variance measure for the other 20 models, so the universal claim is an assertion, not a result. The metadata audit on 82 of 5,632 questions is also too thin to support fine-grained categories; a few tagging errors in a category of 8 or 14 questions directly changes the headline numbers.

Two smaller things: the identical 68.4% for Orthopedics and Social Work looks like a copy-paste error, and the paper promises public release but provides no link or repository. Training-data contamination from public exams is also not discussed, which matters for a benchmark of public questions.

Who is this for? People building or evaluating medical LLMs, especially for Portuguese or allied health contexts, will get value from the resource and the discussion. The conclusion—report granular accuracy, not a single score—holds up even if the exact percentages don't.

Recommendation: accept for peer review, but with revisions. The dataset and question are important enough to warrant referee time. The authors need to release the data, add confidence intervals, show per-model specialty spread, fix the duplicated number, and tighten the language from "systemic" to "observed in the models we examined." As it stands, the paper is a good idea with fragile supporting numbers.

Referee Report

3 major / 5 minor

Summary. The paper introduces HealthQA-BR, a benchmark of 5,632 multiple-choice questions drawn from Brazilian national licensing and residency exams (Revalida and Enare), covering medicine, nursing, dentistry, psychology, social work, and other allied health fields. The authors report zero-shot accuracy for over 20 LLMs, finding that top models such as GPT 4.1 reach 86.6% overall accuracy but show strong variability across specialties, with scores as low as 60.0% in Neurosurgery and 68.4% in Social Work. They argue that aggregate scores create a dangerous illusion of competence and that specialty- and profession-level reporting is necessary for safe deployment of medical AI.

Significance. If its specialty-level claims are supported, HealthQA-BR addresses a genuine gap in LLM evaluation by providing a Portuguese-language, interprofessional healthcare benchmark. The dataset construction from official high-stakes exams, the effort to deduplicate questions, and the public release of the benchmark and evaluation suite are concrete contributions. The aggregate model rankings in Table 4 are based on large question counts and appear internally consistent, so the overall performance hierarchy is likely robust. However, the central narrative of a systematically 'spiky' knowledge profile depends on specialty-level accuracy estimates that are currently not statistically supported, and the claimed universality across models is not directly demonstrated.

major comments (3)

[Table 2 and Figure 2] The specialty-level accuracy estimates for GPT 4.1 are presented to one decimal place without confidence intervals, despite very small category sizes: Neurosurgery has 25 questions, Dermatology 14, Anesthesiology 10, and Rheumatology 8. For n=25, a one-answer change moves accuracy by 4 percentage points; for n=8, by 12.5 points. The headline figure of 60.0% in Neurosurgery therefore has a wide confidence interval and is statistically indistinguishable from many other specialties. The paper should report exact hit counts and confidence intervals (or credible intervals) for every specialty, and should either aggregate low-sample categories further or explicitly mark them as unreliable.
[Section 4.2 and Abstract] The claim that the 'spiky' profile is 'a systemic issue observed across all models' is asserted but not supported by any per-model specialty data. Only GPT 4.1 is analyzed in detail; Table 4 reports only overall and exam-level scores, not specialty breakdowns. To support the universal claim, the authors should provide specialty-level accuracy for all evaluated models, or at least a variance measure per model, and demonstrate that the unevenness is not driven by small-sample noise. Without this, the paper's central generalization remains an unsupported extrapolation from a single model.
[Section 2.2, item 6] The metadata tags are the infrastructure for every specialty-level result, yet the final quality assurance audit validated only 82 of 5,632 questions (~1.5%), and no per-tag error rate or inter-annotator agreement is reported. Because a single mislabeled question in a small specialty can change the reported accuracy by several percentage points, the authors should either expand the audit substantially, provide a detailed breakdown of audit results by specialty, or run the analysis under realistic assumptions about metadata error rates to show the conclusions are unchanged.

minor comments (5)

[Table 2 vs Table 3] The identical reported value of 68.4% for Orthopedics and Traumatology (n=79) and Social Work (n=79) is suspicious; please verify the underlying counts and report the raw hit counts (e.g., 54/79) so readers can check the arithmetic.
[Section 3.3] The 'Most Common Answer' baseline is stated as 21.89% for a five-option MCQ. This value needs a derivation or citation, since random uniform guessing would be 20%.
[Figures 2 and 3] The figures would be more informative if they included sample sizes or error bars, and the captions should state that the percentages are point estimates without confidence intervals.
[Section 5] The statement that findings are 'strongly corroborated' by AfriMed-QA goes beyond what is demonstrated in the paper; the comparison is qualitative and no systematic analysis of variability is provided. Please soften or support this claim.
[Abstract and Section 3.1] The abstract says 'over 20' models while Section 3.1 says 'over 21'; please make the count consistent throughout, and list the exact number of models in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark results are direct empirical measurements against externally authored exams, not derivations from the paper's own assumptions.

full rationale

HealthQA-BR is an empirical evaluation study. The central claims—that GPT 4.1 reaches 86.6% overall but drops to 60.0% in Neurosurgery and 68.4% in Social Work, and that this spiky profile is systemic—are supported by accuracy measurements computed from model responses to multiple-choice questions sourced from Brazil's national licensing and residency exams. There is no fitted parameter, no quantity defined in terms of another claimed result, and no derivation chain that reduces to the paper's inputs. The specialty-level percentages are direct counts of correct answers divided by question counts, and any concern about their statistical reliability is a correctness or external-validity issue, not circularity. The paper's use of AfriMed-QA as external corroboration is an independent comparison, and there are no self-citations carrying load-bearing weight. The acknowledged limitations about MCQ format and accuracy as a metric do not indicate circular reasoning; they are scope statements. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted free parameters or theoretical constructs are needed because the paper is an empirical benchmark evaluation. The load-bearing assumptions are exam validity, reliability of the manual metadata tags, and zero-shot MCQ accuracy as a proxy for clinical knowledge; none of these is independently verified in the manuscript.

assumptions (3)

domain assumption Official Revalida and Enare questions and answer keys are a valid measure of the clinical knowledge needed for professional practice in Brazil.
Invoked in Section 2.1, where the exams are described as high-stakes and vetted by subject matter experts.
domain assumption The semi-automated curation and manual metadata tagging correctly identify the specialty or profession of each question.
Section 2.2 relies on manual review and a final audit of only 82 of 5,632 questions, so tagging errors in the remaining 98.5% could change specialty-level results.
domain assumption Accuracy on multiple-choice questions is an informative proxy for clinical knowledge, and zero-shot prompting is a fair evaluation mode.
Section 3.2 uses zero-shot prompting; Section 5 acknowledges that MCQ format and accuracy are a narrow approach to evaluating LLMs in healthcare.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HealthQA-BR: A System-Wide Benchmark Reveals Critical Knowledge Gaps in Large Language Models." pith.science (2026). https://pith.science/paper/XOJ3ESFG

@misc{pith2026250621578,
  author       = {Pith},
  title        = {Pith review of: HealthQA-BR: A System-Wide Benchmark Reveals Critical Knowledge Gaps in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XOJ3ESFG}},
  note         = {Machine review of arXiv:2506.21578}
}

read the original abstract

The evaluation of Large Language Models (LLMs) in healthcare has been dominated by physician-centric, English-language benchmarks, creating a dangerous illusion of competence that ignores the interprofessional nature of patient care. To provide a more holistic and realistic assessment, we introduce HealthQA-BR, the first large-scale, system-wide benchmark for Portuguese-speaking healthcare. Comprising 5,632 questions from Brazil's national licensing and residency exams, it uniquely assesses knowledge not only in medicine and its specialties but also in nursing, dentistry, psychology, social work, and other allied health professions. We conducted a rigorous zero-shot evaluation of over 20 leading LLMs. Our results reveal that while state-of-the-art models like GPT 4.1 achieve high overall accuracy (86.6%), this top-line score masks alarming, previously unmeasured deficiencies. A granular analysis shows performance plummets from near-perfect in specialties like Ophthalmology (98.7%) to barely passing in Neurosurgery (60.0%) and, most notably, Social Work (68.4%). This "spiky" knowledge profile is a systemic issue observed across all models, demonstrating that high-level scores are insufficient for safety validation. By publicly releasing HealthQA-BR and our evaluation suite, we provide a crucial tool to move beyond single-score evaluations and toward a more honest, granular audit of AI readiness for the entire healthcare team.

Figures

Figures reproduced from arXiv: 2506.21578 by the authors.

**Figure 2.** GPT 4.1 Accuracy by Medical Specialty. The chart displays the accuracy of the [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** GPT 4.1 Accuracy by Health Specialty. The chart displays the accuracy of the top [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering
cs.CL 2026-07 conditional novelty 5.0 of 10

Hybrid RAG over UK public health guidance sharply raises MCQA accuracy and free-form faithfulness, letting smaller open models match larger closed models without retrieval.

Reference graph

Works this paper leans on

15 extracted references · 12 canonical work pages · cited by 1 Pith paper

[1]

How large language models perform on the United States Medical Licensing Examination: a systematic review

Frank E, Lee JWY. How large language models perform on the United States Medical Licensing Examination: a systematic review. medRxiv [Preprint]. 2023 May 26 [cited 2024 Jun 11]. Available from: https://www.medrxiv.org/content/10.1101/2023.05. 26.23290649v1

work page doi:10.1101/2023.05 2023
[2]

AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Bench- mark Dataset

Olatunji T, Nimo C, Owodunni A, Abdullahi T, Ayodele E, Sanni M, et al. AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Bench- mark Dataset. arXiv [Preprint]. 2024 Nov 23 [cited 2025 Jun 11]. Available from: https://arxiv.org/abs/2411.15640

arXiv 2024
[3]

Interprofessional Collaborative Practice

World Health Professions Alliance. Interprofessional Collaborative Practice. [Inter- net]. Geneva: WHPA; [cited 2024 Jun 11]. Available from: https://www.whpa.org/ what-we-do/interprofessional-collaborative-practice

work page 2024
[4]

Artificial Intelligence and Pa- tient Safety: Promise and Challenges

Agency for Healthcare Research and Quality. Artificial Intelligence and Pa- tient Safety: Promise and Challenges. [Internet]. Rockville, MD: PSNet; 2024 Jan [cited 2024 Jun 11]. Available from: https://psnet.ahrq.gov/perspective/ artificial-intelligence-and-patient-safety-promise-and-challenges

work page 2024
[5]

Trust and medical AI: the challenges we face and the expertise needed to overcome them

Vollmer S, Mateen BA, Bohner G, Király FJ, Ghani R, Jonsson P, et al. Trust and medical AI: the challenges we face and the expertise needed to overcome them. J R Soc Med. 2021;114(5):214-218. doi:10.1177/01410768211005992

work page doi:10.1177/01410768211005992 2021
[6]

AI adoption: healthcare or health scare? [Internet]

Frontier Economics. AI adoption: healthcare or health scare? [Internet]. London: Frontier Economics; 2024 Feb [cited 2024 Jun 11]. Available from: https://www.frontier-economics.com/uk/en/news-and-articles/articles/ article-i8738-ai-adoption-healthcare-or-health-scare/

work page 2024
[7]

MedExQA: A Medical Explainability Benchmark for Evaluating Vision-Language Models

Koleilat M, Magu O, Eze O, Agbichi C, Akintayo A. MedExQA: A Medical Explainability Benchmark for Evaluating Vision-Language Models. In: Proceedings of the 23rd Workshop on Biomedical Language Processing (BioNLP 2024). Stroudsburg, PA: Association for Computational Linguistics; 2024. p. 147-159

work page 2024
[8]

Revalida

Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP). Revalida. [Internet]. Brasília: INEP; [cited 2025 Jun 11]. Available from: https://www.gov.br/ inep/pt-br/areas-de-atuacao/avaliacao-e-exames-educacionais/revalida

work page 2025

Show all 15 references

[9]

Exame Nacional de Residência (Enare)

Empresa Brasileira de Serviços Hospitalares (Ebserh). Exame Nacional de Residência (Enare). [Internet]. Brasília: Ebserh; [cited 2025 Jun 11]. Available from:https://www. gov.br/ebserh/pt-br/ensino-e-pesquisa/enare

2025
[10]

Introducing GPT-4.1 in the API [Internet]

OpenAI. Introducing GPT-4.1 in the API [Internet]. San Francisco: OpenAI; 2025 Apr 14 [cited 2025 Jun 11]. Available from:https://openai.com/index/gpt-4-1/ 13

2025
[11]

Provas e Gabaritos

Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP). Provas e Gabaritos. [Internet]. Brasília: INEP; 2020 Aug 31 [updated 2025 Mar 26; cited 2024 Jun 11]. Available from: https://www.gov.br/inep/pt-br/areas-de-atuacao/ avaliacao-e-exames-educacionais/...

2020
[12]

Provas e Gabaritos para residên- cia Médica

Empresa Brasileira de Serviços Hospitalares (Ebserh). Provas e Gabaritos para residên- cia Médica. [Internet]. Brasília: Ebserh; [cited 2025 Jun 11]. Available from: https: //mapa-vagas-enare-ebserh.conhecimento.fgv.br/provas-gabaritos-medica.html

2025
[13]

Large Language Models Encode Clinical Knowledge

Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. Large Language Models Encode Clinical Knowledge. arXiv [Preprint]. 2022 Mar 25 [cited 2024 Jun 11]:2203.14371. Available from: https://arxiv.org/abs/2212.13138

2022 arXiv
[14]

Artificial Intelligence in Medical Education

Howard FM, Dolezal J, Tuffy G, Dolezalova K. Artificial Intelligence in Medical Education. JAMA. 2024;331(23):1981-1982

2024
[15]

Testing and Evaluation of Health Care Applica- tions of Large Language Models: A Systematic Review

Bedi S, Liu Y, Orr-Ewing L, et al. Testing and Evaluation of Health Care Applica- tions of Large Language Models: A Systematic Review. JAMA. 2025;333(4):319–328. doi:10.1001/jama.2024.21700 14

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

How large language models perform on the United States Medical Licensing Examination: a systematic review

Frank E, Lee JWY. How large language models perform on the United States Medical Licensing Examination: a systematic review. medRxiv [Preprint]. 2023 May 26 [cited 2024 Jun 11]. Available from: https://www.medrxiv.org/content/10.1101/2023.05. 26.23290649v1

work page doi:10.1101/2023.05 2023

[2] [2]

AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Bench- mark Dataset

Olatunji T, Nimo C, Owodunni A, Abdullahi T, Ayodele E, Sanni M, et al. AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Bench- mark Dataset. arXiv [Preprint]. 2024 Nov 23 [cited 2025 Jun 11]. Available from: https://arxiv.org/abs/2411.15640

arXiv 2024

[3] [3]

Interprofessional Collaborative Practice

World Health Professions Alliance. Interprofessional Collaborative Practice. [Inter- net]. Geneva: WHPA; [cited 2024 Jun 11]. Available from: https://www.whpa.org/ what-we-do/interprofessional-collaborative-practice

work page 2024

[4] [4]

Artificial Intelligence and Pa- tient Safety: Promise and Challenges

Agency for Healthcare Research and Quality. Artificial Intelligence and Pa- tient Safety: Promise and Challenges. [Internet]. Rockville, MD: PSNet; 2024 Jan [cited 2024 Jun 11]. Available from: https://psnet.ahrq.gov/perspective/ artificial-intelligence-and-patient-safety-promise-and-challenges

work page 2024

[5] [5]

Trust and medical AI: the challenges we face and the expertise needed to overcome them

Vollmer S, Mateen BA, Bohner G, Király FJ, Ghani R, Jonsson P, et al. Trust and medical AI: the challenges we face and the expertise needed to overcome them. J R Soc Med. 2021;114(5):214-218. doi:10.1177/01410768211005992

work page doi:10.1177/01410768211005992 2021

[6] [6]

AI adoption: healthcare or health scare? [Internet]

Frontier Economics. AI adoption: healthcare or health scare? [Internet]. London: Frontier Economics; 2024 Feb [cited 2024 Jun 11]. Available from: https://www.frontier-economics.com/uk/en/news-and-articles/articles/ article-i8738-ai-adoption-healthcare-or-health-scare/

work page 2024

[7] [7]

MedExQA: A Medical Explainability Benchmark for Evaluating Vision-Language Models

Koleilat M, Magu O, Eze O, Agbichi C, Akintayo A. MedExQA: A Medical Explainability Benchmark for Evaluating Vision-Language Models. In: Proceedings of the 23rd Workshop on Biomedical Language Processing (BioNLP 2024). Stroudsburg, PA: Association for Computational Linguistics; 2024. p. 147-159

work page 2024

[8] [8]

Revalida

Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP). Revalida. [Internet]. Brasília: INEP; [cited 2025 Jun 11]. Available from: https://www.gov.br/ inep/pt-br/areas-de-atuacao/avaliacao-e-exames-educacionais/revalida

work page 2025