KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations

Junyu Liu; Kaiqi Yan; Momoko Nagai-Tanima; Qian Niu; Tianyang Wang; Tomoki Aoyama

REVIEW 3 major objections 4 minor 23 references

KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read KokushiMD-10 builds an 11,588-question multimodal benchmark from ten Japanese healthcare licensing exams and finds that no tested LLM passes consistently across all domains.

desk verdict A genuinely useful Japanese multimodal medical QA benchmark undermined by an unspecified pass/fail scoring procedure. read the letter →

arxiv 2506.11114 v1 pith:B765MWWU submitted 2025-06-09 cs.CL cs.AI

Junyu Liu , Kaiqi Yan , Tianyang Wang , Qian Niu , Momoko Nagai-Tanima , Tomoki Aoyama This is my paper

classification cs.CLcs.AI

keywords KokushiMD-10JapanesehealthcarelicensingexamsmultimodalbenchmarkLLMevaluationmedicalquestionansweringclinicalreasoningchain-of-thoughtrationalesvision-languageunderstanding

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces KokushiMD-10, a benchmark built from 11,588 real questions across ten Japanese national healthcare licensing exams, including medicine, dentistry, nursing, pharmacy, and allied health professions. It claims to be the first benchmark spanning ten professions while also including images, with roughly one in five questions carrying clinical images and six exams supplied with chain-of-thought rationales. The authors evaluate 33 LLMs in text-only mode and 11 in multimodal mode, scoring them against the official passing thresholds of the original exams. The central finding is that although leading models such as GPT-4o and Claude 3.5 pass some exams, no model consistently meets passing thresholds across domains. If correct, this gives the research community a reusable Japanese-language, multimodal benchmark for measuring clinical reasoning.

What carries the argument

The load-bearing object is the construction-and-evaluation pipeline. Official exam PDFs are converted to PNG images, each question image is sent to Claude-3.5-Sonnet at temperature 0 with a structured JSON prompt that includes the correct answer from official answer-mapping tables, and a post-processing loop with scripted checks, automated Claude validation, and human review of selected samples cleans the result. The evaluation side ties scores to official passing thresholds, categorizes questions as single-answer, multiple-choice, numerical calculation, or blank, and requires exact option-set matches for multiple-choice items. The benchmark itself, with 11,588 real questions, 19.7 percent image-based, English translations, and chain-of-thought rationales for six exams, is the device that makes comparisons across professions and models possible.

What would settle it

Take a random sample of, say, 500 benchmark questions, extract them manually from the official exam PDFs, and compare question text, options, images, and answer keys against KokushiMD-10; if mismatches appear at a rate that changes any model's pass/fail verdict, the reported conclusions rest on extraction noise rather than model ability.

Watch

Extended reading notes

Core claim

The central claim is that a multi-stage extraction pipeline can turn five years (2020–2024) of official Japanese Ministry of Health, Labour and Welfare licensing exam PDFs into a structured multimodal QA dataset, and that evaluating 33 LLMs on it separates models by profession and modality. The paper reports that GPT-4o and Claude-3.5-Sonnet lead in text-only and multimodal settings, that Pharmacy questions produce the highest absolute scores while Dentistry produce the lowest, and that adding images barely changes model rankings. On official thresholds, 12 of 33 text-only models and 6 of 11 multimodal models passed at least one exam, but none passed consistently across all ten professions. This is presented as evidence that current LLMs have not reached the reliability needed for real-world clinical use across the breadth of healthcare roles.

Load-bearing premise

The results depend on the assumption that the mostly automated pipeline that turned official exam PDFs into question-answer pairs did so accurately; the paper reports no measurement of extraction error, and human review covered only a sample.

Editorial extensions

If this is right

Researchers gain a Japanese-language, multimodal medical QA benchmark with official ground-truth answer keys and chain-of-thought rationales for six of the ten exams.
Model evaluations can be compared across ten healthcare professions and five exam years against the same official passing thresholds used for human candidates.
The multimodal mode shows that adding images to the text does not change model rankings, suggesting current scores are driven mainly by textual reasoning.
The pass/fail results imply that even frontier models are not yet dependable for autonomous clinical decision-making at licensing-exam standard across all healthcare roles.
Because every question is tagged with specialty, year, section, and type, the dataset supports fine-grained analyses of where models lose points, such as dentistry, and where they gain them, such as pharmacy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same PDF-to-structured-JSON extraction loop could in principle be applied to licensing exams in other non-English languages, so the methodological contribution may outlive the specific dataset.
The near-zero benefit of image inputs could be an artifact of Japanese exam design, where images often accompany text that already carries the diagnostic information; an ablation that masks the text of image questions would test whether visual reasoning is really weak.
Because six exams carry chain-of-thought rationales, the benchmark can support studies of whether models that answer correctly reason correctly, not just accuracy comparisons.
The large number of zero or near-zero scoring small open models suggests the benchmark may also be useful as a coarse capability ladder for model development, not only for medical certification readiness.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know about this paper. First, the benchmark idea is genuinely useful: a Japanese, multimodal, ten-profession licensing exam QA set with 11,500+ real questions and a broad 33-model evaluation. That fills a real gap next to MedQA and YakugakuQA. Second, the headline conclusion—no model consistently meets passing thresholds—is not backed up as written, because the paper never specifies the thresholds or how official scoring rules were implemented.

What is new and good: the multi-profession scope, the 19.7% image questions, the Japanese-language focus, and the evaluation breadth across 33 models. The construction pipeline is described in enough detail to reproduce once artifacts are released. Ground truth comes from official answer keys, so there is no equation-fitting circularity. Using Claude 3.5 Sonnet to build the dataset and then evaluating it on that dataset is a mild concern, but only if extraction errors are systematic.

The soft spot is §3.3. The authors say they use 'domain-specific thresholds aligned with official exam standards,' but they never give the thresholds, never list essential questions, never say how forbidden options or composite scoring were handled, and never provide the mapping from raw score to pass/fail. Table 1's pass/fail counts therefore cannot be checked. This is load-bearing: the paper's central claim rests on that table. It may be right, but as written it is not evidenced. That matters more than the unmeasured extraction error rate, which is also a real limitation. The phrase 'expert-annotated rationales' overclaims: the CoT explanations come from public government and prep sources, not expert annotation. The 'first' claim should be supported with a more systematic comparison to prior benchmarks.

If the authors release the dataset and scripts and specify the thresholds, this becomes a solid resource. As it stands, the paper deserves peer review but needs revision before the pass/fail claim can be trusted.

Recommendation: send to review, flagging the threshold specification as the critical revision. I would read a revised version closely.

Referee Report

3 major / 4 minor

Summary. The paper introduces KokushiMD-10, a multimodal QA benchmark built from ten Japanese national healthcare licensing examinations (2020-2024), containing over 11,588 questions with roughly 19.7% image-based items. The construction pipeline converts official exam PDFs to PNGs, extracts structured QA pairs with Claude 3.5 Sonnet at temperature 0, aligns answers from official mapping tables, and applies automated checks with sample-based human review. The authors evaluate 33 models in text-only mode and 11 in multimodal mode across 50 exams, reporting pass/fail counts and per-specialty mean scores. The headline conclusion is that no model consistently meets official passing thresholds across healthcare domains.

Significance. If the dataset is released and validated, KokushiMD-10 would fill a genuine gap: a Japanese, multimodal, multi-profession medical licensing benchmark with expert rationales. The breadth (10 professions, 50 exams, 11,588 questions) and the inclusion of over 30 models make it a potentially useful resource for multilingual and multimodal medical AI evaluation. The paper also deserves credit for attempting human-in-the-loop validation, for reporting per-domain results, and for providing CoT rationales for six exams. However, the scientific value is currently conditional: the evaluation protocol for pass/fail is not specified, the extraction error rate is not measured, and the dataset and scripts are not yet public, so the main empirical claims cannot be independently verified.

major comments (3)

[§3.3, Table 1] The official passing criteria are described only qualitatively. The paper states that models are scored with 'domain-specific thresholds aligned with official exam standards' but never lists the thresholds, the scoring formula, or how special rules (essential questions, forbidden options, composite scoring, sectional thresholds, qualitative grading) are encoded. Since Table 1 and the abstract's claim that 'no model consistently meets passing thresholds' depend on these unspecified thresholds, the central conclusion is not reproducible. Please provide a complete mapping for each of the 50 exams: item weights, required essential-question performance, forbidden-option handling, and the exact numerical pass threshold; if such rules cannot be implemented, the claim should be restricted to raw score comparisons.
[§2.2, Post-processing Loop] No extraction error rate is reported. The pipeline relies on Claude 3.5 Sonnet both to create the dataset and as an evaluated model, and the human review covers only 'selected samples' with no sample size, sampling procedure, or inter-annotator agreement. Without a measured error rate for question text, options, image association, and answer alignment, the benchmark scores could be systematically biased. Please report error rates on a random held-out sample with confidence intervals, describe the correction loop quantitatively, and release the dataset and evaluation scripts; the URLs in the paper currently say 'will be available', so the reproducibility claim is not yet verifiable.
[§3.3, exclusion of blank/invalid questions] The statement that 'a small subset of blank or invalid questions is excluded' is unquantified. The number of excluded items and the criteria for exclusion are not given for any exam, so the denominators and per-exam scores cannot be reconstructed. This interacts with the pass/fail claim: excluding even a few questions can change whether a model crosses a threshold. Please report exclusion counts per exam or make the filtering code and raw dataset available.

minor comments (4)

[Figure 3] Figure 3 contains garbled unicode text and is largely unreadable; please replace it with a clean cumulative-pass plot and define the color mapping for healthcare subjects.
[Table 2 footnote] The footnote says 'Six zero-score models were excluded, leaving 27 text-only models,' which conflicts with the 33 models reported in §3.2 and Table 1; clarify how a zero-score model is defined and why these models are excluded from Table 2 but included in Table 1.
[§3.4, Multimodal comparison] The statement that visual information provides limited additional benefit is based on rank-order stability without any statistical comparison; report paired significance tests or phrase the claim more cautiously.
[§2.2, CoT explanations] The CoT explanations for six exams are described as constructed from public sources, but no annotation protocol, quality checks, or agreement measures are given; since the rationales are a claimed contribution, please document their construction and validation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ground-truth answers come from the official answer key, not from the evaluated models.

full rationale

The paper's construction pipeline converts official exam PDFs to images, uses Claude 3.5 Sonnet to transcribe questions into structured JSON, and aligns answers with an official answer-mapping table extracted from the PDFs. The ground truth for evaluation is therefore the official answer key, not the outputs of any evaluated model. No parameter is fitted to the evaluation data and then renamed a prediction; no equation or derivation reduces one claimed result to another by construction. The fact that Claude 3.5 Sonnet is used both for extraction and later evaluated is a data-quality and potential-bias concern, not a circular-derivation step, because the correct answers are not generated by the extraction model and the extraction errors are not shown to be correlated with evaluation scores. The paper's 'official passing thresholds' are asserted but not specified in detail, and the scoring rubric is not fully reproducible; however, that is a correctness and completeness concern, not circularity. The only self-citation, reference [10] to a prior CMMED benchmark by one of the authors, appears in related-work comparisons and is not load-bearing for any central claim. No load-bearing self-citation chain, imported uniqueness theorem, or ansatz-smuggling pattern is present. The central benchmark result is self-contained with respect to the official exam materials and the implemented evaluation, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted; the benchmark uses official exam answers as ground truth and official thresholds as pass criteria. The main load-bearing assumptions are about the fidelity of the automated extraction and about applying human exam thresholds to model scores.

assumptions (4)

domain assumption Official exam PDFs and answer tables from the Ministry of Health, Labour and Welfare are complete and correct ground truth for all questions.
Used in Sections 2.1 and 2.2; if the official answer keys are misread or the PDFs are incomplete, the whole benchmark inherits the error.
domain assumption Claude 3.5 Sonnet's structured extraction preserves question text, options, images, and answer indices without material errors.
Section 2.2; extraction is the core data production step and is validated only by structural checks and sample-based human review.
ad hoc to paper Excluding a small subset of blank or invalid questions does not bias per-exam scores.
Section 3.3; exclusion criteria are not quantified, so the effect on pass/fail results is unknown.
domain assumption Official human-oriented passing thresholds, including composite or qualitative scoring, can be applied directly to raw LLM accuracy scores.
Section 3.1; official criteria involve essential-question requirements and other rules that are not fully modeled by overall accuracy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations." pith.science (2026). https://pith.science/paper/B765MWWU

@misc{pith2026250611114,
  author       = {Pith},
  title        = {Pith review of: KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B765MWWU}},
  note         = {Machine review of arXiv:2506.11114}
}

read the original abstract

Recent advances in large language models (LLMs) have demonstrated notable performance in medical licensing exams. However, comprehensive evaluation of LLMs across various healthcare roles, particularly in high-stakes clinical scenarios, remains a challenge. Existing benchmarks are typically text-based, English-centric, and focus primarily on medicines, which limits their ability to assess broader healthcare knowledge and multimodal reasoning. To address these gaps, we introduce KokushiMD-10, the first multimodal benchmark constructed from ten Japanese national healthcare licensing exams. This benchmark spans multiple fields, including Medicine, Dentistry, Nursing, Pharmacy, and allied health professions. It contains over 11588 real exam questions, incorporating clinical images and expert-annotated rationales to evaluate both textual and visual reasoning. We benchmark over 30 state-of-the-art LLMs, including GPT-4o, Claude 3.5, and Gemini, across both text and image-based settings. Despite promising results, no model consistently meets passing thresholds across domains, highlighting the ongoing challenges in medical AI. KokushiMD-10 provides a comprehensive and linguistically grounded resource for evaluating and advancing reasoning-centric medical AI across multilingual and multimodal clinical tasks.

Figures

Figures reproduced from arXiv: 2506.11114 by the authors.

**Figure 1.** Dataset statistics across ten healthcare professions: (a) Total and image-based question counts per exam; (b) Overall proportion of image vs. text-only questions; (c) Question type diversity measured by Shannon entropy; (d) Distribution of question types (single-choice, multiple-choice, calculation, blank). – We release a large-scale QA dataset in Japanese that incorporates both textual and visual modalities, facil… view at source ↗

**Figure 2.** The construction pipeline of KokushiMD-10, including PDF-to-image preprocessing, structured QA extraction, answer alignment, human-in-the-loop correction, and structural format verification. more multiple-choice formats. Diversity in question types, quantified via Shannon entropy [18], is highest in Dentistry and lowest in Radiologic Technology. 2.2 Benchmark Construction To ensure structural consistency and scala… view at source ↗

**Figure 3.** Cumulative number of passed exams per model across 50 licensing exams (five years per healthcare profession). Colors indicate different healthcare subjects [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 19 canonical work pages

[1]

Academy, T.: Qualification exam information center.https://www.tokyo-ac.jp/ qualification/info-exam/, accessed: 2025-06-05

work page 2025
[2]

Computer software (2024),https://acrobat

Adobe Inc.: Adobe acrobat pro. Computer software (2024),https://acrobat. adobe.com, version 2024.004.2054, accessed 8 Jun 2025

work page 2024
[3]

Clinical Anatomy38(2), 186–199 (2025)

Al-Khater, K.M.K.: Comparative assessment of three ai platforms in answering usmle step 1 anatomy questions or identifying anatomical structures on radio- graphs. Clinical Anatomy38(2), 186–199 (2025)

work page 2025
[4]

Anthropic: Claude 3 (opus release) (2025),https://www.anthropic.com/claude, accessed 8 Jun 2025

work page 2025
[5]

ishiyaku-dental.jp/, accessed: 2025-06-05

Dental, I.: Overview and trends in dental national examinations.https://www. ishiyaku-dental.jp/, accessed: 2025-06-05

work page 2025
[6]

Google DeepMind: Gemini 1.5 pro (2025),https://gemini.google.com, accessed 8 Jun 2025

work page 2025
[7]

mynavi.jp/conts/kokushi_kouryaku/, accessed: 2025-06-05 KokushiMD-10 9

Healthcare, M.: National exam strategy and career information.https://nurse. mynavi.jp/conts/kokushi_kouryaku/, accessed: 2025-06-05 KokushiMD-10 9

work page 2025
[8]

Applied Sciences11(14), 6421 (2021)

Jin, D., Pan, E., Oufattole, N., Weng, W.H., Fang, H., Szolovits, P.: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences11(14), 6421 (2021)

2021

Show all 23 references

[9]

arXiv preprint arXiv:2303.18027 (2023)

Kasai, J., Kasai, Y., Sakaguchi, K., Yamada, Y., Radev, D.: Evaluating gpt- 4 and chatgpt on japanese medical licensing examinations. arXiv preprint arXiv:2303.18027 (2023)

2023 arXiv
[10]

Advances in Neural Information Processing Systems 36, 52430–52452 (2023)

Liu, J., Zhou, P., Hua, Y., Chong, D., Tian, Z., Liu, A., Wang, H., You, C., Guo, Z., Zhu, L., et al.: Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset. Advances in Neural Information Processing Systems 36, 52430–52452 (2023)

2023
[11]

Media, C.: Overview of radiologic technologist national examination.https:// contact.ne.jp/media/?p=170, accessed: 2025-06-05

2025
[12]

html, accessed 2025-06-07

Ministry of Health, Labour and Welfare: Qualification exam results announcement (2025),https://www.mhlw.go.jp/kouseiroudoushou/shikaku_shiken/goukaku. html, accessed 2025-06-07

2025
[13]

Ministry of Health, Labour and Welfare of Japan: National licensing examina- tion overview and results.https://www.mhlw.go.jp/kouseiroudoushou/shikaku_ shiken/(2024), accessed: 2025-06-05

2024
[14]

JJDEA40, 3–10 (2024)

Morishita, M., Fukuda, H., Muraoka, K., Nakamura, T., Yoshioka, I., Ono, K., Awano, S.: Comparison of the performance on the japanese national dental exam- ination using gpt-3.5 and gpt-4. JJDEA40, 3–10 (2024)

2024
[15]

OpenAI: ChatGPT (gpt-4o model) (2025),https://openai.com/chatgpt, ac- cessed 8 Jun 2025

2025
[16]

In: Conference on health, inference, and learning

Pal, A., Umapathi, L.K., Sankarasubbu, M.: Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In: Conference on health, inference, and learning. pp. 248–260. PMLR (2022)

2022
[17]

guppy.jp/, accessed: 2025-06-05

Platform, G.J.: National exam overview for public health roles.https://www. guppy.jp/, accessed: 2025-06-05

2025
[18]

Bell System Technical Journal27(3), 379–423 (1948)

Shannon, C.E.: A mathematical theory of communication. Bell System Technical Journal27(3), 379–423 (1948)

1948
[19]

Scientific Reports14(1), 9330 (2024)

Shieh, A., Tran, B., He, G., Kumar, M., Freed, J.A., Majety, P.: Assessing chatgpt 4.0’s test performance and clinical diagnostic accuracy on usmle step 2 ck and clinical case reports. Scientific Reports14(1), 9330 (2024)

2024
[20]

arXiv preprint arXiv:2505.16661 (2025)

Sukeda, I., Fujii, T., Buma, K., Sasaki, S., Ono, S.: A japanese language model and three new evaluation benchmarks for pharmaceutical nlp. arXiv preprint arXiv:2505.16661 (2025)

2025 arXiv
[21]

JMIR medical education9(1), e48002 (2023)

Takagi, S., Watari, T., Erabi, A., Sakaguchi, K., et al.: Performance of gpt-3.5 and gpt-4 on the japanese medical licensing examination: comparison study. JMIR medical education9(1), e48002 (2023)

2023
[22]

Advances in Neural Information Processing Systems35, 24824–24837 (2022)

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems35, 24824–24837 (2022)

2022
[23]

arXiv preprint arXiv:2408.02900 (2024),https: //arxiv.org/abs/2408.02900, accessed 8 Jun 2025

Xie, Y., et al.: Medtrinity-25m: A large-scale multimodal dataset with multigran- ular annotations for medicine. arXiv preprint arXiv:2408.02900 (2024),https: //arxiv.org/abs/2408.02900, accessed 8 Jun 2025

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Academy, T.: Qualification exam information center.https://www.tokyo-ac.jp/ qualification/info-exam/, accessed: 2025-06-05

work page 2025

[2] [2]

Computer software (2024),https://acrobat

Adobe Inc.: Adobe acrobat pro. Computer software (2024),https://acrobat. adobe.com, version 2024.004.2054, accessed 8 Jun 2025

work page 2024

[3] [3]

Clinical Anatomy38(2), 186–199 (2025)

Al-Khater, K.M.K.: Comparative assessment of three ai platforms in answering usmle step 1 anatomy questions or identifying anatomical structures on radio- graphs. Clinical Anatomy38(2), 186–199 (2025)

work page 2025

[4] [4]

Anthropic: Claude 3 (opus release) (2025),https://www.anthropic.com/claude, accessed 8 Jun 2025

work page 2025

[5] [5]

ishiyaku-dental.jp/, accessed: 2025-06-05

Dental, I.: Overview and trends in dental national examinations.https://www. ishiyaku-dental.jp/, accessed: 2025-06-05

work page 2025

[6] [6]

Google DeepMind: Gemini 1.5 pro (2025),https://gemini.google.com, accessed 8 Jun 2025

work page 2025

[7] [7]

mynavi.jp/conts/kokushi_kouryaku/, accessed: 2025-06-05 KokushiMD-10 9

Healthcare, M.: National exam strategy and career information.https://nurse. mynavi.jp/conts/kokushi_kouryaku/, accessed: 2025-06-05 KokushiMD-10 9

work page 2025

[8] [8]

Applied Sciences11(14), 6421 (2021)

Jin, D., Pan, E., Oufattole, N., Weng, W.H., Fang, H., Szolovits, P.: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences11(14), 6421 (2021)

2021