Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper presents JPHARMATRON, a 7B-parameter Japanese pharmaceutical language model, and shows it outperforms existing open medical LLMs across three new benchmarks while revealing that even GPT-4o struggles with cross-sentence…

desk verdict Useful first Japanese pharma benchmark suite and a plausible domain-adapted 7B model, but the headline YakugakuQA number is selected on the same benchmark and needs a held-out split. read the letter →

arxiv 2505.16661 v2 pith:3DYFBORH submitted 2025-05-22 cs.CL

Shinnosuke Ono , Issey Sukeda , Takuro Fujii , Kosei Buma , Shunsuke Sasaki This is my paper
classification cs.CL
keywords JapanesepharmaceuticalNLPcontinualpretrainingdomain-specificlanguagemodelbenchmarksuitepharmacistlicensingexamterminologynormalizationinconsistencydetectionmerging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that a modest-sized (7B-parameter) Japanese language model specialized for pharmaceuticals can be built cheaply by continuing to pretrain an open multilingual model on a curated bilingual corpus, and that such a model can outperform existing open medical LLMs and approach commercial ones on pharmaceutical tasks. To make this case, the authors introduce three benchmarks: YakugakuQA (3,021 questions from the Japanese national pharmacist licensing exam), NayoseQA (34,769 multiple-choice items on cross-lingual drug-name and terminology normalization), and SogoCheck (200 paired-statement consistency checks drawn from regulatory documents). Their best model, JPHARMATRON-7B (9B tokens), beats Meditron3-Qwen2.5-7B by 7.9 points on YakugakuQA (62.0% vs 54.1%) and by 5.9 points on IgakuQA, and outperforms all open models of its size on the new benchmarks. The paper also reports that GPT-4o scores only 39.1% on SogoCheck, a task with deliberately simple surface inconsistencies, suggesting that cross-sentence consistency reasoning is an unresolved problem for current LLMs. If correct, the work makes a practical argument that secure, low-cost, domain-specific language models are feasible outside English and provides reusable evaluation resources for pharmaceutical NLP.

What carries the argument

The central mechanism is the JPHARMATRON training recipe: continual pretraining of Qwen2.5-7B on a deduplicated, LLM-filtered bilingual corpus (Japanese pharma texts and English PubMed abstracts), followed by model merging with Qwen2.5-7B-Instruct using the DARE TIES method, with merge weights selected by evolutionary search (EvoLLM) on YakugakuQA accuracy. This recipe injects domain knowledge while restoring instruction-following without supervised fine-tuning. The three benchmarks—YakugakuQA (factual recall), NayoseQA (lexical normalization), and SogoCheck (inconsistency detection)—serve as the measurement apparatus that makes the improvement visible.

What would settle it

Run the JPHARMATRON merge-recipe selection on a training split of YakugakuQA and report the selected model's accuracy on a held-out split (or on a fresh pharmacist exam from a year after 2024). If the model no longer beats Meditron by the reported margin, the claimed superiority on YakugakuQA is an artifact of selection on the same benchmark.

Watch

Extended reading notes

Core claim

The central claim is that continual pretraining on a filtered, domain-specific corpus—2 billion Japanese pharmaceutical tokens plus 8 billion English biomedical tokens—followed by a DARE TIES merge with an instruction-tuned base model (with weights chosen by EvoLLM), yields a 7B model, JPHARMATRON, that is the strongest open Japanese pharmaceutical LLM to date. Across the three new benchmarks and two existing medical benchmarks, JPHARMATRON-7B (9B tokens) outperforms the open Meditron3-Qwen2.5-7B and the general-purpose Swallow models on every task; on YakugakuQA it reaches 62.0% accuracy, and on the terminology-heavy NayoseQA its 10B-token variant reaches 62.6%. The authors interpret these results as evidence that general medical specialization alone is insufficient for pharmaceutical work, and that domain-adaptive continual pretraining can close much of the gap to commercial LLMs at a deployable scale. At the same time, they report that no model, including GPT-4o (39.1%), performs well on SogoCheck, which they take as evidence that fine-grained cross-sentence consistency detection remains an open research challenge.

Load-bearing premise

The model's headline advantage is measured on the same YakugakuQA benchmark that was used to select the best merging recipe, and the paper assumes that this selection did not overfit the benchmark.

Editorial extensions

If this is right

  • JPHARMATRON-7B is the strongest open Japanese pharmaceutical model on JPHARMABENCH and the two existing medical benchmarks, and general medical models like Meditron are not sufficient for pharmaceutical tasks.
  • A practical, locally deployable 7B model can be competitive with commercial APIs on terminology-heavy tasks, which matters for privacy-sensitive and cost-constrained pharmaceutical environments.
  • SogoCheck is a diagnostic that reveals a general weakness in current LLMs: cross-sentence consistency reasoning, even with simple textual clues, remains largely unsolved across model scales.
  • The benchmark suite provides reusable evaluation resources for Japanese pharmaceutical NLP, covering factual recall, lexical variation, and logical consistency, beyond what IgakuQA and JMMLU offer.
  • The merging strategy (DARE TIES with EvoLLM-selected weights) is a lightweight alternative to supervised fine-tuning for restoring instruction-following in domain-pretrained models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the domain corpus is the main driver of the gains, scaling the Japanese pharmaceutical data beyond 2B tokens, or adding structured regulatory text, could further narrow the gap to commercial models on factual-recall tasks at the same 7B scale.
  • The SogoCheck result suggests current LLMs rely on surface-level matching rather than deep semantic alignment; a natural next experiment is to test chain-of-thought prompting or retrieval of the original Japanese Pharmacopoeia entries to see if consistency detection improves.
  • Because the merge recipe was selected on the same YakugakuQA benchmark used for headline results, an independent, freshly collected pharmacist exam (e.g., a later year) would be needed to confirm the open-model leadership claim without selection bias.
  • The small gap between domain and general models on NayoseQA implies that pharmaceutical LLM development should prioritize reasoning and factual recall over further terminology alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents JPHARMATRON, a 7B-parameter Japanese pharmaceutical language model built by continual pretraining of Qwen2.5-7B on a mixture of Japanese pharmaceutical tokens and English biomedical tokens, followed by model merging to restore instruction-following ability. It also introduces three new benchmarks: YakugakuQA (multiple-choice questions from the Japanese national pharmacist licensing exam), NayoseQA (cross-lingual drug-name and terminology normalization), and SogoCheck (paired-statement inconsistency detection). The model is evaluated on these benchmarks plus IgakuQA and a pharmaceutical subset of JMMLU, against open models (Swallow series, Meditron) and commercial models (GPT-4o, o1-preview). The central claims are that JPHARMATRON outperforms existing open models and approaches commercial performance on pharmaceutical tasks, and that even GPT-4o performs poorly on SogoCheck.

Significance. If the evaluation is sound, this is a useful contribution: it provides the first Japanese pharmaceutical LLM benchmark suite, covers complementary competencies (factual recall, lexical normalization, and cross-sentence consistency), and releases model, code, and data. The SogoCheck result is an interesting negative finding that could motivate new research on consistency reasoning. The paper also documents a reproducible data-cleaning and filtering pipeline, which strengthens its practical value. However, the headline model-comparison claim rests on an evaluation protocol that is not yet rigorous enough to establish the claimed margin over open baselines.

major comments (3)
  1. [Section 5.2, Table 4, and Appendix C.5 (Table 9)] The YakugakuQA accuracy is used both as the model-selection criterion and as the headline evaluation. The final merge recipe (DARE TIES via EvoLLM) was selected because it scored 60.7% on YakugakuQA, beating the runner-up by 0.3 points; the same benchmark then provides the primary evidence (62.0% for the 9B-token model) that JPHARMATRON outperforms Meditron3-Qwen2.5-7B by 7.9 points. Because no held-out split of YakugakuQA is reported, the 62.0% value is partly a fitted value, and the 0.3-point selection gap is within the range of selection noise. Please report the comparison on a held-out split of YakugakuQA (or use nested cross-validation) and state the margin with that split. The paper should also state explicitly whether the 9B-token row in Table 4 uses the same DARE TIES recipe selected in Table 9.
  2. [Appendix C.1, Appendix C.3, and Section 5.2] The training corpus includes web-derived data (CC100/FineWeb filtered for pharmaceutical relevance), while YakugakuQA is manually collected from the public yakugakulab.info website. The paper reports D4 deduplication only within the training corpus and does not report any overlap analysis between the training tokens and the evaluation benchmarks. If NPLE questions or their close paraphrases appear in the web-derived training data, the reported gains on YakugakuQA could reflect memorization rather than domain-adaptive pretraining. Please run and report a contamination check (e.g., substring or embedding overlap of benchmark examples against the training corpus) and re-examine the comparison after excluding overlapping documents.
  3. [Section 3.4, Table 4, and Section 5.2] SogoCheck contains only 200 LLM-generated examples, and the abstract and conclusion use the 39.1% GPT-4o score to claim that cross-sentence consistency reasoning remains an open challenge. The paper's own Limitations section acknowledges the small scale, but the claim as stated in the abstract is not supported by a confidence interval or significance test. Please provide bootstrap confidence intervals for the Table 4 accuracies, particularly for SogoCheck, and temper the claim accordingly if the intervals overlap.
minor comments (7)
  1. [Abstract] The abstract in the full text says the model, code, and datasets 'will be released upon acceptance,' while the abstract block at the top lists a GitHub URL as already released; please clarify the release status.
  2. [Section 3.2] The sentence 'The category varies among pharmacy and eight other related areas' is followed by a list of nine categories with pharmacy appearing twice; adjust the count or the list.
  3. [Section 5.3] The phrase 'We analyze the 16.4% of incorrectly answered questions on YakugakuQA' should specify that these are GPT-4o's errors, not a general statement about all models.
  4. [Section 4] The word 'defered' should be 'deferred.'
  5. [Section 6.1] The word 'suprisingly' should be 'surprisingly.'
  6. [Throughout] The model name is written inconsistently as JPHARMATRON, JPHARMATRON, and JPharmatron; standardize the typography.
  7. [Tables 4 and 5] Table 5 includes models (e.g., Llama-3-Swallow-8B-Instruct-v0.1) that do not appear in Table 4; clarify the relationship between the two tables.

Circularity Check

1 steps flagged · score 4.0 of 10

YakugakuQA doubles as merge-recipe selection set and headline evaluation; the 2B-token result is a fitted value.

  1. fitted input called prediction [Appendix C.5, Table 9; Section 5.2, Table 4 (YakugakuQA results)]
    "Accuracy comparison on YakugakuQA across different merging methods. Qwen2.5-7B-Instruct was used as the base model and JPHARMATRON-7B (Ours) was used as the auxiliary model. ... Table 9 shows the superiority of EvoLLM (Akiba et al., 2025) coupled with DARE TIES merging."

    All four merge configurations in Table 9 are scored on YakugakuQA (TIES 8:2=57.2, 7:3=59.0, 6:4=60.4, DARE TIES=60.7), and the highest, DARE TIES by EvoLLM, is adopted. That same 60.7 appears in Table 4 as the YakugakuQA score for JPHARMATRON-7B/2B tokens. A number used to select a model configuration is not a held-out prediction; the 0.3-point margin over the runner-up is the same order as selection noise, and the paper does not report a separate split to validate the recipe on the 9B-token variant whose 62.0% backs the headline 7.9-point gain over Meditron. The YakugakuQA result is therefore partly fitted by construction, though NayoseQA, SogoCheck, IgakuQA, and JMMLU are not used for this selection.

full rationale

This is an empirical systems-and-benchmarks paper rather than a derivation chain, so most forms of circularity do not arise. The one substantive issue is that YakugakuQA is used both to choose the merge recipe (Appendix C.5, Table 9) and as a headline evaluation benchmark (Table 4, Section 5.2). The 2B-token variant's 60.7% is exactly the selection value, so that reported result is fitted by construction; the 9B variant's 62.0% inherits the recipe chosen on the same benchmark without an independent held-out split. This weakens the specific 7.9-point margin over Meditron but does not defeat the overall claim, because NayoseQA, SogoCheck, IgakuQA, and JMMLU are evaluated independently and were not used for that selection. The paper's own Limitations section acknowledges the small scale of SogoCheck and the multiple-choice format of NayoseQA, but those are validity concerns rather than circular steps. A separate contamination concern (the web-derived 9B corpus may overlap with the public yakugakulab.info source of YakugakuQA) is not a circularity finding because the paper neither states nor denies such overlap; it is a correctness risk. The authors' self-citations (Sukeda et al.) appear only as related work and are not load-bearing for the model or benchmark design. Overall, the central contribution retains independent content, so the score is moderate-low rather than severe.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. Its free parameters are training and data curation choices; the main scientific caveat is that the final model configuration was selected on one of the evaluation benchmarks.

free parameters (3)
  • TIES/DARE TIES merge weights and recipe = selected to maximize YakugakuQA (60.7% with EvoLLM DARE TIES)
    The merge recipe was chosen by evaluating on YakugakuQA (Table 9), making the reported YakugakuQA result partly dependent on this hand-chosen configuration.
  • Data deduplication thresholds (epsilon, R) = epsilon = 3e-8, R = 0.95
    Chosen by hand for the D4 pipeline (Appendix C.3); these affect training data composition and thus model performance.
  • LLM relevance/quality filtering thresholds = relevance score >= 1, quality score > 3
    Hand-set thresholds for the classifier-based data filtering (Appendix C.2), influencing which documents entered the 2B-token corpus.
assumptions (3)
  • domain assumption Text-only NPLE questions are a representative proxy for pharmaceutical knowledge
    YakugakuQA excludes image-based questions, which are prevalent in chemistry, so the benchmark may underrepresent calculation and structure-based reasoning.
  • domain assumption LLM-generated SogoCheck pairs capture real consistency-checking ability
    The 200 pairs are synthesized with an LLM and not validated by human pharmacists; the authors acknowledge this in Section 8.
  • domain assumption Exact match on extracted answer choices is a fair evaluation metric
    Outputs are post-processed to extract the choice; this may favor models with specific formatting and may not reflect practical use.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP." pith.science (2026). https://pith.science/paper/3DYFBORH

@misc{pith2026250516661,
  author       = {Pith},
  title        = {Pith review of: A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3DYFBORH}},
  note         = {Machine review of arXiv:2505.16661}
}
read the original abstract

We present a Japanese domain-specific language model for the pharmaceutical field, developed through continual pretraining on 2 billion Japanese pharmaceutical tokens and 8 billion English biomedical tokens. To enable rigorous evaluation, we introduce three new benchmarks: YakugakuQA, based on national pharmacist licensing exams; NayoseQA, which tests cross-lingual synonym and terminology normalization; and SogoCheck, a novel task designed to assess consistency reasoning between paired statements. We evaluate our model against both open-source medical LLMs and commercial models, including GPT-4o. Results show that our domain-specific model outperforms existing open models and achieves competitive performance with commercial ones, particularly on terminology-heavy and knowledge-based tasks. Interestingly, even GPT-4o performs poorly on SogoCheck, suggesting that cross-sentence consistency reasoning remains an open challenge. Our benchmark suite offers a broader diagnostic lens for pharmaceutical NLP, covering factual recall, lexical variation, and logical consistency. This work demonstrates the feasibility of building practical, secure, and cost-effective language models for Japanese domain-specific applications, and provides reusable evaluation resources for future research in pharmaceutical and healthcare NLP. Our model, codes, and datasets are released at https://github.com/EQUES-Inc/pharma-LLM-eval.

Figures

Figures reproduced from arXiv: 2505.16661 by the authors.

Figure 1
Figure 1. JPHARMATRON and JPHARMABENCH. The pipeline for data curation, continued pretraining, and evaluation of JPHARMATRON. understanding and precise terminology handling are required. In these domains, general-purpose LLMs often fall short due to inadequate domain knowledge and difficulty handling complex or spe￾cialized queries. Moreover, while domain-specific fine-tuning can enhance surface-level performance, it has been… view at source ↗
Figure 2
Figure 2. Performance Comparison with Meditron. JPHARMATRON consistently achieves higher scores than Meditron across JPHARMABENCH, IgakuQA, and JMMLU. Qwen2.5 (Yang et al., 2024) model using a cu￾rated corpus consisting of Japanese pharmaceutical journals, web resources, and synthetic data (Ap￾pendix C). Unlike prior work focusing on drug dis￾covery (Chaves et al., 2024; Tsuruta et al., 2024), our model targets real-world ope… view at source ↗
Figure 3
Figure 3. An example question from the Japanese National Pharmacist Licensing Examination. The model is required to output “4” in this case. The ques￾tion is originally in Japanese, but translated into English by ChatGPT for readability. 3.1 Overview of JPHARMABENCH To evaluate language models in the Japanese phar￾maceutical domain, we constructed three novel benchmarks, each reflecting a different type of rea￾soning or knowl… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: A simplest example from SogoCheck. The numbers are inconsistent across two inputs. Originally in Japanese, but translated for readability. 3.4 SogoCheck: Inconsistency Detection in Paired Pharmaceutical Statements SogoCheck is a novel benchmark we introduce to evaluate…
Figure 5
Figure 5. Figure 5: Error analysis on GPT-4o’s responses in YakugakuQA. Our ultimate goal in this field is to achieve a strong and useful pharmaceutical LLM. To this end, we need to further strengthen open models, as commercial models are often unavailable or re￾stricted by regulations. O…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations

    cs.CL 2025-06 conditional novelty 6.0 of 10

    KokushiMD-10 is a Japanese multimodal benchmark spanning ten healthcare licensing exams, on which no tested model passed all exams.

Reference graph

Works this paper leans on

46 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Amro Abbas, Kushal Tirumala, D \'a niel Simig, Surya Ganguli, and Ari S Morcos. 2023. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540

    arXiv 2023

  4. [4]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. https://aclanthology.org/2024.eacl-srw.17/ Large language models for mathematical reasoning: Progresses and challenges . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 225--237, St....

    2024

  5. [5]

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2025. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, pages 1--10

    work page 2025

  6. [6]

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart \' n Bl \'a zquez, Guilherme Penedo, Lewis Tunstall, Andr \'e s Marafioti, Hynek Kydl \' c ek, Agust \' n Piqueres Lajar \' n, Vaibhav Srivastav, et al. 2025. Smollm2: When smol goes big--data-centric training of a small language model. arXiv preprint arXiv:2502.02737

    arXiv 2025

  7. [7]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Z...

    arXiv 2023

  8. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

Show all 46 references
  1. [9]

    Vaccine Supply, and Quality Unit

    Gillian Chaloner-Larsson, Roger Anderson, Anik Egan, Manoel Antonio Da Fonseca Costa Filho, Jorge F Gomez Herrera, World Health Organization. Vaccine Supply, and Quality Unit. 1999. A WHO guide to good manufacturing practice (GMP) requirements / written by Gillian Chaloner-Lar...

    1999

  2. [10]

    Sara Mahdavi, Christopher Semturs, David Fleet, Vivek Natarajan, and Shekoofeh Azizi

    Juan Manuel Zambrano Chaves, Eric Wang, Tao Tu, Eeshit Dhaval Vaishnav, Byron Lee, S. Sara Mahdavi, Christopher Semturs, David Fleet, Vivek Natarajan, and Shekoofeh Azizi. 2024. https://arxiv.org/abs/2406.06316 Tx-LLM: A Large Language Model for Therapeutics . Preprint, arXiv:...

    2024 arXiv

  3. [11]

    Linqing Chen, Weilei Wang, Zilong Bai, Peng Xu, Yan Fang, Jie Fang, Wentao Wu, Lizhi Zhou, Ruiji Zhang, Yubin Xia, et al. 2024. PharmaGPT: Domain-Specific Large Language Models for Bio-Pharmaceutical and Chemistry . arXiv preprint arXiv:2406.18045

    2024 arXiv

  4. [12]

    Zeming Chen, Alejandro Hern \'a ndez Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas K \"o pf, Amirkeivan Mohtashami, et al. 2023. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint a...

    2023 arXiv

  5. [13]

    Alexa Ehlert, Benjamin Ehlert, Binxin Cao, and Kathryn Morbitzer. 2024. Large Language Models and the North American Pharmacist Licensure Examination (NAPLEX) Practice Questions . American Journal of Pharmaceutical Education, 88(11):101294

    2024

  6. [14]

    Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities . In Proceedings of the Fi...

    2024

  7. [15]

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. https://doi.org/10.18653/v1/2024.emnlp-industry.36 Arcee ' s M erge K it: A toolkit for merging large language models . In Proceedi...

    2024 doi

  8. [16]

    Hager, F

    P. Hager, F. Jungmann, R. Holland, et al. 2024. https://doi.org/10.1038/s41591-024-03097-1 Evaluation and mitigation of the limitations of large language models in clinical decision-making . Nature Medicine, 30(11):2613--2622

    2024 doi

  9. [17]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding . Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  10. [18]

    Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. 2022. Artificial intelligence foundation for therapeutic science. Nature Chemical Biology

    2022

  11. [19]

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944

    2023 arXiv

  12. [20]

    Junfeng Jiang, Jiahao Huang, and Akiko Aizawa. 2024. https://arxiv.org/abs/2409.13317 JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models . Preprint, arXiv:2409.13317

    2024 arXiv

  13. [21]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams . arXiv preprint arXiv:2009.13081

    2020 arXiv

  14. [22]

    Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. 2023. https://arxiv.org/abs/2303.18027 Evaluating GPT -4 and ChatGPT on J apanese medical licensing examinations . Preprint, arXiv:2303.18027

    2023 arXiv

  15. [23]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024 a . Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579

    2024 arXiv

  16. [24]

    Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.163 GSM -plus: A comprehensive benchmark for evaluating the robustness of LLM s as mathematical problem solvers . In Proceedings of the 62nd Annual Meeting of t...

    2024 doi

  17. [25]

    Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet \"U st \"u n, Sara Hooker, and Sebastian Ruder. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.935 How does quantization affect multilingual LLM s? In Findings of the Association for Computational Lingui...

    2024 doi

  18. [26]

    Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. 2023. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452

    2023 arXiv

  19. [27]

    Harsha Nori, Naoto Usuyama, Nicholas King, Scott Mayer McKinney, Xavier Fernandes, Sheng Zhang, and Eric Horvitz. 2024. From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond . arXiv preprint arXiv:2411.03590

    2024 arXiv

  20. [28]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training

    2018

  21. [29]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250

    2016 arXiv

  22. [30]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023 a . Large language models encode clinical knowledge. Nature, 620(7972):172--180

    2023

  23. [31]

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023 b . Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617

    2023 arXiv

  24. [32]

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. 2022. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523--19536

    2022

  25. [33]

    Issey Sukeda. 2024. Development and bilingual evaluation of J apanese medical large language model within reasonably low computational resources. arXiv preprint arXiv:2409.11783

    2024 arXiv

  26. [34]

    Issey Sukeda, Risa Kishikawa, and Satoshi Kodera. 2024 a . 70 B -parameter large language models in J apanese medical question-answering. arXiv preprint arXiv:2406.14882

    2024 arXiv

  27. [35]

    Issey Sukeda, Masahiro Suzuki, Hiroki Sakaji, and Satoshi Kodera. 2023. JMedLoRA: medical domain adaptation on Japanese large language models using instruction-tuning . arXiv preprint arXiv:2310.10083

    2023 arXiv

  28. [36]

    Issey Sukeda, Masahiro Suzuki, Hiroki Sakaji, and Satoshi Kodera. 2024 b . Development and analysis of medical instruction-tuning for J apanese large language models. Artificial Intelligence in Health, 1(2):107--116

    2024

  29. [37]

    Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. 2023. D4: Improving llm pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems, 36:53983--53995

    2023

  30. [38]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  31. [39]

    Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. 2024. https://doi.org/10.18653/v1/2024.acl-long.410 R e FT : Reasoning with reinforced fine-tuning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2024 doi

  32. [40]

    Hirofumi Tsuruta, Hiroyuki Yamazaki, Ryota Maeda, Ryotaro Tamura, and Akihiro Imura. 2024. https://openreview.net/forum?id=Na2gnQFkn8 A SARS -cov-2 interaction dataset and VHH sequence corpus for antibody language models . In The Thirty-eight Conference on Neural Information P...

    2024

  33. [41]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093--7115

    2023

  34. [42]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  35. [43]

    Ziqi Yin, Hao Wang, Kaito Horio, Daisuke Kawahara, and Satoshi Sekine. 2024. Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance . arXiv preprint arXiv:2402.14531

    2024 arXiv

  36. [44]

    Jingqing Zhang, Kai Sun, Akshay Jagadeesh, Parastoo Falakaflaki, Elena Kayayan, Guanyu Tao, Mahta Haghighat Ghahfarokhi, Deepa Gupta, Ashok Gupta, Vibhor Gupta, et al. 2024. The potential and pitfalls of using a large language model such as chatgpt, gpt-4, or llama as a clinic...

    2024

  37. [45]

    Yiran Zhao, Jinghan Zhang, I Chern, Siyang Gao, Pengfei Liu, Junxian He, et al. 2023. Felm: Benchmarking factuality evaluation of large language models. Advances in Neural Information Processing Systems, 36:44502--44523

    2023

  38. [46]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006--55021

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.