Pith. sign in

REVIEW 3 major objections 7 minor 71 references

Technical Report: Small Language Model for Japanese Clinical and Medicine

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 1.2B-parameter Japanese clinical model claims top scores on six of eight medical NLP tasks and runs on about 2.2 GB of GPU memory.

desk verdict A useful, honest Japanese clinical SLM report whose headline 'beats GPT-4o' is an artifact of fine-tuning on the benchmark's own training split. read the letter →

arxiv 2412.16423 v1 pith:VRGGJODE submitted 2024-12-21 cs.CL cs.AIcs.LG

Shogo Watanabe This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords smalllanguagemodelJapaneseclinicalNLPinstructiontuningmedicalJMED-LLMmorphologicalanalysissynthetictextbooksnamedentityrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This report argues that a carefully built small language model can be competitive with, and on several tasks superior to, multibillion-parameter general models in Japanese clinical and medical text processing. The author constructs a 1.2B-parameter model, NCVC-slm-1, from filtered Japanese Wikipedia and web text, augmented with synthetic medical textbook content, then instruction-tunes it on the eight-task JMED-LLM benchmark. The tuned model records the highest score on six of the eight tasks, including adverse-event classification, symptom detection, sentence similarity, and three named-entity recognition tasks. The reason to care is practical: a model of this size runs on about 2.2 GB of GPU memory, making fully local, privacy-preserving processing of clinical text realistic.

What carries the argument

The mechanism that carries the result is a domain-tuned tokenization pipeline and a textbooks-style pretraining corpus. Raw text is cleaned, normalized, and then segmented with a Japanese morphological analyzer whose dictionary has been augmented with clinical terms, so medical words survive as whole tokens instead of being fragmented into characters or fallback UTF-8 pieces; a Unigram tokenizer with a 32,768-token vocabulary converts the analyzed text into subword units. The pretraining corpus mixes filtered Wikipedia and Common Crawl text with roughly 0.24B tokens of scraped and synthetic medical textbooks, the synthetic portion being generated by larger language models from disease and drug lists and including exam-like exercises. On top of this, instruction tuning on the JMED-LLM training splits teaches the base model the benchmark's task formats. Each component does specific work: morphological analysis preserves medical vocabulary, the synthetic textbooks supply a domain signal that the 2.6% medical fraction of the corpus would otherwise lack, and instruction tuning converts raw language-modeling competence into usable downstream task performance.

What would settle it

Run the same eight JMED-LLM tasks under one protocol for all models, using a version of NCVC-slm-1 trained only on the allowed training split and comparison models evaluated in the same harness; if the small model no longer tops six tasks, the central claim is falsified. Separately, search for n-gram or embedding-level overlap between the synthesized exam-like exercises and the IgakuQA and JMED-LLM test items; substantial overlap would indicate contamination rather than capability.

Watch

Extended reading notes

Core claim

The paper's central claim is that domain specialization plus instruction tuning can make a 1B-scale model outperform much larger models on a range of Japanese clinical tasks. Concretely, NCVC-slm-1-instruct achieves the top reported score on 6 of the 8 JMED-LLM tasks, namely CRADE, SMDIS, JCSTS, MRNER-disease, MRNER-medicine, and NRNER, beating models such as GPT-4o on those tasks; it remains below GPT-4o on JMMLU-Med and RRTNM, which the author attributes to a higher demand for broad knowledge and inference. The paper also reports that the untuned base model scores near chance on JMED-LLM, which the author reads as evidence that task-specific instruction tuning, rather than scale alone, is what unlocks the small model's performance.

Load-bearing premise

The claim rests on the premise that the rival models' quoted scores were produced under comparable evaluation conditions and that fine-tuning on the benchmark's own training data gives the small model no unfair advantage.

Editorial extensions

If this is right

  • A 1.2B-parameter model is shown capable of top scores on six JMED-LLM tasks, so local inference on a single GPU is sufficient for several practical clinical text tasks, including named-entity recognition from medical reports and nursing records.
  • Because fine-tuned small models beat much larger general models on those six tasks, benchmark-specific instruction tuning is the decisive ingredient, not parameter count.
  • The remaining two tasks, JMMLU-Med and RRTNM, favor GPT-4o, indicating that broad medical knowledge and complex inference are still a scalability limit for 1B-scale models.
  • Adding synthetic exam data to instruction tuning hurt IgakuQA performance and did not clearly help JMED-LLM, so naive data augmentation is not a reliable path to better clinical question answering.
  • If deployed, this kind of model could keep sensitive patient text in a local environment rather than sending it to a server.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The six-task advantage may be partly an artifact of asymmetric training: NCVC-slm-1 was fine-tuned on JMED-LLM training sets, while the comparison scores were quoted from external evaluations of models not similarly fine-tuned; an apples-to-apples run might shrink the margin.
  • The pretraining corpus contains synthetic exam-like exercises, and the report does not test for overlap with IgakuQA or JMED-LLM; contamination is a testable alternative explanation for part of the result.
  • A natural extension is to evaluate the model on held-out clinical documents, such as discharge summaries from institutions not represented in the training web text, to see whether the task-format advantage survives outside benchmark prompts.
  • The architecture choices, including morphological tokenization, grouped query attention, RMSNorm, and SiLU activation, could be ablated independently to identify which one contributes most to the benchmark wins.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper describes the development of NCVC-slm-1, a 1.2B-parameter Japanese language model for clinical and medical text, trained on filtered Wikipedia and OSCAR data supplemented with scraped and synthetic medical textbooks. After instruction tuning on the JMED-LLM training data, the model is evaluated on IgakuQA and JMED-LLM; the authors report that the instruction-tuned model achieves the highest scores on 6 of 8 JMED-LLM tasks compared with larger models including GPT-4o. The paper also reports ablations on training token budgets and on the inclusion of synthetic exercises, and it discusses design choices for tokenization, architecture, and training stability.

Significance. If the comparative claims were properly supported, the work would be a practically useful demonstration that a small, locally deployable model can compete with much larger models on several Japanese clinical and medical NLP tasks after modest fine-tuning. The paper is transparent about many engineering details, including data filtering results, tokenizer design, architecture decisions, and negative results on IgakuQA and on synthetic-data fine-tuning. However, the central '6 of 8 best' claim is currently confounded: the proposed model was instruction-tuned on the JMED-LLM training splits while the comparison models were evaluated zero-shot with scores taken from an external source. The engineering contribution is real, but the headline comparative result needs to be reframed or re-evaluated before it can be accepted.

major comments (3)
  1. [Section 2.4.2 / Section 3.3 / Tables 3-4] The central claim that NCVC-slm-1-instruct 'was the highest score of 6 tasks' is confounded with benchmark exposure. Section 2.4.2 states that the instruction tuning dataset used '8 JMED-LLM dataset except test samples,' so the model was trained on the same task distribution as the evaluation sets. The comparison models' scores in Table 3 and Table 4 are cited from reference [67], and the paper gives no indication that those baselines were fine-tuned on JMED-LLM training data. This is not a minor caveat: the base NCVC-slm-1 models score near zero on most JMED-LLM tasks (e.g., 0.00 F1 on all NER tasks in Table 4), and after fine-tuning they jump to 0.98 kappa on SMDIS and 0.87-0.90 partial F1 on NER tasks. The jump indicates that the results are largely attributable to supervised training on the benchmark itself. To support the comparative conclusion, the authors should either (a) fine-tune the comparison models on the same JMED-LLM training data, (b) evaluate NCVC-slm-1 under the same zero-shot protocol used for the baselines, or (c) explicitly and prominently reframe the claim as demonstrating that fine-tuning an SLM on a target benchmark yields high scores relative to zero-shot baselines.
  2. [Section 2.1.2 / Section 2.5 / Table 2] The pre-training corpus contains synthesized exercises described as 'similar to Japanese national medical licensing examinations,' and the paper evaluates on IgakuQA, which is exactly such examinations. No overlap or contamination analysis is reported between the synthetic exercises and the IgakuQA test items (or the JMED-LLM test items). Given that the synthetic exercises were generated from disease and drug topic lists, it is plausible that near-duplicates of real exam questions appear in the training data. The authors should provide a quantitative overlap analysis (for example, n-gram overlap between the synthetic exercise set and the evaluation sets) and discuss the implications for the reported IgakuQA and JMED-LLM numbers. This is particularly relevant for interpreting the ablation in Table 5, where adding synthetic exercises degrades IgakuQA performance.
  3. [Tables 3-4 / Section 3.3] No confidence intervals, error bars, or significance tests are reported for any JMED-LLM score, and several differences between NCVC-slm-1-instruct and the best baseline are small (for example, MRNER-medicine is 0.65 for NCVC-slm-1-instruct-20B and 0.65 for gemma-2-9b-it; JCSTS is 0.75 versus 0.60). Because the baseline numbers come from an external, unofficial source ([67], a speaker-deck), the score differences may not be reproducible under a single evaluation harness. The authors should provide their evaluation code and run the comparison models under the same protocol, or at minimum report the seed-averaged variance of the scores, before claiming superiority on 6 of 8 tasks.
minor comments (7)
  1. [Abstract / Section 2.3] The abstract says '1B parameters' while Section 2.3 states 'approximately 1 billions (more accurately 1.2B)'; please use one consistent number.
  2. [Section 3.3 / Table 4 legend] The text lists 'MRNER-disease, MRNER-disease, and NRNER'; the second instance should be MRNER-medicine.
  3. [Table 3] The row label 'Llama-3-youko-8b-insturct' contains a typo (insturct instead of instruct).
  4. [Section 3.2 / Table 2] The caption for Table 2 should specify that the NCVC-slm-1-instruct rows are tuned with JMED-LLM only and without the synthetic exercises dataset; the current text reports these rows but the comparison with Table 5 is not immediately clear.
  5. [Section 2.1.2] The sentence 'Their textbooks were 5 versions of previous language models' is grammatically unclear; it should read 'These textbooks were generated by 5 versions of previous language models.'
  6. [Section 5] The conclusion contains a duplicated word: 'This type of SLM is expected to to assist human jobs.'
  7. [Appendix A.4] Figure 6 shows cyclic deterioration and improvement of base-model IgakuQA scores with increasing pretraining tokens, but the main text does not discuss this pattern or its possible causes.

Circularity Check

1 steps flagged · score 4.0 of 10

The '6 of 8 best' JMED-LLM claim is partially circular because the model was instruction-tuned on the same benchmark's training split while the comparison scores came from models not given that training; the rest of the derivation is self-contained.

  1. fitted input called prediction [Section 2.4.2 (Fine-tuning) and Section 3.3 (JMED-LLM), Tables 3-4]
    "Fine-tuning was performed based on instruction tuning [62]. Instruction tuning dataset was used 8 JMED-LLM [63] dataset except test samples. ... instruction tuning models were the highest score of 6 tasks (CRADE, SMDIS, JCSTS, MRNER-disease, MRNER-disease, and NRNER). ... The scores of other LLMs were cited from [67]."

    The headline 'highest scores on 6 of 8 tasks' compares a model fine-tuned on the JMED-LLM training split with external baselines that were not fine-tuned on that split. The base model scores are near zero on these tasks (Tables 3-4: SMDIS kappa 0.01-0.02, NER F1 0.00) and jump to 0.98 kappa and 0.86-0.90 F1 after instruction tuning on the JMED-LLM training set. Thus the reported superiority is not an independent test of small-model capability; it is the expected effect of supervised training on the benchmark's own task distribution. The 'prediction' of best performance is therefore partially forced by the experimental construction rather than by a general SLM advantage.

full rationale

No equation-level circularity, uniqueness theorem, or load-bearing self-citation appears in this paper. The pre-training and tokenizer sections are self-contained empirical engineering. The IgakuQA result is honestly negative (NCVC-slm-1 stays below 23%), so the synthetic exam-like pretraining data do not support a circular claim there. The only circularity-adjacent step is the JMED-LLM evaluation: the model is instruction-tuned on the same benchmark whose test split is then used to declare superiority over external models cited from [67] that did not receive that fine-tuning. This makes the comparative '6 of 8 best' claim partly an artifact of benchmark exposure, but the held-out test scores themselves are real measurements, and the paper transparently describes the fine-tuning procedure. The score is therefore set at 4: partial circularity of the central comparative claim, with independent content remaining in the underlying evaluation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No theoretical entities are introduced. The model and synthetic datasets are engineering artifacts. The free parameters listed are hand-chosen training or data-generation settings that shape the model and its evaluation; none are independently fitted constants.

free parameters (4)
  • Pretraining token budget = 50B tokens seen, 5.5 epochs
    Chosen from Chinchilla scaling law and phi-1 precedent; not fitted to task data but determines model capability.
  • NEFTune noise strength alpha = 5
    Selected by hand following the NEFTune paper; affects fine-tuning performance.
  • Text quality filtering threshold = 68% removed by heuristics, 1% by model
    Ad hoc screening rules; the classifier threshold is implicit and was not tuned to a fixed target.
  • Synthetic textbook volumes = 91,000 diseases, 9,000 drugs, 5 samples per exam topic
    Chosen by hand to augment the scarce 49MB of real clinical text; the paper finds these data degrade IgakuQA scores, so their benefit is questionable.
assumptions (4)
  • domain assumption Scaling laws (Chinchilla) relate model size to optimal token counts.
    Used in Section 2.4.1 to choose the 50B-token training budget.
  • domain assumption Textbook-quality data improves small model performance (phi-1 approach).
    The whole corpus design in Section 2.1 relies on this; the paper's own ablations (Tables 5-7) show synthetic exercises can hurt performance.
  • domain assumption Morphological analysis before tokenization improves Japanese model performance.
    Invoked in Section 2.2 citing refs [35-37]; not revalidated here.
  • domain assumption JMED-LLM is a valid measure of clinical and medical language capability.
    The central evaluation (Section 3.3) assumes benchmark scores reflect real clinical usefulness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Technical Report: Small Language Model for Japanese Clinical and Medicine." pith.science (2026). https://pith.science/paper/VRGGJODE

@misc{pith2026241216423,
  author       = {Pith},
  title        = {Pith review of: Technical Report: Small Language Model for Japanese Clinical and Medicine},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VRGGJODE}},
  note         = {Machine review of arXiv:2412.16423}
}
read the original abstract

This report presents a small language model (SLM) for Japanese clinical and medicine, named NCVC-slm-1. This 1B parameters model was trained using Japanese text classified to be of high-quality. Moreover, NCVC-slm-1 was augmented with respect to clinical and medicine content that includes the variety of diseases, drugs, and examinations. Using a carefully designed pre-processing, a specialized morphological analyzer and tokenizer, this small and light-weight model performed not only to generate text but also indicated the feasibility of understanding clinical and medicine text. In comparison to other large language models, a fine-tuning NCVC-slm-1 demonstrated the highest scores on 6 tasks of total 8 on JMED-LLM. According to this result, SLM indicated the feasibility of performing several downstream tasks in the field of clinical and medicine. Hopefully, NCVC-slm-1 will be contributed to develop and accelerate the field of clinical and medicine for a bright future.

Figures

Figures reproduced from arXiv: 2412.16423 by the authors.

Figure 1
Figure 1. illustrates the pie chart of data proportions for pre-training. OSCAR occupies a large percentage of the whole corpus. Unfortunately, the augmented clinical and medicine textbooks were only 2.6% [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The overview of NCVC-slm-1 model architecture [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The loss logging during self-supervised pre-training [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Token embedding space by t-SNE (perplexity=15) [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Token embedding space by UMAP (neighbor=15) [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: The relationship between seen tokens on pre-training and model performance [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: illustrates the loss logging of instruction tuning. The horizontal and vertical axis is global step and cross entropy loss, respectively [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Attention map layer 1 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Attention map layer 12 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Attention map layer 24 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 23 canonical work pages

  1. [67]

    https://speakerdeck.com/fta98/ri-ben-yu-yi-liao-llmping- jia-bentimakunogou-zhu-toxing-neng-fen-xi

    日本語医療LLMベンチマークの構築と性能分析. https://speakerdeck.com/fta98/ri-ben-yu-yi-liao-llmping- jia-bentimakunogou-zhu-toxing-neng-fen-xi

    work page

  2. [1]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  3. [2]

    Improving language understanding by generative pre-training, 2018

    Alec Radford et al. Improving language understanding by generative pre-training, 2018

    work page 2018

  4. [3]

    Language models are unsupervised multitask learners, 2019

    Alec Radford et al. Language models are unsupervised multitask learners, 2019

    work page 2019

  5. [4]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  6. [5]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. http://arxiv.org/abs/2303.08774, 3 2023

    arXiv 2023

  7. [6]

    Scaling laws for neural language models.http://arxiv.org/abs/2001.08361, 1 2020

    Jared Kaplan and Sam McCandlish et al. Scaling laws for neural language models.http://arxiv.org/abs/2001.08361, 1 2020

    arXiv 2001

  8. [7]

    Unified scaling laws for routed language models

    Aidan Clark et al. Unified scaling laws for routed language models. http://arxiv.org/abs/2202.01169, 2 2022

    arXiv 2022

Show all 71 references
  1. [8]

    Llama: Open and efficient foundation language models

    Hugo Touvron et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  2. [9]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models. https://arxiv.org/abs/2307.09288, 2023

    2023 arXiv

  3. [10]

    The llama 3 herd of models

    Abhimanyu Dubey et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [11]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery et al. Palm: Scaling language modeling with pathways. http://arxiv.org/abs/2204.02311, 4 2022

    2022 arXiv

  5. [12]

    Palm 2 technical report

    Rohan Anil et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    2023 arXiv

  6. [13]

    The falcon series of open language models

    Ebtesam Almazrouei et al. The falcon series of open language models. http://arxiv.org/abs/2311.16867, 11 2023

    2023 arXiv

  7. [14]

    Gemma: Open models based on gemini research and technology

    Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  8. [15]

    Gemma 2: Improving open language models at a practical size

    Team Gemma and Riviere et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  9. [16]

    Qwen technical report

    Jinze Bai et al. Qwen technical report. https://arxiv.org/abs/2309.16609, 2023

    2023 arXiv

  10. [17]

    Qwen2 technical report

    An Yang et al. Qwen2 technical report. https://arxiv.org/abs/2407.10671, 2024

    2024 arXiv

  11. [18]

    The claude 3 model family: Opus, sonnet, haiku anthropic

    Anthropic. The claude 3 model family: Opus, sonnet, haiku anthropic

  12. [19]

    c4ai-command-r-plus (revision 432fac1), 2024

    Cohere For AI. c4ai-command-r-plus (revision 432fac1), 2024

    2024

  13. [20]

    Medalpaca – an open-source collection of medical conversational ai models and training data

    Tianyu Han et al. Medalpaca – an open-source collection of medical conversational ai models and training data. http://arxiv.org/abs/2304.08247, 4 2023

    2023 arXiv

  14. [21]

    Meditron-70b: Scaling medical pretraining for large language models

    Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079, 2023

    2023 arXiv

  15. [22]

    Adapting large language models via reading comprehension

    Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations, 2024

    2024

  16. [23]

    Me llama: Foundation large language models for medical applications

    Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, et al. Me llama: Foundation large language models for medical applications. arXiv preprint arXiv:2402.12749, 2024

    2024 arXiv

  17. [24]

    Capabilities of gemini models in medicine

    Khaled Saab et al. Capabilities of gemini models in medicine. http://arxiv.org/abs/2404.18416, 4 2024

    2024 arXiv

  18. [25]

    https://tech.preferred.jp/ja/blog/llama3-preferred-medswallow-70b/

    Prefered Networks. https://tech.preferred.jp/ja/blog/llama3-preferred-medswallow-70b/

  19. [26]

    Textbooks are all you need

    Suriya Gunasekar et al. Textbooks are all you need. http://arxiv.org/abs/2306.11644, 2023

    2023 arXiv

  20. [27]

    Textbooks are all you need ii: phi-1.5 technical report

    Yuanzhi Li et al. Textbooks are all you need ii: phi-1.5 technical report. http://arxiv.org/abs/2309.05463, 9 2023

    2023 arXiv

  21. [28]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin et al. Phi-3 technical report: A highly capable language model locally on your phone. https://arxiv.org/abs/2404.14219, 2024. 10 2024 NCVC-slm-1

    2024 arXiv

  22. [29]

    Wiki-40b: Multilingual language model dataset, 2020

    Mandy Guo et al. Wiki-40b: Multilingual language model dataset, 2020

    2020

  23. [30]

    A monolingual approach to contextualized word embeddings for mid-resource languages

    Pedro Javier Ortiz Suárez et al. A monolingual approach to contextualized word embeddings for mid-resource languages

  24. [31]

    Random forests, 2001

    Leo Breiman. Random forests, 2001

    2001

  25. [32]

    Elyza-japanese-llama-2-7b, 2023

    Akira Sasaki, Masato Hirakawa, Shintaro Horie, and Tomoaki Nakamura. Elyza-japanese-llama-2-7b, 2023

    2023

  26. [33]

    elyza/llama-3-elyza-jp-8b, 2024

    Masato Hirakawa, Shintaro Horie, Tomoaki Nakamura, Daisuke Oba, Sam Passaglia, and Akira Sasaki. elyza/llama-3-elyza-jp-8b, 2024

    2024

  27. [34]

    Japanese stablelm instruct alpha 7b v2

    Meng Lee, Fujiki Nakamura, Makoto Shing, Paul McCann, Takuya Akiba, and Naoki Orii. Japanese stablelm instruct alpha 7b v2

  28. [35]

    Tokenizerの違いによる日本語 bertモデルの性能評価

    築地俊平 and 新納浩幸. Tokenizerの違いによる日本語 bertモデルの性能評価. 言語処理学会第 27 回年 次大会, 2021

    2021

  29. [36]

    日本語 gptを用いたトークナイザの影響の調査

    井上誠一 et al. 日本語 gptを用いたトークナイザの影響の調査. 言語処理学会第 28 回年次大会, 2022

    2022

  30. [37]

    日本語tokenizerの違いは下流タスク性能に影響を与えるか？ 言語処理学会第 29 回年 次大会, 2023

    藤井巧朗 et al. 日本語tokenizerの違いは下流タスク性能に影響を与えるか？ 言語処理学会第 29 回年 次大会, 2023

    2023

  31. [38]

    Mecab: Yet another part-of-speech and morphological analyzer, 2006

    Taku Kudo. Mecab: Yet another part-of-speech and morphological analyzer, 2006

    2006

  32. [39]

    J-medic: A japanese disease name dictionary based on real clinical usage

    Kaoru Ito et al. J-medic: A japanese disease name dictionary based on real clinical usage. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018

    2018

  33. [40]

    Subword regularization: Improving neural network translation models with multiple subword candidates

    Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959, 2018

    2018 arXiv

  34. [41]

    Roformer: Enhanced transformer with rotary position embedding.http://arxiv.org/abs/2104.09864, 4 2021

    Jianlin Su et al. Roformer: Enhanced transformer with rotary position embedding.http://arxiv.org/abs/2104.09864, 4 2021

    2021 arXiv

  35. [42]

    On layer normalization in the transformer architecture

    Ruibin Xiong et al. On layer normalization in the transformer architecture. http://arxiv.org/abs/2002.04745, 2 2020

    2002 arXiv

  36. [43]

    A study of bfloat16 for deep learning training

    Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja V ooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of bfloat16 for deep learning training. arXiv preprint arXiv:1905.12322, 2019

    1905 arXiv

  37. [44]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). http://arxiv.org/abs/1606.08415, 6 2016

    2016 arXiv

  38. [45]

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. http://arxiv.org/abs/1702.03118, 2 2017

    2017 arXiv

  39. [46]

    Rectified linear units improve restricted boltzmann machines

    Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010

    2010

  40. [47]

    Layer normalization

    Jimmy Lei Ba et al. Layer normalization. http://arxiv.org/abs/1607.06450, 7 2016

    2016 arXiv

  41. [48]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. http://arxiv.org/abs/1910.07467, 10 2019

    1910 arXiv

  42. [49]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie et al. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    2023 arXiv

  43. [50]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. http://arxiv.org/abs/1911.02150, 11 2019

    1911 arXiv

  44. [51]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. http://arxiv.org/abs/1909.08053, 9 2019

    1909 arXiv

  45. [52]

    Nguyen and Julian Salazar

    Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv:1910.05895, 10 2019

    1910 arXiv

  46. [53]

    Spike no more: Stabilizing the pre-training of large language models

    Sho Takase et al. Spike no more: Stabilizing the pre-training of large language models. http://arxiv.org/abs/2312.16903, 12 2023

    2023 arXiv

  47. [54]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010

    2010

  48. [55]

    Training compute-optimal large language models

    Jordan Hoffmann et al. Training compute-optimal large language models. http://arxiv.org/abs/2203.15556, 3 2022. 11 2024 NCVC-slm-1

    2022 arXiv

  49. [56]

    To repeat or not to repeat: Insights from scaling llm under token-crisis

    Fuzhao Xue et al. To repeat or not to repeat: Insights from scaling llm under token-crisis. Advances in Neural Information Processing Systems, 36, 2024

    2024

  50. [57]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. http://arxiv.org/abs/2205.14135, 5 2022

    2022 arXiv

  51. [58]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley et al. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020

    2020

  52. [59]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020

    2020

  53. [60]

    Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R

    Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. http://arxiv.org/abs/1207.0580, 7 2012

    2012 arXiv

  54. [61]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. http://arxiv.org/abs/1711.05101, 11 2017

    2017 arXiv

  55. [62]

    Finetuned language models are zero-shot learners closed-book qa

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le Google Research. Finetuned language models are zero-shot learners closed-book qa. https://github.com/google-research/flan., 2022

    2022

  56. [63]

    Jmed-llm: Japanese medical evaluation dataset for large language models

    LLM-jp. Jmed-llm: Japanese medical evaluation dataset for large language models. https://github.com/sociocom/JMED-LLM, 2024

    2024

  57. [64]

    Neftune: Noisy embeddings improve instruction finetuning

    Neel Jain et al. Neftune: Noisy embeddings improve instruction finetuning. http://arxiv.org/abs/2310.05914, 10 2023

    2023 arXiv

  58. [65]

    Evaluating gpt-4 and chatgpt on japanese medical licensing examinations

    Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. Evaluating gpt-4 and chatgpt on japanese medical licensing examinations. http://arxiv.org/abs/2303.18027, 3 2023

    2023 arXiv

  59. [66]

    A coefficient of agreement for nominal scales

    Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement , 20(1):37–46, 1960

    1960

  60. [68]

    The curse of recursion: Training on generated data makes models forget

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. http://arxiv.org/abs/2305.17493, 5 2023

    2023 arXiv

  61. [69]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008

    2008

  62. [70]

    Umap: Uniform manifold approximation and projection for dimension reduction

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. http://arxiv.org/abs/1802.03426, 2 2018

    2018 arXiv

  63. [71]

    Efficient estimation of word representations in vector space

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. http://arxiv.org/abs/1301.3781, 1 2013. A Appendix A.1 Text Generation Samples Text Generation Samples 脳卒中と循環器病について教えてください。 循 環 器病は、 脳 梗 塞・ 心 筋 梗 塞が 多く、 高 齢 者...

    2013 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.