Pith. sign in

REVIEW 4 major objections 4 minor 52 references

MedLLM: An Open Medical Language Model at the Sub-Billion Scale

T0 review · 4 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 0.1B-parameter medical language model stays competitive on context-grounded question answering while collapsing on knowledge-recall tasks, revealing a task-structured dissociation that compression exposes and 7B models mask.

desk verdict Useful small-model artifacts; the dissociation headline overstates and two abstract/result numbers don't match the table. read the letter →

arxiv 2607.27490 v1 pith:J67KGPM3 submitted 2026-07-29 cs.AI

Maxx Richard Rahman , Asim Ahmed , Mihan Mohagheghzadeh , Wolfgang Maass This is my paper
classification cs.AI
keywords smalllanguagemodelsmedicalquestionansweringdomainadaptationdirectpreferenceoptimizationmodelcompressiontask-structureddissociationreference-guidedcorpusselectionretrievalvsparametricrecall
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that medical competence in language models does not degrade uniformly as parameters shrink; instead, at the sub-billion scale it splits by task type. A 0.1B model trained with a fully open pipeline comes within 2.9 percentage points of a medically adapted 7B model on context-grounded PubMedQA and significantly beats every 7B baseline on MedMCQA, yet stays near random chance on the knowledge-heavy MedQA benchmark. The authors interpret this as evidence that at small scale, the binding constraint is model capacity for storing medical facts, not the adaptation recipe. If true, it implies that sub-billion medical models are best used with retrieval and supplied context rather than as parametric memory. The paper also releases MedFineWeb, a reference-guided medical corpus selected from general web text by embedding similarity to medical QA data, avoiding curated medical sources.

What carries the argument

The central mechanism is MedFineWeb, a reference-guided medical corpus built by splitting general web text into ~1000-word chunks and retaining the top 50% ranked by average cosine similarity to the top-5 nearest examples from MedMCQA, MedQA, and PubMedQA references, then merging and deduplicating. This selection shifts the corpus toward medical topic and style without using curated medical sources or the benchmark QA texts. The training pipeline combines curriculum sequence-length pretraining, domain fine-tuning on MedFineWeb, and per-benchmark SFT followed by DPO with beta=0.1. The paper also ties the embedding and output projection to save parameters at a high vocabulary-to-width ratio.

What would settle it

Train the same three-phase pipeline at a mid-size scale, say 1B or 2B parameters, on MedFineWeb and with the same per-benchmark SFT and DPO, and test whether MedQA accuracy rises while MedMCQA and PubMedQA results stay roughly stable; if the MedQA gap closes only at 7B or does not close at all, the capacity explanation is weakened. Alternatively, evaluate MedLLM on MedQA in a retrieval-augmented setting; if an off-the-shelf retrieval system lifts accuracy well above the 28.1% floor, the limitation is less about inherent memory and more about the interaction between memorized knowledge and form

Watch

Extended reading notes

Core claim

MedLLM, a 0.1B-parameter decoder-only transformer, trained through general pretraining, domain fine-tuning on MedFineWeb, and per-benchmark SFT plus DPO, exhibits a dissociation visible only at sub-billion scale: on context-grounded QA (PubMedQA) it reaches 58.2% after DPO, within 2.9pp of the medically adapted PMC-LLaMA-7B and ahead of instruction-tuned 7B baselines; on the recall benchmark MedMCQA it reaches 34.9%, significantly exceeding every 7B and sub-7B baseline; but on MedQA, a USMLE-style vignette benchmark, it stays at 28.1%, only 3.1pp above the floor. The authors argue that where recall succeeds, the constraint is model capacity rather than adaptation, and where recall fails (Med

Load-bearing premise

The paper infers that the observed task split is caused by model capacity from a comparison of MedLLM against 7B baselines that differ in architecture, training data, and evaluation protocol, rather than from training control models at multiple scales with the identical pipeline.

Editorial extensions

If this is right

  • If the dissociation is real, sub-billion medical models should be paired with retrieval-augmented generation to supply the facts they cannot store.
  • Perplexity is an unreliable model-selection signal at this scale: during domain fine-tuning, validation perplexity halves while benchmark accuracy stays flat.
  • DPO's effect at small scale is to sharpen existing answer discrimination rather than to inject new knowledge, implying preference alignment should be paired with capacity or retrieval for recall-heavy tasks.
  • The task-structured pattern, rather than reference-structured gains, argues against benchmark contamination from MedFineWeb even though the corpus was selected by similarity to benchmark QA data.
  • A 7B-scale dissociation is masked because both capabilities are present; the sub-billion regime is the only place to detect which capability is capacity-bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the authors leave implicit: adding a retrieval step over a medical corpus to MedLLM on MedQA would test whether the MedQA floor is truly a stored-knowledge capacity limit or also a formatting/distance issue; if retrieval closes most of the gap, it would sharpen the paper's central conclusion.
  • The reference-guided corpus construction could be transferable to other specialized domains (legal, financial, scientific) by swapping the QA reference sets; the paper only teases this possibility in its contribution list.
  • The DPO margin amplification result, if generalized, suggests that at sub-billion scale, preference optimization may primarily recalibrate confidence rather than competence, which has implications for calibration-sensitive applications like clinical decision support.
  • A controlled capacity sweep with the exact same data and recipe at 0.1B, 0.5B, and 2B would directly confirm that the task split is a function of parameters rather than of the specific heterogeneous baselines used here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MedLLM, a 0.1B-parameter open medical language model trained with a three-phase pipeline: general pretraining with curriculum sequence-length scheduling, domain fine-tuning on MedFineWeb (a corpus selected from general web text by embedding similarity to medical QA benchmarks), and SFT plus DPO alignment. The central claim is a task-structured dissociation under compression: on context-grounded PubMedQA the model is said to come within 2.9pp of a medically adapted 7B system, while on knowledge-recall benchmarks it is capacity-bound on MedQA yet significantly exceeds all 7B and sub-7B baselines on MedMCQA. The paper also reports ablations showing that domain fine-tuning lowers perplexity without raising benchmark accuracy, and that DPO amplifies the correct-vs-incorrect log-probability margin.

Significance. If the central dissociation claim were supported, this would be a useful contribution to sub-billion-scale medical modeling and to understanding how capacity constraints shape medical capability. The paper has clear strengths: the full pipeline and data are released, the parameter allocation is carefully accounted for, the ablation studies are informative, and the intermediate-checkpoint evaluation showing perplexity-accuracy divergence is a valuable observation. However, the quantitative claims in the abstract and conclusion overreach the evidence in Table 4, the significance claims are contradicted by the paper's own dagger marking, the scale-attribution argument lacks a same-pipeline control, and the contamination defense is circular. These issues are load-bearing for the headline claims.

major comments (4)
  1. [Abstract and Conclusions; Table 4] The abstract and conclusion state that on context-grounded PubMedQA MedLLM comes 'within 2.9pp of a medically adapted 7B model'. Table 4 gives MedLLM DPO 58.2; PMC-LLaMA-7B (the only medically adapted 7B baseline) is 72.9, a 14.7pp gap; Falcon-7B is 65.3, a 7.1pp gap. The 2.6pp gap is to Gemma-2B, a general 2B model, not a medically adapted 7B. Also, 'surpasses the ... general-purpose 7B baselines' on this task is contradicted by Falcon-7B (65.3, marked with \ddagger). The central quantitative claim must be corrected or removed.
  2. [Fine-Tuned Model Evaluation; Table 4] The text claims that MedLLM (DPO) 'significantly exceeding every 7B and sub-7B baseline under McNemar’s test' on MMLU. In Table 4, the MMLU row shows Zephyr-7B at 32.1 with no \dagger mark, while MedLLM (DPO) is 32.4. By the table's stated convention, the absence of a dagger means MedLLM does not significantly exceed Zephyr. This directly contradicts the claim and must be reconciled.
  3. [Baselines; Experimental Settings; Conclusions] The 'visible only at sub-billion scale' claim requires a same-pipeline scaling comparison. No such comparison is provided: MedLLM is 0.1B and baselines are 1.5B-7B models with different architectures, training corpora, fine-tuning procedures, and evaluation protocols (e.g., MedLLM zero-shot top-token vs. 3-shot ICL for pretrained baselines). The observed dissociation could reflect benchmark difficulty or baseline heterogeneity rather than model capacity. A controlled experiment training the same pipeline at, say, 0.5B, 1B, and 7B, or at least a clear statement that the claim is restricted to this comparison, is needed.
  4. [Leakage and Contamination Analysis] The contamination defense is circular. The paper uses MedMCQA, MedQA, and PubMedQA as semantic references to select MedFineWeb and then evaluates on the same benchmarks. The argument that no answer leakage occurred is, in the paper's own words, 'internal to the outcomes': the task-structured pattern is offered as evidence against contamination. But similarity-based selection toward the benchmark distribution could inflate performance without copying exact text, and the predicted reference-structured pattern is not a necessary consequence of leakage. A valid control would evaluate on held-out medical benchmarks not used as references, or use cross-split selection/evaluation. This needs to be addressed for the benchmark results to be interpretable.
minor comments (4)
  1. [Leakage and Contamination Analysis] The n-gram overlap check reports '22 chunks (about 0.0018%)', but if MedFineWeb has ~3M chunks, 22/3M is 0.0007%. Please clarify the denominator and the exact matching procedure.
  2. [Analysis on Continued Pretraining] Table 6 says evaluation is 'few-shot following Chen et al. (2023)' but does not specify the number of shots or the demonstration source. State the protocol explicitly.
  3. [Results; Eq. (8)] The claim that DPO's 5.2x margin amplification is 'the O(1/β) separation the objective predicts' is heuristic; the DPO objective does not by itself predict a precise numerical factor for the change in mean log-probability margin. Reword to avoid implying an exact theoretical prediction.
  4. [Figure 2] Figure 2 caption says 'pretrained model comparison' but MedLLM is evaluated after DFT. Clarify that this is the pretrained+DFT model, not the raw pretrained checkpoint.

Circularity Check

2 steps flagged · score 5.0 of 10

Leakage defense is circular and the DPO margin 'prediction' is a post-hoc match to the chosen objective; the core dissociation claim remains an empirical benchmark observation but its validation is partly self-referential.

  1. other [Reference-Guided Medical Corpus, 'Leakage and Contamination Analysis']
    "However, verbatim checks do not address the deeper concern that selection by similarity to the benchmarks could inject answer knowledge even without copying text. Our results indicate it does not, by an argument internal to the outcomes. Were similarity-based selection leaking answers, the datasets used as references would be the ones to improve, and would improve together; instead the effect is task-structured, not reference-structured: MedMCQA and MedQA, both references, remain near the task floor after adaptation, while the gain concentrates on context-grounded PubMedQA, where the answer is"

    The paper uses its own benchmark outcome pattern to rule out contamination of those benchmarks. The central claim (task-structured dissociation) is simultaneously the evidence that the measurements are valid, so the benchmark results cannot independently support the claim. If the task-structured pattern itself were a selection artifact, the same argument would 'confirm' no leakage, making the defense unfalsifiable by construction.

  2. fitted input called prediction [Results, 'Fine-Tuned Model Evaluation' (paragraph after Table 4)]
    "DPO adds accuracy over SFT (MedMCQA33.9→34.9%, MedQA+1.5pp, PubMedQA+0.3pp, a slight MMLU regression) yet amplifies the mean correct-versus-incorrect log-probability margin 5.2×, from 0.1037 to 0.5403, the O(1/β) separation the objective predicts at β=0.1 (Eq. 8)"

    The 'predicted' quantity is the log-ratio margin that the DPO objective itself is minimized over, at a β selected by the authors. Reporting that this margin grows after optimizing the same margin objective is not an independent prediction; the O(1/β) scaling is a post-hoc match to the chosen hyperparameter rather than a parameter-free test.

full rationale

The paper's central empirical claim—that medical competence splits by task type at 0.1B—is not itself derived from an equation; it is an observed benchmark pattern. However, its validation is partially circular: MedFineWeb is selected by similarity to the very benchmarks on which the model is evaluated, and the only semantic-leakage defense is the outcome pattern that constitutes the finding. The DPO margin observation is likewise a consequence of optimizing the DPO loss rather than a predicted phenomenon. I do not see a load-bearing self-citation chain; citations to prior DPO, architecture, and scaling work are external. The absence of a same-pipeline scaling sweep is a serious control/validity concern for the 'visible only at sub-billion scale' inference, but that is not circularity per se. These considerations warrant a partial circularity score of 5 rather than a higher score, since the model, corpus, and benchmark comparisons are still independently specified and reported.

Assumptions & free parameters 6 free parameters · 6 assumptions · 1 invented entities

The central claim's load-bearing inputs are the reference-guided selection (which uses the evaluation benchmarks themselves), the choice of hyperparameters (α, β, λ_z), and the assumption that heterogeneous 7B baselines are valid controls for scale attribution.

free parameters (6)
  • α (MedFineWeb selection top fraction) = 0.50
    Algorithm 1 keeps the top 50% of chunks by similarity; no sensitivity analysis, and the choice directly determines corpus size and composition.
  • β (DPO temperature) = 0.1
    Chosen hyperparameter; the paper's 'O(1/β) separation' explanation for the observed 5.2× log-probability margin amplification depends on this value.
  • λ_z (z-loss coefficient) = 1e-4
    Z-loss weight in Eq. 6, chosen for output-scale stability at V=32,000.
  • top-5 nearest neighbors in similarity scoring = 5
    Averaging over five neighbors is a design choice to 'robustify' the score; no ablation.
  • reference examples per dataset = 20,000
    Up to 20k references per QA set; for MedQA only 10,178 are available; chosen to 'keep retrieval tractable'.
  • curriculum sequence lengths = 128→256→512
    Three-stage curriculum schedule; chosen for optimization stability, no comparison to fixed-length training.
assumptions (6)
  • domain assumption Sentence Transformer cosine similarity is an adequate proxy for medical relevance of web chunks.
    Used in Algorithm 1 to select MedFineWeb; if the embedding model does not capture medical relevance, the corpus is mis-specified.
  • domain assumption The four benchmarks can be partitioned into context-grounded (PubMedQA) and knowledge-recall (MedMCQA, MedQA, MMLU) task types.
    This partition is the basis of the dissociation claim; MedQA and MedMCQA are both multiple-choice recall yet produce opposite results, so the partition is not clean.
  • domain assumption Heterogeneous 7B baselines, with varying training protocols, represent '7B medical competence'.
    Used throughout Results to conclude that MedLLM's performance 'significantly exceeds every 7B and sub-7B baseline' on recall benchmarks and to infer scale-dependent dissociation.
  • ad hoc to paper Absence of exact text matches and the task-structured outcome pattern rule out benchmark contamination.
    Leakage and Contamination Analysis: the paper's own defense is 'internal to the outcomes,' which is circular.
  • standard math DPO with β=0.1 amplifies the correct-vs-incorrect log-probability margin by O(1/β).
    Used to explain the 5.2× margin amplification; this is a qualitative scaling property of DPO under the Bradley-Terry model, not a quantitative derivation.
  • standard math McNemar's test significance markers in Table 4 are correctly computed.
    The paper's significance claims rely on these; at least one textual claim is inconsistent with the markers.
invented entities (1)
  • MedFineWeb
    purpose: A 3B-token medical plain-text corpus for domain fine-tuning, selected from general web by embedding similarity to medical QA references.
    The paper claims it is 'released' but provides no URL, hash, or downloadable artifact; its existence and contents cannot be independently verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedLLM: An Open Medical Language Model at the Sub-Billion Scale." pith.science (2026). https://pith.science/paper/J67KGPM3

@misc{pith2026260727490,
  author       = {Pith},
  title        = {Pith review of: MedLLM: An Open Medical Language Model at the Sub-Billion Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J67KGPM3}},
  note         = {Machine review of arXiv:2607.27490}
}
abstract

Open medical language models have converged on a single scale: every widely used system runs at 7B parameters or more, leaving the sub-billion regime uncharacterized. We present MedLLM, an open 0.1B-parameter medical language model trained through a fully open three-phase pipeline: general pretraining with curriculum sequence-length scheduling, domain fine-tuning on MedFineWeb, a reference-guided medical corpus we release that is selected from general web data by embedding similarity to medical question-answering (QA) data, and preference-aligned fine-tuning combining SFT with direct preference optimization (DPO). Across medical benchmarks, MedLLM shows a pattern visible only at sub-billion scale: medical competence does not degrade uniformly under compression but splits by task type. On context-grounded QA it comes within $2.9$pp of a medically adapted 7B model and surpasses the instruction-tuned and general-purpose 7B baselines; on knowledge-recall QA it stays near the task floor on clinical-vignette MedQA yet significantly exceeds every 7B and sub-7B baseline on MedMCQA, indicating that where recall fails the constraint is model capacity rather than adaptation. This dissociation is masked at 7B, where both capabilities are present, and surfaces only when capacity is scarce.

Figures

Figures reproduced from arXiv: 2607.27490 by the authors.

Figure 1
Figure 1. MedLLM model: 0.1B-parameter decoder-only transformer with 20 layers. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Pretrained model comparison across scales. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 3 linked inside Pith

  1. [1]

    Proceedings of the Conference on Language Modeling , year =

    Walsh, Pete and Soldaini, Luca and Groeneveld, Dirk and Lo, Kyle and Arora, Shane and Bhagia, Akshita and others , title =. Proceedings of the Conference on Language Modeling , year =

  2. [2]

    2023 , eprint =

    Chen, Zeming and Hern. 2023 , eprint =

    2023

  3. [3]

    and Ermon, Stefano and Finn, Chelsea , title =

    Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Manning, Christopher D. and Ermon, Stefano and Finn, Chelsea , title =. Advances in Neural Information Processing Systems , volume =

  4. [4]

    Singhal, Karan and Azizi, Shekoofeh and Tu, Tao and Mahdavi, S. Sara and Wei, Jason and Chung, Hyung Won and Scales, Nathan and Tanwani, Ajay and Cole-Lewis, Heather and Pfohl, Stephen and Payne, Perry and Seneviratne, Martin and Gamble, Paul and Kelly, Chris and Babiker, Abubakr and Sch. Large Language Models Encode Clinical Knowledge , journal =

  5. [5]

    Towards Expert-Level Medical Question Answering with Large Language Models , year =

    Singhal, Karan and Tu, Tao and Gottweis, Juraj and Sayres, Rory and Wulczyn, Ellery and Hou, Le and Clark, Kevin and Pfohl, Stephen and Cole-Lewis, Heather and Neal, Darlene and Schaekermann, Mike and Wang, Amy and Amin, Mohamed and Lachgar, Sami and Mansfield, Philip and Prakash, Sushant and Green, Bradley and Dominowska, Ewa and Ag. Towards Expert-Level...

    arXiv

  6. [6]

    2023 , eprint =

    Nori, Harsha and King, Nicholas and McKinney, Scott Mayer and Carignan, Dean and Horvitz, Eric , title =. 2023 , eprint =

    2023

  7. [7]

    Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and others , title =. Advances in Neural Information Processing Systems , volume =

  8. [8]

    2023 , eprint =

    Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and others , title =. 2023 , eprint =

    2023

Show all 52 references
  1. [9]

    Don't Stop Pretraining: Adapt Language Models to Domains and Tasks , booktitle =

    Gururangan, Suchin and Marasovi. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks , booktitle =

  2. [10]

    Continual Pre-Training of Large Language Models: How to (Re)warm Your Model? , year =

    Gupta, Kshitij and Th. Continual Pre-Training of Large Language Models: How to (Re)warm Your Model? , year =. 2308.04014 , archivePrefix =

    arXiv

  3. [11]

    Curriculum Learning , booktitle =

    Bengio, Yoshua and Louradour, J. Curriculum Learning , booktitle =

  4. [12]

    Bioinformatics , volume =

    Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo , title =. Bioinformatics , volume =

  5. [13]

    ACM Transactions on Computing for Healthcare , volume =

    Gu, Yu and Tinn, Robert and Cheng, Hao and Lucas, Michael and Usuyama, Naoto and Liu, Xiaodong and Naumann, Tristan and Gao, Jianfeng and Poon, Hoifung , title =. ACM Transactions on Computing for Healthcare , volume =

  6. [14]

    2023 , eprint =

    Wu, Chaoyi and Lin, Weixiong and Zhang, Xiaoman and Zhang, Ya and Xie, Weidi and Wang, Yanfeng , title =. 2023 , eprint =

    2023

  7. [15]

    and Papaioannou, Jens-Michalis and Grundmann, Paul and Oberhauser, Tom and L

    Han, Tianyu and Adams, Lisa C. and Papaioannou, Jens-Michalis and Grundmann, Paul and Oberhauser, Tom and L. 2023 , eprint =

    2023

  8. [16]

    and PourNejatian, Nima and Costa, Anthony B

    Peng, Cheng and Yang, Xi and Chen, Aokun and Smith, Kaleb E. and PourNejatian, Nima and Costa, Anthony B. and Martin, Cheryl and Flores, Mona G. and Zhang, Ying and Magoc, Tanja and Lipori, Gloria and Mitchell, Duane A. and Ospina, Noel S. and Ahmed, Mustafa M. and Hogan, Will...

  9. [17]

    Advances in Neural Information Processing Systems , volume =

    Ouyang, Long and Wu, Jeffrey and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and others , title =. Advances in Neural Information Processing Systems , volume =

  10. [18]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Sun, Yu and Wang, Shuohuan and Li, Yukun and Feng, Shikun and Tian, Hao and Wu, Hua and Wang, Haifeng , title =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =

  11. [19]

    Zephyr: Direct Distillation of

    Tunstall, Lewis and Beeching, Edward and Lambert, Nathan and Rajani, Nazneen and Rasul, Kashif and Belkada, Younes and Huang, Shengyi and von Werra, Leandro and Fourrier, Cl. Zephyr: Direct Distillation of. Proceedings of the First Conference on Language Modeling , year =

  12. [20]

    2024 , eprint =

    Chen, Junying and Gui, Ruyi and Wang, Anningzhe and Zhang, Shunian and Wan, Xidong and Zhu, Benyou , title =. 2024 , eprint =

    2024

  13. [21]

    Applied Sciences , volume =

    Jin, Di and Pan, Eileen and Oufattole, Nassim and Weng, Wei-Hung and Fang, Hanyi and Szolovits, Peter , title =. Applied Sciences , volume =

  14. [22]

    Proceedings of the Conference on Health, Inference, and Learning , pages =

    Pal, Ankit and Umapathi, Logesh Kumar and Sankarasubbu, Malaikannan , title =. Proceedings of the Conference on Health, Inference, and Learning , pages =

  15. [23]

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pages =

    Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William and Lu, Xinghua , title =. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pages =

    2019

  16. [24]

    Proceedings of the International Conference on Learning Representations , year =

    Hendrycks, Dan and Burns, Collin and Basart, Steven and Zou, Andy and Mazeika, Mantas and Song, Dawn and Steinhardt, Jacob , title =. Proceedings of the International Conference on Learning Representations , year =

  17. [25]

    Transactions on Machine Learning Research , year =

    Liang, Percy and Bommasani, Rishi and Lee, Tony and Tsipras, Dimitris and Soylu, Dilara and Yasunaga, Michihiro and Zhang, Yian and Narayanan, Deepak and Wu, Yuhuai and Kumar, Ananya and others , title =. Transactions on Machine Learning Research , year =

  18. [26]

    Advances in Neural Information Processing Systems , volume =

    Zhang, Biao and Sennrich, Rico , title =. Advances in Neural Information Processing Systems , volume =

  19. [27]

    Neurocomputing , volume =

    Su, Jianlin and Ahmed, Murtadha and Lu, Yu and Pan, Shengfeng and Bo, Wen and Liu, Yunfeng , title =. Neurocomputing , volume =

  20. [28]

    2020 , eprint =

    Shazeer, Noam , title =. 2020 , eprint =

    2020

  21. [29]

    and Rosenberg, William M.C

    Sackett, David L. and Rosenberg, William M.C. and Gray, J.A. Muir and Haynes, R. Brian and Richardson, W. Scott , title =. BMJ , volume =

  22. [30]

    Nature Medicine , volume =

    Thirunavukarasu, Arun James and Ting, Darren Shu Jeng and Elangovan, Kabilan and Gutierrez, Laura and Tan, Ting Fang and Ting, Daniel Shu Wei , title =. Nature Medicine , volume =

  23. [31]

    2023 , eprint =

    Ke, Zixuan and Shao, Yijia and Lin, Haowei and Koncel-Kedziorski, Rik and Iyer, Bhargavi and Xiao, Chunting and Neubig, Graham , title =. 2023 , eprint =

    2023

  24. [32]

    and Ba, Jimmy and Krishnan, Rahul G

    Toma, Augustin and Lawler, Patrick R. and Ba, Jimmy and Krishnan, Rahul G. and Rubin, Barry B. and Wang, Bo , title =. 2023 , eprint =

    2023

  25. [33]

    2024 , eprint =

    Zhang, Xinlu and Li, Chenxin and Zong, Xianjun and Tian, Zheng and Yu, Jie and Yan, Fei , title =. 2024 , eprint =

    2024

  26. [34]

    and Leike, Jan and Brown, Tom B

    Christiano, Paul F. and Leike, Jan and Brown, Tom B. and Martic, Miljan and Legg, Shane and Amodei, Dario , title =. Advances in Neural Information Processing Systems , volume =

  27. [35]

    , title =

    Bradley, Ralph Allan and Terry, Milton E. , title =. Biometrika , volume =

  28. [36]

    Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics , pages =

    Press, Ofir and Wolf, Lior , title =. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics , pages =

  29. [37]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =

    Kudo, Taku and Richardson, John , title =. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =

    2018

  30. [38]

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pages =

    Reimers, Nils and Gurevych, Iryna , title =. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pages =

    2019

  31. [39]

    Billion-Scale Similarity Search with

    Johnson, Jeff and Douze, Matthijs and J. Billion-Scale Similarity Search with. IEEE Transactions on Big Data , volume =

  32. [40]

    2023 , howpublished =

    2023

  33. [41]

    , title =

    Raffel, Colin and Shazeer, Noam and Roberts, Adam and Lee, Katherine and Narang, Sharan and Matena, Michael and Zhou, Yanqi and Li, Wei and Liu, Peter J. , title =. Journal of Machine Learning Research , volume =

  34. [42]

    2019 , howpublished =

    Gokaslan, Aaron and Cohen, Vanya , title =. 2019 , howpublished =

    2019

  35. [43]

    Almazrouei, Ebtesam and Alobeidli, Hamza and Alshamsi, Abdulaziz and Cappelli, Alessandro and Cojocaru, Ruxandra and Debbah, M. The. 2023 , eprint =

    2023

  36. [44]

    Jiang, Albert Q. and Sablayrolles, Alexandre and Mensch, Arthur and Bamford, Chris and Singh Chaplot, Devendra and de las Casas, Diego and Bressand, Florian and Lengyel, Gianna and Lample, Guillaume and Saulnier, Lucile and Lavaud, L. 2023 , eprint =

    2023

  37. [45]

    and Zhou, Denny , title =

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed and Le, Quoc V. and Zhou, Denny , title =. Advances in Neural Information Processing Systems , volume =

  38. [46]

    and Chi, Ed and Narang, Sharan and Chowdhery, Aakanksha and Zhou, Denny , title =

    Wang, Xuezhi and Wei, Jason and Schuurmans, Dale and Le, Quoc V. and Chi, Ed and Narang, Sharan and Chowdhery, Aakanksha and Zhou, Denny , title =. Proceedings of the International Conference on Learning Representations , year =

  39. [47]

    and Finn, Chelsea , title =

    Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea , title =. Advances in Neural Information Processing Systems , volume =

  40. [48]

    Advances in Neural Information Processing Systems , volume =

    Limits to Depth Efficiency of Self-Attention , author =. Advances in Neural Information Processing Systems , volume =

  41. [49]

    arXiv preprint arXiv:2403.08295 , year =

    Gemma: Open Models Based on Gemini Research and Technology , author =. arXiv preprint arXiv:2403.08295 , year =

    arXiv

  42. [50]

    , journal =

    Bolton, Elliot and Venigalla, Abhinav and Yasunaga, Michihiro and Hall, David and Xiong, Betty and Lee, Tony and Daneshjou, Roxana and Frankle, Jonathan and Liang, Percy and Carbin, Michael and Manning, Christopher D. , journal =

  43. [51]

    Luo, Renqian and Sun, Liai and Xia, Yingce and Qin, Tao and Zhang, Sheng and Poon, Hoifung and Liu, Tie-Yan , journal =

  44. [52]

    Findings of the Association for Computational Linguistics: NAACL 2023 , year =

    Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions , author =. Findings of the Association for Computational Linguistics: NAACL 2023 , year =

    2023

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.