Pith. sign in

REVIEW 4 major objections 6 minor 33 references

Dissecting Clinical Reasoning in Language Models: A Comparative Study of Prompts and Model Adaptation Strategies

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prompt structure alone explains up to 44% of F1 variance in clinical language inference, and LoRA fine-tuning lets a 3.8B model trail GPT-4o-mini by just 7.1%.

desk verdict Useful controlled study undone by overclaimed headline numbers: the 44% variance figure is partial eta-squared, and the LoRA gains don't hold up in the paper's own Table 16. read the letter →

arxiv 2507.04142 v1 pith:QHEYKFCA submitted 2025-07-05 cs.CL cs.AI

Mael Jullien , Marco Valentino , Leonardo Ranaldi , Andre Freitas This is my paper
classification cs.CLcs.AI
keywords clinicalnaturallanguageinferencepromptstructurechain-of-thoughtlow-rankadaptation(LoRA)NLI4CTreasoningtypesvariancedecomposition(ANOVA)trialreports
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that in clinical natural language inference, the way a prompt is structured is a primary driver of model performance, comparable in strength to which model is chosen. Using the NLI4CT benchmark, the authors compare four structurally distinct prompt families (free-form chain-of-thought, iterative self-critique, typed action loops, and symbolically structured reasoning) under controlled conditions across four small language models, with and without LoRA fine-tuning. They report that prompt type alone accounts for up to 44% of the variance in macro-F1 after controlling for model identity and fine-tuning, and that LoRA adds a consistent 8-12 F1 points while pushing output validity above 97%. If true, the practical consequence is that prompt design deserves equal weight with model selection and adaptation in building clinical NLP systems, and that compact models with well-chosen prompts and lightweight fine-tuning can approach frontier models.

What carries the argument

The load-bearing machinery is threefold: (1) a taxonomy of four prompt categories treated as structural scaffolds that bias the reasoning trajectory (unstructured natural language reasoning, iterative self-refined reasoning, typed action-based reasoning, and symbolically structured reasoning), each instantiated by a single concrete template; (2) a controlled factorial design in which four small language models are evaluated across all prompt categories, with and without LoRA, using demonstrations distilled from GPT-4o-mini and filtered for correctness; and (3) a fixed-effects Type II ANOVA with partial eta-squared as the variance decomposition, which attributes shares of F1 variance to prompt, model identity, and LoRA. The ANOVA is what converts raw score comparisons into the claim that prompt structure is a primary driver; the controlled design is what makes that attribution defensible.

What would settle it

Write several independently designed templates for each prompt category (varying wording, length, and output constraints while keeping the structural mode fixed), run the same four models through the same LoRA protocol, and recompute the ANOVA: if within-category template variation explains a share of F1 variance comparable to the 44% attributed to category, the structural-category claim is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that prompt structure is a first-order determinant of clinical reasoning performance, not a tuning detail. A fixed-effects Type II ANOVA over the full experimental grid shows that the prompt factor explains 44% of the variance in macro-F1 left after controlling for model architecture and LoRA status ($\eta^2_{\text{partial}} = 0.440$, $p < 0.001$), an effect size comparable to model identity (0.507) and larger than LoRA (0.357). Within individual reasoning classes the same factor explains 30-44% of variance. The authors further claim that LoRA fine-tuning on high-precision demonstrations generated by GPT-4o-mini yields consistent gains of +8 to +12 F1, lifts answer validity above 97%, and allows Phi-4-mini-reasoning (3.8B) with the chain-of-thought prompt to reach 0.729 F1, trailing GPT-4o-mini by only 7.1%. On out-of-domain clinical NLI benchmarks (MedNLI and TREC Clinical Trials), LoRA trained on NLI4CT alone improves F1 in 75% of model-prompt combinations.

Load-bearing premise

The entire 44% claim rests on one template standing in for each prompt category; if the specific wording, length, or output format of the template, rather than its structural style of reasoning, is what drives performance, the category-level conclusion gives way.

Editorial extensions

If this is right

  • Prompt strategy should be treated as a first-class design variable in clinical NLP evaluation, on par with model architecture and adaptation.
  • No single prompt wins on every reasoning type: NLR and TAR lead overall, TAR wins on evidence and quantitative derivation, and SSR gives the largest lift on quantitative comparison and derivation, so prompt choice should be matched to the dominant reasoning demand of the application.
  • Prompt selection enables tunable precision-recall trade-offs, with ISRR maximizing recall and SSR maximizing precision.
  • Compact models with strong prompts and LoRA are a viable deployment path in settings where frontier models cannot be used, since a 3.8B model comes within 7.1 F1 points of GPT-4o-mini.
  • LoRA's out-of-domain gains suggest some inference capabilities acquired on NLI4CT transfer to other clinical NLI distributions without task-specific training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 44% variance claim reproduces across multiple templates per category, it would suggest that clinical NLI performance is governed less by model knowledge than by the structural format in which reasoning is elicited, shifting research resources toward prompt scaffolding rather than ever-larger models.
  • Quantitative derivation, the class least helped by LoRA (+0.047 lift versus +0.117 for evidence), may require symbolic or tool-based augmentation rather than more text supervision; the paper hints at this but does not test it.
  • The single-template-per-category design means the paper's categories are confounded with template surface features; a natural extension is a template-variation study that isolates structural category from wording and length.
  • The gap-narrowing result invites a cost-benefit extrapolation the authors do not make: for applications tolerant of a 7% F1 gap, sub-4B locally deployable models could replace API-based frontier models, removing data-privacy and per-call cost barriers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports a controlled comparison of four prompt categories (NLR, ISRR, TAR, SSR) plus a zero-shot baseline, combined with LoRA fine-tuning, for clinical natural language inference on NLI4CT. The authors introduce expert-labeled reasoning-type annotations over the NLI4CT test set, generate filtered demonstrations with GPT-4o-mini, adapt four small models (1.5-3.8B) with LoRA under a uniform training protocol, and evaluate on NLI4CT plus MedNLI and TREC as generalization benchmarks. The central claims are that prompt type explains up to 44% of variance in macro-F1, that LoRA gives consistent +8-12 F1 gains and validity above 97%, that the best small model trails GPT-4o-mini by only 7.1%, and that LoRA improves generalization in 75% of model-prompt pairs. Code, annotations, prompts, demonstrations, and checkpoints are promised for release.

Significance. The study addresses an important and underexplored question: how prompt structure and parameter-efficient adaptation jointly affect clinical NLI. The controlled setup, the multi-model comparison, the reasoning-type annotation effort, and the external generalization checks are valuable assets, and the release plan strengthens reproducibility. If the central claims were supported, the finding that prompt design is comparable in importance to model architecture would be practically significant for clinical NLP deployment. However, the main variance claim rests on a misinterpreted statistic, and several LoRA-specific claims are contradicted by the paper's own tables. The underlying experimental framework is salvageable, but the headline findings are not currently established as stated.

major comments (4)
  1. [Results Overview; Table 25; Figure 2] The repeated claim that "prompt type alone accounts for up to 44% of the variance in macro-F1" is based on partial eta-squared (η²partial = 0.440). Partial eta-squared is SS_prompt/(SS_prompt + SS_error), not SS_prompt/SS_total, and it does not represent the proportion of total variance explained by the prompt factor. The three partial values in Table 25 (0.440, 0.357, 0.507) can overlap and do not partition a common total. The abstract, introduction, results, and conclusion all restate this as "explains 44% of variance," which is unsupported by the reported statistic. Please report a proper variance decomposition (e.g., eta-squared or omega-squared) or explicitly reinterpret the claims in terms of partial eta-squared, and adjust the headline wording accordingly.
  2. [Representative Prompting Strategies; Prompt Categories as Structural Abstractions] The ANOVA's "Prompt" factor is instantiated by exactly one hand-written template per category: CoT for NLR, Self-Critique for ISRR, ReACT for TAR, and QuaSAR for SSR. Prompt category is therefore confounded with template wording, output length, formatting constraints, and instruction specificity. The claim that "prompt structure is a primary driver" is about abstract structural categories, but the design can only support a claim about these specific templates. The manuscript should either add multiple instantiations per category or explicitly reframe all variance and ranking claims as template-level effects.
  3. [The Impact of LoRA Fine-Tuning; Table 16] The claim that "LoRA supplies consistent gains of +8 to 12 F1 for every prompt and model type except zero-shot" is contradicted by Table 16. For example, DeepSeek-R1-Distill-Qwen-1.5B with SSR goes from F1 0.479 to 0.503 (gain +0.024), Phi-4-mini-reasoning-3.8B with TAR goes from 0.633 to 0.635 (gain +0.002), and Phi-4-mini-reasoning with SSR goes from 0.560 to 0.595 (gain +0.035). The companion claim that LoRA "raises answer validity above 97% across all models" is also contradicted: Table 16 shows Phi-4 SSR LoRA validity 0.907 and DeepSeek SSR LoRA validity 0.920. Please report per-cell gains and validity rates accurately, or restrict the claim to overall averages with the exceptions stated.
  4. [Empirical Evaluation; Tables 25 and 22] The ANOVA appears to use a single macro-F1 value per model-prompt-LoRA cell, with no repeated runs, bootstrap confidence intervals, or per-example variance estimates reported anywhere in the paper. Under this design, the p-values and effect sizes treat each point estimate as error-free, so the "44%" headline lacks any quantification of sampling variability. Please provide repeated-seed results or bootstrap intervals for the variance components, or explicitly state that the analysis is on single deterministic evaluations and temper the statistical claims accordingly.
minor comments (6)
  1. [Introduction] The sentence "By holding model scaling (≥4B)" should read "≤4B", since all fine-tuned models are in the 1.5-3.8B range.
  2. [Table 22 caption] The caption names reasoning types as "Clinical, Common Sense, Existence, Numerical Comparison," which do not match the six categories defined in the paper (Clinical, Lexical Equivalence, Evidence, World-Knowledge, Quantitative Comparison, Quantitative Derivation). Please correct the caption.
  3. [Table 24; Fine-Tuning Details] Table 24 reports approximately 1.5k demonstrations per prompt type, while the Fine-Tuning Details section says each model is fine-tuned with 500 randomly selected demonstrations. Please clarify whether 1.5k is the size of the generated pool and 500 is the training sample, and fix the "ZC" label to "Zero-Shot".
  4. [Figure 2] The y-axis label "Partial Eta-Squared (²)" is missing the η symbol; it should read "Partial η²".
  5. [Results Overview; general notation] The abbreviation "ISSR" appears once where "ISRR" is used elsewhere; please standardize. Also, "LoRA" and "LoRa" are used inconsistently across the text and tables.
  6. [Limitations] The annotation-reliability limitation is acknowledged in the Appendix, but the absence of inter-annotator agreement should also be reflected in the strength of the per-reasoning-type variance claims, since those analyses depend on the single expert's labels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical evaluation whose claims rest on measured outcomes, not on self-referential derivation.

full rationale

This is an empirical benchmarking study, not a derivation. The central claims are supported by controlled experiments with held-out test sets and external datasets (MedNLI and TREC). The variance-decomposition statistic reported is partial eta-squared; the abstract's wording that 'prompt type alone accounts for up to 44% of the variance' is a statistical misdescription, but this is a correctness/interpretation issue, not a circularity: the number is computed from independent experimental conditions, not from fitted constants or self-citations. Self-citations to NLI4CT and QuaSAR are present but not load-bearing: NLI4CT is a public benchmark and QuaSAR is one of four prompt templates whose performance is measured, not assumed. No equation reduces to its own inputs, no fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported from the authors' prior work to force the conclusions. The derivation chain is therefore self-contained.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on three non-trivial assumptions: one template per prompt category, single-expert annotation, and teacher-filtered demonstrations. The LoRA hyperparameters and demonstration count are hand-chosen design constants. No new physical or formal entities are introduced.

free parameters (2)
  • LoRA hyperparameter configuration = rank 8, alpha 16, dropout 0.1, learning rate 2e-5, max steps 500
    A single hand-chosen hyperparameter set is applied uniformly across all model-prompt combinations. The paper notes Phi-4 required different target modules, and the uniform protocol may favor some architectures, so it is a design choice that affects the LoRA comparison.
  • Demonstration sample size per prompt type = 500 randomly selected demonstrations
    The number of demonstrations is fixed by hand to balance convergence and overfitting. It is not tuned per model or prompt, and different sample sizes could change the reported LoRA gains.
assumptions (4)
  • ad hoc to paper A single prompt template per category faithfully represents the abstract prompt category.
    Used to interpret the ANOVA 'Prompt' factor as measuring prompt structure rather than template wording, length, or output format constraints. This enters in the section 'Representative Prompting Strategies'.
  • domain assumption Expert reasoning-type labels for the NLI4CT test set are accurate despite being produced by one annotator.
    All per-reasoning-type analyses depend on this annotation. The Limitations section concedes that no inter-annotator agreement measure was computed.
  • domain assumption Demonstrations generated by GPT-4o-mini and filtered by agreement with gold labels are high-quality, unbiased supervision.
    All LoRA training uses these demonstrations. The Limitations section concedes that teacher model biases may be imported into the student models.
  • standard math Type II ANOVA on macro-F1 with one observation per condition provides valid inference for the variance decomposition.
    The variance decomposition in Tables 22 and 25 assumes unit-level observations with independent errors, but no repeated runs or per-cell variance are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dissecting Clinical Reasoning in Language Models: A Comparative Study of Prompts and Model Adaptation Strategies." pith.science (2026). https://pith.science/paper/QHEYKFCA

@misc{pith2026250704142,
  author       = {Pith},
  title        = {Pith review of: Dissecting Clinical Reasoning in Language Models: A Comparative Study of Prompts and Model Adaptation Strategies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QHEYKFCA}},
  note         = {Machine review of arXiv:2507.04142}
}
read the original abstract

Recent works on large language models (LLMs) have demonstrated the impact of prompting strategies and fine-tuning techniques on their reasoning capabilities. Yet, their effectiveness on clinical natural language inference (NLI) remains underexplored. This study presents the first controlled evaluation of how prompt structure and efficient fine-tuning jointly shape model performance in clinical NLI. We inspect four classes of prompting strategies to elicit reasoning in LLMs at different levels of abstraction, and evaluate their impact on a range of clinically motivated reasoning types. For each prompting strategy, we construct high-quality demonstrations using a frontier model to distil multi-step reasoning capabilities into smaller models (4B parameters) via Low-Rank Adaptation (LoRA). Across different language models fine-tuned on the NLI4CT benchmark, we found that prompt type alone accounts for up to 44% of the variance in macro-F1. Moreover, LoRA fine-tuning yields consistent gains of +8 to 12 F1, raises output alignment above 97%, and narrows the performance gap to GPT-4o-mini to within 7.1%. Additional experiments on reasoning generalisation reveal that LoRA improves performance in 75% of the models on MedNLI and TREC Clinical Trials Track. Overall, these findings demonstrate that (i) prompt structure is a primary driver of clinical reasoning performance, (ii) compact models equipped with strong prompts and LoRA can rival frontier-scale systems, and (iii) reasoning-type-aware evaluation is essential to uncover prompt-induced trade-offs. Our results highlight the promise of combining prompt design and lightweight adaptation for more efficient and trustworthy clinical NLP systems, providing insights on the strengths and limitations of widely adopted prompting and parameter-efficient techniques in highly specialised domains.

Figures

Figures reproduced from arXiv: 2507.04142 by the authors.

Figure 1
Figure 1. Example reasoning trajectories for a single NLI4CT instance under four prompting strategies: NLR, ISRR, TAR, and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Partial η 2 values showing the proportion of variance in F1 performance explained by Prompt, LoRA adaptation, and Model architecture, both globally and across reasoning types. 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 F1 Score Zero-Shot NLR ISRR TAR SSR Version Base LoRa gpt-4o-mini llama 3.2 3B Qwen2.5-3B-Instruct DeepSeek-R1-Distill-Qwen-1.5B Phi-4-mini-reasoning-3.8B [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Macro F1 Scores on the NLI4CT Test Set a passage drawn from one of four sections: Eligibility, In￾tervention, Results, or Adverse Events, predict whether the premise entails or contradicts the statement by assigning one of two labels: Entailment or Contradiction. Each NLI4CT test instance was expert-annotated with one or more of six reasoning categories, indicating the types of reasoning needed to solve it. Full det… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Performance by Reasoning Type on the NLI4CT Test Set (Macro F [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Macro F1 Scores on the TREC Test Set formatted label from the task-defined set (i.e., entailment or contradiction). LoRA-tuned models generate well-formed outputs, exceeding 97% validity across all configurations ( [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Impact of LoRA Tuning on F1 Score by Prompt Strategy and Reasoning Type Method Class LoRa Mean F1 Base Mean F1 Lift NLR Clinical 0.656 0.594 0.062 NLR World-Knowledge Inference 0.660 0.589 0.071 NLR Evidence 0.706 0.562 0.144 NLR Quantitative Comp 0.714 0.629 0.085 NLR…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 11 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    A.; Adeli, E.; Altman, R.; Arora, S.; von Arx, S.; Bernstein, M

    Bommasani, R.; Hudson, D. A.; Adeli, E.; Altman, R.; Arora, S.; von Arx, S.; Bernstein, M. S.; Bohg, J.; Bosselut, A.; Brunskill, E.; et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258

    arXiv 2021

  5. [5]

    D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877--1901

    2020

  6. [6]

    DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948

    arXiv 2025

  7. [7]

    Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2): 3

    2022

Show all 33 references
  1. [9]

    E.; Pollard, T

    Johnson, A. E.; Pollard, T. J.; Shen, L.; Lehman, L.-w. H.; Feng, M.; Ghassemi, M.; Moody, B.; Szolovits, P.; Anthony Celi, L.; and Mark, R. G. 2016. MIMIC-III, a freely accessible critical care database. Scientific data, 3(1): 1--9

    2016

  2. [10]

    Jullien, M.; Valentino, M.; and Freitas, A. 2024. SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), 1947--1962

    2024

  3. [11]

    Jullien, M.; Valentino, M.; Frost, H.; O’Regan, P.; Landers, D.; and Freitas, A. 2023 a . NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 16745--16764

    2023

  4. [12]

    Jullien, M.; Valentino, M.; Frost, H.; O’Regan, P.; Landers, D.; and Freitas, A. 2023 b . SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), 2216--2226

    2023

  5. [13]

    M.; and Schockaert, S

    Khalid, I.; Nourollah, A. M.; and Schockaert, S. 2025. Benchmarking Systematic Relational Reasoning with Large Language and Reasoning Models. arXiv preprint arXiv:2503.23487

    2025 arXiv

  6. [14]

    Ma, Y.; Fan, C.; and Jiang, H. 2023. Sci-cot: Leveraging large language models for enhanced knowledge distillation in small models for scientific qa. In 2023 9th International Conference on Computer and Communications (ICCC), 2394--2398. IEEE

    2023

  7. [15]

    Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36: 46534--46594

    2023

  8. [16]

    Min, S.; Lyu, X.; Holtzman, A.; Artetxe, M.; Lewis, M.; Hajishirzi, H.; and Zettlemoyer, L. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Proceedings of the 2022 Conference on Empirical Met...

    2022

  9. [17]

    Mondorf, P.; and Plank, B. 2024. Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models--A Survey. arXiv preprint arXiv:2404.01869

    2024 arXiv

  10. [18]

    T.; Zhang, S.; Carignan, D.; Edgar, R.; Fusi, N.; King, N.; Larson, J.; Li, Y.; Liu, W.; et al

    Nori, H.; Lee, Y. T.; Zhang, S.; Carignan, D.; Edgar, R.; Fusi, N.; King, N.; Larson, J.; Li, Y.; Liu, W.; et al. 2023. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452

    2023 arXiv

  11. [19]

    J.; Gur-Ari, G.; Michalewski, H.; Austin, J.; Bieber, D.; Dohan, D.; Lewkowycz, A.; Bosma, M.; Luan, D.; et al

    Nye, M.; Andreassen, A. J.; Gur-Ari, G.; Michalewski, H.; Austin, J.; Bieber, D.; Dohan, D.; Lewkowycz, A.; Bosma, M.; Luan, D.; et al. 2021. Show your work: Scratchpads for intermediate computation with language models

    2021

  12. [20]

    OpenAI. 2024. GPT‑4o mini: advancing cost‑efficient intelligence. Model card. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence

    2024

  13. [21]

    A.; and Lewis, M

    Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N. A.; and Lewis, M. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350

    2022 arXiv

  14. [22]

    Ranaldi, L.; Valentino, M.; Polonsky, A.; and Freitas, A. 2025. Improving chain-of-thought reasoning via quasi-symbolic abstractions. arXiv preprint arXiv:2502.12616

    2025

  15. [23]

    M.; Bedrick, S.; and Hersh, W

    Roberts, K.; Demner-Fushman, D.; Voorhees, E. M.; Bedrick, S.; and Hersh, W. R. 2022. Overview of the TREC 2022 Clinical Trials Track. In TREC

    2022

  16. [24]

    Romanov, A.; and Shivade, C. 2018. Lessons from Natural Language Inference in the Clinical Domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 1586--1596

    2018

  17. [25]

    D.; Jiang, D.; Wadhwa, M.; Singhal, P.; Zhao, X.; Ye, X.; Mahowald, K.; and Durrett, G

    Sprague, Z.; Yin, F.; Rodriguez, J. D.; Jiang, D.; Wadhwa, M.; Singhal, P.; Zhao, X.; Ye, X.; Mahowald, K.; and Durrett, G. 2024. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409.12183

    2024 arXiv

  18. [26]

    Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  19. [27]

    V.; Zhou, D.; et al

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824--24837

    2022

  20. [28]

    Wen, P.; Ji, J.; Chan, C.-M.; Dai, J.; Hong, D.; Yang, Y.; Han, S.; and Guo, Y. 2025. Thinkpatterns-21k: A systematic study on the impact of thinking patterns in llms. arXiv preprint arXiv:2503.12918

    2025

  21. [29]

    J.; Li, Y.; Ren, L.; Shen, Y.; et al

    Xu, H.; Peng, B.; Awadalla, H.; Chen, D.; Chen, Y.-C.; Gao, M.; Kim, Y. J.; Li, Y.; Ren, L.; Shen, Y.; et al. 2025. Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math. arXiv preprint arXiv:2504.21233

    2025 arXiv

  22. [30]

    Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; Dong, G.; Wei, H.; Lin, H.; Tang, J.; Wang, J.; Yang, J.; Tu, J.; Zhang, J.; Ma, J.; Xu, J.; Zhou, J.; Bai, J.; He, J.; Lin, J.; Dang, K.; Lu, K.; Chen, K.; Yang, K.; Li, M.; Xue, M.;...

    2024 arXiv

  23. [31]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

    2023

  24. [32]

    Yu, T.; Jing, Y.; Zhang, X.; Jiang, W.; Wu, W.; Wang, Y.; Hu, W.; Du, B.; and Tao, D. 2025. Benchmarking reasoning robustness in large language models. arXiv preprint arXiv:2503.04550

    2025 arXiv

  25. [33]

    Zhang, Z.; Zhang, A.; Li, M.; and Smola, A. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493

    2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.