Pith. sign in

REVIEW 4 major objections 6 minor 41 references

Lightweight Language Models are Prone to Reasoning Errors for Complex Computational Phenotyping Tasks

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Lightweight LLMs exhibit ubiquitous reasoning errors—logical inconsistencies and unfaithful explanations—when phenotyping complex cases, and biased few-shot examples can cut accuracy by 5 to 10 percent.

desk verdict Useful empirical demonstration of logical errors in lightweight LLM phenotyping, but the 'unfaithfulness' label outruns what the annotations can support; still worth a serious referee. read the letter →

arxiv 2507.23146 v4 pith:CMGKLRJI submitted 2025-07-30 q-bio.QM

Sarah Pungitore , Shashank Yadav , David Maughan , Vignesh Subbian This is my paper
classification q-bio.QM
keywords LargeLanguageModelsChain-of-ThoughtReasoningErrorsExplanationCorrectnessUnfaithfulnessComputationalPhenotypingElectronicLightweightLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Computational phenotyping—turning electronic health records into patient cohorts—is slow because it requires manual review, so lightweight language models are an attractive shortcut. This paper tries to establish that those shortcuts come with hidden reasoning costs. Across three local models (Mistral Small, Phi-4, and Qwen-distilled DeepSeek-R1) used to classify respiratory-support therapy encounters, reviewers found two kinds of errors in the models' step-by-step explanations: explanation-correctness errors (logically inconsistent arguments) and unfaithfulness errors (reasoning text that does not represent how the answer was reached). The errors appeared in every model even when the prompt was untouched, and adding few-shot examples all pointing to the same wrong phenotype lowered accuracy by 5 to 10 percent. The paper also contributes a reusable evaluation component—extending the authors' PHEONA framework—so future phenotyping studies can audit reasoning instead of only reading final labels.

What carries the argument

The load-bearing mechanism is the expanded reasoning-evaluation component of the paper's PHEONA framework, using two criteria. Explanation Correctness Errors flag statements of the form premise-to-conclusion that contradict the premise, known facts, or the prompt's rules. Unfaithfulness flags cases where the chain-of-thought text does not reflect the true reasoning, operationalized as Restoration Errors (a wrong inference is corrected later without acknowledging the change), Unfaithful Shortcut Errors (an illogical shortcut is used without admitting it), Few-Shot Biasing (all examples share one answer and pull the model), and Hint Biasing (a hint toward the ground truth influences the answer without the model mentioning it). Three chain-of-thought prompt levels (No CoT, Some CoT, Full CoT) are the controlled conditions across which error presence and accuracy are compared.

What would settle it

Re-running the 80-description phenotyping experiment with an independent, pre-registered annotation rubric and at least three annotators would falsify the ubiquity claim if the measured prevalence of explanation-correctness and unfaithfulness errors fell to zero across all model and chain-of-thought conditions, or if biased few-shot examples produced no accuracy loss.

Watch

Extended reading notes

Core claim

On the paper's terms, the central discovery is that reasoning errors are not occasional failures in lightweight LLMs but are pervasive in chain-of-thought responses to a complex phenotyping task. For all three models, manual review found responses containing logically inconsistent premise-to-conclusion statements (explanation correctness errors) or unacknowledged restoration and shortcut behavior (unfaithfulness), even with unbiased prompts; the error profiles differed by model, with DeepSeek showing the most restoration errors and Mistral the fewest explanation-correctness errors. When prompts were modified, biased few-shot examples reduced accuracy by 5–10 percent depending on model and chain-of-thought level, and hints in the prompt generally produced the lowest accuracies. The paper concludes that reasoning errors are ubiquitous and that auditing explanations—not just final answers—is necessary when lightweight LLMs are used for complex phenotyping.

Load-bearing premise

The claim stands on the assumption that manual reviewers can reliably detect logical inconsistencies and unfaithfulness in chain-of-thought text; the paper's own low Cohen's kappa leaves that reliability unproven.

Editorial extensions

If this is right

  • Chain-of-thought explanations from lightweight LLMs cannot be taken at face value in phenotyping workflows; unaudited reasoning can hide errors even when the final label is right.
  • Prompt design is not neutral: selecting few-shot examples that all point to one phenotype can lower accuracy by 5 percent or more, up to 10 percent, across models and chain-of-thought types.
  • Reasoning quality and task accuracy can diverge: DeepSeek showed more reasoning errors but the smallest accuracy swings under biased prompts, so accuracy alone does not measure reasoning trustworthiness.
  • The extended evaluation framework gives phenotyping studies a concrete way to audit explanations by checking for logically inconsistent premise-to-conclusion statements and for shortcuts that are not acknowledged.
  • Full chain-of-thought prompting improved unbiased accuracy for two models but did not eliminate reasoning errors, so more reasoning steps alone is not a fix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the annotation agreement was low, a rule-based or automated check—for example, verifying that a stated premise formally entails the conclusion—would give a harder measure of explanation-correctness errors than manual review.
  • The 5–10 percent accuracy drop from few-shot biasing implies that any LLM phenotyping pipeline using examples should randomize or counterbalance example labels, otherwise reported accuracy may reflect prompt content rather than model skill.
  • A direct extension would test whether larger or closed-source reasoning models show the same error rates; the paper only covers lightweight local models.
  • If the error patterns generalize, omitting or independently verifying chain-of-thought explanations in clinical decision support may be safer than trusting them until interpretability methods can validate them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper extends the PHEONA evaluation framework with a new Reasoning component aimed at detecting two types of faulty reasoning in chain-of-thought responses from lightweight LLMs: explanation correctness errors and unfaithfulness errors. The authors apply this framework to a computational phenotyping task for acute respiratory failure using three open-weight models (Mistral Small 24B, Phi-4 14B, and Qwen-distilled DeepSeek-R1 32B). For unmodified prompts, they report that both error types are present in responses from all models, based on manual review by two annotators. For prompt-modification experiments, they report accuracy changes under few-shot biasing and hint biasing, with Mistral showing the largest impacts. The authors conclude that reasoning errors are ubiquitous across lightweight LLMs for complex phenotyping and that the PHEONA-Reasoning component provides a practical evaluation tool.

Significance. The paper addresses a practically important problem—how to evaluate and troubleshoot LLM reasoning in clinical informatics tasks. The proposed framework extension is a useful organizational contribution, and the study is one of few to examine reasoning errors in lightweight models on a complex phenotyping task. Strengths include the use of multiple models, a real clinical dataset, and publicly available code. However, the central empirical claim that unfaithfulness errors are ubiquitous is not supported by the measurement approach, and the low inter-annotator agreement undermines the reliability of the manual annotations. If the measurement validity and reliability issues are resolved, the framework could be a valuable resource for the community, but the current evidence does not support the strong conclusions drawn.

major comments (4)
  1. [Table 1; Methods, Unfaithfulness] The operational definition of Unfaithfulness in Table 1 is 'The model fails to indicate its reasoning process within the response,' which is a claim about the relationship between the chain-of-thought text and the model's hidden internal computation. However, the detection methods for Restoration Errors and Unfaithful Shortcut Errors rely solely on textual features—self-corrections without acknowledgment and illogical justifications. Neither feature can establish unfaithfulness: a self-correction can be a faithful record of flawed reasoning, and a coherent narrative can be unfaithful. The authors' own Discussion states that 'there is no indication which, if any, of the experiments best correlate to the internal reasoning processes of each model.' Consequently, the abstract's claim that 'both errors were present in responses from all models' is not supported for the unfaithfulness error type. The paper should either reframe these as 'textual markers of possible unfaithfulness' or provide a validation argument that these textual patterns causally indicate hidden-process mismatch.
  2. [Figure 3; Methods, Explanation Correctness and Unfaithfulness] Inter-annotator agreement, as measured by Cohen's kappa, is reported as generally low across all metrics, with the exception of Mistral's No CoT responses. The authors state that because of this, they assessed only the presence rather than the frequency of errors. However, presence is also unreliable when kappa is near zero: a single reviewer's classification of an error is not a stable label. The counts in Figure 4, which drive the 'ubiquity' claim, are themselves averages of reviewer judgments that disagree substantially. The paper should report the raw confusion between reviewers, provide an adjudication procedure, or demonstrate that the presence claim is robust to disagreements (e.g., by showing that all models have at least one response labeled as an error by both reviewers). Without this, the central empirical claim rests on unvalidated subjective assessments.
  3. [Results, Figure 6; Methods, Few-Shot Biasing and Hint Biasing] The accuracy-impact results are presented only as averages across three trials, without confidence intervals, standard deviations, or significance tests. Figure 6 shows that Mistral has higher variation, but there is no statistical comparison of the biased versus unbiased conditions. The abstract (as quoted) claims that specific few-shot examples reduced accuracy by 5–10%, yet the full-text Results do not report these numbers or any measure of uncertainty. The 'mean absolute change in accuracy relative to the unbiased prompt' is also not formally defined in the Methods. The authors should provide the exact formula, report per-condition means and standard deviations, and conduct a paired significance test across the 700 constructed descriptions to support any claim of a biasing effect.
  4. [Methods, Restoration Errors] The protocol for Restoration and Unfaithful Shortcut Errors says responses were generated 'until we had 80 correctly phenotyped constructed descriptions across all CoT prompts.' This sampling scheme does not guarantee comparability across models: if one model requires many attempts to reach 80 correct responses, the composition of encounters (and their phenotype mix) may differ systematically from another model that reaches 80 quickly. Consequently, differences in error prevalence among models could reflect differences in the sampled encounters rather than in reasoning behavior. The authors should report the number of attempts and the phenotype distribution of the 80 correct responses for each model and CoT type, or use a fixed set of encounters for all models.
minor comments (6)
  1. [Abstract] The full-text abstract states that 'Mistral had the highest overall accuracy impact' but does not provide the 5–10% range mentioned in the reader's version. Please ensure the abstract's quantitative claims exactly match the reported results, and report the numeric accuracy changes in the Results if they are to be highlighted.
  2. [Figure 4 caption] The caption says 'The number of responses with each error was determined for all 80 constructed descriptions across 6 total trials, 3 per reviewer.' This is confusing: the Methods says positive instances were summed across each model and CoT type. Clarify whether the plotted values are averaged over reviewers, summed over reviewers, or based on a consensus label, and explain how 6 trials arise from 3 trials and 2 reviewers.
  3. [Methods, Few-Shot Biasing and Hint Biasing] The term 'mean absolute change in accuracy relative to the unbiased prompt' is used without a definition. Specify whether this is the average of |accuracy_biased - accuracy_unbiased| across the experimental combinations, and state whether it is averaged across trials and phenotypes.
  4. [Methods, Data Source and Processing] The constructed descriptions used as input for all models were generated by Mistral Small 24B with Q8 quantization. While this is acknowledged, its potential to introduce a systematic bias into all model evaluations should be discussed more explicitly, since the same model is also one of the evaluated models.
  5. [Figure 3] The kappa values are shown only through a color scale; adding numerical labels to each cell or a supplementary table would make the agreement levels easier to interpret and would support the authors' claims about low agreement.
  6. [References] Several references are dated 2025 or 2026 (e.g., refs. 5, 18) and may be in press or preprint. Please ensure all cited works are publicly accessible and clearly marked as 'in press' or 'preprint' where appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claims are measured against external gold standards and are not forced by the framework or by self-citation.

full rationale

The paper's central claims are empirical and benchmarked against external data. The presence of explanation-correctness and unfaithfulness errors across lightweight LLMs, and the 5-10% accuracy drops under biased few-shot or hint prompts, are measured from model outputs and scored against the external phenotype ground truths of Essay et al. (ref 19), not derived from any fitted parameter or from the PHEONA/SHREC framework itself. The PHEONA extension (ref 18) and SHREC-generated constructed descriptions (ref 5) are inputs and evaluation scaffolding, not constraints that force the outcome. The unfaithfulness construct is admittedly hard to validate: the authors report low Cohen's kappa and state that 'there is no indication which, if any, of the experiments best correlate to the internal reasoning processes of each model.' That is a genuine construct-validity limitation for the unfaithfulness label, but it is not circularity because the label does not reduce by construction to the quantity being predicted, and the accuracy-difference experiments rest on external ground truths. Self-citations are present (refs 5 and 18) but are not load-bearing; no uniqueness theorem or ansatz is imported to forbid alternatives, and no fitted parameter is renamed as a prediction. Accordingly, no circular step is identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical, so the ledger is dominated by measurement and data-generation assumptions rather than fitted constants. No parameter is fitted to make the result appear; the hand-set experimental choices, such as sample sizes, trial counts, quantization, and sampling hyperparameters, affect precision and generalizability but do not encode the outcome.

free parameters (5)
  • Manual review sample size = 80 constructed descriptions per error type
    Hand-selected; 10 per phenotype for explanation correctness and the first 80 correct responses for restoration and shortcut analyses. Sizes affect the precision of error prevalence estimates.
  • Biasing evaluation set size = 700 responses per model per trial
    100 per phenotype across 7 phenotypes, after excluding IMV to HFNI; hand-selected and excludes the biased phenotype, shaping all accuracy-impact claims.
  • Number of trials = 3
    All experiments used 3 trials; no power analysis is provided, and only standard deviations across trials are shown in one figure.
  • Sampling hyperparameters = temperature 0.50, top-p 0.99
    Fixed for all generations; these choices influence output variability and reproducibility.
  • Quantization level = Q4_K_M for all models
    Chosen to reduce GPU requirements and confounding, but the paper acknowledges it lowered classification accuracy; results may not transfer to full-precision models.
assumptions (4)
  • domain assumption Manual review of chain-of-thought responses can reliably identify explanation correctness and unfaithfulness errors.
    The error metrics are operationalized by two human reviewers; Figure 3 shows generally low inter-annotator agreement, and the Discussion admits this limitation.
  • domain assumption The previously developed ARF computable phenotypes provide valid ground-truth labels.
    All error and accuracy judgments are compared against the Essay et al. phenotyping algorithm; errors in that algorithm propagate to every result.
  • domain assumption Constructed descriptions generated by Mistral Q8 in prior work are sufficient, unbiased inputs for all three models.
    The paper standardizes inputs using SHREC outputs from a single model; any classification or ordering errors in those descriptions become shared input errors. Location: Model Selection section.
  • domain assumption The text of a chain-of-thought response is a meaningful trace of the model's reasoning for audit purposes.
    Detecting unfaithfulness and restoration errors requires treating chain-of-thought text as analyzable evidence; the unfaithfulness literature the paper cites questions this premise, and the paper itself says no experiment is known to correlate with internal processes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lightweight Language Models are Prone to Reasoning Errors for Complex Computational Phenotyping Tasks." pith.science (2026). https://pith.science/paper/CMGKLRJI

@misc{pith2026250723146,
  author       = {Pith},
  title        = {Pith review of: Lightweight Language Models are Prone to Reasoning Errors for Complex Computational Phenotyping Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMGKLRJI}},
  note         = {Machine review of arXiv:2507.23146}
}
read the original abstract

Although computational phenotyping is a central informatics activity with resulting cohorts supporting a wide variety of applications, it is time-intensive because of manual data review. We previously assessed the ability of LLMs to perform computational phenotyping tasks using computable phenotypes for ARF respiratory support therapies. They successfully performed concept classification and classification of single-therapy phenotypes but underperformed on multi-therapy phenotypes. To better understand issues with these complex tasks, we expanded PHEONA, a generalizable framework for evaluation of LLMs, to include methods specifically for evaluating faulty reasoning. We assessed the responses of two lightweight non-reasoning LLMs (Mistral Small 24 billion and Phi-4 14 billion) and one lightweight reasoning LLM (Qwen-distilled DeepSeek-r1 32 billion) both with and without prompt modifications to identify explanation correctness errors and unfaithfulness errors during phenotyping. For experiments without prompt modifications, both errors were present in responses from all models. For experiments with prompt modifications, we measured the mean absolute change in accuracy relative to the unbiased prompt across biasing conditions. Adding specific few-shot examples aligned with an incorrect phenotype reduced accuracy by at least 5% and up to 10% depending on the model and CoT type. Since reasoning errors were ubiquitous across models, our enhancement of PHEONA to include a component for assessing faulty reasoning provides a practical framework for evaluating LLM reasoning and empirical evidence that reasoning errors occur during complex computational phenotyping.

Figures

Figures reproduced from arXiv: 2507.23146 by the authors.

Figure 1
Figure 1. Overview of the components and related criteria of PHEONA (Evaluation of PHEnotyping for Observational Health Data), with an updated component, Reasoning, for evaluating faulty reasoning in model responses [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗
Figure 2
Figure 2. Example constructed description for a single encounter generated by ordering and concatenating previously classified concepts for acute respiratory failure respiratory support therapies and relevant medications.(5) In this example, we observe that the individual was intubated at Record 2 and remained on a ventilator for the remainder of the encounter [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 32 canonical work pages

  1. [1]

    Advances in Electronic Phenotyping: From Rule-Based Definitions to Machine Learning Models

    Banda JM, Seneviratne M, Hernandez-Boussard T , Shah NH. Advances in Electronic Phenotyping: From Rule-Based Definitions to Machine Learning Models. Annu Rev Biomed Data Sci. 2018 Jul 20;1(Volume 1, 2018):53–68. doi:10.1146/annurev-biodatasci-080917- 013315

    work page doi:10.1146/annurev-biodatasci-080917- 2018

  2. [2]

    Towards automated phenotype definition extraction using large language models

    Tekumalla R, Banda JM. Towards automated phenotype definition extraction using large language models. Genomics Inform. 2024 Oct 31;22(1):21. doi:10.1186/s44342-024-00023-2 12

    work page doi:10.1186/s44342-024-00023-2 2024

  3. [3]

    Making work visible for electronic phenotype implementation: Lessons learned from the eMERGE network

    Shang N, Liu C, Rasmussen LV, Ta CN, Caroll RJ, Benoit B, et al. Making work visible for electronic phenotype implementation: Lessons learned from the eMERGE network. J Biomed Inform. 2019 Nov 1;99:103293. doi:10.1016/j.jbi.2019.103293

    work page arXiv 2019

  4. [4]

    A general framework for developing computable clinical phenotype algorithms

    Carrell DS, Floyd JS, Gruber S, Hazlehurst BL, Heagerty PJ, Nelson JC, et al. A general framework for developing computable clinical phenotype algorithms. J Am Med Inform Assoc. 2024 Aug 1;31(8):1785–96. doi:10.1093/jamia/ocae121

    work page doi:10.1093/jamia/ocae121 2024

  5. [5]

    SHREC: A framework for advancing next-generation computational phenotyping with large language models

    Pungitore S, Yadav S, Douglas M, Mosier J, Subbian V . SHREC: A framework for advancing next-generation computational phenotyping with large language models. PLOS Digit Health. 2026 Feb 13;5(2):e0001217. doi:10.1371/journal.pdig.0001217

    work page doi:10.1371/journal.pdig.0001217 2026

  6. [6]

    Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond

    Xu F, Lin Q, Han J, Zhao T , Liu J, Cambria E. Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond. IEEE Trans Knowl Data Eng. 2025 Apr;37(4):1620–34. doi:10.1109/TKDE.2025.3536008

    work page arXiv 2025

  7. [7]

    Deductive Verification of Chain-of- Thought Reasoning

    Ling Z, Fang Y , Li X, Huang Z, Lee M, Memisevic R, et al. Deductive Verification of Chain-of- Thought Reasoning. Adv Neural Inf Process Syst. 2023 Dec 15;36:36407–33

    work page 2023

  8. [8]

    Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners [Internet]

    Tang X, Zheng Z, Li J, Meng F, Zhu SC, Liang Y , et al. Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners [Internet]. arXiv; 2023 [cited 2025 Jul 13]. Available from: http://arxiv.org/abs/2305.14825 doi:10.48550/arXiv.2305.14825

    arXiv doi:10.48550/arxiv.2305.14825 2023

Show all 41 references
  1. [9]

    Large language models can be easily distracted by irrelevant context

    Shi F, Chen X, Misra K, Scales N, Dohan D, Chi E, et al. Large language models can be easily distracted by irrelevant context. In: Proceedings of the 40th International Conference on Machine Learning. Honolulu, Hawaii, USA: JMLR.org; 2023. p. 31210–27. (ICML’23)

    2023

  2. [10]

    Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples

    Saparov A, Pang RY , Padmakumar V, Joshi N, Kazemi M, Kim N, et al. Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples. Adv Neural Inf Process Syst. 2023 Dec 15;36:3083–105

    2023

  3. [11]

    Language Models Are Greedy Reasoners

    Saparov A, He H. Language Models Are Greedy Reasoners. In. 11th International Conference on Learning Representations; 2023 [cited 2025 Jul 1]. Available from: http://www.scopus.com/inward/record.url?scp=85199887669&partnerID=8YFLogxK

    2023

  4. [12]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Adv Neural Inf Process Syst. 2022 Dec 6;35:24824–37

    2022

  5. [13]

    Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

    Turpin M, Michael J, Perez E, Bowman SR. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. In. 2023 [cited 2025 Jun 5]. Available from: https://openreview.net/forum?id=bzs4uPLXvi

    2023

  6. [14]

    Chain-of- Thought Reasoning in the Wild is not Always Faithful

    Arcuschin I, Janiak J, Krzyzanowski R, Rajamanoharan S, Nanda N, Conmy A. Chain-of- Thought Reasoning in the Wild is not Always Faithful. In. 2025 [cited 2025 Jun 5]. Available from: https://openreview.net/forum?id=L8094Whth0 13

    2025

  7. [15]

    Reasoning Models Don’t Always Say What They Think [Internet]

    Chen Y , Benton J, Radhakrishnan A, Uesato J, Denison C, Schulman J, et al. Reasoning Models Don’t Always Say What They Think [Internet]. arXiv; 2025 [cited 2025 May 30]. Available from: http://arxiv.org/abs/2505.05410 doi:10.48550/arXiv.2505.05410

    2025 doi arXiv

  8. [16]

    [cited 2025 Jun 10]

    Apple Machine Learning Research [Internet]. [cited 2025 Jun 10]. The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity. Available from: https://machinelearning.apple.com/research/illusion-of-thinking

    2025

  9. [17]

    Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought [Internet]

    Chua J, Rees E, Batra H, Bowman SR, Michael J, Perez E, et al. Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought [Internet]. arXiv; 2025 [cited 2025 Jun 29]. Available from: http://arxiv.org/abs/2403.05518 doi:10.48550/arXiv.2403.05518

    2025 doi arXiv

  10. [18]

    PHEONA: An Evaluation Framework for Large Language Model-based Approaches to Computational Phenotyping

    Pungitore SA, Yadav S, Subbian V . PHEONA: An Evaluation Framework for Large Language Model-based Approaches to Computational Phenotyping. AMIA Annu Symp Proc. 2026 Feb 14;2025:1041–50. PubMed PMID: 41726409; PubMed Central PMCID: PMC12919548

    2026

  11. [19]

    Rule-Based Cohort Definitions for Acute Respiratory Failure: Electronic Phenotyping Algorithm

    Essay P , Mosier J, Subbian V . Rule-Based Cohort Definitions for Acute Respiratory Failure: Electronic Phenotyping Algorithm. JMIR Med Inf. 20200415th ed. 2020 Apr 15;8(4):e18402. Located at: PubMed-not-MEDLINE; 32293579. doi:10.2196/18402

    2020 doi

  12. [20]

    The eICU Collaborative Research Database, a freely available multi-center database for critical care research

    Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci Data. 2018 Sep 11;5(1):180178. doi:10.1038/sdata.2018.178

    2018 doi

  13. [21]

    Ollama; 2024 [cited 2024 Dec 18]

    ollama/ollama [Go] [Internet]. Ollama; 2024 [cited 2024 Dec 18]. Available from: https://github.com/ollama/ollama

    2024

  14. [22]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [Internet]

    DeepSeek-AI, Guo D, Yang D, Zhang H, Song J, Zhang R, et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [Internet]. arXiv; 2025 [cited 2025 Jun 8]. Available from: http://arxiv.org/abs/2501.12948 doi:10.48550/arXiv.2501.12948

    2025 doi arXiv

  15. [23]

    Interrater reliability: the kappa statistic

    McHugh ML. Interrater reliability: the kappa statistic. Biochem Medica. 2012 Oct 15;22(3):276–82. PubMed PMID: 23092060; PubMed Central PMCID: PMC3900052

    2012

  16. [24]

    Emergent Abilities of Large Language Models

    Wei J, Tay Y , Bommasani R, Raffel C, Zoph B, Borgeaud S, et al. Emergent Abilities of Large Language Models. Trans Mach Learn Res [Internet]. 2022 Jun 26 [cited 2025 Jul 1]. Available from: https://openreview.net/forum?id=yzkSU5zdwD

    2022

  17. [25]

    ReAct: Synergizing Reasoning and Acting in Language Models [Internet]

    Yao S, Zhao J, Yu D, Du N, Shafran I, Narasimhan K, et al. ReAct: Synergizing Reasoning and Acting in Language Models [Internet]. arXiv; 2023 [cited 2025 Jul 1]. Available from: http://arxiv.org/abs/2210.03629 doi:10.48550/arXiv.2210.03629

    2023 doi arXiv

  18. [26]

    Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

    Srivastava A, Rastogi A, Rao A, Shoeb AAM, Abid A, Fisch A, et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Trans Mach Learn Res [Internet]. 2023 Jan 19 [cited 2025 Jul 1]. Available from: https://openreview.net/forum?id=uyT...

    2023

  19. [27]

    Large language models are zero-shot reasoners

    Kojima T , Gu SS, Reid M, Matsuo Y , Iwasawa Y . Large language models are zero-shot reasoners. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. Red Hook, NY , USA: Curran Associates Inc.; 2022. p. 22199–213. (NIPS ’22)

    2022

  20. [28]

    Large Language Models Still Can’t Plan (A Benchmark for LLMs on Planning and Reasoning about Change)

    Valmeekam K, Olmo A, Sreedharan S, Kambhampati S. Large Language Models Still Can’t Plan (A Benchmark for LLMs on Planning and Reasoning about Change). In. 2022 [cited 2025 Jul 1]. Available from: https://openreview.net/forum?id=wUU-7XTL5XO

    2022

  21. [29]

    GPT-4 Doesn’t Know It’s Wrong: An Analysis of Iterative Prompting for Reasoning Problems [Internet]

    Stechly K, Marquez M, Kambhampati S. GPT-4 Doesn’t Know It’s Wrong: An Analysis of Iterative Prompting for Reasoning Problems [Internet]. arXiv; 2023 [cited 2025 Jul 1]. Available from: http://arxiv.org/abs/2310.12397 doi:10.48550/arXiv.2310.12397

    2023 doi arXiv

  22. [30]

    A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems [Internet]

    Ke Z, Jiao F, Ming Y , Nguyen XP , Xu A, Long DX, et al. A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems [Internet]. arXiv; 2025 [cited 2025 Jun 22]. Available from: http://arxiv.org/abs/2504.09037 doi:10.48550/arXiv.2504.09037

    2025 doi

  23. [31]

    LLM-based agentic systems in medicine and healthcare

    Qiu J, Lam K, Li G, Acharya A, Wong TY , Darzi A, et al. LLM-based agentic systems in medicine and healthcare. Nat Mach Intell. 2024 Dec;6(12):1418–20. doi:10.1038/s42256-024-00944-1

    2024 doi

  24. [32]

    Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models [Internet]

    Zhou Z, Zhu Z, Li X, Galkin M, Feng X, Koyejo S, et al. Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models [Internet]. arXiv; 2025 [cited 2025 Jun 5]. Available from: http://arxiv.org/abs/2503.22165 doi:10.48550/arXiv.2503.22165

    2025 doi

  25. [33]

    Understanding Reasoning in Thinking Language Models via Steering Vectors

    Venhoff C, Arcuschin I, Torr P , Conmy A, Nanda N. Understanding Reasoning in Thinking Language Models via Steering Vectors. In. 2025 [cited 2025 Jun 5]. Available from: https://openreview.net/forum?id=OwhVWNOBcz&referrer=%5Bthe%20profile%20of%20Phili p%20Torr%5D(%2Fprofile%3F...

    2025

  26. [34]

    Transformer Circuits [Internet]

    Lindsey AJ, Gurnee W, Ameisen E, Chen B, Pearce A, Turner NL, et al. Transformer Circuits [Internet]. [cited 2025 May 27]. On the Biology of a Large Language Model. Available from: https://transformer-circuits.pub/2025/attribution-graphs/biology.html#dives-cot

    2025

  27. [35]

    Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought

    Chen Q, Qin L, Wang J, Zhou J, Che W. Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought. In. 2024 [cited 2025 Jul 13]. Available from: https://openreview.net/forum?id=pC44UMwy2v

    2024

  28. [36]

    Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

    Zhang X, Du C, Pang T , Liu Q, Gao W, Lin M. Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs. Adv Neural Inf Process Syst. 2024 Dec 16;37:333–56

    2024

  29. [37]

    A Methodology for Generating and Optimizing Chain-of-Thought Based on Knowledge Graphs

    Luan Q. A Methodology for Generating and Optimizing Chain-of-Thought Based on Knowledge Graphs. In: Intelligent Computing Technology and Automation [Internet]. IOS Press; 2024 [cited 2025 Jun 30]. p. 313–24. Available from: https://ebooks.iospress.nl/doi/10.3233/ATDE231203 doi...

    2024 doi

  30. [38]

    Training language models to follow instructions with human feedback

    Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C, Mishkin P , et al. Training language models to follow instructions with human feedback. In. 2022 [cited 2025 Jul 23]. Available from: https://openreview.net/forum?id=TG8KACxEON

    2022

  31. [39]

    STaR: Bootstrapping Reasoning With Reasoning

    Zelikman E, Wu Y , Mu J, Goodman N. STaR: Bootstrapping Reasoning With Reasoning. In. 2022 [cited 2025 Jul 23]. Available from: https://openreview.net/forum?id=_3ELRdg2sgI

    2022

  32. [40]

    Leap-of-thought: teaching pre-trained models to systematically reason over implicit knowledge

    Talmor A, Tafjord O, Clark P , Goldberg Y , Berant J. Leap-of-thought: teaching pre-trained models to systematically reason over implicit knowledge. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook, NY , USA: Curran Associ...

    2020

  33. [41]

    On contrastive learning for likelihood-free inference

    Durkan C, Murray I, Papamakarios G. On contrastive learning for likelihood-free inference. In: Proceedings of the 37th International Conference on Machine Learning. JMLR.org; 2020. p. 2771–81. (ICML’20). 16 FIGURES Figure 1. Overview of the components and related criteria of P...

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.