Truth, Trust, and Trouble: Medical AI on the Edge

Ebad Shabbir; Gautam Siddharth Kashyap; Jiechao Gao; Mohammad Anas Azeez; Rafiq Ali; Usman Naseem; Zohaib Hasan Siddiqui

REVIEW 5 major objections 5 minor 24 references

Truth, Trust, and Trouble: Medical AI on the Edge

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Among three open-source medical language models, AlpaCare-13B is both the most accurate and the safest on a 1,077-question anatomy True/False benchmark.

desk verdict The headline comparisons are unsupported by the paper's own numbers; the benchmark itself is real but the reported metrics don't reconcile. read the letter →

arxiv 2507.02983 v3 pith:II3HDDAF submitted 2025-07-01 cs.CL cs.AI

Mohammad Anas Azeez , Rafiq Ali , Ebad Shabbir , Zohaib Hasan Siddiqui , Gautam Siddharth Kashyap , Jiechao Gao , Usman Naseem This is my paper

classification cs.CLcs.AI

keywords medicalquestionansweringlargelanguagemodelsopen-sourceLLMshonestyhelpfulnessharmlessnessAlpaCare-13BBioMistral-7B-DAREMistral-7Bfew-shotprompting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish a usable benchmarking method for open-source medical LLMs and to rank three models on truthfulness, usefulness, and safety. It builds a 1,077-question True/False anatomy benchmark from textbook passages and clinical texts, then applies automated honesty, helpfulness, and harmlessness scores alongside physician review. Its main finding is that AlpaCare-13B is the best performer on accuracy (91.7%) and harmlessness (0.92), while BioMistral-7B-DARE shows that domain-specific tuning can deliver safety close to a larger model. It also claims few-shot prompting raises accuracy from 78% to 85% and that all models struggle with complex reasoning. If the benchmark is trusted, this gives deployers a concrete comparison among open-source clinical QA models.

What carries the argument

The central object is a three-axis honesty, helpfulness, and harmlessness scoring protocol, with honesty defined as agreement with the source corpus, helpfulness as embedding-based relevance and completeness relative to the reference answer, and harmlessness as refusal to affirm unsafe advice on a flagged 210-question safety subset. The protocol is applied to a purpose-built benchmark of 1,077 True/False anatomy questions generated from textbook passages via rule-based templates and GPT-4 prompting, filtered through edge-case patterns, and reviewed by three physician annotators. This machinery carries the argument because every model ranking in the paper is derived from it.

What would settle it

Re-score the same 1,077 responses with a held-out clinician panel that is blinded to model identity and to the paper's labels, asking them to mark each response as correct, relevant, and safe. If AlpaCare-13B's accuracy and harmlessness do not remain highest, or if its 12% to 15% edge-case mistake rate is not confirmed, the paper's ranking is not reproducible. A sharper test is to generate a second 1,077-question True/False set from a different medical corpus and check whether AlpaCare-13B stays ahead.

Watch

Extended reading notes

Core claim

The paper's central discovery is that AlpaCare-13B, a medical-instruction-tuned 13B model, outperforms the 7B models on the constructed benchmark in both accuracy (91.7% versus 88.3% for BioMistral-7B-DARE and 82.5% for Mistral-7B) and harmlessness (0.92 versus 0.90 and 0.81), while BioMistral-7B-DARE shows that domain-specific tuning can nearly match the larger model's safety despite having the same parameter count as the weaker generalist. The paper also claims few-shot prompting improves accuracy from 78% to 85%, and that all models decline in helpfulness as question complexity increases, with AlpaCare dropping from 0.91 on direct recall to 0.80 on negation or compositional queries. It concludes that domain specialization and scale both matter for clinical QA, but that edge cases remain a vulnerability for every model.

Load-bearing premise

The load-bearing premise is that the 1,077 True/False anatomy questions and the automated honesty, helpfulness, and harmlessness metrics are a valid measure of real clinical question answering; the paper itself notes that the reasoning tasks are simple and that safety judgments rest on automated scoring with limited human review, so if these proxies misrepresent clinical practice, the reported rankings lose their practical meaning.

Editorial extensions

If this is right

AlpaCare-13B can serve as an open-source clinical QA model with over 91% accuracy and the strongest safety score among the three tested models on this benchmark.
Domain-specific tuning can deliver near-parity safety at smaller scale, so deployers should not assume larger models are automatically safer.
Few-shot prompting offers a parameter-free accuracy gain from 78% to 85% that can be applied to existing deployments without fine-tuning.
Complex queries involving multi-hop inference or negation and compositional logic will need scaffolding or retrieval support because helpfulness drops sharply on them for all models.
Rare edge cases remain unsafe even for the best model, which affirms unsafe or misleading statements in 12% of edge-case prompts, so an independent validation layer is needed before clinical use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if the benchmark is trusted only as a True/False anatomy-recall test, the ranking should not be read as evidence about open-ended consumer health conversations, where reasoning, follow-up, and elicitation matter more than single-label classification.
Editorial observation: the introduction says pairwise analysis showed BioMistral-7B-DARE and Mistral-7B outperforming AlpaCare-13B on honesty and helpfulness, while Tables 1 and 3 show AlpaCare ahead; a reader should determine which comparison is meant before relying on the rank ordering.
Editorial inference: because harmlessness is measured only on a flagged 210-question subset with pattern-based screening, an independent clinician panel scoring the same outputs would be a stronger check; the paper's own human-correlation result gives harmlessness the lowest agreement (Kappa 0.65).
Editorial inference: a natural testable extension is to run the same three-axis protocol on an open-ended consumer-health dataset; if the ranking changes when questions are no longer True/False, the reported advantage of AlpaCare-13B is specific to this benchmark form.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the thing you need to know: the stress-test note is right, and it's worse than a typo. Table 1's honesty scores cannot be weighted averages of Table 3's subset scores under the paper's stated definitions; Table 2's harmlessness scores exceed the maximum of Table 3's harmlessness for the same models. That means the central ranking—AlpaCare best on accuracy and safety—is not supported by the data as reported. The few-shot aggregate in Table 6 (zero-shot 0.78) also doesn't reconcile with Table 1's per-model accuracy (82.5–91.7).

That's a shame, because the paper has real assets. The 1,077-question anatomy T/F benchmark is new, the three-model comparison is sensible, the code is public, and the stratification by template vs. LLM-generated questions and by reasoning complexity is a reasonable way to probe behavior. If the numbers were trustworthy, this would be a useful practical data point for people choosing among small open-source models.

The soft spots beyond the inconsistencies: the benchmark is entirely self-constructed, with the authors themselves as the licensed annotators, and the honesty/helpfulness metrics are defined as consistency with the same corpus used to generate the questions—so the evaluation is somewhat circular. The safety subset and edge-case set are hand-picked, and the thresholds (embedding similarity 0.80–0.85) are asserted without sensitivity analysis. None of that would be fatal if the numbers were consistent, but the inconsistencies already are.

The introduction's claim that Mistral and BioMistral beat AlpaCare on honesty/helpfulness is directly contradicted by Tables 1 and 3. That's not a nuance issue.

Who should read this? Practitioners who want to know if a new small benchmark exists might scan it, but they should not rely on the reported model rankings. If the authors fix the aggregation and rerun everything, it could be a modestly useful benchmark paper. As it stands, the paper doesn't deserve a serious referee; I'd desk-reject with a clear request for corrected results. It is a good example for teaching why internal consistency checks matter, but not for substantive conclusions.

Referee Report

5 major / 5 minor

Summary. The paper introduces a benchmark of 1,077 anatomy-focused True/False questions generated from textbooks and clinical reports via rule-based templates and LLM prompting, with safety screening and manual review. It evaluates three open-source LLMs—Mistral-7B, BioMistral-7B-DARE, and AlpaCare-13B—on honesty, helpfulness, and harmlessness. The central claims are that AlpaCare-13B achieves the highest accuracy (91.7%) and harmlessness (0.92), that BioMistral-7B-DARE offers competitive safety (0.90) despite smaller scale, that few-shot prompting improves accuracy from 78% to 85%, and that helpfulness declines on complex queries.

Significance. If the reported rankings were reproducible, this would be a useful contribution to benchmarking open medical LLMs, and the attempt to decompose model quality into honesty, helpfulness, and harmlessness with automated metrics plus human correlation is commendable. However, the paper's central numerical claims are internally inconsistent in several places, so the practical significance of the findings is currently not established. The authors do provide a clearly described pipeline, safety screening, and an explicit code repository link, which are positive elements.

major comments (5)

[§1 vs Tables 1 and 3] The introduction states that BioMistral-7B-DARE and Mistral-7B consistently outperform AlpaCare-13B in honesty and helpfulness, but Table 1 reports AlpaCare honesty of 0.89 versus BioMistral 0.84 and Mistral 0.78, and Table 3 reports AlpaCare honesty of 0.89 on Qtempl and 0.86 on Qmodel, both higher than Mistral on the corresponding subsets. This direct contradiction between the narrative and the data must be resolved.
[§4.1 and §5.3, Tables 1 and 3] Under the metric definition in §4.1, Table 1's honesty score is an average over all 1,077 questions, while §5.3 evaluates the subsets Qtempl and Qmodel. For BioMistral-7B-DARE, Table 3 reports honesty of 0.91 on Qtempl and 0.88 on Qmodel; any weighted average of these values is at least 0.88, yet Table 1 reports 0.84. AlpaCare-13B's Table 1 value of 0.89 can match its Qtempl value, but then BioMistral would need to be 0.91, not 0.84. Unless the subsets are not partitions of the benchmark or the metric definitions are mis-stated, at least one set of reported numbers is wrong, and the ranking in the abstract and conclusion is not supported.
[§5.2 and §5.3, Tables 2 and 3] The harmlessness scores in Table 2 (AlpaCare 0.92, Mistral 0.81) cannot be reproduced as weighted averages of the subset harmlessness values in Table 3, where the maximum values are 0.91 for AlpaCare and 0.78 for Mistral. The same aggregation problem that affects the honesty scores thus also affects the safety ranking, which is one of the paper's two headline results.
[Abstract and §5.6 vs Tables 1 and 6] The abstract and §5.6 report that few-shot prompting improves accuracy from 78% to 85%, but Table 1 reports zero-shot accuracies of 82.5%, 88.3%, and 91.7% for the three models, whose average is 87.5%. The few-shot comparison must be reported on the same benchmark and under the same prompting protocol, and the discrepancy must be explained; otherwise the few-shot claim is not interpretable.
[§3.1, §4.1] The honesty metric is defined as consistency with the same source corpus T' used to generate the questions, and the manual review was performed by the authors themselves, described as 'three licensed medical annotators (i.e. the authors)'. This makes the benchmark largely self-referential and undermines the claim of independent double-blind expert annotation. Independent medical review or external validation against an established benchmark is needed before the reported rankings can be interpreted as clinically meaningful.

minor comments (5)

[Table 3] The column header 'Harm↓' is inconsistent with the text and other tables, where higher harmlessness is preferred; it should be 'Harm↑' or the direction should be clarified.
[§2 Related Work] The citations 'Kim et al. (Katz et al., 2024)' and 'Manes et al. (Kumar et al., 2024)' appear mismatched, and 'Samo et al.' is cited for Mistral-7B even though the referenced paper is about Python code generation rather than the Mistral model.
[§5.5 and Figure 2(c)] The caption of Figure 2(c) refers to a 'Rolling TRUE rate', while the text describes 'rolling correctness rates'; please clarify which quantity is actually plotted.
[§4.2] The hyperparameter section describes fine-tuning of the evaluated models, but the study otherwise appears to evaluate pretrained checkpoints; please state explicitly whether any fine-tuning was performed in this work and, if not, remove the fine-tuning description.
[Table 5] The edge-case set E contains only 100 prompts, and the reported percentages in Table 5 are given without confidence intervals or significance tests; given the small sample, please add uncertainty quantification or at least note this limitation.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivational circularity; the 'honesty' metric is a definitional restatement of accuracy against the same source corpus.

self definitional [Section 4.1 (Honesty metric), Section 3.1 Eq. (1), Table 1]
"Honesty. This metric quantifies the factual consistency between the model-predicted answer \hat a_i and the source corpus T′. An answer is considered honest if it can be directly supported by evidence present in the corpus. Formally: Score_honesty = 1/L ∑ 1[ \hat a_i is factually supported by T′]. ... Label(q_templ) = TRUE, if q_templ ∈ P_j, FALSE, otherwise (1)"

Ground truth for every generated question is defined by membership in the source corpus (Eq. 1), and honesty is defined as whether the model's answer is supported by that same corpus. In this binary TRUE/FALSE benchmark, a response is 'supported by T′' exactly when it equals the corpus-derived label, so the honesty score is the accuracy score under a different name. The paper nevertheless presents honesty as an independent axis and reports it in its own column in Table 1, so the honesty ranking is not an independent empirical result; it is forced by the accuracy construct by definition.

full rationale

The paper is a benchmarking study with no derivation chain that reduces a prediction to its inputs: the three model checkpoints are fixed public artifacts, no model parameter is fitted to the benchmark, and the accuracy and harmlessness rankings are empirical measurements. I found no self-citation load-bearing step, no imported uniqueness theorem, and no ansatz smuggled in via citation. The one circularity-adjacent issue is the honesty metric: the paper defines honesty as consistency with the source corpus, while the benchmark's ground-truth labels are also defined by membership in that same corpus, so honesty collapses into accuracy by construction. This is a real definitional redundancy, but it is not central to the main ranking claim, which rests on accuracy and harmlessness. The numerical inconsistencies noted by the reviewer (Table 3 subset scores not reconciling with Tables 1 and 2 aggregates, and the abstract's zero-shot 78% conflicting with Table 1's per-model accuracies) are internal-consistency and reproducibility defects, not circularity: they do not show that a result is equivalent to its inputs, only that some reported numbers cannot all be correct. Those defects weaken trust in the specific numbers but do not raise the circularity score under the rubric. Overall circularity is minor, so a score of 2 is appropriate.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The benchmark and all evaluation metrics are constructed by the authors. The free parameters are thresholds and subset sizes chosen without external validation, and the axioms are assumptions about the source material, the NER pipeline, the verification of LLM-generated questions, annotator reliability, and the validity of the True/False format. No new physical or empirical entities are invented.

free parameters (4)

Semantic similarity threshold for helpfulness = 0.80-0.85
Used to determine whether a response is 'helpful' via embedding similarity; the range is reported as a tuned parameter without external calibration or sensitivity analysis.
Number of few-shot examples = 3
Chosen for the few-shot prompting experiment; no justification or sensitivity analysis is provided.
Safety-critical subset Qflagged size = 210
The subset of questions used for harmlessness scoring is hand-selected after dataset construction, with no published criteria for inclusion.
Edge-case set E size = 100
A curated set of prompts for edge-case generalization testing, chosen by the authors; the specific patterns are not disclosed.

assumptions (5)

domain assumption Standard anatomy textbooks and clinical reports provide authoritative ground truth for medical facts.
Section 3.1 uses these sources as the corpus T; if they are incomplete or outdated, the benchmark's truth labels are wrong.
domain assumption The NER-based passage construction correctly clusters sentences with overlapping anatomical entities to form coherent passages.
Section 3.1 builds passages P_j by overlapping entities; errors in NER would misplace facts and affect question generation.
domain assumption Consistency with the source corpus T' verifies the correctness of GPT-4-generated questions.
Equation (2) accepts q_model only if consistent with T'; this assumes the corpus is the sole and sufficient authority for medical truth.
domain assumption The authors' medical annotation is accurate and unbiased.
Section 3.1 states the three annotators are the authors; disagreements are resolved by majority vote, with no evidence of independence from the study's hypotheses.
domain assumption True/False questions are a valid proxy for consumer medical question answering.
The entire benchmark uses this format; the paper does not validate it against real consumer queries or against established medical QA benchmarks such as MedQA.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Truth, Trust, and Trouble: Medical AI on the Edge." pith.science (2026). https://pith.science/paper/II3HDDAF

@misc{pith2026250702983,
  author       = {Pith},
  title        = {Pith review of: Truth, Trust, and Trouble: Medical AI on the Edge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/II3HDDAF}},
  note         = {Machine review of arXiv:2507.02983}
}

read the original abstract

Large Language Models (LLMs) hold significant promise for transforming digital health by enabling automated medical question answering. However, ensuring these models meet critical industry standards for factual accuracy, usefulness, and safety remains a challenge, especially for open-source solutions. We present a rigorous benchmarking framework using a dataset of over 1,000 health questions. We assess model performance across honesty, helpfulness, and harmlessness. Our results highlight trade-offs between factual reliability and safety among evaluated models -- Mistral-7B, BioMistral-7B-DARE, and AlpaCare-13B. AlpaCare-13B achieves the highest accuracy (91.7%) and harmlessness (0.92), while domain-specific tuning in BioMistral-7B-DARE boosts safety (0.90) despite its smaller scale. Few-shot prompting improves accuracy from 78% to 85%, and all models show reduced helpfulness on complex queries, highlighting ongoing challenges in clinical QA.

Figures

Figures reproduced from arXiv: 2507.02983 by the authors.

**Figure 2.** Semantic and temporal analysis of question-answer behavior. (a) t-SNE shows semantic clustering with [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 16 canonical work pages

[1]

William L Benzon. 2025. From llm mechanisms to ring-composition: A conversation with claude 3.5 working paper. Available at SSRN 5125800

work page 2025
[2]

Edward Y Chang. 2023. Examining gpt-4: Capabilities, implications and future directions. In The 10th international conference on computational science and computational intelligence, pages 1--8

work page 2023
[3]

Gaurav Kumar Gupta, Aditi Singh, Sijo Valayakkad Manikandan, and Abul Ehtesham. 2025. Digital diagnostics: The potential of large language models in recognizing symptoms of common illnesses. Ai, 6(1):13

work page 2025
[4]

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 2567--2577

2019
[5]

Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270)

work page 2024
[6]

Theresa S Kr \"a nzle. 2024. Evaluating Creativity with AI: Comparing GPT Models and Human Experts in Idea Evaluation. Ph.D. thesis, Ph. D. thesis, Copenhagen Business School

work page 2024
[7]

Sushant Kumar, Sumit Datta, Vishakha Singh, Deepanwita Datta, Sanjay Kumar Singh, and Ritesh Sharma. 2024. Applications, challenges, and future directions of human-in-the-loop learning. IEEE Access, 12:75735--75760

work page 2024
[8]

Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. Biomistral: A collection of open-source pretrained large language models for medical domains. In Findings of the association for computational linguistics: acl 2024, pages 5848--5864

work page 2024

Show all 24 references

[9]

Binbin Li, Tianxin Meng, Xiaoming Shi, Jie Zhai, and Tong Ruan. 2023. Meddm: Llm-executable clinical guidance tree for clinical decision-making. arXiv preprint arXiv:2312.02441

2023 arXiv
[10]

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3214--3252

2022
[11]

IM Garc \' a L \'o pez, MS Ram \' rez Monoya, and JM Molina Espinosa. 2024. Design and challenges of open large language model frameworks (open llm): A systematic literature mapping. ICERI2024 Proceedings, pages 10320--10328

2024
[12]

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248--260. PMLR

2022
[13]

Hassan Samo, Kashif Ali, Muniba Memon, Faheem Ahmed Abbasi, Muhammad Yaqoob Koondhar, and Kamran Dahri. 2024. Fine-tuning mistral 7b large language model for python query response and code generation: A parameter efficient approach. VAWKUM Transactions on Computer Sciences, 12...

2024
[14]

Vishram Singh. 2013. Selective Anatomy: Prep Manual for Undergraduates. Elsevier India

2013
[15]

Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172--180

2023
[16]

Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. 2024. Towards generalist biomedical ai. Nejm Ai, 1(3):AIoa2300138

2024
[17]

Raju Vaishya. 2024. Dr bhagwan din chaurasia: A guiding light and a pillar of anatomy education in india. Apollo Medicine, 21(4):381--385

2024
[18]

Jui-I Wang, Hen-Hsen Huang, and Hsin-Hsi Chen. 2025. Mesaqa: A dataset for multi-span contextual and evidence-grounded question answering. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10891--10901

2025
[19]

Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching-Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, and Xin Wang. 2025. Llm-medqa: Enhancing medical question answering through case studies in large language models. In 2025 International Joint Conference on Neural Networks (IJCNN), pages 1--8. IEEE

2025
[20]

Liangliang Zhang, Zhuorui Jiang, Hongliang Chi, Haoyang Chen, Mohammed Elkoumy, Fali Wang, Qiong Wu, Zhengyi Zhou, Shirui Pan, Suhang Wang, et al. 2026. Diagnosing and addressing pitfalls in kg-rag datasets: Toward more reliable benchmarking. Advances in Neural Information Pro...

2026
[21]

Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. 2023. Alpacare: Instruction-tuned large language models for medical application. arXiv preprint arXiv:2310.14558

2023 arXiv
[22]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595--46623

2023
[23]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[24]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

William L Benzon. 2025. From llm mechanisms to ring-composition: A conversation with claude 3.5 working paper. Available at SSRN 5125800

work page 2025

[2] [2]

Edward Y Chang. 2023. Examining gpt-4: Capabilities, implications and future directions. In The 10th international conference on computational science and computational intelligence, pages 1--8

work page 2023

[3] [3]

Gaurav Kumar Gupta, Aditi Singh, Sijo Valayakkad Manikandan, and Abul Ehtesham. 2025. Digital diagnostics: The potential of large language models in recognizing symptoms of common illnesses. Ai, 6(1):13

work page 2025

[4] [4]

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 2567--2577

2019

[5] [5]

Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 382(2270)

work page 2024

[6] [6]

Theresa S Kr \"a nzle. 2024. Evaluating Creativity with AI: Comparing GPT Models and Human Experts in Idea Evaluation. Ph.D. thesis, Ph. D. thesis, Copenhagen Business School

work page 2024

[7] [7]

Sushant Kumar, Sumit Datta, Vishakha Singh, Deepanwita Datta, Sanjay Kumar Singh, and Ritesh Sharma. 2024. Applications, challenges, and future directions of human-in-the-loop learning. IEEE Access, 12:75735--75760

work page 2024

[8] [8]

Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. Biomistral: A collection of open-source pretrained large language models for medical domains. In Findings of the association for computational linguistics: acl 2024, pages 5848--5864

work page 2024