Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs agree with expert OSCE raters within one point on most items but rarely match exactly.

desk verdict A useful first full-MIRS benchmark, but the practical accuracy claims depend on baselines the paper never reports. read the letter →

arxiv 2501.13957 v2 pith:DUR4R3GC submitted 2025-01-21 cs.CL cs.AI

Jadon Geathers , Yann Hicke , Colleen Chan , Niroop Rajashekar , Justin Sewell , Susannah Cornes , Rene F. Kizilcec , Dennis Shung This is my paper · ORCID
classification cs.CLcs.AI
keywords OSCElargelanguagemodelsMasterInterviewRatingScaleautomatedassessmentpromptingstrategiesmedicaleducationcommunicationskillsbenchmarking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models can score medical students' interview skills in Objective Structured Clinical Examinations (OSCEs) reliably enough to be useful. Across 174 expert-consensus ratings from 10 recorded cases, the four tested models matched the expert score exactly only 27–44% of the time, but landed within one point 67–87% of the time and agreed on broad proficiency bands 75–88% of the time. The authors argue this makes LLMs viable for flagging students who need support or for formative feedback, but not for exact summative scoring. They also find that no single prompting technique helps across all rubric items, and that item-specific prompt selection is more promising than a one-size-fits-all approach.

What carries the argument

The central object is the Master Interview Rating Scale (MIRS), a validated 28-item, 5-point rubric for assessing medical communication skills, together with three accuracy metrics the authors define: exact accuracy, off-by-one accuracy, and thresholded accuracy (scores 1–2 vs. 3–5). The prompting techniques—zero-shot, chain-of-thought, few-shot, and multi-step—are the interventions being compared. The measurement protocol uses transcripts generated by Whisper from video, manually diarized dialogue, temperature zero for deterministic outputs, and Krippendorff’s alpha for reliability.

What would settle it

Have a second panel of expert raters independently score the same ten transcripts on the same MIRS items and compute inter-rater exact and off-by-one accuracy; if expert-expert exact accuracy is comparable to the models’ 0.27–0.44, then the low exact accuracy reflects rubric ambiguity rather than LLM deficiency.

Watch

Extended reading notes

Core claim

Using the Master Interview Rating Scale (MIRS), a 28-item rubric for medical communication, the paper benchmarks GPT-4o, Claude 3.5, Llama 3.1, and Gemini 1.5 Pro on transcripts of ten OSCE encounters. Averaging over all applicable items, exact accuracy was low (0.27–0.44), while off-by-one accuracy was moderate to high (0.67–0.87) and thresholded accuracy, which distinguishes below-proficiency scores (1–2) from passing scores (3–5), was high (0.75–0.88). The paper reports that zero-temperature settings produce high intra-rater reliability (Krippendorff’s $\\alpha = 0.98$ for GPT-4o), that chain-of-thought, few-shot, and multi-step prompting do not improve average performance over zero-shot, and that selecting the best prompt per MIRS item does improve performance—though this selection is made on the same data used for evaluation. A separate multimodal test on two non-verbal items with Gemini 1.5 Pro produced systematic disagreement with human raters (Krippendorff’s $\\alpha = -0.47$), indicating current models cannot score non-verbal communication from video.

Load-bearing premise

The entire accuracy comparison rests on treating the 174 expert consensus scores as correct ground truth, but the paper does not report how many raters produced each consensus or how well human raters agreed with each other.

Editorial extensions

If this is right

  • If these results hold, medical schools could use LLMs as a first-pass screening tool to flag students who fall below the proficiency threshold, with human raters confirming borderline and low scores.
  • The finding that uniform prompting techniques do not improve on zero-shot suggests that future automated scoring systems should treat each rubric item as a separate prompt-design problem.
  • The poor multimodal performance on non-verbal items means transcript-based LLM scoring cannot yet replace human judgment for aspects like pacing and non-verbal facilitation.
  • Because the three metrics give different pictures of model quality, any deployment should specify which level of scoring leniency is appropriate for the decision being made.
  • The paper’s in-sample selection of optimal prompts per item is an upper bound on what tailored prompting could achieve, so real-world gains from prompt tuning remain to be shown with held-out data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the 0.27–0.44 exact-accuracy range may not be a pure measure of LLM capability: if human raters themselves disagree on exact MIRS scores at a similar rate, then exact accuracy is partly measuring rubric ambiguity.
  • The item-level results rest on very small counts—some MIRS items were not applicable to certain cases, so a single transcript can drive an item’s accuracy estimate; readers should treat the item-level heatmap as suggestive rather than stable.
  • The paper’s thresholded accuracy result suggests a practical extension: LLM scores could be used to triage which student interviews need detailed human review, while students far from the threshold receive automated feedback.
  • Because the benchmarks use one institution’s cases and rubric anchors, cross-institutional validation would be needed before these accuracy numbers can be generalized to other medical schools’ OSCE formats.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper benchmarks four LLMs (GPT-4o, Claude 3.5, Llama 3.1, Gemini 1.5 Pro) on scoring 174 expert-consensus MIRS ratings drawn from 10 OSCE transcripts, using zero-shot, chain-of-thought, few-shot, and multi-step prompting. It reports low exact accuracy (0.27 to 0.44), moderate to high off-by-one accuracy (0.67 to 0.87), and thresholded accuracy (0.75 to 0.88), with no consistent gain from augmentative prompting unless the prompting technique is tailored per MIRS item; the authors acknowledge that per-item selection is an upper bound. A multimodal run on two non-verbal items showed strong disagreement with human raters. The paper's central claim is that LLMs are useful for broad proficiency classification but not for exact scoring.

Significance. If the findings hold up after the missing baselines are supplied, this is a useful first systematic, multi-model benchmark of LLM scoring across a complete OSCE communication rubric. The study's strengths include a transparent deterministic protocol (temperature 0), a reported intra-rater reliability check, public prompts on GitHub, and evaluation against external expert consensus rather than model self-consistency. The main interpretive gaps—no human-human inter-rater baseline, no score-distribution or majority-class baseline, and post hoc per-item prompt selection—are remediable and do not invalidate the raw measurements, but they do currently weaken the practical conclusions drawn from the coarse accuracy metrics.

major comments (4)
  1. [Section 4.1, Section 3.4] The headline claim that LLMs achieve moderate-to-high off-by-one and thresholded accuracy is not interpretable without a score-distribution baseline. With a 5-point scale and a proficiency threshold at 3, a model that always returns the majority bucket (or the modal score) can achieve high thresholded accuracy if the expert consensus scores are concentrated in the 3-5 bucket. The paper should report the marginal distribution of the 174 consensus scores and the accuracy of trivial baselines (always majority bucket, always modal score, uniform random) on the same items. This is load-bearing because the central practical claim in Section 4.1 rests on these coarse metrics.
  2. [Sections 3.1 and 3.2] The manuscript compares model scores against UConn expert consensus scores but reports neither the number of raters contributing to each consensus score nor the consensus procedure, and it provides no human-human inter-rater reliability on these ten transcripts. Without a human reliability estimate, exact accuracy of 0.27-0.44 cannot be judged poor, and coarse accuracy cannot be judged useful: if human exact agreement on the same transcripts is also around 0.4, the LLM may be at parity, whereas if human agreement is much higher, the LLM is clearly deficient. The authors should report the rater pool and consensus rule and, ideally, provide human-human agreement (e.g., Krippendorff's alpha or exact agreement) on these data or a clearly comparable subsample.
  3. [Section 4.1, Abstract, Section 5] The claim that CoT, few-shot, and multi-step prompting "proved valuable when tailored to specific assessment items" rests on selecting the optimal prompting technique per MIRS item on the evaluation set itself. The paper explicitly acknowledges this is a theoretical upper bound, but the abstract and discussion still present tailored prompting as a demonstrated benefit. This conclusion requires out-of-sample validation, such as cross-validation or a held-out set, and the comparison should be against a fixed per-item strategy or a proper nested selection procedure. As reported, the improvement is a selection artifact and should be either removed from the conclusions or clearly labeled as an upper-bound analysis.
  4. [Section 4.2, Figure 4] The item-level heatmap and the statement that most items were scored with off-by-one accuracy of 0.8 or higher are based on very small per-item sample sizes: 174 scores over 26 items and 10 cases, and the paper notes that some items were not present in all transcripts. With n=10 per item, a single misclassification changes accuracy by 10 percentage points, and for items present in fewer cases the instability is even larger. The authors should report per-item counts and confidence intervals, or restrict the item-level claims to items with adequate coverage. The current presentation does not support the sharp conclusion that performance is consistent across items and independent of encounter phases and communication domains.
minor comments (5)
  1. [Abstract vs. Section 4.1] The abstract reports exact accuracy of 0.27 to 0.44, while Section 4.1 reports 0.27 to 0.52; the discrepancy should be reconciled or the conditions behind each range should be stated explicitly.
  2. [Section 3.5] The text says temperature 0 "removed the need for repeated trials," yet the intra-rater reliability test used five independent evaluations; the wording should be softened to "reduced" the need for repeated trials, since the reported Krippendorff's alpha of 0.98 itself shows that some variability remains.
  3. [Sections 3.2 and 4.3] The two non-verbal MIRS items are never named; the authors should identify them explicitly and describe exactly how the video was presented to the multimodal model (e.g., full video, clipped segments, with or without transcript) so the multimodal result is reproducible.
  4. [Section 3.3] The paper does not specify how the numeric score was extracted from the model output (e.g., structured JSON, regex parsing, or manual reading). This detail is important for reproducibility and for assessing whether parsing errors could affect the accuracy metrics.
  5. [Section 4.3] The multimodal result is based on only two MIRS items and an unspecified number of video ratings; reporting the per-item counts and the human-human agreement on those items would help calibrate the claim of "systematic disagreement" (Krippendorff's alpha = -0.47).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: an empirical benchmark comparing LLM outputs to external expert consensus scores, with no fitted parameter or label-dependent construction.

full rationale

This paper is an empirical benchmarking study, not a derivation. The central quantities (exact, off-by-one, and thresholded accuracy) are standard agreement metrics computed by comparing model-generated MIRS scores against expert consensus scores supplied by the University of Connecticut. No computed accuracy value is defined in terms of a model output or of a fitted parameter; the consensus labels are external ground truth, not outputs of the models under test. The only data-dependent adaptation is the selection of the optimal prompting technique per MIRS item on the same evaluation dataset, which the paper explicitly flags as a theoretical upper bound and recommends cross-validation for future work; this is post-hoc selection and an acknowledged statistical limitation, not circular reasoning. The reported intra-rater reliability test (Krippendorff's alpha = 0.98) is an independent consistency check of GPT-4o's zero-shot scoring, not a masked fit. Related prior work by other groups (Shakur et al., Jamieson et al.) is cited as external context and is not load-bearing for the paper's own measurements. The skeptical concern that off-by-one and thresholded accuracies lack a majority-class or inter-rater baseline bears on interpretability and on the strength of the practical claim, but it does not make any step circular: the numbers are genuine comparisons to external labels. Accordingly, the circularity score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

No numeric free parameters are fitted to data; the only data-driven choice is the per-item prompting technique selected on the evaluation set, which is listed as a free parameter and flagged as selection bias. All other assumptions are domain assumptions about the gold standard and transcription pipeline.

free parameters (1)
  • Optimal prompting technique per MIRS item = one of zero-shot, CoT, few-shot, multi-step selected per item on the evaluation set
    Section 4.1: selecting the best prompt per item improves reported accuracy, but the selection was made on the same data used for evaluation, so the improvement is an upper bound.
assumptions (5)
  • domain assumption Expert consensus MIRS scores from the University of Connecticut are valid ground truth for LLM scoring.
    Sections 3.1 and 3.2 use 174 consensus scores as the reference; no rater count or consensus procedure is reported.
  • domain assumption Whisper transcription and manual diarization preserve enough information to score the 26 verbal MIRS items.
    Section 3.1 describes audio-to-text conversion and diarization; transcription errors could change the evidence available for scoring.
  • ad hoc to paper Author-written anchor statements for scores 2 and 4 are valid extensions of the original 3-anchor rubric.
    Section 3.2 states the authors wrote anchors for scores 2 and 4, validated by team experts but not by an external published standard.
  • domain assumption A temperature of zero makes single-trial LLM responses sufficiently deterministic for benchmarking.
    Section 3.5 notes zero temperature can still produce slight variation; reliability was only tested for GPT-4o zero-shot.
  • domain assumption Ten OSCE cases and 174 scored items are sufficient to draw item-level and phase-level conclusions across 28 MIRS items.
    Section 4.2 notes some items occur in few transcripts; sparse item-level data weaken general claims of consistency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)." pith.science (2026). https://pith.science/paper/DUR4R3GC

@misc{pith2026250113957,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUR4R3GC}},
  note         = {Machine review of arXiv:2501.13957}
}
read the original abstract

Objective Structured Clinical Examinations (OSCEs) are widely used to assess medical students' communication skills, but scoring interview-based assessments is time-consuming and potentially subject to human bias. This study explored the potential of large language models (LLMs) to automate OSCE evaluations using the Master Interview Rating Scale (MIRS). We compared the performance of four state-of-the-art LLMs (GPT-4o, Claude 3.5, Llama 3.1, and Gemini 1.5 Pro) in evaluating OSCE transcripts across all 28 items of the MIRS under the conditions of zero-shot, chain-of-thought (CoT), few-shot, and multi-step prompting. The models were benchmarked against a dataset of 10 OSCE cases with 174 expert consensus scores available. Model performance was measured using three accuracy metrics (exact, off-by-one, thresholded). Averaging across all MIRS items and OSCE cases, LLMs performed with low exact accuracy (0.27 to 0.44), and moderate to high off-by-one accuracy (0.67 to 0.87) and thresholded accuracy (0.75 to 0.88). A zero temperature parameter ensured high intra-rater reliability ({\alpha} = 0.98 for GPT-4o). CoT, few-shot, and multi-step techniques proved valuable when tailored to specific assessment items. The performance was consistent across MIRS items, independent of encounter phases and communication domains. We demonstrated the feasibility of AI-assisted OSCE evaluation and provided benchmarking of multiple LLMs across multiple prompt techniques. Our work provides a baseline performance assessment for LLMs that lays a foundation for future research into automated assessment of clinical communication skills.

Figures

Figures reproduced from arXiv: 2501.13957 by the authors.

Figure 1
Figure 1. The overall flow of the evaluation process (top), along with examples of an annotated transcript, MIRS item prompt, and score/justification pair (bottom). OSCE video transcripts are appended to MIRS item prompts, which are passed into LLMs for scoring and justification. comprises 28 items, each rated on a 5-point scale with three labeled anchor state￾ments, assessing various aspects of the medical interview includin… view at source ↗
Figure 2
Figure 2. Structure of the steps involved for each prompt, depending on the prompting technique, with one such prompt for each MIRS rubric item. In multi-step prompting, the “Provide Transcript” step uses the extracted excerpt. 3.3 LLMs and Prompting Techniques We tested the performance of four state-of-the-art language models in this study: GPT-4o (OpenAI) [29], Claude 3.5 (Anthropic) [3], Llama 3.1 (Meta) [2], and Gemini 1.… view at source ↗
Figure 3
Figure 3. Average performance of each model using different prompting techniques and measured with different accuracy metrics. Error bars represent standard errors calcu￾lated across the 10 OSCE cases, where each case’s accuracy is first computed as the mean of all applicable MIRS items for that case. 4.2 Performance by MIRS Items We analyzed off-by-one accuracy for each MIRS item across the four models (Claude, Gemini, GPT, … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Heatmap of the off-by-one accuracy for each model and MIRS item. While we analyzed a total of 26 text-based MIRS items, this figure displays results for the 21 items where data points were available across all cases. Opening the discussion and building the relationship…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 41 canonical work pages

  1. [1]

    JMIR Medical Education 9(1), e48291 (2023)

    Abd-Alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P.M., Latifi, S., Aziz, S., Damseh, R., Alrazak, S.A., Sheikh, J., et al.: Large language models in medical education: opportunities, challenges, and future directions. JMIR Medical Education 9(1), e48291 (2023)

    work page 2023

  2. [2]

    AI, M.: Llama-v3p1-405b-instruct (2024), model release date: 2024

    work page 2024

  3. [3]

    Anthropic: Claude-3.5-sonnet-20240620 (2024), model release date: June 20, 2024

    work page 2024

  4. [4]

    Attali, Y., Burstein, J.: Automated essay scoring with e-rater® v. 2. The Journal of Technology, Learning and Assessment4(3) (2006)

    work page 2006

  5. [5]

    Journal of Graduate Medical Education 9(5), 645–649 (2017)

    Baldwin, J.D., Cox, J., Wu, Z.H., Kenny, A., Angus, S.: Delivery and measurement of high-value care in standardized patient encounters. Journal of Graduate Medical Education 9(5), 645–649 (2017)

    work page 2017

  6. [6]

    CMAJ: Canadian Medical Association Journal178(12), 1555–1562 (2008)

    Bartlett, G., Blais, R., Tamblyn, R., Clermont, R.J., MacGibbon, B.: Impact of patient communication problems on the risk of preventable adverse events in acute care settings. CMAJ: Canadian Medical Association Journal178(12), 1555–1562 (2008)

    work page 2008

  7. [7]

    Academic Medicine 90(1), 105–111 (Jan 2015)

    Berg, K., Blatt, B., Lopreiato, J., Jung, J., Schaeffer, A., Heil, D., Owens, T., Carter-Nolan, P.L., Berg, D., Veloski, J., Darby, E., Hojat, M.: Standardized pa- tient assessment of medical student empathy: ethnicity and gender effects in a multi-institutional study. Academic Medicine 90(1), 105–111 (Jan 2015). https: //doi.org/10.1097/ACM.0000000000000529

    work page doi:10.1097/acm.0000000000000529 2015

  8. [8]

    Human Resource Man- agement 55(2), 321–340 (2016)

    Bernardin, H.J., Thomason, S., Buckley, M.R., Kane, J.S.: Rater rating-level bias and accuracy in performance appraisals: The impact of rater personality, perfor- mance management competence, and rater accountability. Human Resource Man- agement 55(2), 321–340 (2016)

    work page 2016

Show all 46 references
  1. [9]

    Borchers, C., Yang, K., Lin, J., Rummel, N., Koedinger, K.R., Aleven, V.: Com- bining dialog acts and skill modeling: What chat interactions enhance learning rates during ai-supported peer tutoring? In: Proceedings of the 17th International Conference on Educational Data Mining (2024)

    2024

  2. [10]

    Journal of ed- ucational evaluation for health professions14 (2017)

    Chong, L., Taylor, S., Haywood, M., Adelstein, B.A., Shulruf, B.: The sights and insights of examiners in objective structured clinical examinations. Journal of ed- ucational evaluation for health professions14 (2017)

    2017

  3. [11]

    DeepMind, G.: Gemini-1.5-pro-exp-0801 (2024), model release date: August 1, 2024

    2024

  4. [12]

    In: Proceedings of the Tenth ACM Conference on Learning@ Scale

    Demszky, D., Liu, J.: M-powering teachers: Natural language processing powered feedback improves 1: 1 instruction and student outcomes. In: Proceedings of the Tenth ACM Conference on Learning@ Scale. pp. 59–69 (2023)

    2023

  5. [13]

    Dimari, A., Tyagi, N., Davanageri, M., Kukreti, R., Yadav, R., Dimari, H.: Ai- based automated grading systems for open book examination system: implications forassessmentinhighereducation.In:2024InternationalConferenceonKnowledge Engineering and Communication Systems (ICKECS)...

    2024

  6. [14]

    Cureus15(4) (2023)

    Duran, M., Hendrix, J.M.: The role of videotaped clinical skills aggregated peer evaluations in the enhancement of evaluation skills of individual medical school faculty members. Cureus15(4) (2023)

    2023

  7. [15]

    Journal of General Internal Medicine22(8), 1155–1160 (Aug 2007)

    Fernandez, A., Wang, F., Braveman, P., Finkas, L.K., Hauer, K.E.: Impact of stu- dent ethnicity and primary childhood language on communication skill assessment in a clinical performance examination. Journal of General Internal Medicine22(8), 1155–1160 (Aug 2007). https://doi....

    2007 doi

  8. [16]

    In: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

    Firdaus, M., Singh, G., Ekbal, A., Bhattacharyya, P.: Multi-step prompt- ing for few-shot emotion-grounded conversations. In: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. p. 3886–3891. CIKM ’23, Association for Computing Machine...

    2023

  9. [17]

    British Educa- tional Research Journal (2024)

    Flodén, J.: Grading exams using large language models: A comparison between human and ai grading of exams in higher education using chatgpt. British Educa- tional Research Journal (2024)

    2024

  10. [18]

    Academic Medicine 97(11S), S29–S34 (Nov 2022)

    Fluet, A., Essakow, J., Ju, M.: Standardized patients’ perspectives on bias in student encounters. Academic Medicine 97(11S), S29–S34 (Nov 2022). https: //doi.org/10.1097/ACM.0000000000004925, epub 2022 Aug 9

    2022 doi

  11. [19]

    Advances in Health Sciences Education20, 229–245 (2015)

    Harrison, C.J., Könings, K.D., Schuwirth, L., Wass, V., Van der Vleuten, C.: Bar- riers to the uptake and use of feedback in the context of summative assessment. Advances in Health Sciences Education20, 229–245 (2015)

    2015

  12. [20]

    Medical Teacher37(1), 81–85 (2015)

    Hope, D., Cameron, H.: Examiners are most lenient at the start of a two-day osce. Medical Teacher37(1), 81–85 (2015)

    2015

  13. [21]

    Frontiers in Oncology13, 1265024 (2023)

    Huang, Y., Gomaa, A., Semrau, S., Haderlein, M., Lettmaier, S., Weissmann, T., Grigo, J., Tkhayat, H.B., Frey, B., Gaipl, U., et al.: Benchmarking chatgpt-4 on a radiation oncology in-training exam and red journal gray zone cases: potentials and challenges for ai-assisted medi...

    2023

  14. [22]

    In: Findings of the Association for Computational Linguistics: EMNLP 2023

    Jain, R., Saha, T., Lalwani, J., Saha, S.: Can you summarize my learnings? to- wards perspective-based educational dialogue summarization. In: Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 3158–3173 (2023)

    2023

  15. [23]

    NEJM AI1(12), AIcs2400631 (2024)

    Jamieson, A.R., Holcomb, M.J., Dalton, T.O., Campbell, K.K., Vedovato, S., Shakur, A.H., Kang, S., Hein, D., Lawson, J., Danuser, G., et al.: Rubrics to prompts: Assessing medical student post-encounter notes with ai. NEJM AI1(12), AIcs2400631 (2024)

    2024

  16. [24]

    Journal of graduate medical education2(2), 165–169 (2010)

    Joyce, B.L., Steenbergh, T., Scher, E.: Use of the kalamazoo essential elements communication checklist (adapted) in an institutional interpersonal and commu- nication skills curriculum. Journal of graduate medical education2(2), 165–169 (2010)

    2010

  17. [25]

    Khan, K.Z., Ramachandran, S., Gaunt, K., Pushkar, P.: The objective structured clinical examination (osce): Amee guide no. 81. part i: an historical and theoretical perspective. Medical teacher35(9), e1437–e1446 (2013)

    2013

  18. [26]

    arXiv preprint arXiv:2405.16402 (2024)

    Luo, M., Warren, C.J., Cheng, L., Abdul-Muhsin, H.M., Banerjee, I.: Assessing empathy in large language models with real-world physician-patient interactions. arXiv preprint arXiv:2405.16402 (2024)

    2024 arXiv

  19. [27]

    Medical education43(10), 989–992 (2009)

    McLaughlin, K., Ainslie, M., Coderre, S., Wright, B., Violato, C.: The effect of differential rater function over time (drift) on objective structured clinical exami- nation ratings. Medical education43(10), 989–992 (2009)

    2009

  20. [28]

    Assessing Writing23, 35–59 (2015)

    McNamara, D.S., Crossley, S.A., Roscoe, R.D., Allen, L.K., Dai, J.: A hierarchical classification approach to automated essay scoring. Assessing Writing23, 35–59 (2015)

    2015

  21. [29]

    OpenAI: Gpt-4o-2024-08-06 (2024), model release date: August 6, 2024

    2024

  22. [30]

    com/openai/whisper, accessed: 2024-10-07

    OpenAI: Whisper: Automatic speech recognition system (2024), https://github. com/openai/whisper, accessed: 2024-10-07

    2024

  23. [31]

    Journal of the American Geriatrics Society56(9), 1730–1735 (2008)

    O’Sullivan, P., Chao, S., Russell, M., Levine, S., Fabiny, A.: Development and Implementation of an Objective Structured Clinical Examination to Provide For- Benchmarking GenAI for Scoring OSCEs 15 mative Feedback on Communication and Interpersonal Skills in Geriatric Train- i...

    2008

  24. [32]

    Pfeiffer, C.A., Palten, B., Akins, A.B., Collins, R.T., Aloi, D., Jack- son, B., Barber, T., Smith, B.: Master interview rating scale (mirs), https://health.uconn.edu/principles-clinical-medicine-clinical-skills-assessment/ master-interview-rating-scale-mirs/, accessed: 2024-10-07

    2024

  25. [33]

    Har- vard Public Health Review23, 1–6 (2019)

    Ratna, H.: The importance of effective communication in healthcare practice. Har- vard Public Health Review23, 1–6 (2019)

    2019

  26. [34]

    Safranek, C.W., Sidamon-Eristoff, A.E., Gilson, A., Chartash, D.: The role of large language models in medical education: applications and implications (2023)

    2023

  27. [35]

    arXiv preprint arXiv:2410.12858 (2024)

    Shakur, A.H., Holcomb, M.J., Hein, D., Kang, S., Dalton, T.O., Campbell, K.K., Scott, D.J., Jamieson, A.R.: Large language models for medical osce assessment: A novel approach to transcript analysis. arXiv preprint arXiv:2410.12858 (2024)

    2024 arXiv

  28. [36]

    The University of Tennessee Health Science Center: Master interview rating scale (2024), https://www.uthsc.edu/simulation/resources/documents/ sp-full-communication-rating-scale-mirs.docx, accessed: 2024-02-07

    2024

  29. [37]

    Elsevier, 3rd edn

    Uchida, T., Cornes, S.: Standardized Patients. Elsevier, 3rd edn. (2023)

    2023

  30. [38]

    Medical Education26(2), 99–104 (1992)

    Vu, N.V., Marcy, M.M., Colliver, J.A., Verhulst, S.J., Travis, T.A., Barrows, H.S.: Standardized (simulated) patients’ accuracy in recording clinical performance check-list items. Medical Education26(2), 99–104 (1992)

    1992

  31. [39]

    arXiv preprint arXiv:2410.03017 (2024)

    Wang,R.E.,Ribeiro,A.T.,Robinson,C.D.,Loeb,S.,Demszky,D.:Tutorcopilot:A human-ai approach for scaling real-time expertise. arXiv preprint arXiv:2410.03017 (2024)

    2024 arXiv

  32. [40]

    ACM computing surveys (csur)53(3), 1–34 (2020)

    Wang, Y., Yao, Q., Kwok, J.T., Ni, L.M.: Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur)53(3), 1–34 (2020)

    2020

  33. [41]

    Advances in neural information processing systems35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems35, 24824–24837 (2022)

    2022

  34. [42]

    Teaching of Psychology p

    Wetzler, E.L., Cassidy, K.S., Jones, M.J., Frazier, C.R., Korbut, N.A., Sims, C.M., Bowen, S.S., Wood, M.: Grading the graders: Comparing generative ai and hu- man assessment in essay evaluation. Teaching of Psychology p. 00986283241282696 (2024)

    2024

  35. [43]

    JMIR Medical Informatics 12(1), e57674 (2024)

    Xu, J., Lu, L., Peng, X., Pang, J., Ding, J., Yang, L., Song, H., Li, K., Sun, X., Zhang, S., et al.: Data set and benchmark (medgpteval) to evaluate responses from large language models in medicine: evaluation development and validation. JMIR Medical Informatics 12(1), e57674 (2024)

    2024

  36. [44]

    why my essay received a 4?

    Yang, B., Nam, S., Huang, Y.: “why my essay received a 4?”: A natural language processing based argumentative essay structure analysis. In: International Confer- ence on Artificial Intelligence in Education. pp. 279–290. Springer (2023)

    2023

  37. [45]

    British Journal of Educational Technology56(1), 150–166 (2025)

    Yavuz, F., Çelik, Ö., Yavaş Çelik, G.: Utilizing large language models for efl essay grading: An examination of reliability and validity in rubric-based assessments. British Journal of Educational Technology56(1), 150–166 (2025)

    2025

  38. [46]

    GMS journal for medical education 37(4) (2020)

    Zimmermann,P.,Kadmon,M.:Standardizedexaminees:developmentofanewtool to evaluate factors influencing osce scores and to train examiners. GMS journal for medical education 37(4) (2020)

    2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.