{"id":"f202956f-dcf1-433d-9e0b-23e3edaf7c55","arxiv_id":"2501.13957","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Four LLMs scored OSCE transcripts with low exact agreement (27-44%) but moderate to high broad-band agreement (75-88%) with expert consensus.","lead":"This paper tested four large language models on scoring ten recorded medical student interviews using a 28-item communication rubric, under four prompting strategies, against 174 expert scores. The models often matched experts within one point but rarely exactly, and extra prompting steps did not consistently help, suggesting AI can assist but not replace human graders.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Off-by-one and thresholded accuracy are reported without a score-distribution or majority-class baseline, so 'moderate-to-high' may largely reflect skewed expert labels rather than LLM skill.","rationale":"The reader's weakest assumption concerned the unvalidated expert consensus labels and missing human-human reliability; the concern raised here is adjacent but more specific: the accuracy metrics are not benchmarked against the score distribution or trivial predictors, so the reported thresholded/off-by-one values may overstate model capability. This reinforces the reader's CONDITIONAL verdict rather than changing it, because the paper already needs additional validation before the central proficiency-classification claim can be accepted. I chose UNCHANGED since the final verdict remains conditional; the new concern sharpens the required condition without altering the overall assessment.","tokens_in":10186,"tokens_out":5387,"duration_ms":60312,"concrete_test":"Compute the empirical distribution of the 174 consensus scores (overall and per MIRS item), then recompute the three accuracy metrics for baseline predictors: always predict the modal consensus score; always predict the high-proficiency bucket (3–5); and random draws from the empirical score distribution. If the best LLM's off-by-one or thresholded accuracy exceeds the best trivial baseline by less than about 0.10, the 'moderate to high' interpretation should be downgraded. Additionally, if a subset of transcripts can be double-scored by a second expert, report human-human off-by-one and thresholded agreement to calibrate whether LLM exact accuracy is near human parity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central practical claim (Abstract; §4.1) is that LLMs are useful for broad proficiency classification, supported by thresholded accuracy 0.75–0.88 and off-by-one accuracy 0.67–0.87, while exact accuracy is low. However, all metrics are computed against 174 expert consensus scores whose marginal distribution is never reported, and no trivial baseline is computed. On a 5-point scale with anchors only at 1, 3, and 5, OSCE consensus ratings commonly concentrate at 3/4/5; in that case, a model that always returns the modal score (or always assigns the high-proficiency bucket) can achieve high off-by-one and thresholded accuracy by construction. The reported zero-shot thresholded values (0.83–0.88 in §4.1) are close to plausible majority-class ceilings, so the observed numbers do not by themselves establish that models can distinguish proficiency levels. The human-rater baseline is likewise absent: no inter-rater reliability is reported for the UConn consensus scores (§3.1–3.2), so we do not know whether exact accuracy of 0.27–0.44 is poor relative to human exact agreement or comparable to it. Both missing baselines bear on the same interpretive step: agreement numbers are meaningful only relative to the difficulty of the label distribution and the human scoring process. Because the entire value proposition depends on the coarse-accuracy metrics, this is the most load-bearing gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks four LLMs (GPT-4o, Claude 3.5, Llama 3.1, Gemini 1.5 Pro) on scoring 174 expert-consensus MIRS ratings drawn from 10 OSCE transcripts, using zero-shot, chain-of-thought, few-shot, and multi-step prompting. It reports low exact accuracy (0.27 to 0.44), moderate to high off-by-one accuracy (0.67 to 0.87), and thresholded accuracy (0.75 to 0.88), with no consistent gain from augmentative prompting unless the prompting technique is tailored per MIRS item; the authors acknowledge that per-item selection is an upper bound. A multimodal run on two non-verbal items showed strong disagreement with human raters. The paper's central claim is that LLMs are useful for broad proficiency classification but not for exact scoring.","tokens_in":10403,"tokens_out":5722,"duration_ms":60551,"significance":"If the findings hold up after the missing baselines are supplied, this is a useful first systematic, multi-model benchmark of LLM scoring across a complete OSCE communication rubric. The study's strengths include a transparent deterministic protocol (temperature 0), a reported intra-rater reliability check, public prompts on GitHub, and evaluation against external expert consensus rather than model self-consistency. The main interpretive gaps—no human-human inter-rater baseline, no score-distribution or majority-class baseline, and post hoc per-item prompt selection—are remediable and do not invalidate the raw measurements, but they do currently weaken the practical conclusions drawn from the coarse accuracy metrics.","major_comments":[{"comment":"The headline claim that LLMs achieve moderate-to-high off-by-one and thresholded accuracy is not interpretable without a score-distribution baseline. With a 5-point scale and a proficiency threshold at 3, a model that always returns the majority bucket (or the modal score) can achieve high thresholded accuracy if the expert consensus scores are concentrated in the 3-5 bucket. The paper should report the marginal distribution of the 174 consensus scores and the accuracy of trivial baselines (always majority bucket, always modal score, uniform random) on the same items. This is load-bearing because the central practical claim in Section 4.1 rests on these coarse metrics.","section":"Section 4.1, Section 3.4"},{"comment":"The manuscript compares model scores against UConn expert consensus scores but reports neither the number of raters contributing to each consensus score nor the consensus procedure, and it provides no human-human inter-rater reliability on these ten transcripts. Without a human reliability estimate, exact accuracy of 0.27-0.44 cannot be judged poor, and coarse accuracy cannot be judged useful: if human exact agreement on the same transcripts is also around 0.4, the LLM may be at parity, whereas if human agreement is much higher, the LLM is clearly deficient. The authors should report the rater pool and consensus rule and, ideally, provide human-human agreement (e.g., Krippendorff's alpha or exact agreement) on these data or a clearly comparable subsample.","section":"Sections 3.1 and 3.2"},{"comment":"The claim that CoT, few-shot, and multi-step prompting \"proved valuable when tailored to specific assessment items\" rests on selecting the optimal prompting technique per MIRS item on the evaluation set itself. The paper explicitly acknowledges this is a theoretical upper bound, but the abstract and discussion still present tailored prompting as a demonstrated benefit. This conclusion requires out-of-sample validation, such as cross-validation or a held-out set, and the comparison should be against a fixed per-item strategy or a proper nested selection procedure. As reported, the improvement is a selection artifact and should be either removed from the conclusions or clearly labeled as an upper-bound analysis.","section":"Section 4.1, Abstract, Section 5"},{"comment":"The item-level heatmap and the statement that most items were scored with off-by-one accuracy of 0.8 or higher are based on very small per-item sample sizes: 174 scores over 26 items and 10 cases, and the paper notes that some items were not present in all transcripts. With n=10 per item, a single misclassification changes accuracy by 10 percentage points, and for items present in fewer cases the instability is even larger. The authors should report per-item counts and confidence intervals, or restrict the item-level claims to items with adequate coverage. The current presentation does not support the sharp conclusion that performance is consistent across items and independent of encounter phases and communication domains.","section":"Section 4.2, Figure 4"}],"minor_comments":[{"comment":"The abstract reports exact accuracy of 0.27 to 0.44, while Section 4.1 reports 0.27 to 0.52; the discrepancy should be reconciled or the conditions behind each range should be stated explicitly.","section":"Abstract vs. Section 4.1"},{"comment":"The text says temperature 0 \"removed the need for repeated trials,\" yet the intra-rater reliability test used five independent evaluations; the wording should be softened to \"reduced\" the need for repeated trials, since the reported Krippendorff's alpha of 0.98 itself shows that some variability remains.","section":"Section 3.5"},{"comment":"The two non-verbal MIRS items are never named; the authors should identify them explicitly and describe exactly how the video was presented to the multimodal model (e.g., full video, clipped segments, with or without transcript) so the multimodal result is reproducible.","section":"Sections 3.2 and 4.3"},{"comment":"The paper does not specify how the numeric score was extracted from the model output (e.g., structured JSON, regex parsing, or manual reading). This detail is important for reproducibility and for assessing whether parsing errors could affect the accuracy metrics.","section":"Section 3.3"},{"comment":"The multimodal result is based on only two MIRS items and an unspecified number of video ratings; reporting the per-item counts and the human-human agreement on those items would help calibrate the claim of \"systematic disagreement\" (Krippendorff's alpha = -0.47).","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a medical-education or applied-NLP journal, and the full-MIRS multi-model benchmark is a useful contribution to the small but growing literature on LLM-based OSCE scoring. The authors are commendably transparent about the upper-bound nature of per-item prompt selection and about single-institution data. The two most important revisions are to add simple score-distribution baselines and human-human reliability information; both are feasible and would substantially raise the interpretability of the reported accuracy numbers. I would not reject on novelty grounds, but the current version overstates what the coarse accuracy metrics establish."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid, honestly presented first benchmark across all 28 MIRS items with four LLMs and four prompting strategies. The low exact accuracy numbers are reported straight, and the poor multimodal result is a useful negative finding. Releasing the prompts is a nice touch. It's a real step beyond the single-item and written-note work that came before.\n\nThe soft spot is load-bearing. The off-by-one and thresholded accuracy claims are the heart of the practical story, but they're computed against 174 expert consensus scores with no majority-class baseline and no human-human reliability estimate. On a 5-point scale like MIRS, with scores typically clustered at 3-5, a model that always predicts the modal bucket will look moderately accurate on thresholded and off-by-one metrics by construction. The reported 0.83–0.88 zero-shot thresholded numbers are close to plausible majority-class ceilings, so as presented they don't demonstrate that the models are actually distinguishing proficiency levels. I'd also want to know whether exact accuracy of 0.27–0.44 is far below human exact agreement or roughly comparable—without the human baseline, we can't tell.\n\nThe per-item prompt selection on the evaluation set is a real issue, but the paper acknowledges it as an upper bound. The single-institution dataset and sparse item-level coverage are limitations, not flaws, given the exploratory nature. The main problem is the missing interpretive context for the coarse accuracy metrics.\n\nVerdict: this deserves peer review, but the revision needs to add baselines—score distribution, majority-class accuracy, and human-human reliability on the same transcripts. Ideally also release the data. The paper is most useful as a benchmark for future work; I wouldn't yet take the 'useful for proficiency classification' claim at face value.\n\nI'd bring it to a reading group focused on LLM evaluation or medical education, and I'd cite it as the existing baseline if I worked in that subfield. A serious referee should see it.","headline":"A useful first full-MIRS benchmark, but the practical accuracy claims depend on baselines the paper never reports.","tokens_in":10997,"tokens_out":2328,"would_cite":true,"duration_ms":26378,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs agree with expert OSCE raters within one point on most items but rarely match exactly.","keywords":["OSCE","large language models","Master Interview Rating Scale","automated assessment","prompting strategies","medical education","communication skills","benchmarking"],"falsifier":"Have a second panel of expert raters independently score the same ten transcripts on the same MIRS items and compute inter-rater exact and off-by-one accuracy; if expert-expert exact accuracy is comparable to the models’ 0.27–0.44, then the low exact accuracy reflects rubric ambiguity rather than LLM deficiency.","tokens_in":1741,"feed_emoji":"🩺","tokens_out":2985,"duration_ms":65038,"temperature":0.7,"pith_summary":"This paper asks whether large language models can score medical students' interview skills in Objective Structured Clinical Examinations (OSCEs) reliably enough to be useful. Across 174 expert-consensus ratings from 10 recorded cases, the four tested models matched the expert score exactly only 27–44% of the time, but landed within one point 67–87% of the time and agreed on broad proficiency bands 75–88% of the time. The authors argue this makes LLMs viable for flagging students who need support or for formative feedback, but not for exact summative scoring. They also find that no single prompting technique helps across all rubric items, and that item-specific prompt selection is more promising than a one-size-fits-all approach.","feed_headline":"LLMs flag weak OSCE interviews but miss exact scores","feed_subtitle":"Across 174 expert-scored items, best models land within one point 67-87% of the time but match exactly only 27-44%.","key_machinery":"The central object is the Master Interview Rating Scale (MIRS), a validated 28-item, 5-point rubric for assessing medical communication skills, together with three accuracy metrics the authors define: exact accuracy, off-by-one accuracy, and thresholded accuracy (scores 1–2 vs. 3–5). The prompting techniques—zero-shot, chain-of-thought, few-shot, and multi-step—are the interventions being compared. The measurement protocol uses transcripts generated by Whisper from video, manually diarized dialogue, temperature zero for deterministic outputs, and Krippendorff’s alpha for reliability.","core_discovery":"Using the Master Interview Rating Scale (MIRS), a 28-item rubric for medical communication, the paper benchmarks GPT-4o, Claude 3.5, Llama 3.1, and Gemini 1.5 Pro on transcripts of ten OSCE encounters. Averaging over all applicable items, exact accuracy was low (0.27–0.44), while off-by-one accuracy was moderate to high (0.67–0.87) and thresholded accuracy, which distinguishes below-proficiency scores (1–2) from passing scores (3–5), was high (0.75–0.88). The paper reports that zero-temperature settings produce high intra-rater reliability (Krippendorff’s $\\\\alpha = 0.98$ for GPT-4o), that chain-of-thought, few-shot, and multi-step prompting do not improve average performance over zero-shot, and that selecting the best prompt per MIRS item does improve performance—though this selection is made on the same data used for evaluation. A separate multimodal test on two non-verbal items with Gemini 1.5 Pro produced systematic disagreement with human raters (Krippendorff’s $\\\\alpha = -0.47$), indicating current models cannot score non-verbal communication from video.","pith_inferences":["An implication the paper leaves implicit is that the 0.27–0.44 exact-accuracy range may not be a pure measure of LLM capability: if human raters themselves disagree on exact MIRS scores at a similar rate, then exact accuracy is partly measuring rubric ambiguity.","The item-level results rest on very small counts—some MIRS items were not applicable to certain cases, so a single transcript can drive an item’s accuracy estimate; readers should treat the item-level heatmap as suggestive rather than stable.","The paper’s thresholded accuracy result suggests a practical extension: LLM scores could be used to triage which student interviews need detailed human review, while students far from the threshold receive automated feedback.","Because the benchmarks use one institution’s cases and rubric anchors, cross-institutional validation would be needed before these accuracy numbers can be generalized to other medical schools’ OSCE formats."],"forward_implications":["If these results hold, medical schools could use LLMs as a first-pass screening tool to flag students who fall below the proficiency threshold, with human raters confirming borderline and low scores.","The finding that uniform prompting techniques do not improve on zero-shot suggests that future automated scoring systems should treat each rubric item as a separate prompt-design problem.","The poor multimodal performance on non-verbal items means transcript-based LLM scoring cannot yet replace human judgment for aspects like pacing and non-verbal facilitation.","Because the three metrics give different pictures of model quality, any deployment should specify which level of scoring leniency is appropriate for the decision being made.","The paper’s in-sample selection of optimal prompts per item is an upper bound on what tailored prompting could achieve, so real-world gains from prompt tuning remain to be shown with held-out data."],"supporting_citations":[{"why":"Supplies the MIRS rubric and the expert consensus scores that serve as the ground truth for all accuracy measurements.","marker":"[32]"},{"why":"Whisper generates the transcripts from OSCE videos that are the input to every model score.","marker":"[30]"},{"why":"Provides the closest prior result—GPT-4 grading a single OSCE rubric item with high concordance—that this study extends to the full 28-item scale.","marker":"[35]"},{"why":"Demonstrates an operational GPT-4 grading system for post-encounter notes, establishing the feasibility context for LLM-based assessment.","marker":"[23]"},{"why":"Chain-of-thought prompting is one of the four techniques benchmarked and its performance is compared against zero-shot.","marker":"[41]"},{"why":"Multi-step prompting, in which the model first extracts relevant excerpts before scoring, is the basis for the multi-step technique.","marker":"[16]"},{"why":"Few-shot learning is the third prompting technique, and the paper uses up to five physician-patient interaction examples per item.","marker":"[40]"},{"why":"Validates the MIRS instrument as a measure of medical communication skills, supporting its use as the evaluation rubric.","marker":"[5,31]"},{"why":"Provides evidence that human raters vary in OSCE scoring, motivating the off-by-one and thresholded accuracy metrics.","marker":"[20,27]"}],"fun_headline_variants":["LLMs nail pass/fail but miss exact scores on OSCE","AI grades OSCE within one point, but exact hits are rare","LLM OSCE grading: reliable pass/fail, unreliable exact scores","GPT-4o scores OSCE interviews consistently but not exactly","OSCE scoring by LLMs: off-by-one common, exact rare"],"cache_read_input_tokens":13056,"weakest_assumption_plain":"The entire accuracy comparison rests on treating the 174 expert consensus scores as correct ground truth, but the paper does not report how many raters produced each consensus or how well human raters agreed with each other.","fun_headline_variants_meta":{"raw":{"variants":["LLMs nail pass/fail but miss exact scores on OSCE","AI grades OSCE within one point, but exact hits are rare","LLM OSCE grading: reliable pass/fail, unreliable exact scores","GPT-4o scores OSCE interviews consistently but not exactly","OSCE scoring by LLMs: off-by-one common, exact rare"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000816,"raw_usage":{"total_tokens":3674,"prompt_tokens":1141,"completion_tokens":2533,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":757,"completion_tokens_details":{"reasoning_tokens":2441}},"tokens_in":757,"tokens_out":2533,"duration_ms":17657,"temperature":1.0,"reasoning_tokens":2441,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:46:29.839137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a second panel of expert raters independently score the same ten transcripts on the same MIRS items and compute inter-rater exact and off-by-one accuracy; if expert-expert exact accuracy is comparable to the models’ 0.27–0.44, then the low exact accuracy reflects rubric ambiguity rather than LLM deficiency.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MIRS rubric and the expert consensus scores that serve as the ground truth for all accuracy measurements."},{"cited_title":"com/openai/whisper, accessed: 2024-10-07","cited_arxiv_id":null,"evidence_quote":"Whisper generates the transcripts from OSCE videos that are the input to every model score."},{"cited_title":"NEJM AI1(12), AIcs2400631 (2024)","cited_arxiv_id":null,"evidence_quote":"Demonstrates an operational GPT-4 grading system for post-encounter notes, establishing the feasibility context for LLM-based assessment."}],"review_version":1}