Pith. sign in

REVIEW 3 major objections 9 minor

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Small open-source LLMs can partially automate clinical SDM scoring

2026-07-08 15:45 UTC pith:F4KTKF57

load-bearing objection First evaluation of open-source small LLMs on OPTION12 coding; results are preliminary and internally contradictory on the headline metric. the 3 major comments →

arxiv 2607.06127 v2 pith:F4KTKF57 submitted 2026-07-07 cs.CL

Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability

Tamara Wit , Lifeng Han , Carly Heipon , David Lindevelt , Anne Stiggelbout , Suzan Verberne This is my paper
classification cs.CL
keywords modelsos-sllmsassessmentframeworkgeneral-domainhumanmakingmedical-domain
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper investigates whether open-source, smaller language models (OS-sLLMs) — models small enough to run locally on modest hardware — can automate the scoring of shared decision-making (SDM) in clinical consultations using the OPTION12 instrument, a 12-item observer rating scale. The motivation is twofold: human coding of OPTION12 is slow and prone to inter-rater disagreement, and prior work on automating this task relied on large commercial models (GPT-4o, Gemini) that raise privacy concerns when fed sensitive patient transcripts. The authors evaluate five models on Dutch melanoma consultation transcripts: three general-domain models (Gemma3:12b, Llama3.1:8b, Mistral7b) and two medical-domain models (MedLlama2:7b, Meditron7b). They find that general-domain models consistently outperform medical-domain ones, which suffer from severe hallucination and instruction-following failures. Gemma3:12b achieves the strongest alignment with human annotations, with a Pearson correlation of 0.51 and Spearman correlation of 0.59 across the 12 OPTION12 items. The authors also propose a Judge-LLM consensus framework, in which the best-performing model adjudicates disagreements among the other models, mimicking how human annotators resolve disputes. The paper catalogs systematic error types — temporal reasoning failures, conversational role confusion, evidence hallucination, score-evidence mismatch, and item misinterpretation — providing a taxonomy for future improvement.

Core claim

The central finding is that general-domain OS-sLLMs outperform medical-domain OS-sLLMs on OPTION12 coding, with Gemma3:12b achieving moderate correlation with human annotators (Pearson r=0.51, Spearman ρ=0.59). The medical-domain models (MedLlama, Meditron) failed catastrophically — producing repeated evidence across items, assigning uniform scores regardless of content, and hallucinating justifications — suggesting that domain-specific fine-tuning for medical knowledge does not transfer to the discourse-analysis and conversational-reasoning skills that OPTION12 coding actually requires. Item-level analysis reveals that some OPTION12 items are systematically harder for all models (Items 1, 8

What carries the argument

The OPTION12 instrument is a 12-item observer-based rating scale where each item scores a specific aspect of shared decision-making on a 0–4 Likert scale. The Judge-LLM consensus framework is a proposed mechanism in which the best-performing OS-sLLM acts as a tie-breaker: when multiple models disagree on a score, the judge model adjudicates, analogous to a third human annotator resolving a coding dispute. Chain-of-thought prompting and few-shot examples from the development set are used to steer model outputs.

Load-bearing premise

The paper's conclusions rest on a development set of only 7 patient consultations (after excluding 4 of 11 for formatting failures and unsuitability), yielding 84 data points per model. With this sample size, the correlation coefficients and model rankings have very wide confidence intervals and no significance tests are reported, so the claim that Gemma3:12b is the best model could easily reverse with a different or larger sample.

What would settle it

If the held-out test set of 15 consultations shows that Gemma3:12b's correlation with human annotators drops below r=0.3, or that medical-domain models perform comparably to general-domain models after prompt optimization, the paper's central claims about model ranking and the superiority of general-domain models would not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If OS-sLLMs can be improved to reliably code OPTION12, clinical departments could audit SDM quality at scale without sending patient transcripts to external API providers, preserving data sovereignty.
  • The finding that medical-domain fine-tuning hurts performance on discourse-analysis tasks suggests that domain-specialized models may be counterproductive for clinical communication assessment, redirecting effort toward general models with task-specific instruction tuning.
  • The Judge-LLM consensus framework, if validated on the held-out test set, could generalize to other multi-annotator clinical NLP tasks where model disagreement patterns complement each other across items.
  • The error taxonomy (temporality failures, role confusion, evidence hallucination) provides concrete targets for targeted fine-tuning or architectural improvements in smaller models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The development set of only 7 patient files (84 data points per model) means the correlation differences between models (e.g., Gemma3:12b at r=0.51 vs. Mistral7b at r=0.18) have very wide confidence intervals and could reverse on a different sample. The ranking of models is not yet statistically established.
  • The absence of reported confidence intervals or significance tests for any metric makes it difficult to distinguish genuine performance gaps from sampling noise, especially with 3 of 11 development files excluded for formatting failures.
  • If the Judge-LLM consensus approach works, it implies that model diversity matters more than model size for this task — an ensemble of complementary weak models could approach the reliability of a single strong model at lower cost and with full local deployment.
  • The authors' observation that OPTION12 coding requires discourse analysis rather than medical knowledge suggests the task may be more analogous to dialogue-act recognition or conversational argument mining than to clinical NLP, which could inform model selection and training data construction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 9 minor

Summary. This paper presents LLM4SDM, the first investigation of open-source smaller language models (OS-sLLMs) for automated assessment of shared decision making (SDM) using the OPTION12 framework. The authors evaluate three general-domain models (Gemma3:12b, Llama3.1:8b, Mistral7b) and two medical-domain models (MedLlama2:7b, Meditron7b) on Dutch melanoma consultation transcripts. Using a development set of 7 patient files (after exclusions from an initial 11), they report item-level agreement rates and correlation with human annotations. The key findings are that general-domain models outperform medical-domain models (which exhibit substantial hallucination), and that Gemma3:12b achieves the highest correlation with human scores (Pearson r=0.513, Spearman ρ=0.587). The paper also proposes a Judge-LLM consensus framework for resolving disagreements among models and provides a qualitative error taxonomy.

Significance. The paper addresses a genuine gap: prior work on LLM-based SDM coding (Selvaraj et al., 2025) used commercial models and the shorter OPTION5 instrument, whereas this study evaluates locally deployable models on the more fine-grained OPTION12. The focus on privacy-preserving deployment is well-motivated for clinical data. The qualitative error taxonomy (Table 5) and the finding that medical-domain models underperform general-domain models on this discourse-reasoning task are useful contributions. However, the significance is substantially limited by the very small development set (7 files), the absence of confidence intervals or significance tests, and internally contradictory results between the embedded conference abstract and the body of the paper. The Judge-LLM framework is proposed but not yet evaluated, so its value remains untested.

major comments (3)
  1. The embedded ISDM2026 conference abstract (§'Accepted Abstract,' 'Preliminary results') reports Pearson correlations of (0.83, 0.80, 0.64) and Spearman correlations of (0.81, 0.78, 0.61) for the three general-domain models, and identifies Mistral7b as the best model ('Mistral7b outperformed the other two Gemma3:12b and Llama3.1:8b by 4 consensus'). Table 4 in the body reports completely different values: Pearson (0.513, 0.180, 0.180) and Spearman (0.587, 0.045, 0.101), with Gemma3:12b as the best model. Table 2 also contradicts the abstract's model ranking (Mistral7b has the lowest total consensus at 15, vs. 20 for Gemma3:12b). These contradictions are never acknowledged or reconciled. The authors must either correct one set of results or explain the discrepancy (e.g., different computation methods, different subsets of files). As it stands, the reader cannot determine which results are,
  2. §4.2, Table 4: The unit of analysis for the correlation is unspecified. The caption states 'across OPTION12 items over 7 patient files,' which could mean 84 paired observations (7×12) or 7 file-level means. If n=7, the 95% CI for r=0.513 spans roughly [-0.31, 0.88], making the model ranking statistically meaningless. If n=84, the correlation is confounded by between-item difficulty variance (Items 1 and 8 have 0% agreement for all models), which inflates the apparent correlation without indicating useful per-item predictive power. The paper must specify the unit of analysis, report confidence intervals, and state whether the correlation is computed at the item-level within each file (and if so, how between-file and between-item variance are handled).
  3. The Judge-LLM consensus framework is described as a contribution and is central to the methodology (Figure 1, §3), but it is never evaluated. The paper states that the judge-sLLM will be deployed on the testing set (15 files) in future work, but no results from this framework are reported. Since the framework is listed as a highlighted contribution, the absence of any evaluation undermines its claimed novelty. The authors should either remove it from the contributions list (framing it as future work) or provide at least a preliminary evaluation on the development set.
minor comments (9)
  1. §4.1: The exclusion of 4 of 11 development files (3 for formatting failures, 1 for unsuitability) is described but the formatting failures are not characterized. Were these failures due to transcript formatting, prompt formatting, or model output parsing? This information would help readers assess reproducibility and whether the exclusions introduce systematic bias.
  2. Table 2: The per-item agreement rates show high variability across models and items (e.g., Item 7: Gemma 57.1%, Llama 0%, Mistral 71.4%). The text interprets this as evidence for model complementarity, but with only 7 files per cell, each percentage represents a difference of ~1 file. The authors should note this explicitly.
  3. §4.3.2: The claim that 'OPTION items 2 to 5 never have 0% scores' is presented as a finding, but the table shows Item 5 has rates of 14.3%, 28.6%, 28.6% — these are low and the distinction from 0% (which is 0/7 vs. 1/7) is not meaningful at this sample size.
  4. §5.3: The explanation that general-domain models outperform medical-domain models because 'OPTION12 coding is not primarily medical knowledge' is plausible but speculative. No quantitative evidence (e.g., comparing the types of errors made by medical vs. general models) is provided to support this claim.
  5. The abstract states 'Gemma3:12b achieves the strongest agreement with human annotations (Pearson r=0.51, Spearman ρ=0.59),' while Table 4 reports r=0.513 and ρ=0.587. These are consistent but the rounding should be standardized.
  6. Figure 1 is referenced but difficult to parse; the text in the figure is small and the flow is not immediately clear. Consider redesigning for readability.
  7. §4.3.1, Figures 2–4: The hallucination examples for Meditron and MedLlama are informative but very long. Consider summarizing the key issue in the main text and moving full outputs to an appendix.
  8. The paper uses both 'OPTION12' and 'OPTION 12' (with a space) inconsistently throughout. Standardize to one form.
  9. Table 6 (Appendix A): The speaker-level statistics are presented but the analysis is cursory ('we can not make a conclusion on the impact of speaker turns on SDM scores'). Either expand this analysis or remove the table.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major points: (1) contradictions between the embedded ISDM2026 abstract and the body of the paper, (2) unspecified unit of analysis and missing confidence intervals for the correlation results, and (3) the Judge-LLM framework being listed as a contribution without evaluation. We agree that all three points require revision. For (1), we will reconcile the abstract and body results by explaining that the abstract reported early preliminary correlations computed on a different subset of files before preprocessing exclusions, and we will add an explicit reconciliation note. For (2), we will specify the unit of analysis, report confidence intervals, and discuss the statistical limitations honestly. For (3), we will reframe the Judge-LLM framework as proposed future work rather than a contribution of the current paper. We cannot fully resolve the fundamental limitation of the small sample size (n=7), but we will be transparent about its implications.

read point-by-point responses

  1. Referee: The embedded ISDM2026 conference abstract reports Pearson correlations of (0.83, 0.80, 0.64) and Spearman correlations of (0.81, 0.78, 0.61) for the three general-domain models, and identifies Mistral7b as the best model. Table 4 in the body reports completely different values: Pearson (0.513, 0.180, 0.180) and Spearman (0.587, 0.045, 0.101), with Gemma3:12b as the best model. Table 2 also contradicts the abstract's model ranking. These contradictions are never acknowledged or reconciled.

    Authors: The referee is correct, and we acknowledge this as a genuine oversight. The ISDM2026 abstract was submitted based on early preliminary results computed on an initial subset of development files before preprocessing exclusions (3 files removed due to formatting failures, 1 removed as unsuitable for SDM coding). The correlations reported in the abstract were computed on a different and larger subset of files than the 7 files that survived all preprocessing for the full analysis reported in the body. After excluding those 4 files, the correlations changed substantially, and the model ranking also shifted. We will add an explicit reconciliation note in the revised manuscript explaining that the abstract reports preliminary results on a different subset, and that the body of the paper supersedes the abstract with the final, corrected results on the 7 properly preprocessed files. We will also verify that the abstract's claim that 'Mistral7b outperformed the other two by 4 consensus' referred to an earlier consensus-counting method that was subsequently refined. We agree this should have been acknowledged explicitly from the outset. revision: yes

  2. Referee: The unit of analysis for the correlation in Table 4 is unspecified. The caption states 'across OPTION12 items over 7 patient files,' which could mean 84 paired observations (7x12) or 7 file-level means. If n=7, the 95% CI for r=0.513 spans roughly [-0.31, 0.88], making the model ranking statistically meaningless. If n=84, the correlation is confounded by between-item difficulty variance. The paper must specify the unit of analysis, report confidence intervals, and state whether the correlation is computed at the item-level within each file and how between-file and between-item variance are handled.

    Authors: The referee is correct that the unit of analysis is unspecified and that this is a critical omission. In our computation, the correlation was calculated across the 84 paired observations (7 files x 12 items), treating each file-item score as a single observation. We agree with the referee that this approach confounds between-item difficulty variance with within-file predictive power, and that the n=7 file-level alternative would yield very wide confidence intervals. In the revision, we will: (1) explicitly state the unit of analysis, (2) report 95% confidence intervals for all reported correlations, (3) acknowledge that with n=7 files the statistical power is insufficient for meaningful model ranking, and (4) discuss the limitation that item-level correlations are inflated by between-item difficulty variance. We will also add a note that these results should be interpreted as preliminary pilot findings rather than statistically robust comparisons. We cannot fully resolve the fundamental limitation of small sample size within the scope of this development-phase pilot, but we will be transparent about what the correlations do and do not support. revision: yes

  3. Referee: The Judge-LLM consensus framework is described as a contribution and is central to the methodology (Figure 1, Section 3), but it is never evaluated. The paper states that the judge-sLLM will be deployed on the testing set in future work, but no results from this framework are reported. Since the framework is listed as a highlighted contribution, the absence of any evaluation undermines its claimed novelty.

    Authors: The referee is correct. The Judge-LLM consensus framework is proposed but not evaluated in the current manuscript. We will remove it from the highlighted contributions list and reframe it as proposed future work. The methodology section will be revised to clarify that the framework is a design proposal for the testing phase, not a contribution of the current paper. We believe the design itself has value as a proposal, but we agree that listing it as a contribution without any evaluation overstates what the paper delivers. The contributions list will be revised to include only: (1) the first investigation of OS-sLLMs for OPTION12 coding, (2) the comparison of general-domain and medical-domain models, (3) the finding that medical models hallucinate more, (4) the error taxonomy, and (5) the item-level and correlation analysis on the development set. The Judge-LLM framework will be described as future work in both the methodology and conclusions sections. revision: yes

Circularity Check

0 steps flagged

No circularity: the paper evaluates LLM outputs against independently created human annotations using standard metrics.

full rationale

The paper evaluates OS-sLLM predictions against independently created human consensus annotations (double-coded by two researchers who resolved disagreements). The evaluation metrics (exact agreement counts, Pearson/Spearman correlation) are computed against this external ground truth. Few-shot examples drawn from the development set are standard practice and do not constitute circularity, as the evaluation metric (correlation with held-out human scores) is not the same quantity as the few-shot input. No self-citation chain is used to define the evaluation framework or metrics. The OPTION12 instrument is an externally validated tool (Elwyn et al., 2005). The Judge-LLM framework is proposed as a future method, not used to define the current results. While the paper has legitimate correctness risks (small sample size of 7 files, contradictory values between the conference abstract and body, unspecified unit of analysis for correlation), these are methodological and reporting concerns, not circularity. The derivation chain from human annotations → model evaluation → correlation metrics is self-contained and does not reduce to its own inputs by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 1 invented entities

The paper introduces several tuned parameters (temperature, context length, prompts) without reporting their values. The Judge-LLM framework is an invented entity without independent validation. The assumption that general-domain models have better discourse reasoning is ad hoc.

free parameters (4)
  • Temperature setting = Not specified
    Temperature is mentioned as a tuned parameter in §3 but the value used is not reported.
  • Context length = Not specified
    Context length is mentioned as optimized in §3 but the value is not reported.
  • Chain-of-thought prompt template = Not specified
    CoT prompting is used but the exact prompt template is not provided.
  • Few-shot examples = From development set
    Few-shot examples are selected from the development set for the test phase, but the selection criteria and number of examples are not specified.
axioms (3)
  • domain assumption OPTION12 is a valid instrument for measuring shared decision making
    The paper relies on the OPTION12 framework as a valid gold standard. This is supported by prior literature (Elwyn et al. 2003, 2005) cited in §2.1.
  • domain assumption Human consensus coding is the correct ground truth
    The paper treats human-agreed consensus scores as the reference standard for evaluating LLM outputs (§3, §4.1). Inter-rater reliability between the two human coders is not reported.
  • ad hoc to paper General-domain LLMs have better discourse reasoning than medical-domain LLMs
    §5.3 proposes this as an explanation for why medical models fail, but it is not independently verified. It is a post-hoc rationalization.
invented entities (1)
  • Judge-LLM consensus framework no independent evidence
    purpose: A framework where one LLM resolves disagreements among multiple LLM annotators, mimicking human consensus meetings.
    The Judge-LLM framework is proposed in §3 and described as a contribution, but it has not been tested or validated on any data. No falsifiable prediction is made about its performance.

pith-pipeline@v1.1.0-glm · 16672 in / 2583 out tokens · 314776 ms · 2026-07-08T15:45:54.632306+00:00 · methodology

0 comments
read the original abstract

We present LLM4SDM, the first study of open-source smaller language models (OS-sLLMs) for automated assessment of shared decision making (SDM) using the Observer OPTION12 framework. Unlike previous work that relies on large commercial models and the shorter OPTION5 instrument, our study focuses on privacy-preserving locally deployable models and Dutch melanoma consultation transcripts. Using expert-annotated clinical consultations, we evaluate three general-domain and two medical-domain OS-sLLMs during a development-phase pilot study. Results show that general-domain models outperform medical-domain models, which exhibit substantial hallucination and instruction-following failures. Gemma3:12b achieves the strongest agreement with human annotations (Pearson r=0.51, Spearman \r{ho}=0.59). Item-level and qualitative analyses reveal systematic challenges related to temporal discourse reasoning, conversational role attribution, and evidence grounding. We further introduce a Judge-LLM consensus framework designed to support disagreement resolution among multiple models. Our findings suggest that while current OS-sLLMs cannot replace human annotators, they offer a promising foundation for privacy-preserving human-in-the-loop SDM assessment.

Figures

Figures reproduced from arXiv: 2607.06127 by Anne Stiggelbout, Carly Heipon, David Lindevelt, Lifeng Han, Suzan Verberne, Tamara Wit.

Figure 1
Figure 1. Figure 1: An overview of LLM4SDM framework: upper layer for development phase and lower layer for model testing. split this data set into development/validation and test sets (11, 15). To investigate both general and medi￾cal domain OS-LLMs, we used the follow￾ing comparable-sized lightweight models: 1) general domain pre-trained Gemma3:12b, Llama3.1:8b, and Mistral7b, and 2) medical domain finetuned models MedLlama… view at source ↗
Figure 2
Figure 2. Figure 2: Meditron Model Hallucination Example - repeated evidence and justification across pre￾dicted OPTIOIN12 items using the same score 3. {'item_number': 10, 'item_description': 'Hoeveel pijn voel je?', 'score': 0, 'evidence': 'verbal', 'justification': 'Asking the patient how much pain he ' 'feels is a standard question to ask in ' 'order to assess his general condition. ' 'A score of 1 indicates that the pati… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.