Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Keeping Experts in the Loop: Expert-Guided Optimization for Clinical Data Classification using Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By selecting the lowest-confidence notes for expert review, StructEase lifts clinical-note classification macro-F1 from 0.935 to 0.986 while keeping a clinician in the loop.

desk verdict Useful expert-in-the-loop framework with open code, but the headline gain over DSPy is label-unequal and the bias analysis is unreported. read the letter →

arxiv 2412.02173 v1 pith:QB4PZ5QJ submitted 2024-12-03 cs.AI

Nader Karayanni , Aya Awwad , Chein-Lien Hsiao , Surish P Shanmugam This is my paper · ORCID
classification cs.AI
keywords expert-in-the-looppromptoptimizationlargelanguagemodelsclinicaltextclassificationlow-confidencesamplingchain-of-thoughtpromptinghelmet-usedetectionNEISSnarratives
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the main barrier to using LLMs on unstructured clinical notes is prompt engineering, not model capability, and that the right remedy is to keep a clinician in the loop with a deliberately small number of corrections. The proposed framework, StructEase, iterates a Chain-of-Thought classification prompt: it samples a 10% batch of unlabeled notes, has the model classify them, ranks the notes by token-level confidence, sends the ten least-confident notes per class to a physician for correction, and rewrites the prompt around those corrections. On a dataset of 17,888 de-identified emergency-department narratives about micromobility injuries, two such rounds raise macro-F1, the balanced average of F1 across classes, from 0.935 at the initial prompt to 0.986 at the second revision. The paper's core claim is that the sampling rule, SamplEase, is what makes the expert effort count: selecting low-confidence notes yields a statistically significant F1 advantage over random sampling. A sympathetic reader should care because, if the claim holds, clinical teams could get near-expert classification from an LLM with tens of labels and no prompt-engineering expertise.

What carries the argument

The carrying mechanism is SamplEase, a sampling-and-refinement loop built on a geometric-mean token confidence score, $\exp\left(\frac{1}{n}\sum_{i=1}^n \log p_i\right)$, computed per note from the LLM completion's token log-probabilities. In each iteration the algorithm takes a 10% random sample of the still-unlabeled data, classifies it with the current prompt $P_i$, and selects the ten lowest-confidence notes per class for expert labeling; mismatched predictions become few-shot examples, and an LLM call rewrites the prompt as $P_{i+1}$. The confidence score is the load-bearing piece because it is what claims to find high-value cases where expert feedback drives performance improvements. The prompt rewrite is the second half: it turns those few-shot corrections into an updated Chain-of-Thought prompt rather than retraining the model.

What would settle it

Run the framework with the prompt-update mechanism held fixed but the expert's notes drawn uniformly at random rather than by lowest confidence, matched for the same number of labels per class; if the median macro-F1 gap between the two arms is not statistically significant across repeated runs, the confidence-ranking mechanism is not the causal driver.

Watch

Extended reading notes

Core claim

StructEase's central discovery, stated on the paper's own terms, is that a small number of expert corrections, chosen by an LLM's own uncertainty signal and folded back into a rewritten prompt, can outperform both fully manual and fully automated prompt engineering for clinical-text classification. The authors report macro-F1 of 0.986 (95% CI 0.972–0.997) with the second prompt iteration P2, compared with 0.980 for the strongest automated baseline, 0.962 for a Chain-of-Thought classifier, and 0.735 for a human-authored prompt. Per-class analysis shows the method most helps the class that starts weakest: the 'No Helmet' class advances from F1 0.88 at P0 to 0.98 at P2. The authors attribute this to SamplEase, whose confidence-based selection of ten low-confidence notes per class reduces labeling redundancy and concentrates expert effort on ambiguous cases.

Load-bearing premise

The entire framework rests on trusting that the language model's token-level confidence score identifies the notes whose expert corrections will most improve the prompt; if that ranking is no better than random selection, the reported F1 advantage disappears.

Editorial extensions

If this is right

  • A clinician with no prompt-engineering training can drive an LLM classifier to macro-F1 0.986 on helmet-status extraction after two rounds of reviewing at most 30 notes per round.
  • The second-round prompt beats both the human-authored prompt (macro-F1 0.735) and the strongest fully automated prompt optimizer (0.980), without fine-tuning the model.
  • SamplEase's one-iteration median macro-F1 of 0.974 significantly exceeds random sampling's 0.959 (P = 0.044), so the confidence-based selection rule adds value beyond label quantity.
  • The largest per-class gain appears where the baseline is weakest: the 'No Helmet' class F1 rises from 0.88 at P0 to 0.98 at P2.
  • Macro-level and per-class metrics remain similar across gender and racial groups in this task, indicating no substantial demographic performance disparity in the reported runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not separate the prompt-rewrite effect from the effect of adding the new expert labels; an ablation that adds the same corrections to a fixed prompt would tell whether the rewrite step is essential.
  • A testable extension is to compare SamplEase's geometric-mean token confidence with other uncertainty signals, such as predictive-entropy sampling or self-consistency, to see whether the specific score is the best selector.
  • Because the selection budget is a flat ten notes per class, rare classes with few examples in the random 10% batch may receive little attention; a class-frequency-aware quota would be a direct modification worth testing on higher-cardinality label sets.
  • If the confidence signal is as reliable as the results suggest, the same score could be reused at deployment to flag uncertain notes for human review, not just to refine prompts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes StructEase, a framework for expert-in-the-loop prompt optimization for LLM-based classification of unstructured clinical notes. StructEase starts from a chain-of-thought prompt P0, uses the proposed SamplEase algorithm to select low-confidence examples per class for expert labeling, updates the prompt with expert-corrected few-shot examples, and iterates. The authors evaluate on 17,888 NEISS micromobility injury narratives, with the task of classifying helmet use into three classes, using 2,000 expert-labeled notes as ground truth. They report macro-F1 improving from 0.935 (P0) to 0.973 (P1) to 0.986 (P2), claim that SamplEase outperforms random sampling (median macro-F1 0.974 vs. 0.959, P=0.044), and compare with human prompts and DSPy baselines in Table 1. The paper argues that StructEase achieves 'significant gains in classification performance compared to current methods' while requiring minimal expert effort.

Significance. The manuscript addresses a real and timely problem: how to inject domain expertise into LLM prompt optimization without burdening experts. The framework is concrete, the implementation is open-source, and the evaluation uses a substantial expert-labeled clinical dataset that is not published, reducing leakage concerns. The held-out evaluation protocol, with training few-shot examples excluded from the test set, is a strength. If the comparative claims can be supported under matched label budgets, StructEase would be a useful contribution to human-in-the-loop prompt optimization for clinical text. At present, however, the headline comparison to DSPy Optimized is confounded by unequal label budgets, and the sampling ablation does not isolate the proposed confidence-based selection mechanism, so the significance of the central claims is not yet established.

major comments (4)
  1. [Table 1, Comparison with Baselines] The headline comparison is not label-matched. P2 consumes 60 expert labels (10 per class per iteration for two iterations), while the DSPy Optimized baseline is run with only 30 randomly labeled samples. At the matched 30-label budget, StructEase P1 achieves macro-F1 0.973, which is below DSPy Optimized's 0.980. The abstract's claim of 'significant gains in classification performance compared to current methods' is therefore not supported by the current experimental design. The authors should add fixed-label-budget comparisons (e.g., DSPy Optimized with 60 labels, or P2 with only one iteration) and report confidence intervals for all baselines so that the comparison is meaningful.
  2. [Comparing SamplEase vs. Random Sampling, Figure 3] The sampling comparison conflates confidence-based selection with per-class stratification. SamplEase selects exactly 10 lowest-confidence examples for each class, while the random sampling baseline appears to draw an unstratified 30-note sample. The observed median improvement (0.974 vs. 0.959) could be driven by ensuring class coverage rather than by the token-level confidence heuristic in Equation (1). To support the claim that SamplEase's confidence signal is the operative mechanism, the authors should compare against a class-stratified random sampling baseline and, ideally, an ablation that selects random examples within each class.
  3. [Evaluation Experiments and Performance Metrics] The statistical basis for the main results is weak. Only six runs are performed, and the model is run at temperature 0 with top-p 1, which the paper itself states ensures selection of the most probable token; repeated runs may therefore be near-deterministic, so bootstrap confidence intervals aggregated across runs may not reflect genuine sampling variability. In addition, no confidence interval or significance test is reported for the P2 versus DSPy Optimized comparison in Table 1, despite the paper's language of 'significant gains.' The authors should clarify what varies across runs and provide a direct statistical comparison of P2 against DSPy Optimized at a matched budget.
  4. [Bias Evaluation] The bias evaluation section reports no quantitative results. The claim that the framework performs 'consistently across different demographic groups' with 'minimal expected variation' is unsupported without per-group macro-F1 values, confidence intervals, or statistical tests. Since the section is presented as one of the four evaluation experiments, the authors should include a table with the actual metric values for each gender and racial category.
minor comments (6)
  1. [Algorithm 1] If a class is absent from the 10% random sample drawn in line 4, the per-class selection in line 8 will produce no examples for that class; the algorithm should specify a fallback or a resampling step to guarantee coverage.
  2. [Equation (1)] The paper does not explain how the token-level log probabilities Pi are obtained from the OpenAI API; if the logprobs feature is used, this should be stated explicitly.
  3. [Figure 3] Figure 3 shows individual run points and medians but no confidence intervals; with n=6 per group, a boxplot or full distribution would be more informative than medians alone.
  4. [Framework Implementation] The GitHub link points to 'StructurEase' while the method is called 'StructEase'; the naming should be made consistent.
  5. [Data and Task Definition] The statement that disagreements were 'resolved through discussion until a 100% inter-rater agreement was reached' would benefit from reporting the initial agreement rate, since that is important for assessing label quality.
  6. [Methods, Framework Implementation] The phrase 'We annotate the prompt generation methods as follows' appears to be a typo; the authors likely mean 'denote' or 'define.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: StructEase's expert-guided prompt refinement is evaluated on held-out notes, and no parameter is fitted to the test set; the label-budget mismatch with DSPy Optimized is a fairness-of-comparison issue, not a circularity issue.

full rationale

The claimed derivation chain is self-contained against external ground truth. The 2,000 expert-labeled notes are held out: "The framework was evaluated using 2,000 expert-labeled notes (ground truth), excluding the training data used to refine the prompt." Few-shot corrections fed into the prompt update are mismatched examples drawn from the expert-reviewed sample, and the evaluation excludes those samples; no test-set labels enter Equation 1 or Algorithm 1. Equation 1 is a standard token-level likelihood confidence score, not defined in terms of the final F1, and the "prediction" is the macro-F1 computed from held-out classifications, not a fitted quantity. No load-bearing argument reduces to a self-citation: the cited baselines (DSPy, PromptAgent) are external, and the paper's unique components (SamplEase, iterative expert feedback) are implemented and tested here. One genuine evaluation limitation is that the DSPy Optimized baseline is run with 30 randomly labeled samples while P2 consumes 60 expert labels ("We ran this version with 30 randomly labeled samples to benchmark its performance"), so the headline P2 vs. DSPy Optimized comparison is not label-matched; at the 30-label budget P1 (0.973) is numerically below DSPy Optimized (0.980). That is a confound in the comparative claim, not a circular derivation, because the framework's outputs are not defined by, or fitted to, the baseline numbers.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The framework has no fitted numerical coefficients in a physical sense, but several hyperparameters are hand-chosen and not optimized. The core assumptions concern the reliability of LLM confidence for sample selection and the ability of prompt rewriting to generalize from few examples.

free parameters (4)
  • sample_fraction = 0.10
    In Algorithm 1, a random 10% pool is sampled in each iteration; this value is chosen by hand and not swept.
  • k_per_class = 10
    The algorithm selects the 10 lowest-confidence notes per class for expert review; no sensitivity analysis is provided.
  • iterations = 2 (P2)
    The framework was run for two prompt updates in the main evaluation; the stopping criterion is left to the expert.
  • model = gpt-4o-mini-2024-07-18
    Results depend on this specific model; no other models were tested.
assumptions (3)
  • domain assumption LLM token log-probabilities are a valid confidence measure for classification correctness.
    Equation 1 computes confidence from per-token log probabilities, and Algorithm 1 uses it to select samples; this is a standard active learning assumption that is not tested here.
  • domain assumption An LLM can rewrite a classification prompt from a handful of misclassified examples to improve accuracy on the broader dataset.
    The UpdatePrompt function Section 'Proposed Framework' relies on this; no ablation is provided to confirm the prompt rewrite, rather than the added examples, is the cause of improvement.
  • domain assumption The 2,000 expert-labeled notes are representative of the 17,888-note dataset.
    The paper states they were randomly selected, but the evaluation set is used for both prompt refinement (after removing few-shots) and test, and no distribution comparison is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Keeping Experts in the Loop: Expert-Guided Optimization for Clinical Data Classification using Large Language Models." pith.science (2026). https://pith.science/paper/QB4PZ5QJ

@misc{pith2026241202173,
  author       = {Pith},
  title        = {Pith review of: Keeping Experts in the Loop: Expert-Guided Optimization for Clinical Data Classification using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QB4PZ5QJ}},
  note         = {Machine review of arXiv:2412.02173}
}
read the original abstract

Since the emergence of Large Language Models (LLMs), the challenge of effectively leveraging their potential in healthcare has taken center stage. A critical barrier to using LLMs for extracting insights from unstructured clinical notes lies in the prompt engineering process. Despite its pivotal role in determining task performance, a clear framework for prompt optimization remains absent. Current methods to address this gap take either a manual prompt refinement approach, where domain experts collaborate with prompt engineers to create an optimal prompt, which is time-intensive and difficult to scale, or through employing automatic prompt optimizing approaches, where the value of the input of domain experts is not fully realized. To address this, we propose StructEase, a novel framework that bridges the gap between automation and the input of human expertise in prompt engineering. A core innovation of the framework is SamplEase, an iterative sampling algorithm that identifies high-value cases where expert feedback drives significant performance improvements. This approach minimizes expert intervention, to effectively enhance classification outcomes. This targeted approach reduces labeling redundancy, mitigates human error, and enhances classification outcomes. We evaluated the performance of StructEase using a dataset of de-identified clinical narratives from the US National Electronic Injury Surveillance System (NEISS), demonstrating significant gains in classification performance compared to current methods. Our findings underscore the value of expert integration in LLM workflows, achieving notable improvements in F1 score while maintaining minimal expert effort. By combining transparency, flexibility, and scalability, StructEase sets the foundation for a framework to integrate expert input into LLM workflows in healthcare and beyond.

Figures

Figures reproduced from arXiv: 2412.02173 by the authors.

Figure 1
Figure 1. The proposed framework of StructEase Pi = U pdateP rompt(Pi−1, F ew Shots) Few Shots are the notes that the prompt Pi−1 classified incorrectly compared to the expert labels. The above methods use an LLM call to generate the new prompt. Choosing Data for Expert Review A critical challenge in designing StructEase is selecting data samples that max￾imize the value of expert-labeled data. This is particularly difficult … view at source ↗
Figure 2
Figure 2. Aggregated performance metrics across iterative prompt refinements ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of performance metrics between [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 17 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    G.; Itzkowitz, N

    Burford, K. G.; Itzkowitz, N. G.; Ortega, A. G.; Teitler, J. O.; and Rundle, A. G. 2024. Use of generative AI to identify helmet status among patients with micromobility-related injuries from unstructured clinical notes. JAMA Network Open, 7(8): e2425981--e2425981

    work page 2024

  4. [4]

    Cleary, D. 2024. A Complete Guide to Meta-Prompting. Accessed: 2024-11-28

    work page 2024

  5. [5]

    Ding, N.; Hu, S.; Zhao, W.; Chen, Y.; Liu, Z.; Zheng, H.-T.; and Sun, M. 2021. Openprompt: An open-source framework for prompt-learning. arXiv preprint arXiv:2111.01998

    arXiv 2021

  6. [6]

    Duan, Y. 2024. The Large Language Model (LLM) Bias Evaluation (Age Bias). DIKWP Research Group International Standard Evaluation. DOI, 10

    work page 2024

  7. [7]

    L.; Franco, I.; Kann, B

    Guevara, M.; Chen, S.; Thomas, S.; Chaunzwa, T. L.; Franco, I.; Kann, B. H.; Moningi, S.; Qian, J. M.; Goldstein, M.; Harper, S.; et al. 2024. Large language models to identify social determinants of health in electronic health records. NPJ digital medicine, 7(1): 6

    work page 2024

  8. [8]

    M.; Rong, R.; Nezafati, K.; Treager, C.; Chi, Z.; Wang, S.; Cheng, X.; Guo, Y.; Klesse, L

    Huang, J.; Yang, D. M.; Rong, R.; Nezafati, K.; Treager, C.; Chi, Z.; Wang, S.; Cheng, X.; Guo, Y.; Klesse, L. J.; et al. 2024. A critical assessment of using ChatGPT for extracting structured data from clinical notes. npj Digital Medicine, 7(1): 106

    work page 2024

Show all 26 references
  1. [9]

    S.; Kazerooni, E

    Jabbour, S.; Fouhey, D.; Shepard, S.; Valley, T. S.; Kazerooni, E. A.; Banovic, N.; Wiens, J.; and Sjoding, M. W. 2023. Measuring the impact of AI in the diagnosis of hospitalized patients: a randomized clinical vignette survey study. Jama, 330(23): 2275--2284

    2023

  2. [10]

    J.; Jaffe, I

    Jethani, N.; Jones, S.; Genes, N.; Major, V. J.; Jaffe, I. S.; Cardillo, A. B.; Heilenbach, N.; Ali, N. F.; Bonanni, L. J.; Clayburn, A. J.; et al. 2023. Evaluating ChatGPT in information extraction: a case study of extracting cognitive exam dates and scores

    2023

  3. [11]

    L.; Hall, D.; Liang, P.; Potts, C.; and Zaharia, M

    Khattab, O.; Santhanam, K.; Li, X. L.; Hall, D.; Liang, P.; Potts, C.; and Zaharia, M. 2022. Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive NLP . arXiv preprint arXiv:2212.14024

    2022 arXiv

  4. [12]

    T.; Moazam, H.; Miller, H.; Zaharia, M.; and Potts, C

    Khattab, O.; Singhvi, A.; Maheshwari, P.; Zhang, Z.; Santhanam, K.; Vardhamanan, S.; Haq, S.; Sharma, A.; Joshi, T. T.; Moazam, H.; Miller, H.; Zaharia, M.; and Potts, C. 2024. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

    2024

  5. [13]

    Landi, H. 2024. Microsoft’s Nuance integrates OpenAI’s GPT-4 into voice-enabled medical scribe software. Accessed: 2024-11-29

    2024

  6. [14]

    Merkel, D. 2014. Docker: lightweight linux containers for consistent development and deployment. Linux journal, 2014(239): 2

    2014

  7. [15]

    Mesk \'o , B. 2023. Prompt engineering as an important emerging skill for medical professionals: tutorial. Journal of medical Internet research, 25: e50638

    2023

  8. [16]

    OpenAI. 2024. GPT-4o-Mini Documentation. https://platform.openai.com/docs/models. Accessed: 2024-11-28

    2024

  9. [17]

    gradient descent

    Pryzant, R.; Iter, D.; Li, J.; Lee, Y. T.; Zhu, C.; and Zeng, M. 2023. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495

    2023 arXiv

  10. [18]

    Ruksha, K. 2024. Prompt Engineering: Classification of Techniques and Prompt Tuning. https://medium.com/the-modern-scientist/prompt-engineering-classification-of-techniques-and-prompt-tuning-6d4247b9b64c

    2024

  11. [19]

    Systems, E. 2024. Epic and Microsoft Bring GPT-4 to EHRs. Accessed: 2024-11-29

    2024

  12. [20]

    Toner-Rodgers, A. 2024. Artificial Intelligence, Scientific Discovery, and Product Innovation

    2024

  13. [21]

    Consumer Product Safety Commission

    U.S. Consumer Product Safety Commission . 2024. NEISS Injury Data. Accessed: November 29, 2024

    2024

  14. [22]

    P.; and Hu, Z

    Wang, X.; Li, C.; Wang, Z.; Bai, F.; Luo, H.; Zhang, J.; Jojic, N.; Xing, E. P.; and Hu, Z. 2023. PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization. arXiv preprint arXiv:2310.16427

    2023 arXiv

  15. [23]

    Wang, Y.; Wang, L.; Rastegar-Mojarad, M.; Moon, S.; Shen, F.; Afzal, N.; Liu, S.; Zeng, Y.; Mehrabi, S.; Sohn, S.; et al. 2018. Clinical information extraction applications: a literature review. Journal of biomedical informatics, 77: 34--49

    2018

  16. [24]

    Wong, K.; Paritosh, P.; and Bollacker, K. 2022. Are ground truth labels reproducible? An empirical study. In Proceedings of ML Evaluation Standards Workshop at ICLR, 25--29

    2022

  17. [25]

    Differentiation

    Yuksekgonul, M.; Bianchi, F.; Boen, J.; Liu, S.; Huang, Z.; Guestrin, C.; and Zou, J. 2024. TextGrad: Automatic" Differentiation" via Text. arXiv preprint arXiv:2406.07496

    2024 arXiv

  18. [26]

    J.; Jaffe, I

    Zhang, H.; Jethani, N.; Jones, S.; Genes, N.; Major, V. J.; Jaffe, I. S.; Cardillo, A. B.; Heilenbach, N.; Ali, N. F.; Bonanni, L. J.; et al. 2024. Evaluating large language models in extracting cognitive exam dates and scores. medRxiv, 2023--07

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.