{"id":"78b03ad6-746a-472c-8b7b-151389a5de13","arxiv_id":"2412.02173","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An expert-in-the-loop framework with per-class uncertainty sampling improves LLM classification of clinical notes more than random sampling and most automated baselines.","lead":"StructEase is a framework that lets a medical expert shape an LLM's classification prompt by reviewing a small set of deliberately chosen notes. On a helmet-use classification task from emergency room notes, it reports higher F1 scores than random sampling and automated optimization baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline comparison to DSPy Optimized is confounded by label budget: P2 used 60 expert labels, DSPy Optimized only 30; at equal 30-label budget P1 (0.973) is below DSPy (0.980), so 'significant gains' may reflect extra labels, not the mechanism.","rationale":"The reader identified token-level confidence as the weakest assumption and also mentioned unequal label budgets in the rationale. My stress-test pass finds that the unequal label budget is the more immediately load-bearing flaw: it directly undermines the headline numeric superiority claim. Even if SamplEase's confidence scores are a good uncertainty signal, Table 1 cannot support 'significant gains compared to current methods' when P2 consumed twice as many labels as the DSPy Optimized baseline, and P1 at the same label budget is numerically worse. The paper does report a SamplEase-versus-random comparison at P1, which is a useful check, but that comparison may itself be confounded by class stratification if the random baseline was unstratified. The central contribution is about expert-guided optimization, so the empirical claim must be evaluated at matched label budgets and with a stratification-matched random control. I therefore agree with the reader's conditional verdict, but for a somewhat different primary reason: the comparison against DSPy Optimized is not yet a fair test of the claimed gains. A concrete label-matched experiment would settle whether the concern is fatal or whether StructEase's advantage survives once budgets are equated.","tokens_in":8417,"tokens_out":5269,"duration_ms":60371,"concrete_test":"Rerun DSPy Optimized on the same 2,000-note evaluation set with class-stratified 30-label and 60-label budgets, using at least six independent runs and reporting bootstrap 95% CIs for macro-F1. If DSPy Optimized at 60 labels reaches or exceeds 0.986, or if class-stratified random sampling at 30 labels reaches or exceeds P1's 0.973, the claimed superiority of the expert-guided mechanism is not established. Also rerun the Figure 3 random-sampling baseline as class-stratified random sampling to isolate the confidence-selection component from class balancing.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Table 1's headline result is StructEase P2 at macro-F1 0.986 versus DSPy Optimized at 0.980. The conditions are not label-matched. Algorithm 1 requests 10 lowest-confidence notes per class per iteration, so P1 uses 30 expert labels (10 × 3 classes) and P2 uses 60, while the DSPy Optimized baseline was run with only 30 randomly labeled samples. This is not an accounting nit: at the matched 30-label budget, StructEase P1 achieves 0.973 (95% CI 0.951–0.992), which is numerically below DSPy Optimized's 0.980. The abstract's claim of 'significant gains in classification performance compared to current methods' therefore rests on comparing a method that consumed twice as many expert labels against a baseline given half that budget. Because the paper's stated advantage is 'minimal expert effort,' the appropriate comparison is at fixed label budget or fixed expert time. In addition, SamplEase's per-class selection stratifies by class, and the 'random sampling' baseline in Figure 3 may be class-unstratified, which conflates confidence-based selection with simple class balancing. The current evidence does not separate the contribution of the token-level confidence heuristic, the per-class stratification, the prompt-update mechanism, or simply the larger and differently distributed label set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes StructEase, a framework for expert-in-the-loop prompt optimization for LLM-based classification of unstructured clinical notes. StructEase starts from a chain-of-thought prompt P0, uses the proposed SamplEase algorithm to select low-confidence examples per class for expert labeling, updates the prompt with expert-corrected few-shot examples, and iterates. The authors evaluate on 17,888 NEISS micromobility injury narratives, with the task of classifying helmet use into three classes, using 2,000 expert-labeled notes as ground truth. They report macro-F1 improving from 0.935 (P0) to 0.973 (P1) to 0.986 (P2), claim that SamplEase outperforms random sampling (median macro-F1 0.974 vs. 0.959, P=0.044), and compare with human prompts and DSPy baselines in Table 1. The paper argues that StructEase achieves 'significant gains in classification performance compared to current methods' while requiring minimal expert effort.","tokens_in":8750,"tokens_out":4139,"duration_ms":44399,"significance":"The manuscript addresses a real and timely problem: how to inject domain expertise into LLM prompt optimization without burdening experts. The framework is concrete, the implementation is open-source, and the evaluation uses a substantial expert-labeled clinical dataset that is not published, reducing leakage concerns. The held-out evaluation protocol, with training few-shot examples excluded from the test set, is a strength. If the comparative claims can be supported under matched label budgets, StructEase would be a useful contribution to human-in-the-loop prompt optimization for clinical text. At present, however, the headline comparison to DSPy Optimized is confounded by unequal label budgets, and the sampling ablation does not isolate the proposed confidence-based selection mechanism, so the significance of the central claims is not yet established.","major_comments":[{"comment":"The headline comparison is not label-matched. P2 consumes 60 expert labels (10 per class per iteration for two iterations), while the DSPy Optimized baseline is run with only 30 randomly labeled samples. At the matched 30-label budget, StructEase P1 achieves macro-F1 0.973, which is below DSPy Optimized's 0.980. The abstract's claim of 'significant gains in classification performance compared to current methods' is therefore not supported by the current experimental design. The authors should add fixed-label-budget comparisons (e.g., DSPy Optimized with 60 labels, or P2 with only one iteration) and report confidence intervals for all baselines so that the comparison is meaningful.","section":"Table 1, Comparison with Baselines"},{"comment":"The sampling comparison conflates confidence-based selection with per-class stratification. SamplEase selects exactly 10 lowest-confidence examples for each class, while the random sampling baseline appears to draw an unstratified 30-note sample. The observed median improvement (0.974 vs. 0.959) could be driven by ensuring class coverage rather than by the token-level confidence heuristic in Equation (1). To support the claim that SamplEase's confidence signal is the operative mechanism, the authors should compare against a class-stratified random sampling baseline and, ideally, an ablation that selects random examples within each class.","section":"Comparing SamplEase vs. Random Sampling, Figure 3"},{"comment":"The statistical basis for the main results is weak. Only six runs are performed, and the model is run at temperature 0 with top-p 1, which the paper itself states ensures selection of the most probable token; repeated runs may therefore be near-deterministic, so bootstrap confidence intervals aggregated across runs may not reflect genuine sampling variability. In addition, no confidence interval or significance test is reported for the P2 versus DSPy Optimized comparison in Table 1, despite the paper's language of 'significant gains.' The authors should clarify what varies across runs and provide a direct statistical comparison of P2 against DSPy Optimized at a matched budget.","section":"Evaluation Experiments and Performance Metrics"},{"comment":"The bias evaluation section reports no quantitative results. The claim that the framework performs 'consistently across different demographic groups' with 'minimal expected variation' is unsupported without per-group macro-F1 values, confidence intervals, or statistical tests. Since the section is presented as one of the four evaluation experiments, the authors should include a table with the actual metric values for each gender and racial category.","section":"Bias Evaluation"}],"minor_comments":[{"comment":"If a class is absent from the 10% random sample drawn in line 4, the per-class selection in line 8 will produce no examples for that class; the algorithm should specify a fallback or a resampling step to guarantee coverage.","section":"Algorithm 1"},{"comment":"The paper does not explain how the token-level log probabilities Pi are obtained from the OpenAI API; if the logprobs feature is used, this should be stated explicitly.","section":"Equation (1)"},{"comment":"Figure 3 shows individual run points and medians but no confidence intervals; with n=6 per group, a boxplot or full distribution would be more informative than medians alone.","section":"Figure 3"},{"comment":"The GitHub link points to 'StructurEase' while the method is called 'StructEase'; the naming should be made consistent.","section":"Framework Implementation"},{"comment":"The statement that disagreements were 'resolved through discussion until a 100% inter-rater agreement was reached' would benefit from reporting the initial agreement rate, since that is important for assessing label quality.","section":"Data and Task Definition"},{"comment":"The phrase 'We annotate the prompt generation methods as follows' appears to be a typo; the authors likely mean 'denote' or 'define.'","section":"Methods, Framework Implementation"}],"recommendation":"major_revision","confidential_remarks":"The label-budget confound in Table 1 is the central issue, and it is fixable. If the authors add a matched-budget comparison and an unstratified-versus-stratified ablation for the sampling method, the manuscript could become a solid contribution. I am not recommending rejection because the framework itself is sensible, the open-source implementation is a concrete asset, and the held-out evaluation protocol is a strength. The manuscript would also benefit from a more careful statistical framing given the very small number of runs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. First, this is a solid, practical framework for expert-in-the-loop prompt refinement on clinical text classification, with open-source code and a coherent held-out evaluation on a real dataset. Second, the headline claim of 'significant gains over current methods' is undercut by a label-budget confound: StructEase P2 consumed 60 expert labels, while the DSPy Optimized baseline was given only 30. At the matched 30-label budget, StructEase P1 (macro-F1 0.973) is actually below DSPy Optimized (0.980). So the abstract's claim rests on an unbalanced comparison. The equal-budget SamplEase-vs-random experiment does show a real, albeit small, gain (median F1 0.974 vs 0.959, p=0.044), so the method has something to it.\n\nWhat is actually new: the combination of per-class lowest-confidence sampling with iterative LLM prompt rewriting is not present in the cited literature. The ingredients are known, but the integration is clean and the workflow is simple enough for a clinician to operate without prompt-engineering expertise. The evaluation is not circular: held-out notes, no parameters fitted to the test set, expert labels as ground truth. The code and containerization make the framework easy to adopt.\n\nWhere it is soft, in descending order of importance. The label-budget issue is the main one; the paper never compares at fixed expert effort, which is precisely what the 'minimal expert effort' claim requires. Second, the random-sampling baseline in Figure 3 may not be class-stratified, so the improvement attributed to confidence may actually come from class balancing. Third, the bias evaluation consists of one qualitative sentence with no numbers; that section is effectively unsubstantiated. Fourth, six independent runs per condition is thin, though the bootstrap confidence intervals mitigate that. The iterative improvement from P0 to P2 is statistically significant, so the direction of the effect is credible.\n\nWho is this for: people designing human-in-the-loop NLP systems in healthcare, and anyone building tools that let domain experts steer LLMs without writing prompts. It is a modest extension of known ideas, not a breakthrough. It deserves a serious referee: the problem is real, the code exists, and the equal-budget SamplEase result is worth checking. Recommendation: send it out, but only with a request that the authors fix the label-budget comparison, stratify the random baseline or explain why it was not stratified, and report actual numbers for the bias analysis. With those revisions, the paper would be a useful contribution to the clinical NLP literature.","headline":"Useful expert-in-the-loop framework with open code, but the headline gain over DSPy is label-unequal and the bias analysis is unreported.","tokens_in":9209,"tokens_out":3064,"would_cite":false,"duration_ms":30518,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By selecting the lowest-confidence notes for expert review, StructEase lifts clinical-note classification macro-F1 from 0.935 to 0.986 while keeping a clinician in the loop.","keywords":["expert-in-the-loop","prompt optimization","large language models","clinical text classification","low-confidence sampling","chain-of-thought prompting","helmet-use detection","NEISS narratives"],"falsifier":"Run the framework with the prompt-update mechanism held fixed but the expert's notes drawn uniformly at random rather than by lowest confidence, matched for the same number of labels per class; if the median macro-F1 gap between the two arms is not statistically significant across repeated runs, the confidence-ranking mechanism is not the causal driver.","tokens_in":8276,"feed_emoji":"🩺","tokens_out":12499,"duration_ms":116901,"temperature":0.7,"pith_summary":"This paper claims that the main barrier to using LLMs on unstructured clinical notes is prompt engineering, not model capability, and that the right remedy is to keep a clinician in the loop with a deliberately small number of corrections. The proposed framework, StructEase, iterates a Chain-of-Thought classification prompt: it samples a 10% batch of unlabeled notes, has the model classify them, ranks the notes by token-level confidence, sends the ten least-confident notes per class to a physician for correction, and rewrites the prompt around those corrections. On a dataset of 17,888 de-identified emergency-department narratives about micromobility injuries, two such rounds raise macro-F1, the balanced average of F1 across classes, from 0.935 at the initial prompt to 0.986 at the second revision. The paper's core claim is that the sampling rule, SamplEase, is what makes the expert effort count: selecting low-confidence notes yields a statistically significant F1 advantage over random sampling. A sympathetic reader should care because, if the claim holds, clinical teams could get near-expert classification from an LLM with tens of labels and no prompt-engineering expertise.","feed_headline":"Expert-guided prompt loop hits 0.986 F1 on clinical notes","feed_subtitle":"Physicians correct just a few low-confidence notes per round and beat fully automated prompt optimizers on helmet-status classification.","key_machinery":"The carrying mechanism is SamplEase, a sampling-and-refinement loop built on a geometric-mean token confidence score, $\\exp\\left(\\frac{1}{n}\\sum_{i=1}^n \\log p_i\\right)$, computed per note from the LLM completion's token log-probabilities. In each iteration the algorithm takes a 10% random sample of the still-unlabeled data, classifies it with the current prompt $P_i$, and selects the ten lowest-confidence notes per class for expert labeling; mismatched predictions become few-shot examples, and an LLM call rewrites the prompt as $P_{i+1}$. The confidence score is the load-bearing piece because it is what claims to find high-value cases where expert feedback drives performance improvements. The prompt rewrite is the second half: it turns those few-shot corrections into an updated Chain-of-Thought prompt rather than retraining the model.","core_discovery":"StructEase's central discovery, stated on the paper's own terms, is that a small number of expert corrections, chosen by an LLM's own uncertainty signal and folded back into a rewritten prompt, can outperform both fully manual and fully automated prompt engineering for clinical-text classification. The authors report macro-F1 of 0.986 (95% CI 0.972–0.997) with the second prompt iteration P2, compared with 0.980 for the strongest automated baseline, 0.962 for a Chain-of-Thought classifier, and 0.735 for a human-authored prompt. Per-class analysis shows the method most helps the class that starts weakest: the 'No Helmet' class advances from F1 0.88 at P0 to 0.98 at P2. The authors attribute this to SamplEase, whose confidence-based selection of ten low-confidence notes per class reduces labeling redundancy and concentrates expert effort on ambiguous cases.","pith_inferences":["The paper does not separate the prompt-rewrite effect from the effect of adding the new expert labels; an ablation that adds the same corrections to a fixed prompt would tell whether the rewrite step is essential.","A testable extension is to compare SamplEase's geometric-mean token confidence with other uncertainty signals, such as predictive-entropy sampling or self-consistency, to see whether the specific score is the best selector.","Because the selection budget is a flat ten notes per class, rare classes with few examples in the random 10% batch may receive little attention; a class-frequency-aware quota would be a direct modification worth testing on higher-cardinality label sets.","If the confidence signal is as reliable as the results suggest, the same score could be reused at deployment to flag uncertain notes for human review, not just to refine prompts."],"forward_implications":["A clinician with no prompt-engineering training can drive an LLM classifier to macro-F1 0.986 on helmet-status extraction after two rounds of reviewing at most 30 notes per round.","The second-round prompt beats both the human-authored prompt (macro-F1 0.735) and the strongest fully automated prompt optimizer (0.980), without fine-tuning the model.","SamplEase's one-iteration median macro-F1 of 0.974 significantly exceeds random sampling's 0.959 (P = 0.044), so the confidence-based selection rule adds value beyond label quantity.","The largest per-class gain appears where the baseline is weakest: the 'No Helmet' class F1 rises from 0.88 at P0 to 0.98 at P2.","Macro-level and per-class metrics remain similar across gender and racial groups in this task, indicating no substantial demographic performance disparity in the reported runs."],"supporting_citations":[{"why":"Supplies the automatic prompt-optimization baseline that StructEase's F1 0.986 result must beat, and defines the fully automated alternative to expert-in-the-loop refinement.","marker":"Khattab et al. 2022, 2024"},{"why":"Automated prompt-optimization work whose search-based runtime limits motivate incorporating domain-expert feedback during prompt refinement.","marker":"Wang et al. 2023"},{"why":"Establishes the helmet-status extraction task from unstructured emergency narratives and provides the NEISS-based dataset and prior generative-AI baseline used for evaluation.","marker":"Burford et al. 2024"},{"why":"Cited as evidence that human expertise interprets and contextualizes AI outputs, grounding the paper's premise that expert corrections improve prompt quality.","marker":"Toner-Rodgers 2024"},{"why":"Supplies the NEISS injury data from which the 17,888 clinical narratives are filtered, grounding the dataset used in all experiments.","marker":"U.S. Consumer Product Safety Commission 2024"}],"fun_headline_variants":["Expert-guided prompt tweaks beat full automation in clinical LLMs","Small expert corrections boost clinical classification F1 to 0.986","SamplEase: uncertainty selects few notes, expert fixes lift F1","Minimal expert effort, maximal gain: StructEase for clinical prompts","LLM prompt optimization: a few expert calls outperform auto-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire framework rests on trusting that the language model's token-level confidence score identifies the notes whose expert corrections will most improve the prompt; if that ranking is no better than random selection, the reported F1 advantage disappears.","fun_headline_variants_meta":{"raw":{"variants":["Expert-guided prompt tweaks beat full automation in clinical LLMs","Small expert corrections boost clinical classification F1 to 0.986","SamplEase: uncertainty selects few notes, expert fixes lift F1","Minimal expert effort, maximal gain: StructEase for clinical prompts","LLM prompt optimization: a few expert calls outperform auto-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2149,"prompt_tokens":1021,"completion_tokens":1128,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":1036}},"tokens_in":637,"tokens_out":1128,"duration_ms":9571,"temperature":1.0,"reasoning_tokens":1036,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:44:55.674745+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework with the prompt-update mechanism held fixed but the expert's notes drawn uniformly at random rather than by lowest confidence, matched for the same number of labels per class; if the median macro-F1 gap between the two arms is not statistically significant across repeated runs, the confidence-ranking mechanism is not the causal driver.","supporting_citations":[{"cited_title":"G.; Itzkowitz, N","cited_arxiv_id":null,"evidence_quote":"Establishes the helmet-status extraction task from unstructured emergency narratives and provides the NEISS-based dataset and prior generative-AI baseline used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited as evidence that human expertise interprets and contextualizes AI outputs, grounding the paper's premise that expert corrections improve prompt quality."}],"review_version":1}