{"id":"2687f652-6dc9-4f5e-a4e9-4da59adc91fe","arxiv_id":"2412.19346","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper shows that iterative pseudo-labeling with confidence, class-adaptive, or GPT-4o filtering can improve fine-grained PICO NER under 10% labeled data, though gains are modest and code is not released.","lead":"This paper tests whether a semi-supervised learning method, combining a small labeled set of clinical-trial abstracts with a large unlabeled set, improves extraction of fine-grained PICO entities (participants, interventions, comparisons, outcomes). In the 10%-labeled setting, the method raises macro F1 from 0.437 to about 0.60 versus a supervised baseline, but the result rests on standard pseudo-labeling and comes with reporting inconsistencies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 16% gain may be an artifact of selecting the best iteration and quality variant on the test set with a single 10% label mask; a validation-fixed, multi-mask rerun is needed.","rationale":"The paper's claim is an empirical one: with 10% labels, FinePICO reaches macro F1 0.60 versus 0.437 for the baseline, and the gap is presented as statistically significant. For that claim to hold, the comparison must be fair: the reported model must come from a pre-specified training and selection rule, and the result must not hinge on one favorable draw of labeled sentences. Two features of the paper threaten this. First, Figure 3 labels 'highest F1' per method and Table 2 reports the best of three quality-enhancement variants; if the iteration is selected using test-set performance, the reported 0.600 is an upper envelope rather than an expected performance. Section 2.1 mentions validation convergence, but it does not state that the final model was chosen by validation F1, and the selected iteration numbers are not reported. The absence of validation curves makes it impossible to rule out test-set peeking. Second, all experiments use a single random 10% label mask; no seed-level variance is reported. A 0.16 F1 difference could be consistent with substantial mask-to-mask variability, especially with only about 981 labeled sentences. The reader correctly flags pseudo-label reliability as an assumption, but the selection/mask issue is more decisive because it can invalidate the quantitative headline even if the pseudo-label quality filters work perfectly. The proposed concrete test addresses exactly this: fix selection on validation, run multiple masks, and report the distribution of gains. If the gain persists, the central claim is materially strengthened; if not, the abstract's 'outperforming by more than 16%' is not established. This does not change the reader's CONDITIONAL verdict; it sharpens the condition under which the claim can be trusted.","tokens_in":19229,"tokens_out":5202,"duration_ms":48707,"concrete_test":"Re-run the 10%-annotation experiment with at least 5 independent random masks. For each mask, train the baseline and all three FinePICO variants, and fix the stopping rule as 'select the iteration with the highest macro F1 on the held-out validation set of 1,091 sentences; then evaluate that single selected model once on the PICO-Corpus test set.' Report the mean and 95% CI across masks for the baseline and for each FinePICO variant, and also report the best-of-three-variants mean. If the mean gain over the baseline is not consistently positive across masks, or if best-of-variants selection explains most of the gap, the central claim of robust SSL benefit is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—0.600 macro F1 with 10% annotations versus 0.437 baseline, a >16% gain—is threatened by how the reported number was selected. Figure 3 reports 'highest F1' per method and Table 2 highlights the best of three quality-enhancement variants, but the paper never states whether the selected iteration was chosen on the validation set or by peeking at the test set. Section 2.1 says training stops when performance converges on validation, which is not the same as taking the maximum over iterations; the two rules can differ materially in self-training loops, where early iterations are often better. Additionally, only one random 10% labeled subset was created, with no variance across masks reported. If the 0.437-to-0.600 gap partly reflects test-set-based iteration selection and a favorable label draw, the improvement would not reproduce under a pre-specified validation-stopping rule or across other 10% masks. The pseudo-label confidence assumption in Eq. 8 is also relevant, but the selection/mask issue is more decisive because it can invalidate the quantitative claim even if pseudo-labels are well calibrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FinePICO, an iterative self-training semi-supervised NER pipeline for fine-grained PICO entity extraction from clinical trial abstracts. The method trains a BiomedBERT baseline on a small annotated subset, generates pseudo-labels on a larger unlabeled set, and filters pseudo-labels using one of three quality-enhancement strategies: confident-based masking, class-adaptive thresholding, or GPT-4o-based confirmation. The main empirical claim is that with only 10% annotated training sentences, FinePICO reaches macro F1 0.600 on the PICO-Corpus test set, outperforming a supervised baseline at F1 0.437 by more than 16%. The paper also reports generalization to a revised PICO annotation scheme and to external AD/COVID-19 corpora, with p-values below 0.001 for several comparisons.","tokens_in":19374,"tokens_out":3677,"duration_ms":36106,"significance":"If the central quantitative claim holds, the paper offers practical evidence that semi-supervised self-training can reduce annotation cost for fine-grained PICO extraction while approaching fully supervised performance. The paper has several strengths: it uses publicly available corpora, evaluates against held-out test sets, reports bootstrapped confidence intervals, compares three pseudo-label selection mechanisms, and includes external-corpus generalization tests. However, the headline result is vulnerable to two evaluation-design issues: the best iteration appears to be selected from training curves without a clear validation-based stopping rule, and all 10%-annotation results are based on a single randomly masked subset with no variance across label subsamples. These issues must be resolved before the reported effect size can be considered reliable.","major_comments":[{"comment":"The paper reports the 'highest F1' observed over iterations for each method, but it does not specify whether the iteration achieving that F1 was chosen on the validation set or by inspecting test-set performance. Section 2.1 states that training stops when performance converges on validation, which is a different rule from selecting the maximum over iterations; in self-training loops early iterations are often better than later ones, so taking the maximum over the displayed curves can materially inflate the reported gain. Please report the F1 obtained under a pre-specified validation-based stopping rule, or otherwise demonstrate that the reported numbers were not selected by peeking at the test set.","section":"§2.1, §3.1, Figure 3"},{"comment":"All 10%-annotation results, including the headline F1 of 0.600 versus baseline 0.437, appear to come from a single randomly selected 10% labeled subset. No variance over different label masks is reported, so it is unclear whether the observed improvement reflects a favorable label draw or a systematic effect of the method. Please rerun the key comparison over at least five different random masks and report the mean and standard deviation of the performance gap, together with a paired significance test across masks.","section":"§2.4, Supplementary Table 3"},{"comment":"Several reported confidence intervals are internally inconsistent. For example, the class-adaptive in-domain row reports F1 = 0.596 with 95% CI (0.609, 0.664), and the GPT cross-domain row reports F1 = 0.600 with 95% CI (0.609, 0.664); in both cases the interval lies entirely above the point estimate and even excludes it. This suggests a column misalignment or a bootstrapping error. Because the statistical claims in Section 3.2 depend on these intervals, they need to be corrected and recomputed.","section":"§3.2, Supplementary Table 5"},{"comment":"The class-adaptive threshold in Eq. (9) is not fully specified: P(k|t_ij) is not defined, and the formula appears to divide a sum of maximum token probabilities by the total count of pseudo-labels per class, rather than computing a per-token confidence threshold. As written, it is unclear how a token is accepted or rejected under this rule, and the method is therefore not reproducible from the description. Please state the precise filtering condition, including how the threshold is applied to individual tokens.","section":"§2.1.3, Eq. (9)"}],"minor_comments":[{"comment":"The phrase 'by more than 16%' should be clarified: (0.600 - 0.437) is 16.3 percentage points, but the relative improvement is approximately 37%. Please state explicitly whether the comparison is in percentage points or relative percent to avoid misleading readers.","section":"Abstract, §4"},{"comment":"The caption contains a typo: 'Lower bound performance is detonated as the baseline model' should read 'denoted.'","section":"Figure 3 caption"},{"comment":"The counts in Table 1 appear to mix abstract-level and sentence-level units: PICO-Corpus is listed as 1,011 abstracts, but Training/Validation/Test are given as 1,010/645/944, and Supplementary Table 3 later states the test corpus has 2,717 sentences. Please clarify the units and ensure the reported splits are consistent across tables.","section":"Table 1, Supplementary Table 3"},{"comment":"Two important hyperparameters are never given concrete values: the confidence threshold used in confident-based masking and the unsupervised loss weight alpha in Eq. (4). Reporting the chosen values would improve reproducibility.","section":"§2.1, §4"},{"comment":"The statement that 'data and codes underlying the study will be available upon request' is insufficient for a reproducible empirical paper; a permanent repository link would be expected.","section":"Data availability"}],"recommendation":"major_revision","confidential_remarks":"The core methodological idea is reasonable and the evaluation covers useful ground, but the headline effect size is not yet trustworthy because of test-set-based iteration selection and the absence of variance across label masks. If the authors can provide validation-selected results, multi-mask variability, and corrected confidence intervals, the paper could become publishable. I also note that the data/code availability statement is weak for a computational paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid, useful application paper with a real reporting problem in its headline number. The claim is that iterative self-training with pseudo-label quality filters lifts fine-grained PICO NER from 0.437 to about 0.60 macro F1 when only 10% of training sentences are labeled. I believe the qualitative finding—SSL helps in the low-label regime—but the exact magnitude is not established, because the paper appears to select the best iteration from the test set and uses a single random label mask.\n\nWhat is genuinely new: applying SSL to fine-grained PICO extraction, the class-adaptive threshold in Eq. 9, the GPT-4o-based pseudo-label verification, and the systematic comparison of in-domain vs. cross-domain unlabeled data. The revised-scheme and external-corpus tests are worthwhile extras, and the error analysis is honest and useful. The authors also cite the relevant prior work on both PICO extraction and SSL. On scope and effort, this is a well-executed empirical study rather than a new method.\n\nThe soft spots are real but not fatal. The stress-test concern lands: Figure 3 reports \"highest F1\" per curve, and the paper never states that the selected iteration was chosen by a validation rule. Section 2.1 says training stops when performance converges on validation, which is not the same as taking the max over ten iterations. Max-over-test is a different procedure and can inflate scores. Second, only one 10% labeled subset was used; there is no variance over masks. Third, the supplementary confidence intervals in Table S5 exclude the point estimates for several rows (e.g., GPT cross-domain F1 0.600 with CI 0.609–0.664), and the abstract's \"p<0.001\" conflicts with Table 4's p=0.014 and p=0.025. Also, alpha and the confidence threshold for confident-based masking are never reported. These problems do not overturn the qualitative direction, but they make the headline \">16%\" improvement unverifiable as stated.\n\nThis paper is for researchers in biomedical NLP, especially those working on PICO extraction or SSL for NER. It deserves a serious referee: the problem is important, the empirical scope is large, and the flaws are fixable. My recommendation is to send it to peer review with a strong request for revision, not to desk-reject. The authors need to release code and data, correct the reporting errors, run multiple label masks, and specify how the iteration was selected before the exact numbers can be trusted.","headline":"Useful SSL-for-PICO study whose headline gain is likely real in direction but inflated by test-set iteration selection and single-mask evaluation.","tokens_in":19996,"tokens_out":3917,"would_cite":false,"duration_ms":34405,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An iterative self-training pipeline with pseudo-label quality filters reaches macro F1 0.60 for fine-grained PICO entity recognition using only 10% of the annotated training data, outperforming the supervised baseline by more than 16%.","keywords":["fine-grained PICO extraction","semi-supervised learning","named entity recognition","self-training","pseudo-labeling","clinical trial abstracts","BiomedBERT","GPT-4o label selection"],"falsifier":"Take the same 2,511-abstract corpus and repeat the 10% labeled-data experiment over 10 or more random masks; if the bootstrapped 95% confidence interval for FinePICO's macro F1 overlaps the baseline's 0.437, the annotation-saving claim fails. Also rerun the loop with the quality filter replaced by random acceptance of pseudo-labels at the same volume; if F1 does not fall below 0.60, the filter is not the source of the gain.","tokens_in":18967,"feed_emoji":"🩺","tokens_out":8433,"duration_ms":71575,"temperature":0.7,"pith_summary":"This paper aims to show that a semi-supervised self-training loop can reduce the annotation burden for fine-grained PICO extraction from clinical trial abstracts without sacrificing accuracy. Using 2,511 abstracts from four public corpora, the authors train a BiomedBERT NER model on a small labeled subset, generate pseudo-labels for a larger unlabeled pool, and selectively retrain on the most confident pseudo-labels. With only 10% of the training sentences labeled, FinePICO reaches macro F1 0.60, compared with 0.437 for the fully supervised baseline, and the gains hold across a revised annotation scheme and external test corpora. The result matters because meta-analysts need fine-grained entities such as arm-specific sample sizes and outcome values, which are expensive to annotate and inconsistently defined across existing datasets.","feed_headline":"PICO extraction reaches F1 0.60 with only 10% labeled data","feed_subtitle":"Iterative pseudo-labeling with confidence filters beats the supervised baseline by 16%, using unlabeled clinical trial abstracts.","key_machinery":"The load-bearing mechanism is an iterative self-training loop guarded by a pseudo-label quality function. A BiomedBERT model trained on labeled sentences predicts labels for unlabeled sentences; the quality function f in Eq. (8) decides which pseudo-labeled tokens enter the next training pool, and the model is retrained on the combined supervised and filtered unsupervised loss until validation performance converges. Three filters are tested: confident-based masking, which keeps tokens above a fixed softmax threshold; class-adaptive threshold masking, which sets a per-class threshold tau_k from the mean maximum probability of tokens predicted in that class; and GPT-4o-based selection, which prompts a large language model with entity-specific annotation guidelines to confirm each pseudo-label. The filter is what prevents the self-training loop from amplifying its own errors, and it is the component the paper varies to show the gain over the baseline is real.","core_discovery":"On the paper's own terms, the central discovery is that iterative self-training with pseudo-label quality control lets fine-grained PICO NER approach the performance of fully supervised training with a fraction of the annotations. In the original PICO scheme with 10% of training sentences labeled, the best FinePICO configurations reach macro F1 0.606 (in-domain, class-adaptive threshold) and 0.60 (cross-domain, GPT-based selection), against a BiomedBERT baseline of 0.437; the improvement is statistically significant at p<0.001. The method also transfers to a revised PICO scheme, where the class-adaptive threshold reaches F1 0.653 with in-domain unlabeled data, and it improves partial-match F1 on the AD and COVID-19 corpora over the baseline. The authors interpret this as evidence that abundant unlabeled RCT abstracts can substitute for much of the missing manual annotation in fine-grained PICO extraction.","pith_inferences":["The near-tie among the three filters suggests the pseudo-label loop, not the filter's sophistication, is the main driver; a testable extension is fixing the filter and varying retraining frequency to see whether one or two iterations capture most of the gain.","The most frequent error is boundary detection, and the model also confuses intervention-arm and control-arm values; this points to a document-level or section-aware extension that uses context beyond a single sentence.","If the result transfers to other clinical NER tasks, the same recipe—small labeled slice plus large unlabeled pool plus class-adaptive filtering—could lower annotation costs for eligibility criteria, adverse events, or drug-regimen extraction, where annotated corpora are similarly sparse.","GPT-4o confirmation is expensive per token; since confidence-based filters match it in F1, a practical next test is comparing cost-adjusted throughput of the three filters on a much larger unlabeled pool."],"forward_implications":["Annotation effort can drop by roughly 90%: with 10% labeled data, FinePICO's F1 of 0.60 exceeds the fully supervised baseline's 0.437, making fine-grained PICO extraction feasible for teams without large annotation budgets.","Cheap confidence filters can substitute for LLM-based label checking: the class-adaptive threshold matches or beats GPT-4o selection in the revised scheme and shows no statistically significant loss in the original scheme.","Cross-domain unlabeled data is at least as useful as in-domain data in the original scheme, so publicly available RCT abstracts from other specialties can serve as the unlabeled pool when in-domain abstracts are scarce.","The performance curve saturates as labeled data approaches 100%, indicating that the SSL gain is concentrated in low-resource regimes and that adding more unlabeled data cannot fully replace the last increments of annotation.","Because the method works under a revised PICO scheme, downstream users can define their own fine-grained entity taxonomy and still benefit from the same semi-supervised pipeline."],"supporting_citations":[{"why":"Supplies the PICO-Corpus of 1,011 breast-cancer RCT abstracts with fine-grained PICO annotations that define the labeled training and test data.","marker":"[29]"},{"why":"Provides the EBM-NLP corpus used as the large cross-domain unlabeled pool after annotations are removed.","marker":"[22]"},{"why":"Contributes the AD and COVID-19 RCT abstracts used as external test corpora and the section-specific learning workflow that motivates hyperparameter choices.","marker":"[23]"},{"why":"BiomedBERT is the foundation model selected because it outperforms other BERT variants on fine-grained PICO extraction.","marker":"[39]"},{"why":"Prior span-based PICO model establishes the preprocessing workflow and reinforces the choice of BiomedBERT as the baseline.","marker":"[44]"},{"why":"Prompt-engineering method for clinical NER underlies the GPT-4o label-selection strategy.","marker":"[43]"},{"why":"Prior work on pseudo-labeling supports confident-based masking as an effective filter for low-confidence predictions.","marker":"[40]"},{"why":"Consistency-training work supports masking low-confidence examples in semi-supervised learning.","marker":"[41]"},{"why":"Motivates the class-adaptive threshold by showing fixed confidence thresholds bias against classes with lower-quality pseudo-labels.","marker":"[42]"}],"fun_headline_variants":["Fine-grained PICO NER with just 10% labeled data gains 16% F1","Semi-supervised PICO NER: 10% labels, unlabeled data, 16% boost","Unlabeled RCT abstracts slash labeling needs: PICO NER +16% F1","PICO NER: Small labeled + large unlabeled = 0.60 F1, 16% over baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that tokens whose pseudo-labels pass the confidence filter (or GPT-4o confirmation) are correct often enough that adding them to the training pool improves the model, and it assumes the single random 10% label split behind the headline number is representative; if either gives way, the reported margin over the baseline shrinks or disappears.","fun_headline_variants_meta":{"raw":{"variants":["Fine-grained PICO NER with just 10% labeled data gains 16% F1","Semi-supervised PICO NER: 10% labels, unlabeled data, 16% boost","Unlabeled RCT abstracts slash labeling needs: PICO NER +16% F1","PICO NER: Small labeled + large unlabeled = 0.60 F1, 16% over baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000937,"raw_usage":{"total_tokens":4078,"prompt_tokens":1084,"completion_tokens":2994,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":2889}},"tokens_in":700,"tokens_out":2994,"duration_ms":18489,"temperature":1.0,"reasoning_tokens":2889,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:41:01.667717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 2,511-abstract corpus and repeat the 10% labeled-data experiment over 10 or more random masks; if the bootstrapped 95% confidence interval for FinePICO's macro F1 overlaps the baseline's 0.437, the annotation-saving claim fails. Also rerun the loop with the quality filter replaced by random acceptance of pseudo-labels at the same volume; if F1 does not fall below 0.60, the filter is not the source of the gain.","supporting_citations":[{"cited_title":"Pico cor- pus: a publicly available corpus to support automatic data extraction from biomedical literature","cited_arxiv_id":null,"evidence_quote":"Supplies the PICO-Corpus of 1,011 breast-cancer RCT abstracts with fine-grained PICO annotations that define the labeled training and test data."},{"cited_title":"A corpus with multi-level annotations of patients, interventions and outcomes to sup- port language processing for medical literature","cited_arxiv_id":null,"evidence_quote":"Provides the EBM-NLP corpus used as the large cross-domain unlabeled pool after annotations are removed."},{"cited_title":"Towards precise pico ex- traction from abstracts of randomized controlled trials using a section-specific learning approach","cited_arxiv_id":null,"evidence_quote":"Contributes the AD and COVID-19 RCT abstracts used as external test corpora and the section-specific learning workflow that motivates hyperparameter choices."},{"cited_title":"A span-based model for extracting overlapping pico entities from randomized controlled trial publications","cited_arxiv_id":null,"evidence_quote":"Prior span-based PICO model establishes the preprocessing workflow and reinforces the choice of BiomedBERT as the baseline."},{"cited_title":"Improving large language models for clinical named entity recognition via prompt engineering","cited_arxiv_id":null,"evidence_quote":"Prompt-engineering method for clinical NER underlies the GPT-4o label-selection strategy."},{"cited_title":"Using pseudo-labeling to improve performance of deep neural networks for animal identification","cited_arxiv_id":null,"evidence_quote":"Prior work on pseudo-labeling supports confident-based masking as an effective filter for low-confidence predictions."},{"cited_title":"Unsupervised data augmentation for consistency training","cited_arxiv_id":null,"evidence_quote":"Consistency-training work supports masking low-confidence examples in semi-supervised learning."},{"cited_title":"De-biasing Distantly Supervised Named Entity Recognition via Causal Intervention","cited_arxiv_id":"2106.09233","evidence_quote":"Motivates the class-adaptive threshold by showing fixed confidence thresholds bias against classes with lower-quality pseudo-labels."}],"review_version":1}