{"id":"6c73118f-721c-4c8e-ae43-bf090abf38d5","arxiv_id":"2508.19467","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuned DeBERTa-large outperforms LLMs on extracting clinical and social impacts from opioid-use Reddit posts (relaxed token F1 0.61 vs 0.44), yet remains below human agreement (kappa 0.81).","lead":"This paper introduces RedditImpacts 2.0, a refined dataset of first-person Reddit posts about opioid use, labeled for clinical and social impacts. It reports that fine-tuned DeBERTa-large detects these impacts better than large language models, but all models still lag behind human agreement.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Overlapping confidence intervals undermine the claim that DeBERTa-large significantly and consistently outperforms GPT-4o; the 17-point F1 gap may be statistical noise on a 278-post test set.","rationale":"The reader's verdict is CONDITIONAL and identifies the single-annotator test set as the weakest assumption. My stress-test finds a different, more directly load-bearing concern: the statistical reliability of the headline performance gap. The paper's own confidence intervals overlap, and no significance test is provided, so the central claim of 'significantly outperforming' is not established. This reinforces the reader's call for revision (addressing overclaims) rather than changing the verdict. The reader did mention overlapping CIs in the rationale, so there is partial agreement, but the weakest assumption differs. The proposed concrete test—a paired bootstrap on the F1 difference—would settle whether the gap is real. I do not see a reason to move the verdict away from CONDITIONAL; the concern strengthens the need for the requested revisions but does not by itself reject the paper.","tokens_in":15160,"tokens_out":3271,"duration_ms":39761,"concrete_test":"Perform a paired bootstrap hypothesis test on the test set: resample posts with replacement (e.g., 10,000 iterations), compute the relaxed token-level F1 difference between DeBERTa-large and GPT-4o 3-shot for each resample, and report the 95% CI of the difference and a two-sided p-value. If the CI includes zero, the claim of significant, consistent outperformance is unsupported. As a secondary check, recompute both models' scores using exact span-level F1 (strict boundary matching) to verify that the gap persists under a more standard NER metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that fine-tuned DeBERTa-large (relaxed F1 0.61) consistently outperforms the best LLM (GPT-4o 3-shot, F1 0.44). The reported 95% CIs are [0.43, 0.62] and [0.39, 0.51]; these overlap substantially, and no formal significance test is reported. With only 278 test posts and class imbalance (SocialImpacts entities 256 vs ClinicalImpacts 108), the observed 17-point gap is not established as statistically reliable. The paper's conclusion (Section 6) uses 'significantly outperforming' but the evidence presented does not support that adverb. Entity-specific CIs also overlap (e.g., SocialImpacts: DeBERTa 0.50 [0.25, 0.62] vs GPT-4o 0.26 [0.18, 0.34]). Additionally, the relaxed token-level metric lacks a specified one-to-one span matching procedure, which could distort absolute F1 scores, though the relative comparison is less affected. The load-bearing premise—that the gap is real and consistent—is therefore insecure without a proper paired significance test or a strict span-level comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RedditImpacts 2.0, a refined named-entity-recognition dataset for identifying self-reported ClinicalImpacts and SocialImpacts in first-person Reddit narratives about nonmedical opioid use. The authors describe revised annotation guidelines, report an inter-annotator agreement of Cohen's kappa 0.81, and evaluate a range of fine-tuned encoder-based models (with and without CRF) and LLMs under zero-, 3-, and 5-shot in-context learning. On a 278-post test set, the best fine-tuned DeBERTa-large model achieves a relaxed token-level F1 of 0.61 [0.43, 0.62], while the best LLM, GPT-4o with 3-shot prompting, achieves 0.44 [0.39, 0.51]. The paper concludes that fine-tuned encoders significantly outperform prompting-based LLMs for this domain, presents a qualitative error analysis, and reports a data-efficiency experiment suggesting that 50% of the training data suffices. The dataset, guidelines, and training scripts are publicly released.","tokens_in":15485,"tokens_out":5016,"duration_ms":54509,"significance":"If the empirical claims hold, this is a useful contribution to clinical NLP and social-media public-health surveillance. The dataset fills a gap by focusing on first-person, self-reported consequences of opioid use, and the refined guidelines address known annotation-noise problems in the earlier Reddit-Impacts resource. The systematic comparison of PLMs, CRF-augmented PLMs, and modern LLMs is relevant, and the public release of data and code supports reproducibility. The qualitative error analysis is informative. However, the central quantitative claim—that DeBERTa-large 'consistently outperforms' or 'significantly outperforms' LLMs—is not currently supported by the reported statistics, and the relaxed metric has a technical ambiguity that can affect the reported F1 values. These issues are fixable but require additional analysis.","major_comments":[{"comment":"The headline claim that DeBERTa-large 'significantly outperforms' GPT-4o is not supported by the reported confidence intervals. DeBERTa-large has F1 0.61 [0.43, 0.62] and GPT-4o has F1 0.44 [0.39, 0.51] in Table 3; these intervals overlap substantially. The entity-specific intervals in Table 4 also overlap (e.g., SocialImpacts: 0.50 [0.25, 0.62] vs. 0.26 [0.18, 0.34]; ClinicalImpacts: 0.66 [0.45, 0.67] vs. 0.51 [0.44, 0.60]). No significance test is reported, and the test set is small (278 posts, 364 entities). The conclusion in Section 6 uses the word 'significantly,' but the evidence presented does not establish statistical significance. Please add a paired test (e.g., paired bootstrap over posts, or McNemar-type test on token/span decisions) or soften the claim to 'outperformed in this evaluation.' Reporting a strict span-level F1 would also strengthen the claim.","section":"§4, Table 3; §6 Conclusion"},{"comment":"The relaxed F1 definition is underspecified because no one-to-one matching between predicted and gold spans is defined. The text says M_T is the set of span pairs with non-zero token-level overlap and then computes TPT as the sum of Overlap(g,p) over all such pairs. As written, a single predicted span overlapping multiple gold spans (or vice versa) contributes multiple times, inflating true positives relative to a standard span-to-span match. This can distort the absolute F1 values and, depending on how models produce overlapping/duplicate spans, may also affect the relative comparison. Please specify the matching procedure (e.g., greedy matching, maximum-weight matching, or an explicit token-level TP definition that counts each gold and predicted token at most once).","section":"§3.5, Eq. for relaxed F1"},{"comment":"The reliability of the test-set labels is not fully established. Cohen's kappa of 0.81 was computed only on the co-annotated subset used for guideline alignment; after agreement, the remaining data were divided between the two annotators, meaning each test post received a single annotation. Systematic annotator bias in the test split could distort the reported F1 and the model comparison. Please report the distribution of the two annotators' posts in the test split, provide a per-split agreement estimate, or carry out adjudication on the test set. This is load-bearing because the main comparison is an empirical measurement against this ground truth.","section":"§3.2, Data Annotation; Table 2"}],"minor_comments":[{"comment":"The GPT-4o 3-shot 95% CI is given as [0.37, 0.51] in the abstract but as [0.39, 0.51] in Table 3(b). Please make these consistent.","section":"Abstract; Table 3(b)"},{"comment":"Table 4 reports DeBERTa-large ClinicalImpacts F1 as 0.66, while the text in Section 4 says the F1 is 0.60. One of these is wrong; please correct and ensure the corresponding CI is aligned.","section":"Table 4; §4"},{"comment":"The sentence 'A notable effort in this area is the work by Ge et al. [16]...' is duplicated verbatim. Please remove the duplicate.","section":"Appendix A"},{"comment":"There is a typo: 'SociallImpacts' should be 'SocialImpacts' in the example under 'Ambiguous context (assumed impact).'","section":"Appendix C"},{"comment":"The prompt text contains 'ONL Y' instead of 'ONLY' in rule 1. Please fix.","section":"Table D2"},{"comment":"The abstract claims that the fine-tuned model outperforms LLMs 'in ... adherence to task-specific guidelines,' but this is supported only by qualitative examples in Section 4.1, not by a quantitative metric. If this is a headline claim, define and report a guideline-violation rate; otherwise soften the wording.","section":"§4.1 / Abstract"},{"comment":"The data-efficiency experiment does not describe how the subsampled training sets were selected, how many random seeds were used, or how the CIs in Figure 2 were computed. Please add these details so the plateau claim is reproducible.","section":"§5.4, Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The contribution is within scope and is potentially publishable, but the main quantitative claim needs a proper significance test or a substantially softened conclusion. The reviewers should also ask for a strict span-level metric and clarification of the relaxed matching procedure. I do not see evidence of a circular argument; the issue is statistical support, not derivation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper's real value is RedditImpacts 2.0, a carefully refined NER dataset with first-person annotation guidelines, released with scripts. That alone makes it worth a look for anyone working on social-media clinical NLP. The empirical comparison of fine-tuned encoders against LLM prompting is competent, and the data-efficiency finding (performance plateaus around 50% of data) is a genuinely useful insight for low-resource settings.\n\nThe soft spot is the headline. The abstract and conclusion say DeBERTa-large 'significantly outperforms' GPT-4o, but the reported 95% confidence intervals overlap (0.61 [0.43–0.62] vs 0.44 [0.39–0.51]), and no paired significance test is reported. The gap is large in absolute terms, and the entity-specific numbers point in the same direction, so I don't think the result is nothing. But 'significantly' is not supported by the evidence as presented. That is a fixable problem: a paired bootstrap or McNemar-style test would settle it, along with reporting strict span-level F1 rather than only the relaxed token-level metric. The relaxed metric is described clearly, but without a strict comparison it's hard to know how much the partial-credit scoring is driving the gap.\n\nTwo smaller concerns. First, the test set is only 278 posts and each post has a single annotator after the agreement phase; systematic annotator bias could shift results. That is a common limitation, but worth acknowledging. Second, the paper describes prior work on the original Reddit-Impacts as having issues but does not rerun the new models on the old data to show a clean apples-to-apples improvement. Minor, given the annotation changes.\n\nNone of this undermines the dataset contribution. The annotation guidelines, detailed appendix, and public release are exactly what the community needs. The paper is honest about its limitations, and the error analysis is genuinely informative—the qualitative differences between encoder and LLM behavior are more convincing than the CI-overlapping F1 gap.\n\nWho is this for? People building NER systems for clinical or substance-use social media text, and anyone evaluating LLMs on structured token-level tasks. It deserves serious peer review, but the authors need to revise the statistical claims and add strict-span evaluation. I'd send it to review and push for those changes.","headline":"A useful new dataset and a fair empirical comparison, but the 'significantly outperforming' claim outruns the statistics.","tokens_in":15923,"tokens_out":1642,"would_cite":true,"duration_ms":18600,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned DeBERTa-large finds opioid impacts on Reddit with F1 0.61, beating GPT-4o prompts at 0.44.","keywords":["named entity recognition","opioid use","social media","Reddit","in-context learning","fine-tuning","DeBERTa","clinical NLP"],"falsifier":"Re-annotate the held-out test posts with both annotators plus adjudication, then recompute relaxed token-level F1 for DeBERTa-large and GPT-4o 3-shot against the consensus labels; if DeBERTa's margin over GPT-4o disappears or becomes statistically insignificant, the central comparison collapses.","tokens_in":15093,"feed_emoji":"💊","tokens_out":4398,"duration_ms":44188,"temperature":0.7,"pith_summary":"The paper claims that for extracting self-reported clinical and social consequences of opioid use from Reddit posts, a domain-specific fine-tuned encoder model performs substantially better than prompting large language models. It introduces RedditImpacts 2.0, a first-person-focused annotated dataset with refined guidelines, and evaluates BERT, RoBERTa, DeBERTa, BioBERT and GPT-4o, Llama, and Gemma under zero- and few-shot in-context learning. The best model, DeBERTa-large, reaches relaxed token-level F1 0.61 (95% CI 0.43–0.62) versus 0.44 for the best LLM, GPT-4o with 3-shot prompting. The paper also reports that about half the training data is enough to reach plateau performance, and that even the best model remains below human inter-annotator agreement (Cohen's kappa 0.81). These findings matter because social media is an underused signal for addiction surveillance, and the comparison clarifies when fine-tuning is still needed rather than prompting alone.","feed_headline":"Fine-tuned model beats GPT-4o on Reddit opioid-impact extraction","feed_subtitle":"DeBERTa-large scores F1 0.61 vs 0.44 for few-shot LLMs on social and clinical consequence spans.","key_machinery":"The task is framed as sequence labeling with BIO tags, where each token is labeled O, B-ClinicalImpacts, I-ClinicalImpacts, B-SocialImpacts, or I-SocialImpacts. The top-performing system is a DeBERTa-large encoder with a linear token-level classification head, fine-tuned with cross-entropy, optionally augmented with a conditional random field layer that enforces valid tag transitions. Evaluation uses a relaxed token-level F1 that credits partial span overlap, appropriate for noisy social-media boundaries; LLM prompts are built by retrieving semantically similar training examples with sentence embeddings and feeding them as few-shot demonstrations.","core_discovery":"The central claim is that a fine-tuned DeBERTa-large model trained as a BIO token tagger on RedditImpacts 2.0 extracts ClinicalImpacts and SocialImpacts from first-person opioid narratives more accurately than state-of-the-art LLMs used with zero-, three-, or five-shot in-context learning. On the held-out test set, DeBERTa-large achieves relaxed token-level F1 0.61 (precision 0.75, recall 0.52), with stronger performance on ClinicalImpacts (F1 0.66) than SocialImpacts (F1 0.50); GPT-4o with three-shot prompting reaches F1 0.44 overall, with SocialImpacts much weaker at 0.26. The paper further claims the gap is not simply about LLM capacity, since error analysis shows GPT-4o mislabels social-","pith_inferences":["An untested extension suggested by the error analysis is a two-stage pipeline that first classifies whether a post is first-person and then runs NER; such a filter could reduce many of GPT-4o's guideline violations, but the paper did not evaluate it.","Because the test set has only 278 posts and SocialImpacts entities are less frequent than ClinicalImpacts, the reported 17-point gap may be sensitive to how social-impact spans are drawn; re-running on a larger or re-annotated test set would sharpen the comparison.","The data-efficiency plateau implies that further gains may come more from better annotation guidelines or model architecture than from labeling more examples; a direct test would compare a second-round guideline revision against doubling the training set.","The relaxed token-level F1 metric credits partial span overlap, so the model-vs-human comparison is not directly on exact spans; a strict-boundary evaluation could change the practical assessment of deployability."],"forward_implications":["On this task, domain-specific fine-tuning of an encoder beats LLM prompting by about 17 F1 points, so fine-tuning remains the stronger default for token-level clinical NER.","Training on roughly 50% of the labeled data reaches performance comparable to the full dataset, so smaller annotation budgets can support deployment in low-resource settings.","Few-shot in-context learning gives only marginal, not statistically significant, gains over zero-shot; adding more examples can slightly hurt performance.","A substantial gap remains to human expertise, indicating headroom for models that better capture implicit impacts and first-person-only disclosures."],"supporting_citations":[{"why":"Supplies the prior Reddit-Impacts dataset and annotation effort that this work refines into RedditImpacts 2.0.","marker":"[16]"},{"why":"DeBERTa is the architecture of the best-performing fine-tuned model.","marker":"[42]"},{"why":"BERT is one of the baseline pre-trained language models fine-tuned for comparison.","marker":"[40]"},{"why":"RoBERTa is one of the baseline pre-trained language models fine-tuned for comparison.","marker":"[41]"},{"why":"BioBERT is the clinical-domain baseline that underperforms on informal social media text.","marker":"[43]"},{"why":"GPT-4o is the best-performing LLM under in-context learning and the main comparison point.","marker":"[28]"},{"why":"Llama 3-70B is one of the open LLMs evaluated under zero- and few-shot prompting.","marker":"[26]"},{"why":"Gemma 3-27B is one of the open LLMs evaluated under zero- and few-shot prompting.","marker":"[27]"},{"why":"Sentence-BERT embeddings are used to retrieve semantically similar exemplars for few-shot prompting.","marker":"[46]"},{"why":"Cohen's kappa is the inter-annotator agreement measure used to define the human-expert performance baseline.","marker":"[34]"}],"fun_headline_variants":["Small tuned model outperforms GPT-4o on opioid impact NER","DeBERTa-large beats GPT-4o on extracting opioid harms from Reddit","Tuned model beats LLMs at spotting opioid consequences in social posts","Gap remains: fine-tuned NER still lags expert agreement on opioid text","Domain-tuned model closes part of AI-expert gap in opioid NER"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The test-set labels are treated as ground truth even though most test posts were annotated by a single human expert after the annotators agreed on a shared subset; if the two annotators systematically differ on the remaining posts, the measured F1 scores and the DeBERTa-versus-GPT-4o gap could shift.","fun_headline_variants_meta":{"raw":{"variants":["Small tuned model outperforms GPT-4o on opioid impact NER","DeBERTa-large beats GPT-4o on extracting opioid harms from Reddit","Tuned model beats LLMs at spotting opioid consequences in social posts","Gap remains: fine-tuned NER still lags expert agreement on opioid text","Domain-tuned model closes part of AI-expert gap in opioid NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000657,"raw_usage":{"total_tokens":2931,"prompt_tokens":919,"completion_tokens":2012,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":1911}},"tokens_in":663,"tokens_out":2012,"duration_ms":13116,"temperature":1.0,"reasoning_tokens":1911,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:46:20.271337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the held-out test posts with both annotators plus adjudication, then recompute relaxed token-level F1 for DeBERTa-large and GPT-4o 3-shot against the consensus labels; if DeBERTa's margin over GPT-4o disappears or becomes statistically insignificant, the central comparison collapses.","supporting_citations":[{"cited_title":"Reddit-Impacts: A Named Entity Recognition Dataset for Analyzing Clinical and Social Effects of Substance Use Derived from Social Media","cited_arxiv_id":"2405.06145","evidence_quote":"Supplies the prior Reddit-Impacts dataset and annotation effort that this work refines into RedditImpacts 2.0."},{"cited_title":"Devlin, M.-W","cited_arxiv_id":null,"evidence_quote":"BERT is one of the baseline pre-trained language models fine-tuned for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BioBERT is the clinical-domain baseline that underperforms on informal social media text."},{"cited_title":"Cohen, A coefficient of agreement for nominal scales, Educational and psychological measure- ment 20, 37 (1960)","cited_arxiv_id":null,"evidence_quote":"Cohen's kappa is the inter-annotator agreement measure used to define the human-expert performance baseline."}],"review_version":1}