{"id":"f26858a5-9cb6-438f-823d-6190e62d0295","arxiv_id":"2411.17967","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"QuaLLM-Health claims GPT-4o-mini can extract clinical variables from GLP-1 Reddit discussions with macro F1 above 0.90, but the evaluation uses the same gold standard for prompt tuning and testing.","lead":"Researchers adapted an existing LLM tool to pull structured cancer-related data from Reddit posts about GLP-1 weight-loss drugs. The reported high accuracy is compromised because the prompts were tuned on the same 100 posts used for evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation is in-sample: prompts were iteratively tuned on the same 100-entry gold standard used for Table 3, so the reported accuracy does not establish generalization to new data.","rationale":"The reader's weakest-assumption analysis identifies the correct load-bearing vulnerability: the evaluation set is the same 100-entry gold standard used for prompt refinement. My reading of the manuscript—especially Section 5.2 and Table 3—confirms that the reported metrics are in-sample. The disclaimer about not using examples directly from the evaluation dataset does not close the loop, because aggregate error feedback and manual inspection of discrepancies are still a form of tuning to that specific set. A held-out test is the minimal remedy. Given that the central claim is framed as demonstrating effective extraction, and no external or held-out validation is provided, the current verdict of REJECT is appropriate. I recommend no change to the reader's verdict.","tokens_in":5307,"tokens_out":2344,"duration_ms":22831,"concrete_test":"Hold out a fresh random sample of at least 100 entries from the 2,059-entry dataset that were not used in any prompt iteration, have two domain experts annotate them with the same guidelines and adjudication, then run the final frozen optimized prompt on this held-out sample exactly once. Report per-variable precision, recall, and F1, plus macro averages, and compare them with Table 3. If macro F1 drops by more than about 0.05, or if any key variable falls below 0.85, the reported performance is substantially in-sample and the generalization claim should be revised. To keep the test decisive, do not modify any prompts after seeing held-out results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that QuaLLM-Health can effectively extract clinically relevant quantitative data from unstructured text. For that claim to hold, the final performance numbers must estimate performance on data not used to develop the prompts. Section 5.2 describes iterative prompt engineering: each refinement is followed by re-evaluation against the gold standard, and discrepancies are used to adjust prompts and create edge-case examples. Section 5.2 also says 'Importantly, we did not use examples directly from the evaluation dataset,' but that does not break the in-sample loop. Prompt wording can be tuned to aggregate error patterns, to specific difficult cases discovered during evaluation, and to the distribution of the 100 annotated entries, even without copying those entries as few-shot examples. Table 3 reports metrics on exactly the same gold-standard set that drove the tuning. The large improvements from Table 2 to Table 3 (e.g., cancer_type F1 from 0.539 to 0.906, mentions_cancer_risk F1 from 0.741 to 0.822) are consistent with overfitting to the evaluation set rather than genuine generalization. The stability testing in Section 5.3 measures run-to-run agreement, not accuracy on new data, so it cannot address this concern. The application to the full 2,059-entry dataset in Section 6 is not an evaluation: without labels for those entries, processing them efficiently says nothing about correctness. Therefore the paper's conclusion, as stated, is not supported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes QuaLLM-Health, an adaptation of the QuaLLM framework for extracting cancer-related quantitative variables from Reddit discussions about GLP-1 receptor agonists. The authors collect about 410,710 posts and comments from five subreddits, filter to 2,059 cancer-related entries, and create a gold-standard dataset by having two domain experts annotate a random sample of 100 entries. They then use iterative prompt engineering with GPT-4o-mini on this gold-standard set, reporting macro-averaged precision, recall, and F1 above 0.90 after optimization, a 95% run-to-run stability match rate, and a deployment on the full dataset costing under $3. The central claim is that the framework can effectively and efficiently extract clinically relevant quantitative data.","tokens_in":5605,"tokens_out":4950,"duration_ms":40962,"significance":"If the performance numbers were estimates of generalization, the framework would be a useful and highly practical contribution to health informatics: it is cheap, fast, and the authors release their code, data, and annotation guidelines. The efficiency and cost aspects are credible. However, the evaluation design, in which the same 100-entry gold standard is used both to iteratively refine the prompts and to report final metrics, means the accuracy claims are currently unvalidated, and the small number of positive instances for several variables makes those claims statistically fragile. With a proper held-out evaluation, this could be a solid methodology paper.","major_comments":[{"comment":"The reported performance is computed on the same 100-entry gold-standard dataset that was used to iteratively refine the prompts. Section 5.2 states that each iteration of prompt refinement was followed by a re-evaluation against the gold standard and that discrepancies were used to craft new edge-case examples and adjust prompts. The claim that no examples were taken directly from the evaluation set does not break the in-sample loop, because prompt wording can be tuned to the aggregate error patterns and to the specific distribution of the 100 entries. The large improvements from Table 2 to Table 3 (e.g., cancer_type F1 from 0.539 to 0.906, mentions_cancer_risk F1 from 0.741 to 0.822) are consistent with overfitting to the evaluation set. The application to the full 2,059-entry set in Section 6 is not an evaluation, as there are no labels for those entries. The authors should report metrics on a held-out test set that was not used in any prompting decision, or use cross-validation with prompt development restricted to training folds.","section":"§5.2, Table 3"},{"comment":"Several variables have very few positive instances in the 100-entry gold standard (e.g., seeking_cancer_risk_data N=9, cancer_diagnosis_after_medication N=13, discussion_GLP1_decreasing_cancer_risk N=13). The macro-averaged F1 of 0.904 is therefore an average over estimates with very small denominators, and no confidence intervals or significance tests are provided. A single false negative among 9 positive instances changes the variable's F1 by more than 0.05, so the claim of 'balanced performance across all variables' is not statistically supported. The authors should provide bootstrap confidence intervals for all metrics and temper conclusions about rare variables.","section":"Table 1, Table 3"},{"comment":"The stability testing shows an average pairwise match rate of 95% across five runs on the gold-standard dataset, but this measures run-to-run agreement of the model's outputs, not their correctness. It cannot address the in-sample evaluation problem, because a model that consistently reproduces the same (possibly overfit) extractions would still show high stability. This section should be repositioned as a reproducibility check, with the caveat that stability does not imply accuracy.","section":"§5.3"}],"minor_comments":[{"comment":"The text contains typographical errors such as 'fromQuaLLM intoQuaLLM-Health' and 'dataset in question'; these should be corrected.","section":"Abstract, §1"},{"comment":"The overview refers to 'Deployment of a fine-tuned pipeline' but no model fine-tuning is performed; the pipeline is prompt-engineered, and the wording should be changed accordingly.","section":"§1.1"},{"comment":"The limitations section mentions small sample size and stability but does not acknowledge that the evaluation is in-sample; adding an explicit statement about the need for held-out validation would improve transparency.","section":"§7.1"},{"comment":"Figure 1 is referenced in the text but does not appear in the manuscript; please include the figure or remove the reference.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising framework and the release of code and resources is appreciated, but the in-sample evaluation is a serious flaw. I recommend major revision rather than reject because the issue is fixable with a held-out evaluation set, though the small annotated sample may require additional annotation to obtain stable estimates. The authors should also clarify the relationship to the original QuaLLM framework and check the accuracy of their stability claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper is a clear and honest adaptation of QuaLLM to GLP-1 cancer discussions on Reddit, but the reported accuracy is in-sample. The iterative prompt engineering loop in Section 5.2 used the same 100-entry gold standard both to refine prompts and to produce the final metrics in Table 3. That does not support the claim of effective extraction on new data.\n\nWhat's genuinely useful here: a new annotated dataset of 100 entries with two-expert labels and kappa scores, detailed annotation guidelines, a reproducible prompt setup on GitHub, and cost/time estimates ($3, about an hour) that other researchers will appreciate. The filtering and preprocessing pipeline is straightforward and described clearly. The authors are also upfront about limitations like small sample size and run-to-run variability.\n\nThe main problem is the evaluation loop. The authors note they did not copy examples directly from the evaluation set into the few-shot prompts, but that does not break the loop. They crafted examples around edge cases discovered on the gold standard, and prompt wording can be tuned to aggregate error patterns and the specific distribution of those 100 entries. The jump in cancer_type F1 from 0.539 to 0.906 is consistent with overfitting. Stability testing (95% pairwise match) only shows run-to-run consistency, not correctness. Applying the pipeline to all 2,059 entries is not an evaluation because there are no labels for those entries. In short, the conclusion that the framework \"demonstrates\" effective extraction goes beyond the evidence.\n\nThat said, the flaw is correctable. A held-out test set—even 20–30 entries set aside before prompt development—would give an unbiased estimate of performance. I would not walk away from this paper; I'd want a revision. The dataset and guidelines have real utility for future work.\n\nThis paper is for health informatics researchers who want a concrete example of LLM-based variable extraction from patient forums, and for anyone designing annotation guidelines for similar constructs. It is not a rigorous validation study as it stands.\n\nIn peer review, I would send it out rather than desk-reject. The evaluation design is a major concern that requires revision, but the work ships code, data, and detailed methodology—enough to warrant a serious referee's time.","headline":"A transparent, useful domain adaptation of QuaLLM for health forums, but the headline accuracy numbers are in-sample because prompts were tuned on the same 100-entry gold standard used for final metrics—so generalization is unproven, though the flaw is fixable.","tokens_in":6109,"tokens_out":2402,"would_cite":true,"duration_ms":22408,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a large language model, refined through human-in-the-loop prompt engineering, can extract clinically relevant quantitative variables from unstructured Reddit discussions about GLP-1 medications and cancer, with…","keywords":["large language models","quantitative data extraction","Reddit health discussions","GLP-1 receptor agonists","prompt engineering","gold-standard annotation","cancer survivorship","human-in-the-loop"],"falsifier":"Draw a fresh random sample from the same five subreddits, have the two annotators label it with the same guideline without showing the LLM pipeline designers, run the final optimized prompt on it, and compare. If macro F1 drops substantially below the reported 0.904, the claimed reliability is an artifact of in-sample tuning.","tokens_in":5169,"feed_emoji":"🩺","tokens_out":6445,"duration_ms":51011,"temperature":0.7,"pith_summary":"The paper sets out to show that an LLM-based pipeline, adapted from an existing extraction framework, can convert messy social media text into structured, clinically relevant quantitative data. It reports that after iterative prompt engineering on a 100-entry gold standard, the model reaches macro-averaged F1 of 0.904 and accuracy above 0.85 on every variable, with 95% run-to-run stability. The authors argue this makes large-scale analysis of patient-generated health discussions practical: the full 2,059-entry corpus was processed for under $3 in about an hour. If the claim holds, researchers could mine patient forums for cancer survivorship, risk perception, and physician communication without manual reading of every post.","feed_headline":"LLM extracts cancer data from Reddit with >90% F1","feed_subtitle":"Adapted pipeline turns unstructured GLP-1 discussions into quantitative variables for under $3.","key_machinery":"The load-bearing object is the extraction pipeline built on GPT-4o-mini, driven by a JSON output schema and a prompt derived from the annotation guideline. The key mechanism is iterative prompt engineering: chain-of-thought reasoning, few-shot examples targeting edge cases, and temperature 0.0, with each revision re-scored against the 100-entry gold standard until performance exceeded the reported thresholds. The gold standard itself, double-annotated by two domain experts with Fleiss' kappa at least 0.8 on key variables, is the target the prompts are aligned to.","core_discovery":"QuaLLM-Health's central discovery is that a zero-shot LLM, once aligned to a human-created annotation guideline through iterative prompt engineering, can extract variables such as cancer survivorship, family cancer history, cancer type, risk concerns, and physician discussions from Reddit posts at a level the authors call balanced and reliable. The optimized model achieved macro-averaged precision 0.911, recall 0.909, and F1 0.904 on the gold-standard set, and stability testing across five runs showed a 95% average pairwise match. The authors conclude that LLMs can effectively and efficiently translate unstructured patient-generated content into structured data suitable for downstream health research.","pith_inferences":["The reported metrics come from the same 100 entries used to refine the prompts, so performance on genuinely unseen posts is likely to be lower than the headline F1 of 0.904.","The stability test was also run on the gold-standard set; on the broader, noisier corpus, run-to-run variation may be larger.","Porting this to a new domain would require a fresh round of human annotation and prompt tuning, because the prompts are tightly fitted to the GLP-1/cancer guideline.","The cost estimate excludes the human annotation effort, which remains the practical bottleneck for scaling to new topics."],"forward_implications":["If the reported performance holds, researchers can run quantitative analyses of patient experiences on thousands of forum posts without manual reading of every entry.","The framework can be ported to other medications, conditions, or platforms by writing a new annotation guideline and re-running the prompt-refinement loop.","The full 2,059-entry GLP-1/cancer dataset becomes a usable source for downstream studies of survivorship, risk perception, and physician communication.","The roughly $3 cost and one-hour runtime lower the barrier for small research teams to adopt LLM extraction.","The reported 95% run-to-run stability suggests the pipeline can support reproducible extraction workflows."],"supporting_citations":[{"why":"Supplies the base QuaLLM framework that this paper adapts to health-related Reddit data.","marker":"[2]"},{"why":"Provides the Reddit API wrapper used to collect the 410,710 posts and comments.","marker":"[3]"},{"why":"Supports the chain-of-thought prompting technique employed during iterative refinement.","marker":"[4]"},{"why":"Supplies the few-shot prompting method used to improve extraction on edge cases.","marker":"[5]"},{"why":"Motivates the iterative prompt refinement approach for chain-of-thought.","marker":"[6]"},{"why":"Used to argue that human annotators are an imperfect comparator, framing the evaluation's limitations.","marker":"[7]"}],"fun_headline_variants":["Reddit cancer insights: LLM extracts data with 90% F1","LLM turns Reddit GLP-1 posts into cancer data for <$3","QuaLLM-Health: LLM mines cancer details from Reddit cheaply","From Reddit to dataset: LLM hits 90% F1 on health posts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that optimizing prompts against the 100-entry gold-standard set and then measuring performance on that same set gives an unbiased estimate of extraction quality on new, unseen posts.","fun_headline_variants_meta":{"raw":{"variants":["Reddit cancer insights: LLM extracts data with 90% F1","LLM turns Reddit GLP-1 posts into cancer data for <$3","QuaLLM-Health: LLM mines cancer details from Reddit cheaply","From Reddit to dataset: LLM hits 90% F1 on health posts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1643,"prompt_tokens":1006,"completion_tokens":637,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":548}},"tokens_in":622,"tokens_out":637,"duration_ms":5675,"temperature":1.0,"reasoning_tokens":548,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:38:11.756130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Draw a fresh random sample from the same five subreddits, have the two annotators label it with the same guideline without showing the LLM pipeline designers, run the final optimized prompt on it, and compare. If macro F1 drops substantially below the reported 0.904, the claimed reliability is an artifact of in-sample tuning.","supporting_citations":[{"cited_title":"PRAW: The python reddit api wrapper.URL: https: // praw","cited_arxiv_id":null,"evidence_quote":"Provides the Reddit API wrapper used to collect the 410,710 posts and comments."},{"cited_title":"Compared with what? Measuring AI against the health care we have.N Engl J Med","cited_arxiv_id":null,"evidence_quote":"Used to argue that human annotators are an imperfect comparator, framing the evaluation's limitations."}],"review_version":1}