{"id":"637cbfd4-2e49-449a-8c1f-fa14b175e143","arxiv_id":"2411.14720","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"In-context learning beats fine-tuning for HPV vaccine stance detection on this dataset, with six stratified examples and a detailed prompt giving the highest accuracy.","lead":"This paper tests seven large language models on classifying HPV vaccine tweets as in favor, against, or neutral, comparing in-context learning with fine-tuning. It finds that in-context learning generally works better, and that the best setup uses six balanced examples and a detailed prompt.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-processing in §3.1 manually assigns the gold label to any LLM output that does not cleanly contain one stance, and correction rates are unreported; the reported F1 scores and the ICL-vs-fine-tuning comparison may therefore measure human repair, not model performance.","rationale":"I read the paper in good faith as a careful empirical comparison of ICL and fine-tuning for stance detection. The strongest claim is plausible, but the post-processing step in §3.1 is a genuine measurement flaw: it imputes correct labels for model outputs that failed to produce a clean label, and the correction rate is unreported. Since the paper explicitly says zero-shot prompts increased format failures, between-condition differences in F1 may be artifacts of human repair rather than model behavior. This is the same concern as the reader's weakest_assumption, so I agree with the reader's identification. The proposed concrete test would settle it: strict parsing removes the human repair from the measurement, and correction-rate tabulation reveals whether the post-processing was balanced across conditions. If the strict scores still show ICL above fine-tuning and correction rates are small and balanced, the claims stand; otherwise the central claim is unsupported. Because the paper supplies enough detail (prompt templates, Appendix D categories, full F1 tables) that the concern is testable without new experiments, and the authors may have the raw outputs to disclose, I keep the reader's CONDITIONAL verdict unchanged rather than escalating to REJECT.","tokens_in":19044,"tokens_out":4391,"duration_ms":43820,"concrete_test":"Re-run the evaluation on the saved raw model outputs (or, if unavailable, re-prompt a sample of conditions) with a strict deterministic parser: a prediction is counted only if exactly one of 'in favor', 'against', 'neutral or unclear' appears in the output; all other outputs are scored as errors. Recompute macro F1 for every model/prompt/shot condition and compare against the post-processed values in Tables C1–C4; also tabulate the human-correction rate per condition. If the strict F1 for the 6-shot stratified detailed GPT-4 Turbo condition drops materially (e.g., below 0.85) or the ICL-vs-fine-tuning ordering reverses, the central claim is unsupported as reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (ICL outperforms fine-tuning; GPT-4 Turbo with six stratified detailed shots reaches macro F1 0.90) depends entirely on the F1 scores in §4.1 and Tables C1–C4. Section 3.1 ('Post-processing the LLM outputs') states that whenever a completion does not 'explicitly begin[] with or include[] only one stance label'—including 'missing initial label', 'dual stances', and 'irrelevant stances'—the authors 'manually inspected the raw output and assigned the correct label.' Because the 'correct' label is the gold-standard stance, every ill-formatted output is converted into a correct prediction. The paper never reports how many outputs fell into each ill-format category, nor whether these rates differ across conditions. It does state in §4.1 that zero-shot prompts increased 'missing initial label' outputs. If correction rates are substantial or uneven, the macro F1 values are not clean measures of model classification, and the comparison between in-context learning and fine-tuning is confounded: fine-tuned models, producing logits, receive no such repair. A secondary but related issue is that the 'optimal' configuration was selected as the maximum on the same 378-tweet test set with no error bars, so small differences such as 0.90 versus 0.89 may reflect selection noise. The manuscript itself acknowledges the human-in-the-loop intervention and the restriction to unambiguous tweets, but it does not quantify the effect of either on the reported scores.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an experimental comparison of in-context learning (ICL) and fine-tuning for stance detection of HPV vaccination tweets, using seven LLMs (GPT-4 Turbo, GPT-4o-mini, Mixtral-8x7B, Mistral-7B, Llama-3-70B, Llama-3-8B, Flan-UL2). The ICL experiment varies prompt template complexity (basic vs. detailed), shot sampling (random vs. stratified), and shot quantity (0–30), evaluated by macro F1 on a 378-tweet test set. The fine-tuning experiment applies LoRA to three open-weight models. The headline claims are that ICL generally outperforms fine-tuning, that increasing shot quantity does not necessarily help, and that the best configuration is GPT-4 Turbo with six stratified shots and a detailed prompt, achieving macro F1 0.90.","tokens_in":19310,"tokens_out":4185,"duration_ms":38703,"significance":"If the evaluation were clean, the paper would provide useful practical evidence for social scientists applying LLMs to stance annotation, with a broad sweep of models, prompt designs, and sampling strategies. The appendices give detailed per-configuration F1 tables, and the human-in-the-loop post-processing is transparently described. However, the central comparison is currently undermined by the manual repair step in Section 3.1, which assigns the gold label to ill-formatted outputs and is reported to be more frequent under zero-shot prompting. Because the paper does not quantify the repair rate or provide a strict-parse evaluation, the reported F1 scores, and the ICL-versus-fine-tuning and optimal-configuration claims, cannot be taken as clean measures of model performance.","major_comments":[{"comment":"The evaluation of in-context learning is contaminated by the manual repair step in 'Post-processing the LLM outputs': every completion that does not cleanly contain a single stance label—including 'missing initial label', 'dual stances', and 'irrelevant stances'—is manually assigned the 'correct' label. This converts ambiguous model outputs into correct predictions by construction, so the macro F1 scores in Tables C1–C4 and Figure 2 measure a mixture of model classification and human repair, not model performance alone. Section 4.1 states that zero-shot prompts increased 'missing initial label' outputs, indicating that the repair rate is condition-dependent, which biases the comparison between zero-shot and few-shot conditions and between ICL and fine-tuning (the fine-tuned models output logits and receive no such repair). The manuscript does not report how many outputs were repaired per condition. Please report per-condition correction rates and a strict pattern-matching-only evaluation (or an evaluation that treats repaired outputs as errors) alongside the current numbers; the central claims in the Abstract depend on this.","section":"3.1, 4.1"},{"comment":"The claim that the optimal configuration is 'six stratified shots paired with detailed contextual prompts' is not supported as stated. Table C1 shows that GPT-4 Turbo reaches macro F1 = 0.90 under at least four configurations (stratified 3-shot basic/detailed and 6-shot basic/detailed), and the zero-shot detailed prompt achieves 0.89. The maximum is selected on the same 378-tweet test set without any estimate of variance or a separate validation set, so a 0.01 difference is within selection noise. Please provide confidence intervals or a bootstrap analysis for the optima, or switch to a validation/train split for configuration selection.","section":"Abstract, 4.1"},{"comment":"The general claim that 'in-context learning outperforms fine-tuning' is drawn from a comparison in which only three open-weight models are fine-tuned, while the model driving the top ICL result, GPT-4 Turbo, is excluded from fine-tuning because it is proprietary. The fine-tuned models are also not matched to the ICL counterparts in size or architecture. The superior performance of ICL may therefore reflect a comparison of stronger base models against weaker fine-tuned ones rather than a within-model effect. Please either restrict the ICL-versus-fine-tuning comparison to the three fine-tuned model families, or add an explicit caveat in the Abstract and Conclusion that the comparison is across model families, not a controlled within-model experiment.","section":"3.2, 4.2"}],"minor_comments":[{"comment":"The text says that 40 few-shot prompts and two zero-shot prompts were generated per test tweet and that this results in 15,876 unique prompts; showing the multiplication (42 × 378 = 15,876) would improve clarity.","section":"3.1"},{"comment":"The legend does not clearly distinguish random versus stratified sampling when the figure is printed in grayscale; using distinct marker shapes in addition to line types would make the panels legible.","section":"Figure 2"},{"comment":"The list of ill-formatted output categories (e.g., 'misindexing', 'creating new stance', 'infinite repetitions') is not defined; adding one-sentence examples of each would help future researchers reproduce the post-processing step.","section":"Appendix D"},{"comment":"The Discussion's 'cognitive overload' interpretation for GPT-4 Turbo should note that the zero-shot detailed prompt attains macro F1 0.89, only 0.01 below the claimed optimum, making the shot-dependent decline a small effect that may not warrant the cognitive-load framing.","section":"4.1, 5"}],"recommendation":"major_revision","confidential_remarks":"The experimental design is valuable and the practical guidance is potentially useful, but the post-processing step as implemented is a serious evaluative flaw. If the authors supply a strict-parse evaluation and per-condition correction rates, or alternatively recast the paper as an evaluation of a human-in-the-loop annotation system, the headline claims may become supportable. The 'optimal configuration' claim also needs statistical support given the tied top scores and the 0.01 margin over zero-shot. I recommend major revision rather than rejection because the issues are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things up front. First, this is a genuinely systematic empirical study: seven models, two prompt templates, two shot-sampling strategies, and shot counts from 0 to 30, with full F1 tables and the actual prompts in appendices. That level of transparency is rare and useful. Second, the stress-test concern is correct and load-bearing: the paper manually assigns the gold label to every LLM output that does not cleanly contain one stance, and it never reports how often that happens or whether the rate differs across conditions. It even says zero-shot prompts increased the 'missing initial label' problem. So the reported F1 scores are not cleanly attributable to the models, and the comparison against fine-tuned models—which get no such repair—is contaminated. This is not a minor caveat; it is the central measurement on which the headline claims rest.\n\nThe paper does several things well. It addresses a real need in computational social science: cheap, reliable annotation of vaccine-related social media. The factorial design is wide and the documentation is detailed, which makes the findings independently checkable if the authors ever release their data and outputs. The discussion is also honest about the restriction to unambiguous tweets and the inability to attribute the detailed-prompt effect to any single component. Those are real strengths.\n\nThe soft spots, in addition to the correction issue: the 'optimal' configuration is selected as the maximum on the same 378-tweet test set with no error bars or holdout, so the 0.90 versus 0.89 differences are likely selection noise. The fine-tuning comparison also has an unfairness of scale—only three models were fine-tuned, and the proprietary GPT-4 Turbo, the best ICL performer, was excluded. None of these are fatal on their own, but together they mean the paper's specific claims—that six stratified shots with a detailed prompt is optimal and that ICL generally beats fine-tuning—are not established by the evidence as reported.\n\nWho is this for? Health communication and computational social science researchers who want practical guidance on LLM annotation will find the setup and prompts valuable, but they should treat the numeric results as upper bounds. The paper deserves peer review because the question is important and the experimental structure is salvageable; a serious referee should demand correction statistics per condition, a version of the analysis that does not silently repair outputs, and a more defensible selection procedure.\n\nMy recommendation: send it to review, but expect a major revision before it can be relied on.","headline":"A systematic, practically useful LLM-annotation study whose central ICL-vs-fine-tuning comparison is undermined by unreported human repair of ill-formatted LLM outputs.","tokens_in":19881,"tokens_out":1416,"would_cite":false,"duration_ms":17099,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a systematic comparison across seven LLM variants, in-context learning outperforms fine-tuning for classifying HPV vaccine stance, with GPT-4 Turbo reaching 0.90 macro F1 using six stratified shots and a detailed prompt.","keywords":["large language models","in-context learning","stance detection","HPV vaccine","social media annotation","fine-tuning","prompt engineering","misinformation"],"falsifier":"Count the ill-formatted outputs (missing initial label, dual stance, irrelevant stance) per condition from the prompt logs; if the manual-correction rate is nontrivial or differs between in-context learning and fine-tuning, the reported F1 gap is not a pure model-performance gap. Re-running the evaluation while discarding or penalizing manually corrected outputs would settle whether the headline comparison holds.","tokens_in":18807,"feed_emoji":"💉","tokens_out":7637,"duration_ms":67756,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models, prompted with a small set of carefully chosen examples, can annotate the stance of HPV vaccine tweets as reliably as conventional supervised pipelines, and that in-context learning does better than fine-tuning for this annotation task. The authors vary prompt detail, example-sampling method, and shot count across seven model variants, and compare those configurations with LoRA fine-tuned open models. Their headline result is that GPT-4 Turbo with six stratified examples and a detailed prompt reaches a macro F1 of 0.90 on a 378-tweet test set. If that result holds, health and communication researchers can scale up annotation of vaccine skepticism and misinformation at a fraction of the cost of human coding.","feed_headline":"Six-shot prompting beats fine-tuning on HPV vaccine stance detection","feed_subtitle":"A 378-tweet test suggests LLMs can label vaccine skepticism at scale without costly human annotation.","key_machinery":"The central object is the in-context learning configuration, defined by prompt template complexity (basic vs. detailed), shot sampling method (random vs. stratified), and shot quantity (0 to 30 in steps of 3), applied to seven LLM variants. The experiment builds 15,876 prompt variants from a 756-tweet ground-truth set split evenly into training and test halves, and evaluates them with macro F1 on 378 held-out tweets. The comparison arm fine-tunes three open models with Low-Rank Adaptation (LoRA), a parameter-efficient method that changes only a small fraction of model weights. The argument is carried by this factorial design, which isolates which prompt dimensions move performance and which models are sensitive to them.","core_discovery":"On the paper's own terms, the discovery is that the best stance detection for HPV vaccine tweets comes not from fine-tuning a model on labeled data, but from giving GPT-4 Turbo six balanced example tweets inside a detailed prompt that assigns the role of 'expert content analyst' and defines each stance category with HPV-specific claims. This configuration reached a macro F1 of 0.90, the highest score among all tested settings, while increasing shot counts generally did not help and in some cases hurt. In the fine-tuning arm, models adapted with LoRA (Mixtral-8x7B, Llama-3-70B, and Flan-UL2) did not consistently match their in-context learning counterparts. The paper also reports that different LLMs respond differently to prompt complexity and shot selection, so the optimal recipe is model-specific rather than universal.","pith_inferences":["If the manual post-processing step was applied often, the reported F1 gap between conditions could reflect human correction rather than model skill; a strict-output evaluation would settle that.","The 'neutral or unclear' class contains only 62 of 756 tweets, so macro F1 is computed over a very small third class; a balanced test set would be a useful robustness check.","The optimal recipe of six stratified shots with a detailed prompt is plausibly transferable to other vaccine debates, but transfer remains untested.","The 'cognitive overload' explanation for GPT-4 Turbo's decline at high shot counts is speculative; separating total prompt length from shot count would test it."],"forward_implications":["Researchers can replace most human coding of HPV vaccine stance with a six-shot GPT-4 Turbo prompt and obtain macro F1 around 0.90.","Adding more examples does not automatically improve accuracy, so annotation pipelines should tune shot count per model rather than defaulting to many-shot.","Fine-tuning open models with LoRA did not consistently beat in-context learning, which favors prompt-based annotation for small domain datasets.","Because models differ in sensitivity, results from one LLM should not be assumed to transfer to another without testing prompt conditions."],"supporting_citations":[{"why":"Supplies the many-shot learning result whose assumption that more shots help is tested and qualified by the paper.","marker":"Agarwal and colleagues (2024)"},{"why":"Defines in-context learning as the emergent few-shot capability that the prompt-design experiment sets out to optimize.","marker":"Brown et al. (2020)"},{"why":"Motivates the shot-selection problem and grounds the paper's comparison of random vs. stratified sampling.","marker":"Liu et al. (2021)"},{"why":"Frames the fair comparison between fine-tuning and in-context learning that the paper replicates in a health domain.","marker":"Mosbach et al. (2023)"},{"why":"Provides the claim that LLMs outperform supervised classifiers for social media annotation, which the paper evaluates for vaccine stance.","marker":"Törnberg (2024)"},{"why":"Recommends LLM-based annotation for computational social science, the practice the paper operationalizes and scales.","marker":"Ziems et al. (2024)"},{"why":"Supplies LoRA, the parameter-efficient fine-tuning method used in the comparison arm.","marker":"Hu et al. (2021)"},{"why":"Cited as prior evidence that fine-tuned LLMs underperform in-context learning on small datasets, supporting the paper's comparison result.","marker":"Bertsch et al. (2024)"}],"fun_headline_variants":["Six-shot LLM prompts top fine-tuning on HPV tweets","Six stratified shots beat fine-tuning for HPV stance","HPV vaccine stance: six-shot beats fine-tuning","Six-shot prompting outperforms fine-tuning on HPV vaccine"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes that the reported macro F1 scores measure the models' own classification performance, but a human post-processing step assigned the correct label to every output that did not cleanly contain one stance label, and the paper does not report how many outputs that involved.","fun_headline_variants_meta":{"raw":{"variants":["Six-shot LLM prompts top fine-tuning on HPV tweets","Six stratified shots beat fine-tuning for HPV stance","HPV vaccine stance: six-shot beats fine-tuning","Six-shot prompting outperforms fine-tuning on HPV vaccine"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001484,"raw_usage":{"total_tokens":5948,"prompt_tokens":919,"completion_tokens":5029,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":4967}},"tokens_in":535,"tokens_out":5029,"duration_ms":33602,"temperature":1.0,"reasoning_tokens":4967,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:58:37.933574+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the ill-formatted outputs (missing initial label, dual stance, irrelevant stance) per condition from the prompt logs; if the manual-correction rate is nontrivial or differs between in-context learning and fine-tuning, the reported F1 gap is not a pure model-performance gap. Re-running the evaluation while discarding or penalizing manually corrected outputs would settle whether the headline comparison holds.","supporting_citations":[],"review_version":1}