{"id":"3cf17725-bcff-45c2-8023-61f50af350dd","arxiv_id":"2505.13706","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GPT-4, GPT-3.5, and Claude 3 Opus all underperform a RoBERTa-CRF baseline on fine-grained propaganda detection, though they beat a weaker baseline on some techniques.","lead":"Large language models detect propaganda techniques in news articles far less reliably than a fine-tuned RoBERTa-CRF baseline, scoring F1=0.16 versus F1=0.67. The findings help practitioners choose systems for media manipulation detection and caution against assuming frontier LLMs beat task-specific models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline negative result assumes PTC gold labels are complete; low LLM precision may reflect correct detections missing from gold, as the paper's Discussion itself concedes.","rationale":"The most load-bearing condition for the paper's comparative claim is that PTC gold spans are a complete and correct target. The paper's own Discussion identifies this as an open question, so I flag it explicitly rather than treating it as an artifact. The concern is not merely a labeling-quality caveat because the two systems being compared are asymmetric with respect to annotation noise: the RoBERTa-CRF baseline is trained to imitate human labels, while the LLMs are asked to apply definitions. Hence low LLM precision can be an artifact of the gold standard rather than a real deficiency. The recommended adjudication experiment would decide whether the headline gap survives a corrected gold set. Other weaknesses (no error bars, best-prompt selection on the test set, the abstract/body mismatch on appeal-to-fear) are real but less decisive: best-prompt selection favors the LLMs, so it cannot rescue the baseline; the abstract mismatch concerns a secondary claim; and the large effect size makes the absence of significance tests less threatening to the primary result. Since the reader already assigned CONDITIONAL and identified this same weakest assumption, my stress-test does not change the verdict.","tokens_in":10308,"tokens_out":5212,"duration_ms":52687,"concrete_test":"Select a random sample from the PTC test set (e.g., 100 sentences stratified by technique and by model false-positive/hit status). Have at least two trained annotators independently label all six techniques at phrase level following the original annotation guideline, then adjudicate disagreements. Treat the adjudicated labels as gold and recompute macro-F1 for the RoBERTa-CRF baseline, GPT-4, GPT-3.5, and Claude 3 Opus under their best prompt settings from Table 2. If the baseline still beats all LLMs by a comparable margin, the concern is resolved; if LLM macro-F1 rises substantially or surpasses the baseline, the paper's headline needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To conclude that LLMs underperform the RoBERTa-CRF baseline, the evaluation must use a gold set that contains the true set of propaganda instances. The paper's comparison relies on the original PTC annotations in Table 2, but the Discussion explicitly concedes the risk: '...whether the LLMs are identifying instances of propaganda techniques that were missed by human annotators in the original dataset.' This is load-bearing because the LLMs are prompted with the technique definitions, while the RoBERTa-CRF baseline is trained on the same human labels; the baseline can achieve high F1 by reproducing annotator decisions, including omissions, whereas LLMs that detect additional valid instances are penalized as false positives. The paper also reports low initial inter-annotator agreement (0.24 and 0.28), making annotation incompleteness plausible. The macro-F1 gap (0.16 vs. 0.67) is large, but the gap itself does not make the assumption harmless: if a substantial fraction of LLM 'false positives' are correct detections, precision and F1 for LLMs are systematically underestimated, and the comparative claim may not hold. No error analysis or adjudicated relabeling is provided to rule this out.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether large language models can detect propaganda techniques in news articles. Using the Propaganda Techniques Corpus (PTC) and focusing on six techniques, the authors evaluate GPT-3.5, GPT-4, and Claude 3 Opus under five prompting settings (zero-shot, one-shot, chain-of-thought, generated knowledge, and self-consistency), plus a fine-tuned GPT-3.5 model. They compare these against two transformer baselines: a RoBERTa-CRF ensemble (Jurkiewicz et al. 2020) and a Multi-Granularity Network (MGN, Da San Martino et al. 2019). The central finding is that none of the LLMs reach the macro-F1 of the RoBERTa-CRF baseline (0.67), with GPT-4 achieving the highest LLM macro-F1 at 0.16. The paper also reports that GPT-3.5 and GPT-4 outperform the MGN baseline on a few individual techniques, and GPT-4 outperforms the MGN model in macro-F1 under some prompting strategies.","tokens_in":10567,"tokens_out":2143,"duration_ms":20570,"significance":"If the central result is accepted, the paper provides a useful, if sobering, data point: general-purpose LLMs are not yet competitive with fine-tuned supervised models on fine-grained propaganda technique detection, despite strong performance on many other NLP tasks. The study also contributes a comparison across prompting strategies and across three LLMs, and it is careful to compare against a strong SemEval-winning baseline and a dataset-author baseline. The reporting of precision and recall in the appendix is a strength. However, the validity of the headline comparison depends on methodological choices that need to be addressed, in particular the selection of the best prompt per setting and the completeness of the gold annotation set.","major_comments":[{"comment":"The paper reports, for each LLM and prompt setting, the F1 score of the prompt variation that gave the highest macro-F1 (Discussions: 'The F1 scores reported in Table 2 correspond to prompts that gave us the highest macro-F1 across these variations'). This is a post-hoc selection on the test set, which inflates the reported scores and makes the comparison with the baselines unfair in an unpredictable direction. The central claim that LLMs underperform the RoBERTa-CRF baseline is large enough to survive this bias, but the narrower claims about which prompting strategy is best, and the detailed GPT-4-vs-MGN comparisons, are not supported without reporting the full distribution of results across prompt variations or an averaged score with standard deviation.","section":"Discussions"},{"comment":"The comparison assumes the PTC human annotations are the complete and correct set of propaganda instances. The paper itself concedes in the Discussions that low precision 'raises the question of whether the LLMs are identifying instances of propaganda techniques that were missed by human annotators in the original dataset.' This is load-bearing: the RoBERTa-CRF baseline is trained on the same gold labels and can achieve high F1 by reproducing annotator decisions, including omissions, while an LLM that correctly identifies an instance absent from the gold set is penalized as a false positive. The reported low inter-annotator agreement (0.24 and 0.28 in the initial annotation stage) makes incomplete annotations plausible. The paper should include an error analysis or an adjudicated/relabeled sample to distinguish genuine false positives from potentially correct detections missing from the gold set; without this, the size of the gap (0.16 vs 0.67) does not by itself rule out a systematic underestimation of LLM precision.","section":"Results / Discussions"},{"comment":"No error bars, confidence intervals, or significance tests are reported, and it is not stated whether the LLM API calls were run at temperature 0 or with multiple seeds, nor how many runs were averaged. Several claimed individual-technique improvements over the MGN baseline are numerically small (e.g., GPT-4 macro-F1 0.16 vs MGN 0.14; GPT-3.5 generated-knowledge appeal-to-fear F1 0.16 vs MGN 0.09). Without run-to-run variance, these differences cannot be distinguished from noise. At minimum, the temperature/decoding settings and number of runs should be given, and for the close comparisons the authors should report variance or a small statistical test.","section":"Results / Appendix Tables 4-6"}],"minor_comments":[{"comment":"The abstract states 'we find that all three LLMs outperform a MultiGranularity Network (MGN) baseline in detecting instances of one out of six propaganda techniques (name-calling)' while the Results say all versions of the LLMs outperformed the MGN for name-calling. The abstract phrasing is slightly awkward; consider saying 'for name-calling' instead of 'in detecting instances of one out of six techniques'.","section":"Abstract"},{"comment":"The paper does not specify how the LLM JSON outputs were parsed, how malformed outputs were handled, or whether the model was allowed to abstain. These details affect precision and recall and should be reported for reproducibility.","section":"Methods / Models"},{"comment":"Table 2 is dense and does not include precision and recall, which are central to the annotation-completeness concern. Consider moving precision and recall into the main table, or at least highlighting the low-precision cells in the discussion.","section":"Table 2"},{"comment":"There are minor grammatical issues, e.g., 'GPT-3.5 and GPT-4 performs better' should be 'perform better', and 'GPT-3.5 and GPT-4 outperforms' should be 'outperform'. These do not affect the technical content but should be corrected.","section":"Discussions"}],"recommendation":"major_revision","confidential_remarks":"The paper's central negative result appears robust, but the manuscript currently reports only best-prompt scores and does not address the gold-label completeness issue that the authors themselves raise. I would ask for a revised version that reports the full prompt-variation distribution and includes at least a small adjudication or error analysis. The paper is within scope for a CS/CL venue and the topic is timely; with these revisions it could become a solid empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe short version: clean, well-documented negative result. On the PTC benchmark, the best LLM (GPT-4 with a couple of prompting strategies) tops out at 0.16 macro-F1 against 0.67 for a RoBERTa-CRF ensemble. That gap is large enough that the headline is probably robust, but read it as a result about the PTC gold labels, not about LLM capability in general.\n\nWhat's actually new: a systematic comparison of six prompting strategies across three LLMs on fine-grained propaganda detection, with a fine-tuned GPT-3.5 thrown in. The paper does this honestly—full precision/recall tables in the appendix, prompt templates included, and comparisons against two published baselines plus other SemEval submissions. It also names its own biggest threat in the Discussion: the PTC annotations are incomplete (initial inter-annotator agreement was 0.24-0.28), so some of the LLMs' 'false positives' could be real detections that human annotators missed. That is the exact right caveat, though the paper doesn't then go and check it.\n\nSoft spots, in proportion: (1) The gold-label issue is real but probably not enough to flip the 0.16-vs-0.67 gap—the recall values are too low to recover the deficit. Still, per-technique conclusions could shift, and any strong claim that LLMs 'can't' detect propaganda needs an adjudication pass first. (2) They report the best macro-F1 over several prompt variations with no error bars or significance tests. That's a mild selection bias, common in this literature, but it should be acknowledged more bluntly. (3) The abstract overstates the appeal-to-fear comparison: GPT-4 is 'on par or above' the MGN baseline in the body, but 'outperforming' in the abstract. Minor. (4) Fine-tuning was only done for GPT-3.5, so the 'fine-tuning doesn't help' conclusion is model-limited.\n\nBottom line: this is a useful, honest empirical study for anyone working on content moderation or propaganda detection, and a good cautionary data point for LLM-as-specialist claims. It deserves a serious referee; with an error analysis or at least an explicit caveat that the comparison is to the gold labels as given, it's conditionally acceptable.\n\nBest,\n[Your name]","headline":"A transparent and largely convincing negative result on LLMs for fine-grained propaganda detection, but the gold-label completeness issue and a few reporting choices keep it from being the final word.","tokens_in":11075,"tokens_out":6085,"would_cite":true,"duration_ms":50151,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Despite advanced prompting, none of the GPT-3.5, GPT-4, and Claude 3 Opus models outperforms a RoBERTa-CRF ensemble baseline on phrase-level propaganda detection in news articles.","keywords":["propaganda detection","large language models","prompt engineering","Propaganda Techniques Corpus","fine-grained text classification","macro-F1","GPT-4"],"falsifier":"Take a random sample of LLM predictions scored as false positives, have fresh annotators judge whether each span matches the published technique definitions, and compare the adjudicated labels with the original gold set. If a substantial share of the false positives are genuine missed annotations, the reported F1 gap would shrink or disappear.","tokens_in":10126,"feed_emoji":"📰","tokens_out":4767,"duration_ms":36847,"temperature":0.7,"pith_summary":"This paper asks whether general-purpose large language models can detect six propaganda techniques in news articles at phrase level. It finds that none of the tested LLMs—GPT-3.5, GPT-4, and Claude 3 Opus, under zero-shot, one-shot, chain-of-thought, generated-knowledge, and self-consistency prompting—matches the RoBERTa-CRF ensemble baseline on macro-F1 or on any individual technique. GPT-4's best macro-F1 is 0.16, against the baseline's 0.67. The LLMs do beat the weaker Multi-Granularity Network baseline on name-calling, and GPT-4 also beats it on macro-F1 under several prompt settings. A sympathetic reader would take this as evidence that current prompting strategies do not close the gap to fine-tuned models on fine-grained propaganda detection.","feed_headline":"GPT-4 can't beat a 2020 model at spotting propaganda","feed_subtitle":"Best GPT-4 macro-F1 of 0.16 versus 0.67 for the RoBERTa-CRF ensemble on six techniques.","key_machinery":"The central object is the evaluation setup: the PTC dataset's phrase-level gold annotations for six propaganda techniques, combined with a macro-F1 metric that averages per-technique F1. The comparison is anchored by two baselines—a RoBERTa-CRF model with an ensemble classifier from the SemEval 2020 task, used as the upper bound, and the Multi-Granularity Network released with the dataset, used as the lower bound. The prompting strategies (zero-shot, one-shot, chain-of-thought, generated knowledge, self-consistency) are the intervention being tested. The metric and baselines together determine the claim: any LLM score is interpreted relative to those anchors.","core_discovery":"The central claim is that large language models, even with advanced prompting, are not yet competitive with a fine-tuned transformer baseline for detecting propaganda techniques in news articles. On the Propaganda Techniques Corpus, the strongest LLM result, GPT-4 with generated-knowledge prompting, reaches a macro-F1 of 0.16, far below the RoBERTa-CRF ensemble's 0.67. The paper reports this gap for every one of the six techniques: none of the LLMs exceeds the baseline on name-calling, loaded language, doubt, appeal to fear, flag-waving, or exaggeration/minimization. Against a lower-bound baseline, the MGN model, GPT-4 wins on macro-F1 under one-shot, generated-knowledge, and self-consistency prompting, and all LLMs beat it on name-calling.","pith_inferences":["The low precision values might mean LLMs flag correct propaganda instances the PTC annotators missed; if so, the headline gap is partly an artifact of treating the gold set as complete. Re-annotating the disagreements would settle this.","A hybrid pipeline—LLMs proposing span candidates and the RoBERTa-CRF model confirming labels—could combine the LLM's recall on some techniques with the baseline's precision.","Because the study covers only six of eighteen techniques, the results may not transfer to more subtle or context-dependent techniques such as repetition or whataboutism."],"forward_implications":["Fine-tuned transformer models remain the practical choice for automated propaganda annotation; LLM prompting alone is not a drop-in replacement.","For name-calling, and for appeal to fear and flag-waving with certain prompts, LLMs beat the weaker MGN baseline, so the technology has partial traction on emotionally loaded techniques.","Prompting strategy matters: GPT-4's best macro-F1 comes only under one-shot, generated-knowledge, and self-consistency prompting, not zero-shot.","The large precision-recall imbalance in LLM outputs means any deployment would need a separate verification step to be usable."],"supporting_citations":[{"why":"Supplies the PTC dataset, the six technique definitions, and the MGN lower-bound baseline.","marker":"Da San Martino et al. (2019)"},{"why":"The RoBERTa-CRF ensemble that serves as the upper-bound baseline; the paper's main negative result is measured against it.","marker":"Jurkiewicz et al. (2020)"},{"why":"Defines the SemEval 2020 Task 11 evaluation protocol from which the baselines and metric come.","marker":"Martino et al. (2020)"},{"why":"Introduces chain-of-thought prompting, one of the five strategies evaluated.","marker":"Wei et al. (2022)"},{"why":"Introduces self-consistency prompting, another evaluated strategy.","marker":"Wang et al. (2022)"},{"why":"Introduces generated knowledge prompting, the strategy that gives GPT-4 its best macro-F1.","marker":"Liu et al. (2021)"}],"fun_headline_variants":["LLMs flop against fine-tuned baseline on propaganda detection","Propaganda detection: GPT-4 scores 0.16, baseline 0.67","Why GPT-4 fails to beat a 2020 model at propaganda spotting","GPT-4 F1 0.16 vs RoBERTa's 0.67 on propaganda detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The PTC human annotations are treated as the complete set of correct propaganda spans, so every LLM detection that the annotators did not record is counted as an error.","fun_headline_variants_meta":{"raw":{"variants":["LLMs flop against fine-tuned baseline on propaganda detection","Propaganda detection: GPT-4 scores 0.16, baseline 0.67","Why GPT-4 fails to beat a 2020 model at propaganda spotting","GPT-4 F1 0.16 vs RoBERTa's 0.67 on propaganda detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000776,"raw_usage":{"total_tokens":3409,"prompt_tokens":896,"completion_tokens":2513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":2423}},"tokens_in":512,"tokens_out":2513,"duration_ms":16353,"temperature":1.0,"reasoning_tokens":2423,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:10:13.678419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of LLM predictions scored as false positives, have fresh annotators judge whether each span matches the published technique definitions, and compare the adjudicated labels with the original gold set. If a substantial share of the false positives are genuine missed annotations, the reported F1 gap would shrink or disappear.","supporting_citations":[],"review_version":1}