{"id":"75ad4de2-4d08-4bda-bd6e-1b107674d8fb","arxiv_id":"2412.16486","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"On 1,847 COVID-19 tweets, GPT-4 scored highest at detecting and classifying scientific claims (F1 0.65-0.76), but the evaluation lacks error bars, baselines, and open artifacts.","lead":"This study tested five AI models on whether COVID-19 tweets contain scientific claims and whether those claims are verifiable, using human-labeled tweets as the benchmark. GPT-4 scored highest, but the evaluation is too thin to prove that result, with no error bars, baselines, code, or data provided.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's central ranking is self-contradictory: §7 says GPT-3.5 Verifiable F1=0.87, Table 5 says 0.70; the correct value decides whether GPT-4 actually has the top F1.","rationale":"I read the paper's central claim as an empirical ranking: GPT-4 achieves the highest F1 on both tasks. For that claim to be supported, the reported F1 values must be a faithful, internally consistent record of the evaluation. The paper violates this in a way that directly affects the ranking: §7 assigns GPT-3.5 a Verifiable F1 of 0.87, while Table 5 shows 0.70. The arithmetic from P and R in Table 5 favors 0.70, but the text's 0.87 is not flagged as an error; the sentence 'It outperformed the Llama 2 models' is only true under 0.87. If the true value is 0.70, the central claim survives but the paper contains a false comparative statement and a factor-of-0.17 discrepancy with no errata. If the true value is 0.87, the central claim is false because GPT-4's 0.76 would be the second-best Verifiable score. The paper's lack of raw outputs, unspecified prompt aggregation, and absence of repeated runs compound the problem, but the self-contradiction is the point where the argument first breaks. The reader's verdict of REJECT is appropriate; no change.","tokens_in":13048,"tokens_out":5112,"duration_ms":38456,"concrete_test":"Obtain the raw model predictions for GPT-3.5 and GPT-4 (from the authors or by re-running the same 1,847 tweets with the identical prompts) and recompute the Verifiable-task F1 for GPT-3.5. Also identify the prompt selection rule: which of the five §5.1 prompts was used and how outputs were aggregated. If GPT-3.5's Verifiable F1 is 0.87, GPT-4 is not the best in that task and the headline ranking fails. If it is 0.70, the table is arithmetically consistent but the text's 'outperformed' sentence is false, and the reported evidence is still too unreliable to support the central claim without correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in Section 7 and the abstract, is that GPT-4 registered the highest F1-scores of 0.65 (Claim Existence) and 0.76 (Verifiable). This claim depends directly on the Verifiable-task scores being reported consistently. They are not. Section 7 text for GPT-3.5 says: 'significantly better performance in the Verifiable task with an F1-score of 0.87' and 'It outperformed the Llama 2 models in detecting verifiable claims.' Table 5, however, reports GPT-3.5 Verifiable P=0.77, R=0.66, A=0.59, F1=0.70. Recomputing from P and R, 2*0.77*0.66/(0.77+0.66) ≈ 0.711, which supports the table's 0.70 and contradicts the text's 0.87. Under the table, GPT-3.5 (0.70) is not above Llama-2 7B (0.72) or 13B (0.71); the text's comparative statement is false. Under the text's 0.87, GPT-3.5 would beat GPT-4's 0.76, so 'GPT 4 outshone all other models' is false. Since the paper does not resolve this and no raw predictions or code are provided, the numerical evidence for the ranking is self-contradictory. Furthermore, it is impossible to tell which prompt version (or aggregation) produced the reported scores, so a reader cannot even decide which of the two numbers is the intended one. Thus the most load-bearing condition — reliable, internally consistent outcome measures for the model comparison — is not met.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates five large language models (GPT-3.5, GPT-4, Llama 2 7B/13B/70B) on two binary classification tasks derived from a published, manually annotated COVID-19 tweet dataset: detecting whether a claim exists (Claim Existence) and whether it is scientifically verifiable (Verifiable). Five system-prompt variants are described, and per-model accuracy, precision, recall, and F1 are tabulated. The central claim is that GPT-4 achieves the highest F1 on both tasks and that LLMs have significant potential as automated fact-checking tools.","tokens_in":13432,"tokens_out":5876,"duration_ms":49015,"significance":"If the reported comparison were reliable, the paper would offer a modest empirical data point on zero-shot claim detection and classification by current LLMs on a specialized, manually annotated dataset. The manuscript is transparent about its dataset source, model families, and decoding hyperparameters, which are strengths. However, the reliability of the headline ranking is undermined by an internal numerical contradiction in the GPT-3.5 Verifiable F1 value, the unspecified protocol for aggregating results across five prompt variants, and the complete absence of variance or significance analysis. These issues make the core comparative claim unsupported as stated.","major_comments":[{"comment":"The text of the GPT 3.5 paragraph states 'a significantly better performance in the Verifiable task with an F1-score of 0.87', but Table 5 reports F1 = 0.70 for the same task. Recomputing F1 from Table 5's precision (0.77) and recall (0.66) gives 2*0.77*0.66/(0.77+0.66) ≈ 0.711, which supports the table and contradicts the text. This inconsistency is load-bearing: if 0.70 is correct, the sentence 'It outperformed the Llama 2 models in detecting verifiable claims' is false (Llama 2 7B and 13B both have higher Verifiable F1); if 0.87 is correct, then GPT-4's Verifiable F1 of 0.76 is not the highest, so the claim that 'GPT 4 outshone all other models' is false. The paper does not resolve this contradiction, and no raw predictions or code are provided, so the reader cannot determine which value is intended.","section":"Section 7 and Table 5"},{"comment":"The paper introduces five distinct system prompt versions (FSP, Guidelines+FSP, Guidelines+FSP+Emotional Stimuli, Chain of Thought, CARP) but reports only a single set of metrics per model-task in Section 7. It is never specified whether the reported numbers correspond to the best-performing prompt per model, an average across prompts, or some other aggregation rule. Because prompt design is known to have a large effect on LLM classification performance, the absence of an aggregation protocol makes the results impossible to reproduce and prevents the reader from assessing whether the reported ranking is robust to prompt choice.","section":"Section 5.1 and Section 7"},{"comment":"All reported metrics derive from a single evaluation pass with no repeated runs, confidence intervals, or statistical significance tests. In this setup, the differences between GPT-4 and GPT-3.5 are small: 0.04 F1 on Claim Existence (0.65 vs 0.61) and, under Table 5, 0.06 F1 on Verifiable (0.76 vs 0.70). Given the nondeterminism inherent in LLM decoding and API serving (even at temperature 0.2), these gaps are within plausible random fluctuation. The paper's assertion that GPT-4 'outshone' other models is therefore not statistically justified.","section":"Sections 7 and 8"},{"comment":"The paper motivates LLMs as superior to traditional machine learning models, citing SciBERT- and RoBERTa-based detectors in the related work, but it includes no non-LLM baseline in the experiments. Without comparing against such a baseline, the paper cannot support its broader claim that LLMs 'bypass' overfitting or offer a more adaptable solution. The empirical contribution is limited to a comparison among a small set of LLMs, and the stronger positioning in the abstract and introduction is untested.","section":"Entire paper (Related Work and Results)"}],"minor_comments":[{"comment":"The F1 entry for the Verifiable task is written as '0.0.72'; this appears to be a typographical error that should read '0.72'.","section":"Table 2"},{"comment":"The prompt code blocks contain LaTeX macro artifacts such as 'Var' and 'quotedbl.Var', which render the example prompts as corrupted and difficult to read. These should be cleaned up for presentation.","section":"Section 5.1"},{"comment":"The paper lists both Top P = 0.4 and Top K = 50 in the parameter settings but does not state whether both were applied jointly or whether one overrode the other. Many inference APIs treat top-p and top-k as alternative sampling strategies, so the actual decoding behavior is ambiguous.","section":"Section 5.3"},{"comment":"The model descriptions lack precise version identifiers (e.g., which GPT-3.5 snapshot or Llama 2 checkpoint was used), which is important for reproducibility since provider-side updates can change behavior.","section":"Section 4.3"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be a course project (header 'CS 533, September 2023') and contains significant formatting artifacts. The internal inconsistency in the GPT-3.5 Verifiable F1 value between Section 7 text and Table 5 is a serious data-integrity concern; combined with the lack of baselines and statistical analysis, the central comparison is not reliable. If the authors could provide the raw outputs and clarify the aggregation, a revised version might be reconsidered, but as submitted the paper does not meet the standard for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this as a course-project report, not a research paper. The headline result—GPT-4 outshines all models on COVID-19 claim detection—is undercut by a direct internal contradiction. Section 7 says GPT-3.5's Verifiable F1 is 0.87; Table 5 reports 0.70. Recomputing from the precision and recall in the table gives 0.71, so the table value is internally consistent and the text's 0.87 is likely an error. But this isn't a minor typo: if 0.87 is right, GPT-3.5 beats GPT-4's 0.76 and the abstract's conclusion is false; if 0.70 is right, GPT-3.5 ranks below the Llama-2 7B and 13B models, and the text's 'outperformed the Llama 2 models' is false. The paper never acknowledges or resolves this.\n\nWhat the paper does well: it uses an externally annotated dataset (Zuo et al., 2022) with the SciTweets annotation framework, so there's no circularity. It documents five prompt variants and the decoding hyperparameters (temperature 0.2, top-p 0.4, etc.) in enough detail to reproduce. It also reports precision, recall, and F1, which is better than only accuracy. The observation that Llama 2 7B and 13B beat the 70B on the Verifiable task is counterintuitive and genuinely interesting if it survives a proper protocol.\n\nThe soft spots are load-bearing. The evaluation is a single run: no repeated trials, no error bars, no significance tests. The gaps between GPT-4 and GPT-3.5 are 0.04–0.06 F1, which is exactly the kind of difference that sampling or API nondeterminism can produce. The paper never says how scores from the five prompt versions were combined—one prompt? best-of? average?—so the reported numbers are ambiguous even setting the 0.87/0.70 issue aside. No code or raw predictions are released, and Table 2 has a typo ('0.0.72'). These are fixable, but they mean the paper's only real finding—the model ranking—is not currently supported by the evidence.\n\nWho this is for: someone wanting a compact example of prompt templates for claim classification, or a pointer to the dataset. Not for anyone needing reliable evidence about LLM capability.\n\nMy recommendation: don't publish as is. The experiment is cheap to repair: rerun with multiple seeds, report mean and standard deviation, specify prompt aggregation, correct the internal inconsistency, and release the predictions. A fixed version might be acceptable for a workshop or as a negative result in a broader study. As submitted, it doesn't meet the standard for a serious peer-reviewed venue, but the underlying work isn't worthless—it needs to be redone carefully.","headline":"The paper's central claim collapses on its own tables: GPT-3.5's Verifiable F1 is 0.87 in the text and 0.70 in Table 5, and the correct number determines whether GPT-4 actually wins.","tokens_in":13926,"tokens_out":5130,"would_cite":false,"duration_ms":40932,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4, prompted but not fine-tuned, detects and classifies COVID-19 scientific claims in tweets better than GPT-3.5 and three Llama 2 models, the paper reports.","keywords":["Large Language Models","scientific claim detection","COVID-19 misinformation","fact-checking","GPT-4","Llama 2","Twitter dataset","prompt engineering"],"falsifier":"Run each model multiple times on the same 1,847 tweets with different random seeds at the same temperature, compute 95% confidence intervals for the $F_1$ scores; if the intervals for GPT-4 and GPT-3.5 overlap, the paper's claim that GPT-4 'outshone' the others collapses to 'statistically indistinguishable.'","tokens_in":12850,"feed_emoji":"🦠","tokens_out":5175,"duration_ms":39944,"temperature":0.7,"pith_summary":"The paper sets out to show that off-the-shelf large language models can spot scientific claims about COVID-19 in tweets and judge whether those claims are verifiable, all without task-specific training. Using 1,847 manually annotated tweets and prompt-engineering techniques like few-shot examples and chain-of-thought reasoning, the study reports GPT-4 as the top performer, with $F_1$ scores of 0.65 for claim existence and 0.76 for verifiability. Because the setup requires no fine-tuning, the authors argue LLMs could be deployed quickly as automated fact-checking tools during public-health crises.","feed_headline":"GPT-4 beats Llama 2 at spotting COVID-19 claims in tweets","feed_subtitle":"No fine-tuning needed: a prompt-driven LLM pipeline flags verifiable COVID-19 claims with F1 up to 0.76.","key_machinery":"The load-bearing mechanism is the two-part label scheme of claim existence and scientific verifiability, taken from an existing annotation framework, combined with system prompts that direct the models to act as a 'COVID-19 Tweets Classifier.' The prompts mount few-shot examples, explicit guidelines, emotional cues, chain-of-thought steps, and clue-reasoning under a single deterministic decoding configuration (temperature 0.2, top-p 0.4), extracting labels directly from generative LLMs without any fine-tuning.","core_discovery":"In the paper's own terms, the central discovery is that GPT-4 outperforms every other tested model on both tasks, reaching an $F_1$ of 0.65 for deciding whether a tweet contains a scientific claim and 0.76 for deciding whether that claim is verifiable against scientific evidence. GPT-3.5 is close behind (0.61 and 0.70), while the Llama 2 family trails, with the 70B model surprisingly weaker on verifiability (0.63) than the 7B and 13B variants (0.72 and 0.71). Across all models, precision runs higher than recall, so the systems rarely raise false alarms but routinely miss actual claims.","pith_inferences":["A single evaluation pass with fixed decoding settings cannot distinguish genuine model differences from API nondeterminism; rerunning GPT-3.5 and GPT-4 with multiple seeds would test whether the reported $F_1$ gaps are stable.","Extending the best model with retrieval-augmented generation, as the paper's future-work section suggests, could convert a claim detector into a claim verifier that cites the scientific literature for each flagged claim.","For public-health practice, the low recall numbers imply these tools are best used as a first-pass triage to queue tweets for human review, rather than as an autonomous filter."],"forward_implications":["A prompt-only LLM pipeline could categorize COVID-19 tweets in hours, far faster than training a supervised classifier.","The measured ranking suggests model size is not a reliable predictor: the 70B Llama scored lowest on verifiability.","Deployments would benefit from recall-boosting post-processing because all models miss many true claims.","The successful use of an established annotation framework means existing claim datasets can be reused as few-shot examples for new LLM applications."],"supporting_citations":[{"why":"Supplies the 1,847-tweet COVID-19 Twitter dataset with manual claim-existence and verifiability labels that all models are evaluated against.","marker":"[18]"},{"why":"Defines the annotation criteria for scientific claim existence and verifiability that shape both the dataset labels and the system prompts.","marker":"[3]"},{"why":"Provides the CARP (Clue and Reasoning Enhanced Prompting) technique used as one of the five prompt versions.","marker":"[13]"}],"fun_headline_variants":["LLMs spot COVID claims: GPT-4 leads, no fine-tuning needed","GPT-4 tops LLM fact-checker test on COVID tweets","No training? LLMs still catch COVID claims in tweets","GPT-4 best at flagging verifiable COVID claims on Twitter","LLM fact-checking: GPT-4 wins without any fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured differences between models are treated as real even though each model was run once with no repeated runs, no confidence intervals, and no significance tests, so the 0.04–0.06 $F_1$ gaps between GPT-4, GPT-3.5, and the top Llama could be sampling or API randomness.","fun_headline_variants_meta":{"raw":{"variants":["LLMs spot COVID claims: GPT-4 leads, no fine-tuning needed","GPT-4 tops LLM fact-checker test on COVID tweets","No training? LLMs still catch COVID claims in tweets","GPT-4 best at flagging verifiable COVID claims on Twitter","LLM fact-checking: GPT-4 wins without any fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000612,"raw_usage":{"total_tokens":2807,"prompt_tokens":866,"completion_tokens":1941,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":1849}},"tokens_in":482,"tokens_out":1941,"duration_ms":11635,"temperature":1.0,"reasoning_tokens":1849,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:31:40.415158+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each model multiple times on the same 1,847 tweets with different random seeds at the same temperature, compute 95% confidence intervals for the $F_1$ scores; if the intervals for GPT-4 and GPT-3.5 overlap, the paper's claim that GPT-4 'outshone' the others collapses to 'statistically indistinguishable.'","supporting_citations":[],"review_version":1}