{"id":"98c5aa3e-ea73-4613-8c5c-a97a3fcdfadb","arxiv_id":"2501.13802","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuned GPT-3.5-turbo agrees with expert climate coders on social media claims as often as the experts agree with each other (alpha=0.89), but the study's open-source benchmark is weakened by a flawed prompt and random replacement of invalid responses.","lead":"This paper benchmarks 16 LLMs and a fine-tuned GPT-3.5-turbo on classifying false or misleading climate claims, then compares the best model to two expert climate communicators. The fine-tuned model matches expert labels closely, but the open-source versus proprietary comparison rests on a prompt that appears to mislabel a claim category.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random substitution of invalid LLM responses in §5.1 conflates non-compliance with classification error, so the claimed open-source underperformance in Table 1 is not established.","rationale":"I agree with the reader's conditional verdict but locate the most load-bearing weakness differently. The expert-annotator overlap is a real external-validity concern, but the fine-tuned model's performance on the social media sample is at least measured by an internally consistent protocol; the comparison against independent experts is a generalizability question. By contrast, the random substitution protocol in §5.1 directly undermines the benchmark that supports the paper's first and second headline claims. If invalid responses are replaced with random labels, the model's output distribution is no longer a function of the text; it is a mixture of the model's classifier and a uniform random draw. For models with high invalid rates, Table 1 scores are mostly noise. The corrupted 2_1 label in Appendix A.8 gives open-source models an erroneous taxonomy, further biasing the comparison. The proposed test—recomputing F1 with valid responses only and simulating the random-replacement baseline—would determine whether the open-source gap survives. Until then, the paper's comparative claims should be treated as conditional.","tokens_in":20775,"tokens_out":5402,"duration_ms":49295,"concrete_test":"Recompute Table 1 under a corrected protocol: (i) exclude invalid responses and report the valid-response rate for every model; (ii) re-run open-source models with the corrected A.8 prompt (fix 2_1 to 'It's natural cycles/variation'); (iii) run a Monte Carlo baseline that replaces invalid responses with random labels at each model's observed invalid rate and compare to Table 1. If open-source macro-F1 values remain within one standard error of the random-replacement baseline, or if any open-source F1 rises above 0.40 after the corrections, the central underperformance claim is an artifact of the evaluation protocol and should be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's first empirical contribution—that open-source models substantially underperform proprietary ones in classifying climate misinformation—rests on Table 1, which is computed after replacing every invalid or unparseable model output with a label drawn uniformly at random from 28 possible labels (Section 5.1). This protocol does not measure classification skill: a model that refuses to answer on a large fraction of the 2,904 test paragraphs (see Figure 2) will have its score driven toward a random baseline by the imputed labels. The reported F1 values for open-source models (e.g., Mistral-7B-Instruct 0.28, Llama-3.1-8B 0.10, Llama-2-7B-chat 0.06) are therefore uninterpretable as 'underperformance' at the task. A fair evaluation must report compliance rates separately and either (a) score only valid responses, (b) treat invalid responses as an explicit 'unparseable' class, or (c) use the model's abstention as an informative signal. Additionally, the open-source prompt in Appendix A.8 contains a mislabeled category: '2_1' is listed as 'They changed the name from global warming to climate change', which duplicates '1_8' instead of the correct 'It's natural cycles/variation'. Thus even valid open-source responses were generated under a corrupted taxonomy. These two issues together make the open-source vs proprietary comparison in RQ1/RQ2 unreliable as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates LLMs for classifying false or misleading climate claims. Using the CARDS test set (N=2,904), it compares zero-shot proprietary and open-source LLMs against the RoBERTa-based CARDS model, then fine-tunes GPT-3.5-turbo on CARDS training data and evaluates it both on that test set and on a curated social-media sample (914 paragraphs) labeled by two climate-communication experts. The paper reports that (i) open-source models substantially underperform proprietary models, (ii) CARDS outperforms several proprietary models, and (iii) fine-tuned GPT-3.5-turbo matches expert-level agreement on the social-media sample (macro F1=0.88, Krippendorff's alpha=0.89). It argues that this demonstrates the value of expert-annotated data and human oversight for LLM-based governance tools.","tokens_in":21000,"tokens_out":9394,"duration_ms":83217,"significance":"If fully supported, the paper would provide a useful demonstration that fine-tuning a small, inexpensive proprietary model on expert-annotated data can outperform larger general-purpose models and a task-specific BERT model on an external test set, and can approach expert-level coding on out-of-distribution social-media text. The use of the external CARDS held-out test set for the fine-tuning evaluation, the explicit comparison to expert inter-coder reliability, and the release of the expert-coded sample are strengths that make the core fine-tuning result plausible and non-circular. However, the two headline comparisons—open-source versus proprietary underperformance and expert-level equivalence—currently rest on methodological choices that need repair. With those repairs, the paper could be a valuable case study for human-oversight alignment in governance applications.","major_comments":[{"comment":"Replacing every invalid or unparseable LLM output with a label drawn uniformly at random from 28 possible labels means that models with high non-compliance rates (Figure 2) are scored near a random baseline. The F1 values for open-source models in Table 1 therefore conflate instruction-following with classification skill, and the paper's first contribution—that open-source models substantially underperform—is not established as stated. Please report valid-response rates per model and either (a) evaluate on valid responses only, (b) introduce an explicit 'unparseable/refusal' class, or (c) model abstention separately; in all cases, report uncertainty around the metrics.","section":"§5.1, Table 1, Figure 2"},{"comment":"The prompt template for open-source models contains a corrupted taxonomy: category '2_1' is defined as 'They changed the name from global warming to climate change', which duplicates '1_8', while the actual CARDS category 'It's natural cycles/variation' is absent. Open-source valid responses were therefore generated under a different label space than the proprietary models and the CARDS ground truth, independently compromising the RQ1/RQ2 comparison even for compliant outputs. The open-source evaluation must be rerun with a corrected prompt, or the paper should report results only for a corrected subset.","section":"Appendix A.8"},{"comment":"The expert labels used as ground truth for the social-media evaluation were produced by two of the paper's authors applying the same CARDS taxonomy used to fine-tune the model, and the final labels are a reconciled consensus rather than independent codings. The reported model-expert agreement (alpha=0.89, F1=0.88) is therefore not directly comparable to the reported inter-expert alpha=0.89, which measures agreement between two independent codings; a model can match a consensus label more easily than it can match either expert. Please report agreement with each expert separately, state whether annotators were blind to model outputs, and discuss the shared-taxonomy and author-labeling threat to the RQ4 claim of expert-level performance.","section":"§4.4, §5.2, Table 2"},{"comment":"The open-source and proprietary models were evaluated with different prompt templates (the full coding-manual JSON prompt versus the shortened classes prompt), different inference temperatures (0 versus 0.001), and different decoding paths. The reported substantial gap in Table 1 cannot be cleanly attributed to model family or openness because prompt format and compliance are confounded with model choice. The paper should use an as-identical-as-possible evaluation protocol, or explicitly test whether the open-source results change under the proprietary-style prompt.","section":"§4.1, Appendices A.4 and A.8"}],"minor_comments":[{"comment":"The paper refers to '28 possible labels' (Section 5.1) and '27 false or misleading claim labels' (Section 4.3), but the taxonomy in Appendices A.4 and A.8 contains 26 claim labels plus the no-claim class; please correct these counts.","section":"§5.1 and §4.3"},{"comment":"Please clarify whether the macro-averaged F1 includes the no-claim class; Table 1 says 'five categories of super-claims' while Table 2 includes a no-claim category, and the two tables should use a consistent evaluation scope.","section":"Table 1"},{"comment":"There are numerous typos, including 'reproducability' (§4.1), 'intercoder-relability' (§5.2), 'Krippendorf' (§5.2), and 'Videgen' (reference [46]); please proofread carefully.","section":"Throughout"},{"comment":"The alignment and misalignment symbols in the table appear to be missing or not rendered, making the illustrative examples hard to interpret; please ensure the glyphs display correctly.","section":"Appendix A.11"},{"comment":"The performance differences among the proprietary models (e.g., GPT-4o 0.75 versus GPT-4 0.74) are likely within noise; reporting bootstrap confidence intervals would make the comparison more informative.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The strongest citable result is the fine-tuned model's performance on the CARDS test set; the open-source comparison in its current form should not be used as a headline. If the authors can re-run the open-source evaluation with a corrected prompt and compliance-aware scoring, and if the expert-labeling protocol is clarified, the paper would be publishable. I would also ask the editor to weigh whether the workshop-format page limit contributed to the imprecision in Appendix A.8; in a journal version, this must be corrected before the comparison can be taken seriously."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the fine-tuned GPT-3.5-turbo result on the social media sample is the real contribution; the open-source versus proprietary benchmark is not trustworthy as reported. The paper does something genuinely new: it benchmarks 16 LLMs on the CARDS taxonomy, fine-tunes a small commercial model on expert labels, and evaluates the fine-tuned model against two senior experts on a fresh social-media sample. That last piece is the most valuable. The authors curate 71k low-credibility articles, sample 914 paragraphs, and show the fine-tuned model agrees with expert labels at alpha=0.89, the same as the experts' own intercoder agreement, with macro F1=0.88. That is a concrete, reproducible result worth taking seriously.\n\nThe soft spots are real and they hit the first headline claim. Section 5.1 replaces every invalid or unparseable model output with a randomly sampled label from 28 classes. For models with high invalid-response rates (Figure 2 shows several open-source models failing on a large fraction of the 2,904 test paragraphs), this drives scores toward a random baseline. Those F1 numbers in Table 1 for Mistral, Llama, etc. are not measuring classification skill; they are measuring a mix of compliance and chance. The stress-test note also catches a genuine error in Appendix A.8: category 2_1 is listed as 'They changed the name from global warming to climate change', duplicating 1_8 instead of 'It's natural cycles/variation'. So even valid open-source responses were generated under a corrupted prompt. These two issues jointly invalidate RQ1/RQ2 as reported.\n\nThe expert-alignment result is less damaged but not clean. The two expert coders are also authors, and they applied the same CARDS taxonomy the model was fine-tuned on. That introduces a mild self-referential element. The alpha=0.89 could partly reflect shared coding preferences rather than independent expert judgment. No significance tests or confidence intervals are provided. For a 914-paragraph sample with 491 'no claim' items, the macro F1 of 0.88 is a single point estimate.\n\nWho is this for: anyone working on LLM-assisted content moderation, AI governance, or climate misinformation. The paper would benefit from a careful referee who insists on reporting compliance rates separately, fixing the prompt, and treating the expert-alignment claim more cautiously. I'd send it to review—the core idea is worth developing—but it needs substantial revision before the comparative claims.","headline":"A promising fine-tuning result buried under an unreliable open-source comparison; the expert-alignment result is suggestive but needs harder evidence.","tokens_in":21574,"tokens_out":3481,"would_cite":true,"duration_ms":27385,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned GPT-3.5-turbo classifies climate misinformation at expert-level agreement.","keywords":["climate misinformation","large language models","fine-tuning","expert annotation","human oversight","content moderation","AI governance","CARDS taxonomy"],"falsifier":"Have an independent panel of climate-communication experts, none involved in this study, code the same 914 paragraphs from the raw CARDS rubric and compute Krippendorff's $\\alpha$ between the fine-tuned model and the panel. If that agreement falls well below 0.89, or below the panel's own intercoder agreement, the expert-equivalence claim fails; a complementary check would add health-impact claims outside the CARDS taxonomy, where the paper already reports the model misses many cases.","tokens_in":20529,"feed_emoji":"🌍","tokens_out":7823,"duration_ms":63544,"temperature":0.7,"pith_summary":"The paper asks whether LLMs can help govern climate misinformation rather than amplify it. The authors benchmark 16 open- and closed-source models on the expert-annotated CARDS test set, then fine-tune GPT-3.5-turbo on the same expert data and test it on 914 paragraphs drawn from low-credibility climate articles shared on Facebook and X in 2022. The central result is that the fine-tuned model labels false or misleading climate claims with the same measured reliability as two senior climate-communication experts: Krippendorff's $\\alpha = 0.89$ for both the expert pair and the model against their labels, with a macro F1 of 0.88. On the same sample it clearly outperforms the specialist CARDS RoBERTa model ($\\alpha = 0.66$, macro F1 0.74), and on the benchmark it also beats larger proprietary zero-shot models. The authors take this as evidence that expert-annotated training data, as a form of human oversight, is what makes LLMs usable for governance tasks such as content moderation at scale.","feed_headline":"Fine-tuned GPT-3.5-turbo equals climate-expert coding","feed_subtitle":"On 914 social-media paragraphs, the model matched senior experts (α=0.89) and beat a specialist RoBERTa tool.","key_machinery":"The load-bearing mechanism is the CARDS taxonomy, a set of 27 false-or-misleading claim labels grouped under five super-claims about climate change, together with the expert-annotated paragraphs used to train and evaluate it. The authors convert the CARDS coding manual into prompts for zero-shot evaluation, then fine-tune GPT-3.5-turbo on the CARDS training set using system, user, and assistant message templates so the model emits one of the 27 labels. On the social-media sample, alignment with expert annotations is measured with Krippendorff's $\\alpha$ and macro-averaged precision, recall, and F1 at the super-claim level, with the reconciled labels of two senior experts serving as ground truth.","core_discovery":"On the paper's own terms, the discovery is that a relatively small proprietary model fine-tuned on a public expert-annotated taxonomy can classify false or misleading climate claims in wild social-media text at the same level of agreement as senior experts. The fine-tuned GPT-3.5-turbo reaches Krippendorff's $\\alpha = 0.89$ against the reconciled expert labels on 914 paragraphs, matching the two experts' intercoder agreement of $\\alpha = 0.89$, and a macro F1 of 0.88 at the super-claim level. This exceeds the CARDS RoBERTa model on the same paragraphs ($\\alpha = 0.66$, macro F1 0.74) and exceeds GPT-4o and RoBERTa on the original CARDS test set (macro F1 0.84 vs. 0.75 and 0.77). The authors also find that open-source 2B–9B models lag far behind in zero-shot classification, and that existing expert-trained tools outperform most proprietary zero-shot models, including GPT-4o.","pith_inferences":["Extension: because the ground-truth labels and the fine-tuning data come from the same CARDS rubric and two of the expert coders are also authors, the reported $\\alpha = 0.89$ is best read as alignment with this specific expert coding scheme; an independent coding panel could yield a lower number.","Extension: if the fine-tuning recipe transfers, the same human-oversight pattern could be applied to other governance domains such as health or political misinformation, provided an expert-annotated taxonomy exists for each domain.","Extension: a useful follow-up would be per-claim calibration on rare sub-claims such as species and plant impacts, where the paper reports systematic misfires; class-imbalance correction or targeted additional expert data would be a direct test of whether the bias is in the training distribution rather than the model."],"forward_implications":["Fine-tuning a mid-size proprietary LLM on an expert taxonomy can produce a claim classifier whose agreement with experts on in-the-wild social media text matches, rather than merely approaches, the experts' own intercoder agreement.","Expert-trained specialist tools remain strong baselines: CARDS outperforms most proprietary zero-shot models, including GPT-4o, on the benchmark, so domain-specific training data still matters.","Open-source models in the 2B–9B parameter range are far behind on this classification task, implying that resource-constrained civil-society organizations would currently need API access or better open training data to use LLMs for this governance task.","The paper's own limitation sections imply that expert-level performance on low-credibility sources should not be assumed for high-credibility sources, claims outside the CARDS taxonomy, non-English text, or health-impact claims, where the model misses many cases."],"supporting_citations":[{"why":"supplies the CARDS taxonomy, the expert-annotated training/test paragraphs, the coding manual used to craft prompts, and the RoBERTa baseline the paper compares against.","marker":"[15]"},{"why":"supplies Media Bias Fact Check categories used to select articles from right-bias, questionable, and conspiracy-pseudoscience domains for the social-media sample.","marker":"[38]"},{"why":"supplies NewsGuard trust scores used alongside MBFC to identify low-credibility domains in the social-media corpus.","marker":"[39]"},{"why":"documents invalid responses from LLMs on NLP tasks, the evidence the paper relies on to justify replacing invalid zero-shot labels with randomly sampled ones before scoring.","marker":"[45]"}],"fun_headline_variants":["Fine-tuned GPT-3.5 matches climate experts at spotting lies","GPT-3.5-turbo fine-tuned to match expert climate coding","Small fine-tuned LLM equals veteran climate experts on misinformation","Expert-level climate lie detection with fine-tuned GPT-3.5"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the reconciled labels of the two senior experts are valid ground truth for what counts as false or misleading climate claims, rather than a local coding preference; if those labels are systematically tied to the same taxonomy and training examples the model was fine-tuned on, the reported expert-level agreement could overstate the model's ability to match independent expert judgment.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned GPT-3.5 matches climate experts at spotting lies","GPT-3.5-turbo fine-tuned to match expert climate coding","Small fine-tuned LLM equals veteran climate experts on misinformation","Expert-level climate lie detection with fine-tuned GPT-3.5"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1900,"prompt_tokens":1049,"completion_tokens":851,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":772}},"tokens_in":665,"tokens_out":851,"duration_ms":6354,"temperature":1.0,"reasoning_tokens":772,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:34:42.902987+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent panel of climate-communication experts, none involved in this study, code the same 914 paragraphs from the raw CARDS rubric and compute Krippendorff's $\\alpha$ between the fine-tuned model and the panel. If that agreement falls well below 0.89, or below the panel's own intercoder agreement, the expert-equivalence claim fails; a complementary check would add health-impact claims outside the CARDS taxonomy, where the paper already reports the model misses many cases.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the CARDS taxonomy, the expert-annotated training/test paragraphs, the coding manual used to craft prompts, and the RoBERTa baseline the paper compares against."},{"cited_title":"URL: https://www.mediabiasfactcheck.com, ac- cessed: 2024-09-09","cited_arxiv_id":null,"evidence_quote":"supplies Media Bias Fact Check categories used to select articles from right-bias, questionable, and conspiracy-pseudoscience domains for the social-media sample."},{"cited_title":"URL: https://www.newsguardtech.com/solutions/newsguard/, ac- cessed: 2024-09-09","cited_arxiv_id":null,"evidence_quote":"supplies NewsGuard trust scores used alongside MBFC to identify low-credibility domains in the social-media corpus."}],"review_version":1}