{"id":"f3347414-4295-4bde-8c5f-8bb951054b28","arxiv_id":"2502.02191","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs can usefully process and classify climate and sustainability policy texts, but their outputs still depend on human judgment and face reproducibility and evaluation gaps.","lead":"This paper applies large language models to climate and sustainability policy documents and finds the tools work well for summarizing and classifying text, but need heavy human oversight. It offers a practical map of where AI helps and where it fails in policy workflows.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central empirical claim rests on an 80% SDG-match figure (Sec. 3.2) with no evaluation protocol; the matching criterion, denominator, and annotator agreement are unspecified, so 'successful' is unverifiable.","rationale":"The reader's weakest_assumption already targets the unvalidated evaluation of the 80% match and the consistency-as-quality model selection. My stress-test identifies the same load-bearing point with more specificity: the 80% figure in Section 3.2 is the only quantitative anchor for the paper's central claim, and it is presented without the protocol needed to interpret it. This is not a stylistic complaint; it is a correctness risk. If the 80% was computed as lenient paragraph-level overlap, the true performance could be much lower, and the conclusion that LLMs are 'successful' at classifying policy texts would not follow. The paper itself is transparent about many limitations and is best read as a process-oriented position paper, which supports a CONDITIONAL verdict. My concern does not move the verdict; it strengthens the condition. The authors should either make the evaluation protocol and data available or soften the numerical claim to 'preliminary and unverified.' The concrete test I propose would settle the concern by forcing an exact, reproducible definition of the match and recomputing the headline number. I also give credit for the paper's reflexive documentation of workflow stages and its explicit calls for evaluation methods; these are valuable independent of the unverified 80% figure.","tokens_in":15020,"tokens_out":2780,"duration_ms":27897,"concrete_test":"Request the full evaluation record from the authors: the set of documents, the LLM's predicted SDG assignments, the KnowSDGs expert assignments, and the matching script. Recompute the match with a pre-specified strict protocol: a predicted SDG counts as a match only if it is among the up-to-two SDGs assigned to that same document in KnowSDGs, and report precision, recall, F1, and a 95% confidence interval for the match rate. If F1 falls below 0.7 or the match rate drops below 60% under this strict document-level criterion, the empirical claim of successful SDG classification is not supported by the reported evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.2, the authors state that 'approximately 80 percent of the predicted SDGs match with the experts' assignation' when comparing their LLM routine against the KnowSDGs database. This single quantitative figure is the main evidence for the abstract's positive claim that LLMs are 'successful at processing, classifying and summarizing heterogeneous text-based data.' Yet the paper provides no evaluation protocol. The unit of prediction is a paragraph with multiple SDG assignments, while KnowSDGs assigns up to two SDG targets per document. It is unclear whether the 80% is paragraph-level precision, document-level overlap, a top-k match, or something else. No precision, recall, or F1 are reported; no inter-annotator reliability for the expert baseline is given; no sample size or confidence interval is stated; and no data or code are shipped. The authors honestly acknowledge that 'automatic evaluation is not feasible because of the lack of benchmark datasets' and that their manual screening 'has its limits,' but this leaves the headline empirical claim unsupported. The model-selection rationale in Section 3.1, choosing Gemini because it 'maintains consistency across all runs and is not subject to randomness,' is also a concern because consistency is not accuracy, but that concern is secondary; even if consistency were valid, the 80% figure would still be the load-bearing evidence. Without a defined and reproducible evaluation metric, the central claim cannot be independently checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports on two LLM-based text analysis workflows for climate and sustainability policy documents. The first workflow (Section 3.1) processes Nationally Determined Contributions (NDCs) by splitting them into paragraphs, assigning SDGs via semantic similarity against UN SDG Fast Facts, and applying ClimateBERT for sentiment classification; a parallel prompt-based strategy using Gemini 1.0 is also described. The second workflow (Section 3.2) uses a multi-stage prompt on Gemini to assign SDGs to documents and identify synergies, trade-offs, and directionality between SDG pairs, with a comparison against the KnowSDGs database, where the authors report 'approximately 80 percent' agreement. The paper frames these as lessons-learned from two pilot studies, emphasizing the need for human intervention, and discusses limitations grouped by workflow stage (data collection, processing, model selection, evaluation, and translation to policy). The central positive claim—that LLMs are 'successful' at processing, classifying, and summarizing heterogeneous policy text—rests on the 80% figure and on manual screening, neither of which is accompanied by a detailed evaluation protocol.","tokens_in":15384,"tokens_out":4070,"duration_ms":40543,"significance":"The paper addresses a timely and practically important question: whether LLMs can support climate and sustainability policy analysis in a reliable and usable way. Its main strengths are the honest documentation of two concrete workflows—including prompt-engineering iterations, text-cleaning rules, and explicit acknowledgment of evaluation gaps—and the discussion of workflow-stage-specific barriers such as model retirement, market-driven model selection, and the useful-to-usable translation problem. The paper also productively advocates for transparent output evaluation and for hybrid human-machine processes. If the reported 80% agreement with KnowSDGs were backed by a clear evaluation protocol, the paper would provide a useful benchmark for future work. As it stands, the quantitative evidence is too thinly specified to independently validate the abstract's positive claim; the value of the paper is therefore mainly qualitative and methodological, which is still a legitimate contribution if the empirical claims are appropriately qualified.","major_comments":[{"comment":"The paper reports that 'approximately 80 percent of the predicted SDGs match with the experts' assignation' when comparing the LLM routine against KnowSDGs, but it does not define the matching criterion (paragraph-level, document-level, top-k, or exact-match), the denominator, the sample size, or any confidence interval. Since KnowSDGs assigns up to two SDG targets per document while the LLM assigns multiple SDGs per paragraph, the comparison is confounded by unit mismatch; without a precise protocol and ideally precision/recall/F1, this single quantitative claim cannot be independently verified and cannot support the abstract's 'successful' assessment.","section":"Section 3.2"},{"comment":"The selection of Gemini 1.0 over GPT3.5 is justified by the statement that Gemini 'maintains consistency across all runs and is not subject to randomness.' This equates output consistency with model quality, but consistency does not imply accuracy, and no comparison of the two models' SDG assignments against a labeled benchmark is provided. The paper also does not report the decoding parameters (e.g., temperature, top-p, seed) or the number of runs used to establish this consistency, so the model-choice rationale is not reproducible.","section":"Section 3.1"},{"comment":"The NDC sentiment and SDG classification results were evaluated by manual screening, which the authors acknowledge 'has its limits' and that 'automatic evaluation is not feasible because of the lack of benchmark datasets.' However, the abstract and Section 4 rely on these manual results to assert that LLMs are 'successful' at classifying and summarizing texts. No inter-annotator agreement statistics, screening sample size, or criteria for resolving disagreements are given, so the central positive claim cannot be independently assessed.","section":"Section 3.1, Section 4"},{"comment":"The multi-stage prompt that is claimed to be a 'fully replicable alternative' is described only at the prompt-text level; the paper does not specify how the output of the first prompt (a variable-length list of main and secondary SDGs) is fed into the second prompt, how the relationship extraction is structured, or how the directionality labels are aggregated. This omission is load-bearing because the central contribution includes the claim of a replicable AI-based routine for detecting synergies and trade-offs; without this pipeline detail, the workflow cannot be re-implemented.","section":"Section 3.2, Table 3"}],"minor_comments":[{"comment":"The sentence 'Transformers understand logical intra-sequence relationships between different parts and produce outputs based on those... trained on a predefined large corpus on a predefined large corpus of diverse texts' contains a duplicated phrase ('on a predefined large corpus on a predefined large corpus'); please fix this typo.","section":"Section 2"},{"comment":"In the trade-off definition, the example 'Climate change will increase poverty by 64%' is labeled 'An example of a synergic relationship,' which is a copy-paste error from the synergy example and could mislead readers implementing the prompt; the label should be corrected.","section":"Table 3, second prompt"},{"comment":"The paper states that Gemini 1.0 'maintains consistency across all runs and is not subject to randomness,' but no decoding parameters (temperature, top-p, random seed) or API version details are reported; please add these for reproducibility.","section":"Section 3.1"},{"comment":"The paper describes the workflow in Figure 4 but does not provide a data or code availability statement. Since Section 3 claims a 'fully replicable alternative' to expert-based coding, the authors should state whether cleaned corpora, prompts, and analysis scripts will be released, even in a repository.","section":"Section 4, Figure 4"},{"comment":"Reference 18 is cited both for TF-IDF and for the 'attention mechanisms' that introduced the transformer architecture, but the reference list does not include the original Transformer paper (Vaswani et al., 2017); the citations should be corrected and the appropriate reference added.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's qualitative discussion and the documentation of two real-world workflows are likely to be of interest to the computational social science and policy-oriented AI communities. The authors' self-citations (e.g., Vinuesa et al. 2020, Fuso Nerini et al. 2019, Larosa et al. 2023) are appropriate in context and do not constitute a problem. The central concern is the unvalidated 80% agreement figure and the manual screening that supports the main empirical claim. If the authors can provide a clear evaluation protocol—even a small-scale annotated sample with agreement statistics—or alternatively remove or strongly qualify the quantitative claim in the abstract, this would be a publishable contribution. The current version is not acceptable without that revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is the kind of paper that reads better than it cites. The two case studies are real: they apply ClimateBERT and Gemini to NDC paragraphs and to SDG interlinkage detection using a Delphi-based corpus. What is genuinely new is the reflective workflow documentation—the trial-and-error prompting, the comparison of model consistency across runs, and the four-step limits framework in Figure 4. If you work on LLMs for policy text, this gives you a grounded sense of where the human still sits in the loop.\n\nThe qualitative claims are credible and consistent with what I know of the field. The authors are upfront that automatic evaluation isn't feasible without benchmarks and that their manual screening has limits. They don't oversell the generalizability.\n\nThe soft spot is exactly the one the stress-test flags. The “approximately 80 percent” match to KnowSDGs in Section 3.2 is the only quantitative anchor for the abstract's “successful” claim, and it is under-specified. No denominator, no precision/recall, no inter-annotator agreement, no confidence interval. The unit of prediction (paragraph with multiple SDGs) doesn't obviously align with KnowSDGs' document-level up-to-two-SDG assignments. And the model-selection rationale—Gemini is consistent, therefore better—conflates determinism with accuracy. None of this invalidates the qualitative story, but it means the central numerical claim cannot be independently checked as written.\n\nThere is also no shipped code or data, which is a shame for a paper whose selling point is replicability. The prompts are given, which helps, but the evaluation pipeline is not.\n\nWho is this for? Researchers building similar pipelines and policy analysts who want a realistic map of where LLMs help. It is a decent peer-review submission, provided the editor sends it to someone who will ask for the evaluation protocol to be defined and, ideally, a small reproducibility appendix.\n\nMy recommendation: engage with it, but treat the 80% figure as a placeholder until the authors supply the missing details.","headline":"A genuinely useful workflow paper whose one headline number (80% SDG match) is under-specified; worth peer review, but don't cite the number as it stands.","tokens_in":15818,"tokens_out":2018,"would_cite":false,"duration_ms":18533,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-based text pipelines can classify and summarize climate and sustainability policy documents, but the paper argues they succeed only with substantial human oversight and still lack robust evaluation methods.","keywords":["large language models","climate policy","sustainable development goals","natural language processing","Nationally Determined Contributions","SDG interlinkages","policy workflows","human oversight"],"falsifier":"A concrete test would be to run the paper's SDG-assignment prompt on a fresh set of policy documents that have been independently labelled by a panel of experts who did not help build the prompt, and compute agreement on a document-by-document basis; if agreement on unseen documents falls to chance levels, the central claim about LLM usefulness for policy analysis would be undercut.","tokens_in":14805,"feed_emoji":"🌍","tokens_out":7763,"duration_ms":72158,"temperature":0.7,"pith_summary":"The paper tries to establish that large language models can already perform useful work on climate and sustainability policy texts, processing, classifying, and summarizing documents that are heterogeneous in format and purpose, and that the main barrier to using them in policy is no longer raw capability but human oversight, reproducibility, and evaluation. It supports this with two pilot workflows: one maps paragraphs of Nationally Determined Contributions to the Sustainable Development Goals and classifies their sentiment toward climate action, and the other detects synergies, trade-offs, and directionality among SDGs across a broad corpus of policy and research documents. The strongest quantitative evidence is a reported 80 percent agreement between the LLM routine and expert-assigned SDGs in the second pilot. A sympathetic reader would care because the paper turns the debate away from whether AI can read policy documents toward the practical conditions under which its outputs become usable for policymakers.","feed_headline":"LLM routine matches experts on 80% of SDG links","feed_subtitle":"Two policy-text pilots show real gains from AI, but human oversight and missing benchmarks still block policy use.","key_machinery":"The machinery that carries the argument is the two-pilot workflow itself. The first pilot uses a static semantic-similarity step to assign each NDC paragraph to the SDG with the highest score against UN SDG Fast Facts, then passes the paragraph to ClimateBERT, a climate-domain-adapted model, for sentiment classification on a 0-1-2 scale covering negative, neutral, and positive stances on climate adaptation and mitigation. The second pilot uses a multi-stage prompt on the general-purpose model Gemini 1.0 that first lists main and secondary SDGs for each text and then asks the model to label each SDG pair as synergy, trade-off, or neutral, with directionality stated as inward, outward, or both. The paper's contribution is not a new model but a documented, replicable routine built from these components, plus an explicit account of where the routine fails.","core_discovery":"The central claim, stated on the paper's own terms, is that a structured LLM-based workflow can reproduce a meaningful portion of expert judgement in sustainability-policy text analysis. When the authors compared their multi-stage prompt routine with the KnowSDGs database of expert-assigned SDG targets, approximately 80 percent of the predicted SDGs matched the experts' assignment, although SDG17 was chronically underdetected and SDG1 appeared in more excerpts than expected. In the NDC pilot, the authors found that combining semantic similarity with UN SDG Fast Facts and ClimateBERT sentiment classification yields a workable paragraph-level mapping of climate pledges to SDGs and to positive, neutral, or negative stances on climate adaptation and mitigation. The paper also documents that the choice of prompt wording changes results for some models, that Gemini 1.0 was selected because its outputs were consistent across runs, and that no labelled benchmark exists to formally evaluate these tasks. The conclusion is that LLMs are useful for nowcasting, large-scale monitoring, and discovering non-obvious SDG connections, but that human intervention at data collection, model selection, and sense-making stages remains necessary.","pith_inferences":["Because the paper used the KnowSDGs database to test successive prompt attempts, the reported 80 percent match may overstate how well the routine would do on genuinely unseen documents; a held-out evaluation against independent expert labels would be a stronger test.","Selecting Gemini 1.0 because it is consistent across runs treats determinism as a proxy for quality, which can hide systematic errors; comparing deterministic and stochastic models on the same labelled set would separate stability from accuracy.","The trial-and-error prompt process is evidence that prompt wording is becoming a new researcher degree of freedom in computational social science, much like coding schemes were in manual content analysis.","A direct extension would be to rerun the same two-pilot workflow on newer NDC submissions or on local-level policy documents to test whether the 80 percent match and the underdetection of SDG17 replicate outside the original corpus."],"forward_implications":["If the results hold, LLM-based routines can act as a reliable first-pass screening tool for NDC and SDG-related policy documents, freeing analysts to focus manual coding on flagged paragraphs.","The approximate 80 percent match with expert assignment implies that AI-assisted SDG mapping can complement, but not replace, expert judgement in sustainability policy analysis.","The documented sensitivity to prompt wording and model choice implies that published results in this area need to include exact prompts and model versions to be reproducible.","The absence of labelled benchmarks means that progress in computational policy analysis will stay difficult to compare until evaluation datasets are built.","The workflow demonstrates that grey literature, news items, and national reports can be brought into SDG interlinkage analysis without relying solely on expert-selected academic sources."],"supporting_citations":[{"why":"grounds the motivating claim that AI can help achieve the Sustainable Development Goals, which this paper tests in a policy-document setting.","marker":"[2]"},{"why":"provides the expert-based mapping of climate-SDG interlinkages that the NDC workflow extends and partially automates.","marker":"[34]"},{"why":"supplies the climate-domain LLM used for paragraph-level sentiment classification in the NDC pipeline.","marker":"[35]"},{"why":"supplies the general-purpose LLM used for prompt-based SDG assignment and interlinkage detection.","marker":"[36]"},{"why":"underpins the paper's caution about unreliability and reproducibility limits of large language models.","marker":"[16]"},{"why":"frames the reflexive and integrative approach to computational climate communication that the paper adopts.","marker":"[31]"},{"why":"serves as the expert-assigned baseline against which the reported 80 percent match is measured.","marker":"KnowSDGs database"}],"fun_headline_variants":["LLMs hit 80% expert match on SDG mapping","AI matches experts on 80% of SDG links—with caveats","LLM policy tool: 80% expert alignment, human gaps remain","SDG mapping: LLMs near experts but need human checks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's load-bearing premise is that the two pilot corpora and the manual screening used to judge outputs are representative of climate and sustainability policy texts generally, and that an LLM's run-to-run consistency is a meaningful sign of quality.","fun_headline_variants_meta":{"raw":{"variants":["LLMs hit 80% expert match on SDG mapping","AI matches experts on 80% of SDG links—with caveats","LLM policy tool: 80% expert alignment, human gaps remain","SDG mapping: LLMs near experts but need human checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1406,"prompt_tokens":923,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":407}},"tokens_in":539,"tokens_out":483,"duration_ms":4867,"temperature":1.0,"reasoning_tokens":407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:58:35.427259+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to run the paper's SDG-assignment prompt on a fresh set of policy documents that have been independently labelled by a panel of experts who did not help build the prompt, and compute agreement on a document-by-document basis; if agreement on unseen documents falls to chance levels, the central claim about LLM usefulness for policy analysis would be undercut.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the expert-based mapping of climate-SDG interlinkages that the NDC workflow extends and partially automates."},{"cited_title":"Gemini : A Family of Highly Capable Multimodal Models","cited_arxiv_id":null,"evidence_quote":"supplies the general-purpose LLM used for prompt-based SDG assignment and interlinkage detection."},{"cited_title":"M., Gebru, T., McMillan-Major, A","cited_arxiv_id":null,"evidence_quote":"underpins the paper's caution about unreliability and reproducibility limits of large language models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"frames the reflexive and integrative approach to computational climate communication that the paper adopts."}],"review_version":1}