{"id":"db49a7dc-d74a-48d5-bda8-571eb390c60a","arxiv_id":"2411.17598","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Three small LLMs classify SDG-related abstracts into 'Relevant' and 'Non-Relevant' with large disagreements, but the paper provides no ground truth to show which classifications are correct.","lead":"This paper tests whether small language models can tell the difference between research abstracts that genuinely contribute to United Nations Sustainable Development Goals and abstracts that only mention related keywords. It finds that three different models disagree strongly on what counts as relevant, and the study does not check which judgments are correct.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim depends on LLM self-labels as ground truth; with no gold-standard validation or keyword baseline, 'improved precision' is unmeasured.","rationale":"The reader's weakest assumption—that LLM classifications themselves are taken as valid measures of substantive SDG contribution—is exactly the load-bearing point. The paper shows only inter-model variation, which does not establish that any model's label tracks true relevance. A model could consistently apply a prompt while being consistently wrong relative to expert judgment. The absence of a keyword baseline is equally damaging: even if the LLM labels were plausible, the paper never quantifies the precision of the existing Scopus queries, so the central claim of 'improving precision' cannot be evaluated. I agree with the reader's REJECT verdict, with high confidence, because the gap between evidence and claim is fundamental rather than cosmetic. The paper does have some merit: it documents that different small LLMs apply the same prompt differently, which is a real empirical observation that motivates ensemble or prompt-refinement work. The proposed multi-agent ensemble is a reasonable direction, but it is speculative future work, not a demonstrated result. Since the reader's verdict already reflects this, no adjustment is needed.","tokens_in":4377,"tokens_out":2524,"duration_ms":23068,"concrete_test":"Sample 300 abstracts from the SDG 1 retrieval set; have at least two domain experts, blind to model labels, independently annotate each abstract as relevant or non-relevant to SDG 1 targets using the paper's criteria; compute inter-annotator agreement and a majority-vote gold standard; then compare each LLM's labels to the gold standard using precision, recall, and F1, and also compute the precision of the original Scopus keyword query on the same sample. If no LLM achieves significantly higher precision than the keyword baseline at matched recall, the central claim of improved precision fails; if one does, the claim gains empirical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that small, locally-hosted LLMs can differentiate substantive SDG contributions from incidental keyword matches, thereby improving precision. The evidence in Section III, however, consists solely of the models' own 'Relevant'/'Non-Relevant' labels (Fig. 2: 52%, 70%, 15%) and pairwise Venn overlaps (Fig. 3). No comparison is made against human expert judgments, an externally validated SDG-tagged corpus, or the original Scopus keyword retrieval's precision. Precision is a ground-truth-relative metric; without a reference standard, the claimed precision improvement is not measured. The observed divergence across models is at least as consistent with prompt underspecification as with meaningful semantic differentiation. The paper's own Section IV limitations—prompt sensitivity, abstract-only evidence, and SDG 1 focus—further undercut generalizability, but the more fundamental issue is that the outcome variable itself is unvalidated. The proposed pipeline's usefulness rests entirely on the unverified assumption that LLM relevance labels correspond to true substantive contributions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using small, locally-hosted LLMs as an 'evaluation agent' to filter Scopus abstracts retrieved by SDG-related keyword queries, with the goal of improving the precision of SDG research identification. The authors describe a prompt-based binary classification procedure and report, for three LLMs (Phi-3.5-mini, Mistral-7B-v0.3, Llama-3.2-3B), the percentage of abstracts labeled 'Relevant' versus 'Non-Relevant' to SDG 1, along with pairwise Venn-diagram agreement. They observe wide inter-model variation (52%, 70%, 15% relevant) and suggest future ensemble and multi-agent frameworks. The manuscript's central claim is that LLMs can differentiate substantive SDG contributions from incidental keyword matches, thereby improving precision over Boolean search.","tokens_in":4616,"tokens_out":2314,"duration_ms":23008,"significance":"If properly validated, an LLM-based filter that improves precision over keyword-based SDG retrieval would be genuinely useful for institutional reporting and bibliometric benchmarking, since false positives from Boolean queries are a known problem in this area. The paper does identify a real gap and provides a clearly described prompting pipeline with three locally-hosted models, which is a reasonable starting point. However, the current manuscript provides no measurement of precision at all: there is no gold standard, no human annotation, no recall or precision metric, and no comparison to the original Scopus keyword retrieval. The central claim therefore rests entirely on the unvalidated assumption that the LLMs' self-reported labels correspond to true substantive relevance. The paper's strengths are its clear problem framing and the explicit description of prompt components (Table I), but the evidence presented in Section III cannot support the abstract's quantitative claim of improved precision.","major_comments":[{"comment":"The paper's only evidence for the ability to differentiate substantive contributions is the models' own 'Relevant'/'Non-Relevant' labels. There is no ground-truth set of abstracts annotated by human experts or any externally validated SDG-tagged corpus, so precision and recall cannot be computed. The statement in the Abstract that the approach 'improves precision' is therefore unmeasured: a claim about precision requires a reference standard against which retrieved labels are judged, and none is provided.","section":"Section III, Figs. 2 and 3"},{"comment":"The observed inter-model divergence (52%, 70%, and 15% relevant labels) is presented as evidence of 'differences in the way each model applies the evaluation criteria,' but this variation is equally consistent with prompt underspecification or model unreliability. Without external validation, the differences do not support a claim about semantic differentiation; they only document that the three LLMs disagree. In particular, the sentence in Section III describing Llama-3.2 as 'retaining the precision of Llama-3.2' presupposes that its stricter labels are correct, which is never established.","section":"Section III, Fig. 2"},{"comment":"No baseline precision is reported for the keyword-based Scopus retrieval that constitutes the input to the evaluation agent. The paper asserts that Boolean queries 'conflate incidental keyword matches with genuine contributions,' but it never measures the precision of the initial retrieval set or compares it with the precision of the filtered set. Since both the input and output are unlabeled with respect to true relevance, the claimed improvement over traditional methods is not demonstrated. The limitations acknowledged in Section IV (prompt sensitivity, abstract-only evidence, SDG 1 focus) do not address this missing baseline, and the statement that the study focused 'primarily on SDG 1' makes it unclear whether Figs. 2 and 3 describe SDG 1 alone or all 17 SDGs.","section":"Section II and Section IV"},{"comment":"The proposed ensemble and 'multi-agent conversation' are speculative and have not been implemented or evaluated. The paper describes a single-pass prompting procedure, not an agentic system with iterative evaluation or inter-agent feedback. As a result, the conclusion that 'an ensemble approach... could be a novel solution' is not supported by any experimental evidence, and the title's 'Agentic AI' overstates what was actually tested.","section":"Section III, last paragraph"}],"minor_comments":[{"comment":"The figure caption contains a typo: 'Absracts' should be 'Abstracts'.","section":"Fig. 1"},{"comment":"The reported percentages in Fig. 2 are not accompanied by the number of abstracts or the SDG subset used; providing sample sizes and clarifying whether the analysis covers SDG 1 only would improve reproducibility.","section":"Section III"},{"comment":"The manuscript does not state whether the data or prompt templates will be made available; a data/code availability statement would be needed for the results to be independently reproduced.","section":"General"}],"recommendation":"reject","confidential_remarks":"The central claim is not supported by the presented evidence, and the missing ground-truth validation and baseline comparison are fundamental rather than cosmetic. The interesting question—whether small LLMs can improve SDG retrieval precision—would require a properly annotated evaluation set, which is beyond a typical revision of this manuscript. The paper might be better positioned as a short position or work-in-progress paper if the authors wish to disseminate the prompt design and preliminary observations, but in its current form it does not meet the evidentiary standards of a journal article."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a pilot that conflates \"models can classify\" with \"models classify correctly.\" The idea is reasonable, the execution is transparent, but the central claim of precision improvement is unmeasured. No ground truth, no baseline, no precision/recall; the three models' divergent labels demonstrate prompt sensitivity more than semantic differentiation. The paper's own limitations section admits most of this, but the abstract and title still overclaim.\n\nWhat's new: using small, locally-hosted LLMs as a post-filter after Boolean keyword retrieval, with instructions to distinguish substantive SDG contributions from incidental mentions. There's a real gap here: institutional SDG reporting relies on keyword queries that over-retrieve. The paper frames the problem well and the prompt design (explicit target list, binary output, reasoning) is sensible. The use of three locally-hostable models is also pragmatic.\n\nWhat's not solid: everything about validation. Fig. 2 shows percentages of relevant labels (52/70/15) and Fig. 3 shows overlap. None of that tells us whether any model is right. Precision is a ground-truth-relative metric; without a gold standard you cannot claim you improved it. You can't even claim the models \"differentiate\" substantive from incidental—they differentiate in the sense that they assign different labels, but so would three random prompts. The reported inter-model divergence could be prompt ambiguity, not a signal. There is also no comparison against the original Scopus keyword retrieval, so the \"improving precision\" claim is entirely unsupported. The paper acknowledges prompt sensitivity, abstract-only evidence, and SDG 1 focus, but those are secondary; the primary issue is the missing outcome measure.\n\nWho's this for? People working on SDG mapping in bibliometrics might find the prompt template and the cautionary inter-model divergence useful as a starting point. But this is a workshop note, not a complete paper. If the authors added a human-annotated benchmark (even 200-300 abstracts) and reported precision/recall against a keyword baseline, it would be a solid contribution.\n\nI wouldn't send this to a full journal as-is. I'd desk reject with encouragement to resubmit after validation, or route it to a workshop where preliminary work is acceptable.","headline":"A plausible pilot that overclaims precision because it never measures it against anything.","tokens_in":5060,"tokens_out":2468,"would_cite":false,"duration_ms":22636,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that small, locally hosted large language models can separate substantive contributions to SDG targets from documents that merely match SDG keywords, improving the precision of institutional research assessments.","keywords":["large language models","sustainable development goals","semantic relevance classification","evaluation agent","bibliometric precision","SDG research mapping","multi-agent ensemble"],"falsifier":"Take a random sample of three hundred abstracts from the SDG 1 retrieval set, have domain experts independently label each as a genuine contribution to an SDG 1 target or not, then run the three LLM prompts on the same sample; if the models' agreement with the expert labels is no better than random, the claimed precision improvement over keyword retrieval collapses.","tokens_in":4117,"feed_emoji":"🎯","tokens_out":5476,"duration_ms":45255,"temperature":0.7,"pith_summary":"The paper claims that small, locally hosted large language models can act as evaluation agents that separate abstracts making real contributions to a Sustainable Development Goal (SDG) target from abstracts that merely contain SDG-related keywords. It builds a two-stage pipeline: a keyword search retrieves a large set of candidate abstracts, then a prompted LLM classifies each abstract as 'Relevant' or 'Non-Relevant' to a specific SDG target, with reasoning tied to textual evidence. Testing on SDG 1 with three compact models, the paper reports that the models do distinguish such cases and that their differing thresholds could be combined in a multi-agent ensemble. If correct, this would let research institutions measure SDG output with higher precision than Boolean queries, supporting more accurate rankings and benchmarking.","feed_headline":"Small LLMs can spot real SDG research in keyword piles","feed_subtitle":"Three compact local models separate substantive contributions from incidental matches, sharpening SDG metrics.","key_machinery":"The load-bearing mechanism is the structured evaluation prompt, a set of instructions that gives the LLM the role of distinguishing superficial mentions from substantive contributions, supplies the official SDG target definitions (for example SDG 1 Target 1.2 on halving poverty and Target 1.5 on building resilience), provides example abstracts of both classes, and demands a binary 'Relevant' or 'Non-Relevant' verdict with reasoning quoting the abstract. This prompt converts a general-purpose text generator into a classifier, and the model choice (Phi-3.5-mini, Mistral-7B, Llama-3.2) supplies different inductive biases that the paper treats as a resource for ensemble design.","core_discovery":"On the paper's own terms, the central finding is that prompt-guided, locally hosted LLMs can differentiate semantically relevant contributions to SDG targets from documents retrieved due to incidental keyword matches. The evidence is a set of 20,000 abstracts per SDG retrieved from a scholarly database using established SDG queries; on SDG 1 the three models labeled 52%, 70%, and 15% of abstracts as 'Relevant,' respectively. The authors interpret the disagreement as complementary classification thresholds rather than failure, with Phi-3.5-mini's balanced output as a potential middle ground, Mistral-7B as a recall-oriented filter, and Llama-3.2 as a precision-oriented gate. The claim is that even a single small LLM, and especially a combination of them, improves on the context-blindness of keyword-based retrieval.","pith_inferences":["The decisive test this paper does not perform is comparing the models' labels against human expert judgment; until that comparison exists, the reported 'precision improvement' is actually a claim about model self-consistency rather than measured relevance.","The wide spread among models (15% to 70% 'Relevant') suggests the classification threshold is driven by prompt sensitivity as much as by document semantics; in a real deployment, one would calibrate the threshold against a labeled sample.","A practical extension would use the three-model disagreement as an active sampling signal: abstracts on which strict and lenient models disagree could be routed to human review, concentrating human effort where it matters.","The same architecture could apply to any multi-label research-classification task with a keyword-first retrieval stage, such as detecting alignment with national research priorities."],"forward_implications":["If the claim holds, institutions can filter their keyword-retrieved SDG literature to substantive contributions and base rankings and reports on higher-precision counts.","The local hosting requirement means the evaluation can be run on complete institutional corpora without sending proprietary publications to external services.","Because the three models occupy different points on the recall-precision spectrum, a staged multi-agent pipeline (broad filter first, strict filter last) could balance coverage and specificity.","The same prompt-as-evaluation-agent pattern could be extended from SDG 1 to all 17 goals, in the authors' future work."],"supporting_citations":[{"why":"Supplies the SDG-specific keyword query set used to build the retrieval data of 20,000 abstracts per goal.","marker":"[7]"},{"why":"Introduces Phi-3.5-mini-instruct, one of the three small language models used as an evaluation agent.","marker":"[9]"},{"why":"Introduces Mistral-7B-Instruct-v0.3, the model that produced the most expansive 'Relevant' classifications.","marker":"[10]"},{"why":"Introduces Llama-3.2-3B-Instruct, the model with the strictest relevance threshold.","marker":"[11]"},{"why":"Documents inconsistencies among Boolean bibliometric approaches to mapping SDG publications, motivating the need for semantic filtering.","marker":"[1]"},{"why":"Proposes query expansion with LLMs within the keyword paradigm, the baseline the paper argues still fails to discern substantive contributions.","marker":"[4]"}],"fun_headline_variants":["Small LLMs separate real SDG research from keyword matches","Locally hosted LLMs sharpen SDG metrics with semantic filtering","Tiny AI models cut SDG keyword noise for better reporting","LLM agents refine SDG contributions beyond Boolean queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the models' 'Relevant' labels correspond to true substantive SDG contributions, yet the study provides no human judgment, gold standard, or external benchmark to verify that equivalence.","fun_headline_variants_meta":{"raw":{"variants":["Small LLMs separate real SDG research from keyword matches","Locally hosted LLMs sharpen SDG metrics with semantic filtering","Tiny AI models cut SDG keyword noise for better reporting","LLM agents refine SDG contributions beyond Boolean queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1255,"prompt_tokens":849,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":338}},"tokens_in":465,"tokens_out":406,"duration_ms":4168,"temperature":1.0,"reasoning_tokens":338,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:54:54.344300+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of three hundred abstracts from the SDG 1 retrieval set, have domain experts independently label each as a genuine contribution to an SDG 1 target or not, then run the three LLM prompts on the same sample; if the models' agreement with the expert labels is no better than random, the claimed precision improvement over keyword retrieval collapses.","supporting_citations":[{"cited_title":"ULA, a Bibliometric Method to Identify Sustainable Development Goals Using Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Proposes query expansion with LLMs within the keyword paradigm, the baseline the paper argues still fails to discern substantive contributions."}],"review_version":1}