Pith. sign in

REVIEW 4 major objections 3 minor 12 references

Agentic AI for Improving Precision in Identifying Contributions to Sustainable Development Goals

T0 review · 4 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that small, locally hosted large language models can separate substantive contributions to SDG targets from documents that merely match SDG keywords, improving the precision of institutional research assessments.

desk verdict A plausible pilot that overclaims precision because it never measures it against anything. read the letter →

arxiv 2411.17598 v1 pith:2QNKWVGF submitted 2024-11-26 cs.DL cs.AIcs.IR

William A. Ingram , Bipasha Banerjee , Edward A. Fox This is my paper · ORCID
classification cs.DLcs.AIcs.IR
keywords largelanguagemodelssustainabledevelopmentgoalssemanticrelevanceclassificationevaluationagentbibliometricprecisionSDGresearchmappingmulti-agentensemble
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that small, locally hosted large language models can act as evaluation agents that separate abstracts making real contributions to a Sustainable Development Goal (SDG) target from abstracts that merely contain SDG-related keywords. It builds a two-stage pipeline: a keyword search retrieves a large set of candidate abstracts, then a prompted LLM classifies each abstract as 'Relevant' or 'Non-Relevant' to a specific SDG target, with reasoning tied to textual evidence. Testing on SDG 1 with three compact models, the paper reports that the models do distinguish such cases and that their differing thresholds could be combined in a multi-agent ensemble. If correct, this would let research institutions measure SDG output with higher precision than Boolean queries, supporting more accurate rankings and benchmarking.

What carries the argument

The load-bearing mechanism is the structured evaluation prompt, a set of instructions that gives the LLM the role of distinguishing superficial mentions from substantive contributions, supplies the official SDG target definitions (for example SDG 1 Target 1.2 on halving poverty and Target 1.5 on building resilience), provides example abstracts of both classes, and demands a binary 'Relevant' or 'Non-Relevant' verdict with reasoning quoting the abstract. This prompt converts a general-purpose text generator into a classifier, and the model choice (Phi-3.5-mini, Mistral-7B, Llama-3.2) supplies different inductive biases that the paper treats as a resource for ensemble design.

What would settle it

Take a random sample of three hundred abstracts from the SDG 1 retrieval set, have domain experts independently label each as a genuine contribution to an SDG 1 target or not, then run the three LLM prompts on the same sample; if the models' agreement with the expert labels is no better than random, the claimed precision improvement over keyword retrieval collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central finding is that prompt-guided, locally hosted LLMs can differentiate semantically relevant contributions to SDG targets from documents retrieved due to incidental keyword matches. The evidence is a set of 20,000 abstracts per SDG retrieved from a scholarly database using established SDG queries; on SDG 1 the three models labeled 52%, 70%, and 15% of abstracts as 'Relevant,' respectively. The authors interpret the disagreement as complementary classification thresholds rather than failure, with Phi-3.5-mini's balanced output as a potential middle ground, Mistral-7B as a recall-oriented filter, and Llama-3.2 as a precision-oriented gate. The claim is that even a single small LLM, and especially a combination of them, improves on the context-blindness of keyword-based retrieval.

Load-bearing premise

The method assumes that the models' 'Relevant' labels correspond to true substantive SDG contributions, yet the study provides no human judgment, gold standard, or external benchmark to verify that equivalence.

Editorial extensions

If this is right

  • If the claim holds, institutions can filter their keyword-retrieved SDG literature to substantive contributions and base rankings and reports on higher-precision counts.
  • The local hosting requirement means the evaluation can be run on complete institutional corpora without sending proprietary publications to external services.
  • Because the three models occupy different points on the recall-precision spectrum, a staged multi-agent pipeline (broad filter first, strict filter last) could balance coverage and specificity.
  • The same prompt-as-evaluation-agent pattern could be extended from SDG 1 to all 17 goals, in the authors' future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The decisive test this paper does not perform is comparing the models' labels against human expert judgment; until that comparison exists, the reported 'precision improvement' is actually a claim about model self-consistency rather than measured relevance.
  • The wide spread among models (15% to 70% 'Relevant') suggests the classification threshold is driven by prompt sensitivity as much as by document semantics; in a real deployment, one would calibrate the threshold against a labeled sample.
  • A practical extension would use the three-model disagreement as an active sampling signal: abstracts on which strict and lenient models disagree could be routed to human review, concentrating human effort where it matters.
  • The same architecture could apply to any multi-label research-classification task with a keyword-first retrieval stage, such as detecting alignment with national research priorities.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes using small, locally-hosted LLMs as an 'evaluation agent' to filter Scopus abstracts retrieved by SDG-related keyword queries, with the goal of improving the precision of SDG research identification. The authors describe a prompt-based binary classification procedure and report, for three LLMs (Phi-3.5-mini, Mistral-7B-v0.3, Llama-3.2-3B), the percentage of abstracts labeled 'Relevant' versus 'Non-Relevant' to SDG 1, along with pairwise Venn-diagram agreement. They observe wide inter-model variation (52%, 70%, 15% relevant) and suggest future ensemble and multi-agent frameworks. The manuscript's central claim is that LLMs can differentiate substantive SDG contributions from incidental keyword matches, thereby improving precision over Boolean search.

Significance. If properly validated, an LLM-based filter that improves precision over keyword-based SDG retrieval would be genuinely useful for institutional reporting and bibliometric benchmarking, since false positives from Boolean queries are a known problem in this area. The paper does identify a real gap and provides a clearly described prompting pipeline with three locally-hosted models, which is a reasonable starting point. However, the current manuscript provides no measurement of precision at all: there is no gold standard, no human annotation, no recall or precision metric, and no comparison to the original Scopus keyword retrieval. The central claim therefore rests entirely on the unvalidated assumption that the LLMs' self-reported labels correspond to true substantive relevance. The paper's strengths are its clear problem framing and the explicit description of prompt components (Table I), but the evidence presented in Section III cannot support the abstract's quantitative claim of improved precision.

major comments (4)
  1. [Section III, Figs. 2 and 3] The paper's only evidence for the ability to differentiate substantive contributions is the models' own 'Relevant'/'Non-Relevant' labels. There is no ground-truth set of abstracts annotated by human experts or any externally validated SDG-tagged corpus, so precision and recall cannot be computed. The statement in the Abstract that the approach 'improves precision' is therefore unmeasured: a claim about precision requires a reference standard against which retrieved labels are judged, and none is provided.
  2. [Section III, Fig. 2] The observed inter-model divergence (52%, 70%, and 15% relevant labels) is presented as evidence of 'differences in the way each model applies the evaluation criteria,' but this variation is equally consistent with prompt underspecification or model unreliability. Without external validation, the differences do not support a claim about semantic differentiation; they only document that the three LLMs disagree. In particular, the sentence in Section III describing Llama-3.2 as 'retaining the precision of Llama-3.2' presupposes that its stricter labels are correct, which is never established.
  3. [Section II and Section IV] No baseline precision is reported for the keyword-based Scopus retrieval that constitutes the input to the evaluation agent. The paper asserts that Boolean queries 'conflate incidental keyword matches with genuine contributions,' but it never measures the precision of the initial retrieval set or compares it with the precision of the filtered set. Since both the input and output are unlabeled with respect to true relevance, the claimed improvement over traditional methods is not demonstrated. The limitations acknowledged in Section IV (prompt sensitivity, abstract-only evidence, SDG 1 focus) do not address this missing baseline, and the statement that the study focused 'primarily on SDG 1' makes it unclear whether Figs. 2 and 3 describe SDG 1 alone or all 17 SDGs.
  4. [Section III, last paragraph] The proposed ensemble and 'multi-agent conversation' are speculative and have not been implemented or evaluated. The paper describes a single-pass prompting procedure, not an agentic system with iterative evaluation or inter-agent feedback. As a result, the conclusion that 'an ensemble approach... could be a novel solution' is not supported by any experimental evidence, and the title's 'Agentic AI' overstates what was actually tested.
minor comments (3)
  1. [Fig. 1] The figure caption contains a typo: 'Absracts' should be 'Abstracts'.
  2. [Section III] The reported percentages in Fig. 2 are not accompanied by the number of abstracts or the SDG subset used; providing sample sizes and clarifying whether the analysis covers SDG 1 only would improve reproducibility.
  3. [General] The manuscript does not state whether the data or prompt templates will be made available; a data/code availability statement would be needed for the results to be independently reproduced.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the paper's limitation is lack of external validation, not circularity.

full rationale

The paper's pipeline is explicit and self-contained: abstracts from Scopus are retrieved with Elsevier SDG keyword queries and then labeled by one of three prompted LLMs. There are no fitted parameters, no quantity is derived from another by construction, and no load-bearing self-citation or imported uniqueness theorem is used. The reported evidence (Fig. 2 and Fig. 3) is raw label distributions and inter-model overlap. The central claim that LLMs can differentiate genuine contributions from superficial mentions is not measured against any independent gold standard, so precision is asserted rather than demonstrated; this is an external-validity or soundness concern, not circularity, because the labels are not fitted to the conclusion and the argument does not reduce to its inputs by definition. The paper's own Section IV limitations, including prompt sensitivity, abstract-only evidence, and the SDG 1 focus, further highlight that the weakness is unvalidated measurement rather than a circular derivation chain.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted. The central claim rests on the unvalidated assumption that LLM labels correspond to genuine substantive SDG contributions, and on the use of keyword queries and abstracts as the sole input. These are domain assumptions or ad hoc assumptions with no external validation.

assumptions (3)
  • domain assumption Elsevier's SDG Research Mapping Initiative queries are an appropriate initial retrieval baseline.
    The paper uses these queries to collect 20,000 abstracts per SDG (Section II, [7]), but does not validate their precision or recall.
  • domain assumption Abstracts contain enough information to decide substantive SDG contribution.
    The models classify abstracts only; the authors acknowledge in Section IV that full text may be needed for accurate relevance determination.
  • ad hoc to paper The LLM output labels can be treated as evidence of capability to distinguish substantive contributions.
    The entire demonstration in Section III rests on the unvalidated assumption that the models' labels are correct. Without ground truth, the outputs only show inter-model disagreement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agentic AI for Improving Precision in Identifying Contributions to Sustainable Development Goals." pith.science (2026). https://pith.science/paper/2QNKWVGF

@misc{pith2026241117598,
  author       = {Pith},
  title        = {Pith review of: Agentic AI for Improving Precision in Identifying Contributions to Sustainable Development Goals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2QNKWVGF}},
  note         = {Machine review of arXiv:2411.17598}
}
read the original abstract

As research institutions increasingly commit to supporting the United Nations' Sustainable Development Goals (SDGs), there is a pressing need to accurately assess their research output against these goals. Current approaches, primarily reliant on keyword-based Boolean search queries, conflate incidental keyword matches with genuine contributions, reducing retrieval precision and complicating benchmarking efforts. This study investigates the application of autoregressive Large Language Models (LLMs) as evaluation agents to identify relevant scholarly contributions to SDG targets in scholarly publications. Using a dataset of academic abstracts retrieved via SDG-specific keyword queries, we demonstrate that small, locally-hosted LLMs can differentiate semantically relevant contributions to SDG targets from documents retrieved due to incidental keyword matches, addressing the limitations of traditional methods. By leveraging the contextual understanding of LLMs, this approach provides a scalable framework for improving SDG-related research metrics and informing institutional reporting.

Figures

Figures reproduced from arXiv: 2411.17598 by the authors.

Figure 1
Figure 1. Absracts retrieved from Scopus via keyword-based [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Classification outcomes for Phi-3.5-mini, Mistral-7B [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Venn diagrams illustrating model agreement for [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

  1. [1]

    Mapping scholarly publications related to the Sustainable Development Goals: Do independent bibliometric approaches get the same results?

    C. S. Armitage, M. Lorenz, and S. Mikki, “Mapping scholarly publications related to the Sustainable Development Goals: Do independent bibliometric approaches get the same results?” Quantitative Science Studies , vol. 1, no. 3, pp. 1092–1108, Aug. 2020. [Online]. Available: https://doi.org/10.1162/qss a 00071

    doi:10.1162/qss 2020

  2. [2]

    A computational approach to evaluating curricular alignment to the United Nations Sustainable Development Goals,

    P. Lemarchand, M. McKeever, C. MacMahon, and P. Owende, “A computational approach to evaluating curricular alignment to the United Nations Sustainable Development Goals,” Frontiers in Sustainability , vol. 3, 2022. [Online]. Available: https://doi.org/10.3389/frsus.2022. 909676

    work page doi:10.3389/frsus.2022 2022

  3. [3]

    Classifying economics for the common good: a note on the links between sustainable development goals and JEL codes,

    J. T. Heikkil ¨a, “Classifying economics for the common good: a note on the links between sustainable development goals and JEL codes,” Journal of Documentation , vol. 77, no. 2, p. 305–319, Jan. 2021. [Online]. Available: https://doi.org/10.1108/jd-08-2020-0146

    work page doi:10.1108/jd-08-2020-0146 2021

  4. [4]

    ULA, a Bibliometric Method to Identify Sustainable Development Goals Using Large Language Models,

    L. Bergeron, J. Franc ¸ois, R. State, and J. Hilger, “ULA, a Bibliometric Method to Identify Sustainable Development Goals Using Large Language Models,” in 2023 IEEE International Humanitarian Technology Conference (IHTC) , 2023, pp. 1–6. [Online]. Available: https://doi.org/10.1109/IHTC58960.2023.10508849

    work page arXiv 2023

  5. [5]

    Evaluating the Performance of Large Language Models for SDG Mapping (Technical Report),

    H. Yin, A. Aryani, and N. Nambiar, “Evaluating the Performance of Large Language Models for SDG Mapping (Technical Report),” CoRR,

    work page

  6. [6]

    SDG target detection in environmental reports using Retrieval-augmented Generation with LLMs,

    D. Garigliotti, “SDG target detection in environmental reports using Retrieval-augmented Generation with LLMs,” in Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024) , D. Stammbach et al. , Eds. Bangkok, Thailand: Association for Computational Linguistics, Aug. 2024, pp. 241–250. [Online]. Available: http...

    work page 2024

  7. [7]

    Identifying Research Supporting the United Nations Sustainable Development Goals,

    B. Jayabalasingham, R. Boverhof, K. Agnew, and L. Klein, “Identifying Research Supporting the United Nations Sustainable Development Goals,” Elsevier Data Repository, V1, 2019, v1. [Online]. Available: https://doi.org/10.17632/87txkw7khs.1

    doi:10.17632/87txkw7khs.1 2019

  8. [8]

    (2024, June) Impact Rankings 2024: Method- ology

    THE reporters. (2024, June) Impact Rankings 2024: Method- ology. [Online]. Available: https://www.timeshighereducation.com/ world-university-rankings/impact-rankings-2024-methodology

    work page 2024

Show all 12 references
  1. [9]

    Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,

    M. Abdin et al. , “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone,” CoRR, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2404.14219

    2024 doi arXiv

  2. [10]

    Mistral 7B,

    A. Q. Jiang et al. , “Mistral 7B,” CoRR, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2310.06825

    2023 doi arXiv

  3. [11]

    The Llama 3 Herd of Models,

    A. Dubey et al., “The Llama 3 Herd of Models,” CoRR, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2407.21783

    2024 doi arXiv

  4. [2024]

    Available: https://doi.org/10.48550/arXiv.2408.02201

    [Online]. Available: https://doi.org/10.48550/arXiv.2408.02201

    doi arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.