Research Entity Extraction and Topic Detection from UKRI Grant Proposals

Alexandru Marcoci; Angelo Salatino; Gemma Derrick; Kara Moraw; Rosa Filgueira; Sarah Callaghan; Xingran Ruan

arxiv: 2606.30304 · v1 · pith:CRPGDC6Mnew · submitted 2026-06-29 · 💻 cs.DL · cs.AI· cs.IR

Research Entity Extraction and Topic Detection from UKRI Grant Proposals

Xingran Ruan , Angelo Salatino , Rosa Filgueira , Kara Moraw , Alexandru Marcoci , Gemma Derrick , Sarah Callaghan This is my paper

Pith reviewed 2026-06-30 03:23 UTC · model grok-4.3

classification 💻 cs.DL cs.AIcs.IR

keywords entity extractiontopic classificationLLMgrant proposalsUKRIMistralresearch entitiesmetascience

0 comments

The pith

Mistral LLM reaches 90.5% accuracy classifying topics from UKRI grant proposals, beating the DSIT-Taxonomies pipeline at 71.4%.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares three approaches—GPT-4o, Mistral, and DSIT-Taxonomies—for extracting research entities from funding proposals and mapping them to topics. It finds Mistral and GPT-4o produce comparable high-quality entity sets with substantial overlap, while the Mistral pipeline maps these entities to the OpenAlex Topics taxonomy more accurately than the full DSIT-Taxonomies method. Evaluated on abstracts from 42 proposals, the Mistral approach reaches 90.5% topic classification accuracy against 71.4% for the alternative. A sympathetic reader would care because this points to a practical way to scan large volumes of grant data for early signals of emerging research areas that could guide public funding. The authors conclude Mistral supplies a high-performance, efficient, and secure option for such analysis.

Core claim

A three-stage pipeline that employs Mistral for primary entity extraction and subsequent mapping against the OpenAlex Topics taxonomy achieves 90.5% topic classification accuracy across 42 grant proposal abstracts, outperforming the DSIT-Taxonomies pipeline at 71.4% while GPT-4o yields entity sets of similar quality and semantic overlap.

What carries the argument

The three-stage pipeline that uses Mistral for primary entity extraction followed by mapping to the OpenAlex Topics taxonomy.

If this is right

Mistral supplies a high-performance, operationally efficient solution for large-scale processing of grant proposals.
The method supports secure handling of sensitive funding data without relying on fragmented taxonomy pipelines.
Improved topic classification can surface early signals of emerging research areas to guide public investment decisions.
GPT-4o and Mistral produce entity sets with significant semantic overlap that both exceed the quality of the DSIT-Taxonomies output.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same pipeline could be run on grant records from other national funders to compare topic emergence patterns across countries.
Embedding the extraction step into ongoing monitoring systems might allow funders to adjust priorities in near real time as new proposals arrive.
Replacing manual or rule-based taxonomies with LLM mapping may lower the cost of metascience studies that track research trends over time.
Extending the evaluation to full proposal texts rather than abstracts alone would test whether accuracy improves or remains stable.

Load-bearing premise

The accuracy figures rest on a reliable evaluation protocol applied to 42 proposals whose topic labels can be independently verified.

What would settle it

A re-run of the evaluation on the same 42 proposals or a fresh independent sample of comparable size where the Mistral pipeline accuracy drops below 80%.

Figures

Figures reproduced from arXiv: 2606.30304 by Alexandru Marcoci, Angelo Salatino, Gemma Derrick, Kara Moraw, Rosa Filgueira, Sarah Callaghan, Xingran Ruan.

**Figure 2.** Figure 2: Workflow of the Topic Classification. In practice, let T0 denote the DSIT-Taxonomies-assigned taxonomy node for a proposal p. This node may correspond to a domain, field, subfield, or topic within OpenAlex Topics. Instead of accepting T0 as final, our algorithm restricted the DSIT-Taxonomies relevance score table to all topic-level descendants of T0, which was denoted as Desc(T0). This produced a topic-lev… view at source ↗

**Figure 3.** Figure 3: Interface developed via Jupyter Notebook, displaying comparative annotations for DSIT-Taxonomies (left), GPT-4o (middle), and Mistral (right). The green box displays the approach. A primary observation from our analysis is the fragmented nature of the DSIT-Taxonomies approach. As shown in the left panel of [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** Figure 4: Interface developed via Jupyter Notebook, displaying the annotated proposal, [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

read the original abstract

This paper presents preliminary findings from a UKRI-funded Metascience project comparing three LLM-based approaches, GPT-4o, Mistral, and a bespoke algorithm, DSIT-Taxonomies, for extracting and classifying research entities from funding proposals. Our project "Tracking Stars and Unicorns" aims to identify early signals of emerging research areas to inform public investment. Our methodology employed a three-stage pipeline, leveraging Mistral for primary entity extraction and mapping against the OpenAlex Topics taxonomy. We evaluated our approach across 42 proposals' abstracts from different areas and observed that Mistral and GPT-4o produce comparable, high-quality entity sets with significant semantic overlap, outperforming the fragmented DSIT-Taxonomies approach. Crucially, the Mistral-based approach achieved superior topic classification accuracy (90.5%) compared to the full DSIT-Taxonomies pipeline (71.4%). We conclude that Mistral offers a high-performance, operationally efficient, and secure solution for large-scale analysis of sensitive grant data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The 90.5% accuracy claim for Mistral on topic classification cannot be evaluated because the paper gives no account of how the ground-truth labels for the 42 abstracts were produced or verified.

read the letter

The one thing to know is that the headline accuracy numbers (90.5% for Mistral versus 71.4% for the DSIT pipeline) rest on topic labels whose origin is not described. Without that information it is impossible to tell whether the 19-point gap reflects real differences or post-hoc labeling choices.

The paper applies three existing approaches—GPT-4o, Mistral, and the DSIT-Taxonomies pipeline—to entity extraction and topic mapping on UKRI grant abstracts, using the OpenAlex taxonomy. It reports that the two LLMs produce comparable entity sets with substantial semantic overlap and that Mistral is operationally preferable for sensitive data. That last point is the concrete contribution: a direct comparison on real grant text that flags efficiency and security considerations.

The work is incremental rather than foundational. It tests off-the-shelf models on a new corpus instead of deriving a new method or proving a general property. The sample is fixed at 42 abstracts drawn from different areas, with no stated selection criteria or power analysis.

The main weakness is the evaluation protocol. The abstract states only that accuracy was measured “across 42 proposals’ abstracts” but supplies no detail on label provenance, blinding, inter-annotator agreement, or whether labels were fixed before the model outputs were inspected. If the labels were influenced by the model runs, the reported superiority is not interpretable. The stress-test note correctly flags this gap; nothing in the provided abstract resolves it.

This is for metascience teams already building grant-analysis pipelines who want a quick head-to-head on UKRI text. A reader looking for a reusable benchmark or a validated workflow will find the current evidence too thin.

I would not cite the paper as it stands. It deserves peer review once the labeling procedure, sample construction, and any error analysis are added, because the underlying task—scalable analysis of funding proposals—matters even if the present numbers need stronger grounding.

Referee Report

1 major / 1 minor

Summary. The paper presents preliminary findings from a UKRI-funded Metascience project comparing three LLM-based approaches (GPT-4o, Mistral, and a bespoke DSIT-Taxonomies pipeline) for extracting research entities and detecting topics from funding proposals. The methodology uses a three-stage pipeline with Mistral for primary entity extraction mapped to the OpenAlex Topics taxonomy. Evaluation on 42 proposal abstracts from different areas shows Mistral and GPT-4o yielding comparable high-quality entity sets with significant semantic overlap, outperforming the fragmented DSIT-Taxonomies approach; crucially, Mistral achieves 90.5% topic classification accuracy versus 71.4% for the full DSIT-Taxonomies pipeline. The authors conclude that Mistral offers a high-performance, efficient, and secure option for large-scale analysis of sensitive grant data.

Significance. If the accuracy comparison rests on a sound, pre-specified evaluation protocol, the result would provide actionable evidence favoring open-source LLMs over bespoke taxonomy pipelines for entity and topic extraction in grant analysis, with direct relevance to metascience and research-funding policy. The work is preliminary and limited by sample size, but it supplies a concrete head-to-head comparison that could guide operational choices for processing sensitive UKRI data.

major comments (1)

[Abstract] Abstract: The headline accuracy comparison (Mistral 90.5% vs. DSIT-Taxonomies 71.4%) is presented without any description of how the ground-truth topic labels for the 42 abstracts were obtained, whether labeling occurred before model outputs were inspected, what inter-annotator agreement was achieved, or how the 42 proposals were selected. This information is load-bearing for the central claim of superiority; without it the reported gap cannot be interpreted as evidence.

minor comments (1)

[Abstract] Abstract: The phrasing 'across 42 proposals' abstracts from different areas' is grammatically awkward and should be clarified.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their careful review and for identifying a key gap in the description of our evaluation protocol. We agree this information is necessary to support the central accuracy comparison and will revise the manuscript accordingly.

read point-by-point responses

Referee: [Abstract] Abstract: The headline accuracy comparison (Mistral 90.5% vs. DSIT-Taxonomies 71.4%) is presented without any description of how the ground-truth topic labels for the 42 abstracts were obtained, whether labeling occurred before model outputs were inspected, what inter-annotator agreement was achieved, or how the 42 proposals were selected. This information is load-bearing for the central claim of superiority; without it the reported gap cannot be interpreted as evidence.

Authors: We agree that the current manuscript does not provide these details, which limits the interpretability of the reported accuracy figures. In the revised version we will add a new subsection in Methods that explicitly describes: the stratified selection of the 42 proposals across UKRI funding areas; that ground-truth topic labels were assigned by two independent domain experts prior to generation or inspection of any model outputs; and the inter-annotator agreement obtained. We will also insert a brief reference to this protocol in the abstract. These additions will allow readers to assess the validity of the Mistral vs. DSIT-Taxonomies comparison. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical head-to-head comparison

full rationale

The paper reports direct empirical results from applying GPT-4o, Mistral, and DSIT-Taxonomies to 42 grant proposal abstracts and measuring topic classification accuracy (90.5% vs 71.4%). No equations, derivations, fitted parameters, or self-citations are present that reduce any claim to its own inputs by construction. The evaluation is presented as an observational comparison against an external benchmark set, satisfying the condition for a self-contained result with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review yields no explicit free parameters, invented entities, or non-standard axioms; evaluation implicitly assumes LLM outputs can be scored for accuracy against an external taxonomy without detailing the scoring procedure.

axioms (1)

domain assumption LLM-generated entity sets can be reliably compared for semantic overlap and mapped to OpenAlex topics with measurable accuracy
The 90.5% and 71.4% figures rest on this unelaborated assumption.

pith-pipeline@v0.9.1-grok · 5724 in / 1161 out tokens · 46933 ms · 2026-06-30T03:23:19.111456+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

9 extracted references · 1 canonical work pages · 1 internal anchor

[1]

As such, governments are increasingly seeking to optimise returns on public research investment by identifying and prioritising areas of emerging strength

Introduction In the global knowledge economy, publicly competitively funded research plays a central role in driving innovation, national productivity and economic growth (Sussex et al, 2016). As such, governments are increasingly seeking to optimise returns on public research investment by identifying and prioritising areas of emerging strength. In the U...

2016
[2]

Tracking Stars and Unicorns

and semantic analysis across large corpora of scientific text (Mansour et al, 2025). Despite this, the question remains in whether the reliability, consistency and comparability of different LLM approaches when applied to complex, domain-specific texts such as ex-ante research proposals. This paper presents preliminary findings from “Tracking Stars and Un...

2025
[3]

and Bibliometrix (Aria et al., 2017). Subsequent ontology-driven systems, such as the CSO Classifier (Salatino et al., 2022), SciNoBo (Gialitsis et al., 2022), QuickUMLS (Soldaini & Goharian, 2016), and MetaMap (Aronson & Lang, 2010), introduced more structured annotation by mapping text to predefined taxonomies. However, these systems are often domain-sp...

2017
[4]

Building on these, recent LLMs offer superior semantic reasoning and broader cross-domain generalisation

have demonstrated strong performance in scientific text analysis, through contextual embeddings. Building on these, recent LLMs offer superior semantic reasoning and broader cross-domain generalisation. Unlike ontology-driven systems, LLMs support entity extraction and zero-shot classification without relying on fixed vocabularies. Pretrained on vast scie...

2025
[5]

Finite Element Method

National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs, n=5), a specialist scientific organisation primarily funded by the MRC and the BBSRC. 3.2. Extracting Research Entities To map the conceptual landscape of the research proposals, this study employed an automated entity extraction pipeline leveraging the Mistral la...

2023
[6]

stem cells

Results 4.1. Evaluation Dataset For the evaluation, 42 proposal applications were randomly selected, ensuring a broad distribution across the various source councils and different time periods, to capture a wide range of academic disciplines and evolving research themes. This selection process also accounted for variations in text length, with the sample'...

2019
[7]

By processing thousands of proposals, the aggregate dataset achieves a comprehensive level of thematic coverage that naturally offsets minor individual inaccuracies

Discussion and conclusions The results of our evaluation suggest that while Mistral and GPT-4o demonstrate high performance in entity extraction, they are most effectively deployed at scale (i.e., for large datasets which limit human evaluation). By processing thousands of proposals, the aggregate dataset achieves a comprehensive level of thematic coverag...

2026
[8]

Leveraging Large Language Models for Generating Research Topic Ontologies: A Multi-Disciplinary Study

to conduct fine-grained analyses and identify emerging trends. To ensure a comprehensive view, we will extend our classification beyond titles and abstracts to include secondary outputs such as “Impact Summaries” and “Key Findings”. Open science practices The analyses presented in this manuscript fully adhere to the open science practices. The dataset and...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.5281/zenodo.18937183 2024
[9]

social impact of technology

Van Eck, N., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. scientometrics, 84(2), 523-538. Yin, Y., Dong, Y., Wang, K., Wang, D., & Jones, B. F. (2022). Public use and public funding of science. Nature human behaviour, 6(10), 1344-1350. Appendix A Here we report the prompt we used to extract entities from t...

2010

[1] [1]

As such, governments are increasingly seeking to optimise returns on public research investment by identifying and prioritising areas of emerging strength

Introduction In the global knowledge economy, publicly competitively funded research plays a central role in driving innovation, national productivity and economic growth (Sussex et al, 2016). As such, governments are increasingly seeking to optimise returns on public research investment by identifying and prioritising areas of emerging strength. In the U...

2016

[2] [2]

Tracking Stars and Unicorns

and semantic analysis across large corpora of scientific text (Mansour et al, 2025). Despite this, the question remains in whether the reliability, consistency and comparability of different LLM approaches when applied to complex, domain-specific texts such as ex-ante research proposals. This paper presents preliminary findings from “Tracking Stars and Un...

2025

[3] [3]

and Bibliometrix (Aria et al., 2017). Subsequent ontology-driven systems, such as the CSO Classifier (Salatino et al., 2022), SciNoBo (Gialitsis et al., 2022), QuickUMLS (Soldaini & Goharian, 2016), and MetaMap (Aronson & Lang, 2010), introduced more structured annotation by mapping text to predefined taxonomies. However, these systems are often domain-sp...

2017

[4] [4]

Building on these, recent LLMs offer superior semantic reasoning and broader cross-domain generalisation

have demonstrated strong performance in scientific text analysis, through contextual embeddings. Building on these, recent LLMs offer superior semantic reasoning and broader cross-domain generalisation. Unlike ontology-driven systems, LLMs support entity extraction and zero-shot classification without relying on fixed vocabularies. Pretrained on vast scie...

2025

[5] [5]

Finite Element Method

National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs, n=5), a specialist scientific organisation primarily funded by the MRC and the BBSRC. 3.2. Extracting Research Entities To map the conceptual landscape of the research proposals, this study employed an automated entity extraction pipeline leveraging the Mistral la...

2023

[6] [6]

stem cells

Results 4.1. Evaluation Dataset For the evaluation, 42 proposal applications were randomly selected, ensuring a broad distribution across the various source councils and different time periods, to capture a wide range of academic disciplines and evolving research themes. This selection process also accounted for variations in text length, with the sample'...

2019

[7] [7]

By processing thousands of proposals, the aggregate dataset achieves a comprehensive level of thematic coverage that naturally offsets minor individual inaccuracies

Discussion and conclusions The results of our evaluation suggest that while Mistral and GPT-4o demonstrate high performance in entity extraction, they are most effectively deployed at scale (i.e., for large datasets which limit human evaluation). By processing thousands of proposals, the aggregate dataset achieves a comprehensive level of thematic coverag...

2026

[8] [8]

Leveraging Large Language Models for Generating Research Topic Ontologies: A Multi-Disciplinary Study

to conduct fine-grained analyses and identify emerging trends. To ensure a comprehensive view, we will extend our classification beyond titles and abstracts to include secondary outputs such as “Impact Summaries” and “Key Findings”. Open science practices The analyses presented in this manuscript fully adhere to the open science practices. The dataset and...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.5281/zenodo.18937183 2024

[9] [9]

social impact of technology

Van Eck, N., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. scientometrics, 84(2), 523-538. Yin, Y., Dong, Y., Wang, K., Wang, D., & Jones, B. F. (2022). Public use and public funding of science. Nature human behaviour, 6(10), 1344-1350. Appendix A Here we report the prompt we used to extract entities from t...

2010