Application of LLMs to Threat Assessment of Foreign Peacekeeping Missions

Christian Schmidt; Diego Pilutti; Gerhard Backfried; Michael Suker

arxiv: 2606.27106 · v1 · pith:JWQHAZJGnew · submitted 2026-06-25 · 💻 cs.CR · cs.AI

Application of LLMs to Threat Assessment of Foreign Peacekeeping Missions

Gerhard Backfried , Christian Schmidt , Diego Pilutti , Michael Suker This is my paper

Pith reviewed 2026-06-26 03:34 UTC · model grok-4.3

classification 💻 cs.CR cs.AI

keywords LLMsthreat assessmentpeacekeeping missionsOSINTmedia analysisrisk modelthreat extraction

0 comments

The pith

LLMs extract peacekeeping mission threats from media with high agreement to human judgment.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a workflow that applies large language models to open-source media to identify threats relevant to foreign peacekeeping missions. It integrates an existing risk model with media collection and a sequence of LLM steps that map content to threats, structure the information, and refine outputs for relevance. Evaluation on extracted threats shows strong alignment with human analysts specifically on whether a threat exists and whether it relates to the mission. This alignment is presented as evidence that the approach can assist analysts handling media volume in such contexts. The work focuses on the EU Monitoring Mission in Georgia as its concrete use case.

Core claim

The paper claims that its LLM pipeline for mapping media contents to mission-relevant threats produces results with high agreement to human judgment on core aspects including threat presence and mission relevance, indicating that LLMs offer a workable method to support threat assessment in peacekeeping operations.

What carries the argument

The LLM-supported threat extraction pipeline that combines an interdisciplinary risk-model with OSINT media collection and additional processing steps to map contents to structured threats.

If this is right

The workflow can generate structured threat information from media that aligns with analyst assessments on key dimensions.
Additional LLM processing steps after initial extraction improve the grounding and relevance of the outputs.
The method is demonstrated on the EU Monitoring Mission in Georgia and can be applied to similar foreign peacekeeping contexts.
High agreement on core aspects suggests the pipeline can reduce the manual effort required for initial threat screening.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same pipeline structure could be tested on missions in different regions to check if agreement levels remain stable.
Integrating the output with existing analyst tools might allow faster updating of risk assessments as new media appears.
If the mapping holds across larger document sets, it could support monitoring of multiple missions simultaneously.
Extending the evaluation to measure time savings for analysts would clarify operational impact beyond agreement scores.

Load-bearing premise

Media contents can be reliably mapped to mission-relevant threats by the LLM pipeline without substantial hallucination, bias, or loss of context.

What would settle it

A fresh collection of media documents on the same mission where independent human reviewers find low agreement with the pipeline outputs on threat identification and relevance.

Figures

Figures reproduced from arXiv: 2606.27106 by Christian Schmidt, Diego Pilutti, Gerhard Backfried, Michael Suker.

**Figure 2.** Figure 2: Evaluation Scores a high level of agreement between automatically generated information and human judgment. As such, these results are promising in terms of future (automated) support for further missions. 9 Conclusion and Outlook We have presented a semi-automatic approach for the application of LLMs to support threat assessment in foreign peacekeeping missions. Based on an indicator-based risk model and… view at source ↗

read the original abstract

We present a novel approach for applying Large Language Models (LLMs) to threat assessment in the context of foreign peacekeeping missions. Building on the PINPOINT project and its use case, the EU Monitoring Mission in Georgia, we combine an interdisciplinary risk-model with OSINT-based media collection and LLM-supported threat extraction. The proposed workflow maps media contents to mission-relevant threats, extracts structured information and applies several additional LLM-based processing steps to improve relevance and grounding. An evaluation of threats extracted from media documents shows high agreement between automatically generated results and human judgment for core aspects such as threat and mission relevance. These results indicate that LLMs provide a promising approach to support analysts in the context of peacekeeping missions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Routine LLM application to peacekeeping OSINT with an unevidenced 'high agreement' claim that lacks sample size, metrics, or hallucination controls.

read the letter

The core takeaway is that this paper describes a standard LLM pipeline for extracting threats from media about the EU Monitoring Mission in Georgia, then asserts high agreement with human judgment without showing the numbers or protocol that would let anyone check it.

What it does is combine an existing risk model with OSINT collection and several LLM steps for structuring and filtering. That workflow is described clearly enough at a high level and targets a real operational niche where analyst time is limited. The interdisciplinary angle and the specific use case are the parts that feel grounded.

The soft spot is the evaluation. The abstract says the extracted threats show high agreement on threat and mission relevance, but supplies no document count, no agreement statistic, no inter-rater details, and no mention of how they handled LLM hallucination or prompt sensitivity. Without those, the agreement number cannot be interpreted. The stress-test concern holds because the claim is load-bearing and the supporting evidence is missing.

This is the kind of paper that might interest a small group of security analysts or OSINT practitioners who want to see how LLMs are being tried in their domain. It does not reorganize methods or add new technical machinery, so it is unlikely to be cited outside that narrow circle.

I would not bring it to a reading group. I would not cite it. It does not yet deserve peer review in its current form because the central result cannot be verified from what is presented.

Referee Report

1 major / 0 minor

Summary. The manuscript presents a workflow for threat assessment in foreign peacekeeping missions that integrates an interdisciplinary risk model, OSINT media collection, and a sequence of LLM-supported steps for mapping media to mission-relevant threats, structured extraction, and relevance refinement. The case study is the EU Monitoring Mission in Georgia (building on the PINPOINT project). The central empirical claim is that an evaluation of threats extracted from media documents demonstrates high agreement between the LLM pipeline outputs and human judgment on core aspects such as threat identification and mission relevance, suggesting LLMs as a promising support tool for analysts.

Significance. If the evaluation can be substantiated with the missing quantitative details, the work could provide a concrete demonstration of LLM utility in a high-stakes applied security setting, potentially aiding efficiency in processing open-source information for peacekeeping operations. The grounding in a specific real-world use case and the multi-step LLM pipeline are strengths that distinguish it from generic LLM application papers.

major comments (1)

[Abstract] Abstract (and Evaluation section): The claim that 'an evaluation of threats extracted from media documents shows high agreement between automatically generated results and human judgment for core aspects such as threat and mission relevance' supplies no sample size, document count, agreement metric (e.g., Cohen's kappa, F1, or percentage), inter-rater reliability, human annotation protocol, exclusion criteria, or controls for LLM hallucination, prompt sensitivity, or context loss. This information is required to establish that the observed agreement reflects pipeline fidelity rather than evaluation artifacts, and its absence is load-bearing for the paper's primary conclusion.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback. The primary concern regarding missing quantitative details in the abstract and evaluation section is valid, and we will revise the manuscript to address it directly.

read point-by-point responses

Referee: [Abstract] Abstract (and Evaluation section): The claim that 'an evaluation of threats extracted from media documents shows high agreement between automatically generated results and human judgment for core aspects such as threat and mission relevance' supplies no sample size, document count, agreement metric (e.g., Cohen's kappa, F1, or percentage), inter-rater reliability, human annotation protocol, exclusion criteria, or controls for LLM hallucination, prompt sensitivity, or context loss. This information is required to establish that the observed agreement reflects pipeline fidelity rather than evaluation artifacts, and its absence is load-bearing for the paper's primary conclusion.

Authors: We agree that the current abstract and evaluation section do not provide the requested quantitative details. In the revised manuscript we will expand the evaluation section (and update the abstract) to report the sample size (number of media documents and extracted threats), the agreement metric(s) employed, inter-rater reliability, the human annotation protocol, exclusion criteria, and any controls implemented for LLM hallucination, prompt sensitivity, or context loss. These additions will allow readers to assess whether the observed agreement reflects pipeline performance. revision: yes

Circularity Check

0 steps flagged

No circularity: evaluation claim rests on independent human comparison with no fitted parameters or self-referential definitions

full rationale

The paper presents an LLM-based workflow for threat extraction from media and reports an empirical evaluation showing high agreement with human judgment on threat and mission relevance. No equations, fitted parameters, or predictions appear. The central result is framed as an external human comparison rather than a quantity derived from the pipeline itself by construction. No self-citation chains or uniqueness theorems are invoked to support the agreement claim. The derivation is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Only the abstract is available; no free parameters, axioms, or invented entities are explicitly stated or derivable from the provided text.

axioms (1)

domain assumption The interdisciplinary risk-model from the PINPOINT project correctly identifies mission-relevant threats
The workflow is built on this model; its validity is presupposed for the mapping step to be meaningful.

pith-pipeline@v0.9.1-grok · 5642 in / 1157 out tokens · 40784 ms · 2026-06-26T03:34:07.513417+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 4 canonical work pages · 1 internal anchor

[1]

Open source intelligence investigation: From strategy to implementation , pages=

Fusion of OSINT and non-OSINT data , author=. Open source intelligence investigation: From strategy to implementation , pages=. 2016 , publisher=

2016
[2]

2018 , publisher=

Defining second generation open source intelligence (OSINT) for the defense enterprise , author=. 2018 , publisher=

2018
[3]

Security and Human Rights , volume=

Analysis for Peace The Evolving Data Tools of UN and OSCE Field Operations , author=. Security and Human Rights , volume=. 2020 , publisher=

2020
[4]

Proceedings of the 10th International Conference on e-Business , year=

Open Source Intelligence for Traditional- and Social Media Sources , author=. Proceedings of the 10th International Conference on e-Business , year=
[5]

CROWD OSInfo

Wissensentwicklung mit "CROWD OSInfo" : eine Innovation des Cyber Documentation & Research Center (CDRC) der Zentraldokumentation (ZentDok), Landesverteidigungsakadmie (LVAK) , author=. 2015 , publisher=

2015
[6]

Proceedings of the Intelligence 2023 Conference of the Australian Institute of Professional Intelligence Officers, forthcoming , year=

Open Source Intelligence for Traditional- and Social Media Sources , author=. Proceedings of the Intelligence 2023 Conference of the Australian Institute of Professional Intelligence Officers, forthcoming , year=

2023
[7]

Advances in Neural Information Processing Systems , volume=

Judging LLM-as-a-judge with MT-bench and Chatbot Arena , author=. Advances in Neural Information Processing Systems , volume=
[8]

Procedia Engineering , volume =

New Artificial Intelligence Tools for Deep Conflict Resolution and Humanitarian Response , author =. Procedia Engineering , volume =. 2015 , publisher =. doi:10.1016/j.proeng.2015.06.083 , url =

work page doi:10.1016/j.proeng.2015.06.083 2015
[9]

Pasligh , title =

Hendrik A. Pasligh , title =
[10]

2025 , eprint =

On the Military Applications of Large Language Models , author =. 2025 , eprint =

2025
[11]

Stat , volume=

On Large Language Models in National Security Applications , author=. Stat , volume=. 2025 , publisher=

2025
[12]

2024 , publisher =

Hands-On Large Language Models: Language Understanding and Generation , author =. 2024 , publisher =

2024
[13]

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Nils Reimers and Iryna Gurevych , title =. CoRR , volume =. 2019 , url =. 1908.10084 , timestamp =

work page internal anchor Pith review Pith/arXiv arXiv 2019
[14]

Proceedings of Conference on Cognitive and Computational Aspects of Situation Management 2023 , editor =

Gerhard Backfried and Dorothea Thomas-Aniola and Diego Pilutti and Martin Boyer and Rüdiger Hein and Amir Tatabaei and Michael Zinkanell and Michael Suker and Philipp Agathonos , title =. Proceedings of Conference on Cognitive and Computational Aspects of Situation Management 2023 , editor =. doi:10.29007/vnzb , pages =

work page doi:10.29007/vnzb 2023
[15]

doi:10.1787/9789264043466-en , isbn =

Handbook on Constructing Composite Indicators: Methodology and User Guide , year =. doi:10.1787/9789264043466-en , isbn =

work page doi:10.1787/9789264043466-en

[1] [1]

Open source intelligence investigation: From strategy to implementation , pages=

Fusion of OSINT and non-OSINT data , author=. Open source intelligence investigation: From strategy to implementation , pages=. 2016 , publisher=

2016

[2] [2]

2018 , publisher=

Defining second generation open source intelligence (OSINT) for the defense enterprise , author=. 2018 , publisher=

2018

[3] [3]

Security and Human Rights , volume=

Analysis for Peace The Evolving Data Tools of UN and OSCE Field Operations , author=. Security and Human Rights , volume=. 2020 , publisher=

2020

[4] [4]

Proceedings of the 10th International Conference on e-Business , year=

Open Source Intelligence for Traditional- and Social Media Sources , author=. Proceedings of the 10th International Conference on e-Business , year=

[5] [5]

CROWD OSInfo

Wissensentwicklung mit "CROWD OSInfo" : eine Innovation des Cyber Documentation & Research Center (CDRC) der Zentraldokumentation (ZentDok), Landesverteidigungsakadmie (LVAK) , author=. 2015 , publisher=

2015

[6] [6]

Proceedings of the Intelligence 2023 Conference of the Australian Institute of Professional Intelligence Officers, forthcoming , year=

Open Source Intelligence for Traditional- and Social Media Sources , author=. Proceedings of the Intelligence 2023 Conference of the Australian Institute of Professional Intelligence Officers, forthcoming , year=

2023

[7] [7]

Advances in Neural Information Processing Systems , volume=

Judging LLM-as-a-judge with MT-bench and Chatbot Arena , author=. Advances in Neural Information Processing Systems , volume=

[8] [8]

Procedia Engineering , volume =

New Artificial Intelligence Tools for Deep Conflict Resolution and Humanitarian Response , author =. Procedia Engineering , volume =. 2015 , publisher =. doi:10.1016/j.proeng.2015.06.083 , url =

work page doi:10.1016/j.proeng.2015.06.083 2015

[9] [9]

Pasligh , title =

Hendrik A. Pasligh , title =

[10] [10]

2025 , eprint =

On the Military Applications of Large Language Models , author =. 2025 , eprint =

2025

[11] [11]

Stat , volume=

On Large Language Models in National Security Applications , author=. Stat , volume=. 2025 , publisher=

2025

[12] [12]

2024 , publisher =

Hands-On Large Language Models: Language Understanding and Generation , author =. 2024 , publisher =

2024

[13] [13]

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Nils Reimers and Iryna Gurevych , title =. CoRR , volume =. 2019 , url =. 1908.10084 , timestamp =

work page internal anchor Pith review Pith/arXiv arXiv 2019

[14] [14]

Proceedings of Conference on Cognitive and Computational Aspects of Situation Management 2023 , editor =

Gerhard Backfried and Dorothea Thomas-Aniola and Diego Pilutti and Martin Boyer and Rüdiger Hein and Amir Tatabaei and Michael Zinkanell and Michael Suker and Philipp Agathonos , title =. Proceedings of Conference on Cognitive and Computational Aspects of Situation Management 2023 , editor =. doi:10.29007/vnzb , pages =

work page doi:10.29007/vnzb 2023

[15] [15]

doi:10.1787/9789264043466-en , isbn =

Handbook on Constructing Composite Indicators: Methodology and User Guide , year =. doi:10.1787/9789264043466-en , isbn =

work page doi:10.1787/9789264043466-en