Pith. sign in

REVIEW 3 major objections 2 minor 33 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

LLMs do not forecast conflict escalation but instead apply coverage-based categories from training data.

2026-07-02 22:46 UTC pith:LIEZQ7XY

load-bearing objection LLMs match opposite trivial baselines by coverage tier in conflict forecasting and lose to a simple LR, but escalation labels may not be independent of the coverage variable used to stratify. the 3 major comments →

arxiv 2607.00018 v1 pith:LIEZQ7XY submitted 2026-05-29 cs.CY

The Limits of LLM Forecasting: Parametric Knowledge Gaps Across Conflict Zones

Poli Nemkova This is my paper
classification cs.CY
keywords LLM forecastingconflict escalationmedia coverage asymmetryzero-shot predictionparametric knowledgecountry categorical priorACLED data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper evaluates zero-shot forecasting of armed conflict escalation across 22 countries on a held-out set of 660 cases. It finds that Llama-3.3-70B predicts escalation for every under-covered zone, matching a trivial always-yes baseline, while GPT-4o predicts no escalation for every over-covered zone and misses all five actual events. A logistic regression using only eleven observation-window features and no country identifiers outperforms both models. Adding structured evidence from ACLED worsens LLM performance on under-covered zones. The core issue is that models interpret temporal signals through a country-categorical prior shaped by asymmetric media coverage.

Core claim

LLMs do not forecast conflict but categorize it: Llama predicts escalation on every under-covered case and matches the always-yes baseline to three decimals, while GPT-4o predicts no escalation on every over-covered case and misses all five actual events. A logistic regression with eleven observation-window features and no country information achieves higher F1 than either model, and adding ACLED evidence degrades LLM results on under-covered zones by a factor of roughly two.

What carries the argument

Coverage-stratified zero-shot evaluation that isolates the LLM's reliance on a country-categorical prior instead of temporal signal from observation windows.

Load-bearing premise

The escalation labels on the held-out test set are independent of the media coverage variable used to group countries.

What would settle it

An LLM correctly predicting escalation in an under-covered zone on the basis of the eleven observation-window features alone, without defaulting to the always-yes pattern.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Under-covered populations receive qualitatively different AI outputs that cannot separate stable from escalating periods.
  • Adding external structured evidence such as ACLED degrades rather than improves LLM performance on low-coverage zones.
  • Standard benchmarks without coverage stratification will miss this categorical failure mode.
  • Training data documentation must record geographic representation of conflict events.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar coverage-driven defaults may appear in other domains where media attention is highly uneven, such as disaster forecasting or economic risk assessment.
  • Retraining or fine-tuning on balanced conflict data from under-covered zones could be tested as a direct countermeasure.
  • The finding implies that parametric knowledge gaps are not uniform but concentrated in regions with sparse training examples.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The manuscript documents a 224× disparity in English-language media coverage of armed conflicts across 22 countries (2020–2026) and evaluates zero-shot escalation forecasting by Llama-3.3-70B and GPT-4o on a 660-case held-out test set. It reports that LLMs exhibit a qualitative failure mode rather than a performance gradient: Llama predicts escalation on every under-covered case (matching the Always-YES baseline to three decimals) while GPT-4o predicts no escalation on every over-covered case (missing all five actual events). A logistic regression using eleven observation-window features containing no country information achieves F1 ≈ 0.402 and outperforms both LLMs across tiers; adding structured ACLED evidence further degrades LLM performance on under-covered zones. The paper concludes that LLMs categorize conflicts by coverage prior rather than forecast temporal signal and calls for coverage-stratified benchmarking and improved datasets for under-covered zones.

Significance. If the central claim holds after addressing label independence, the work provides concrete empirical evidence of parametric knowledge gaps in LLMs for conflict forecasting, with direct implications for equitable AI deployment in under-covered regions. The explicit comparison against structured baselines (logistic regression and ACLED-augmented prompts) and the held-out test set construction are strengths that allow falsifiable assessment of the categorization hypothesis.

major comments (3)
  1. [Methods (escalation labeling and test-set construction)] Methods (label construction and test-set definition): The central claim that LLMs categorize by coverage prior rather than forecast requires that the five missed escalation events and the 660-case labels are independent of the 224× coverage variable used for stratification. The manuscript must demonstrate that escalation detection does not draw from the same English-language media sources used to bin countries, or else the uniform YES/NO behaviors could be an artifact of label construction. This is load-bearing for the qualitative-failure interpretation.
  2. [Results (feature construction and logistic regression baseline)] Results (eleven observation-window features): The paper asserts these features contain “no country information” and capture temporal signal without leakage from the coverage prior, yet provides no explicit test (e.g., correlation analysis or ablation) showing orthogonality to coverage intensity or to the escalation threshold itself. Without this, the outperformance of the logistic regression over LLMs cannot be cleanly attributed to the absence of a country-categorical prior.
  3. [Results (structured evidence experiments)] Results (ACLED augmentation): The reported degradation (GPT-4o F1 0.323 → 0.168 on under-covered zones) is presented as evidence that the bottleneck is interpretation rather than data availability, but the manuscript does not analyze whether the added ACLED evidence itself correlates with coverage tiers or introduces new selection effects.
minor comments (2)
  1. [Abstract and Results] Abstract and §4: The phrase “matching the trivial Always-YES baseline to three decimals” should be accompanied by the exact numerical values for transparency.
  2. [Methods] The manuscript should clarify the exact temporal windows and feature definitions for the eleven variables to allow replication.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their thorough review and insightful comments on our manuscript. We address each of the major comments point by point below, indicating the revisions we plan to make to address the concerns raised.

read point-by-point responses

  1. Referee: Methods (escalation labeling and test-set construction): The central claim that LLMs categorize by coverage prior rather than forecast requires that the five missed escalation events and the 660-case labels are independent of the 224× coverage variable used for stratification. The manuscript must demonstrate that escalation detection does not draw from the same English-language media sources used to bin countries, or else the uniform YES/NO behaviors could be an artifact of label construction. This is load-bearing for the qualitative-failure interpretation.

    Authors: We agree that independence between the escalation labels and the coverage stratification is essential for the interpretation. The labels are derived from ACLED, which compiles data from diverse international and local sources, not limited to English-language media. However, to strengthen this, we will add a detailed description of the label construction process and an analysis showing that the sources for escalation events do not overlap significantly with the media coverage data used for binning countries. This will be included in a revised Methods section. revision: yes

  2. Referee: Results (feature construction and logistic regression baseline): The paper asserts these features contain “no country information” and capture temporal signal without leakage from the coverage prior, yet provides no explicit test (e.g., correlation analysis or ablation) showing orthogonality to coverage intensity or to the escalation threshold itself. Without this, the outperformance of the logistic regression over LLMs cannot be cleanly attributed to the absence of a country-categorical prior.

    Authors: The eleven features are constructed solely from temporal observation windows (e.g., event counts in prior periods) without any country identifiers or coverage metrics. We did not perform an explicit orthogonality test in the submitted version. We will add such an analysis, including correlations with coverage intensity, to demonstrate that the features do not encode the coverage prior, thereby supporting the attribution of the logistic regression's superior performance to its use of temporal signals. revision: yes

  3. Referee: Results (structured evidence experiments): The reported degradation (GPT-4o F1 0.323 → 0.168 on under-covered zones) is presented as evidence that the bottleneck is interpretation rather than data availability, but the manuscript does not analyze whether the added ACLED evidence itself correlates with coverage tiers or introduces new selection effects.

    Authors: We recognize the value of examining whether ACLED data correlates with coverage tiers. In the revision, we will include an analysis of ACLED event distribution and coverage across tiers, along with a discussion of any potential selection effects. This will help clarify that the performance degradation is due to the LLM's interpretation challenges rather than data issues. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark with independent baselines

full rationale

The paper is a held-out empirical comparison of LLM zero-shot forecasting against structured baselines on a 660-case test set stratified by media coverage. The logistic regression baseline is fitted on eleven observation-window features and reported as an external comparator (F1=0.402), not derived from or reduced to the LLM outputs. No equations, uniqueness theorems, ansatzes, or self-citations are invoked that would make any reported result equivalent to its inputs by construction. The central claims rest on direct performance observations (e.g., Llama matching Always-YES on under-covered cases) that remain falsifiable against the test labels and external data sources.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that the held-out test set escalation labels are ground truth independent of media coverage and that the logistic regression features contain no implicit country signal. No new entities are postulated. The 11 features are treated as given rather than derived.

free parameters (1)
  • eleven observation-window features
    The logistic regression uses eleven unspecified features extracted from observation windows; their exact construction and any thresholds are not derived from first principles in the abstract.
axioms (1)
  • domain assumption Escalation events in the test set can be labeled independently of English-language media coverage volume.
    The stratification by coverage and the claim that LLMs follow a country-categorical prior both presuppose that the ground-truth escalation labels are not themselves shaped by the same coverage variable.

pith-pipeline@v0.9.1-grok · 5803 in / 1498 out tokens · 25506 ms · 2026-07-02T22:46:29.445079+00:00 · methodology

0 comments
read the original abstract

Media coverage of armed conflict is deeply asymmetric: we document a 224$\times$ gap between the most and least covered conflict zones in English-language media across 22 countries (2020--2026). We evaluate zero-shot conflict escalation forecasting across all 22 countries on a 660-case held-out test set, comparing Llama-3.3-70B and GPT-4o against three structured baselines. The central finding is not a performance gradient but a qualitative failure: LLMs do not forecast conflict -- they categorize it. Llama predicts escalation on every under-covered case, matching the trivial Always-YES baseline to three decimals; GPT-4o predicts NO on every over-covered case, missing all five actual escalation events. A logistic regression using only eleven observation-window features with \emph{no country information} achieves F1~=~0.402, outperforming both LLMs in every measurable tier. This failure cannot be resolved at inference time: adding structured ACLED evidence degrades performance on under-covered zones (GPT-4o F1: 0.323~$\to$~0.168) and falls below LR by a factor of 2.4. The bottleneck is not data availability but the LLM's interpretation of temporal signal under a country-categorical prior. Under-covered populations receive not just less accurate AI, but qualitatively different AI that cannot distinguish stable from escalating periods. We call for coverage-stratified benchmarking, conflict NLP datasets for under-covered zones, and training data documentation standards for geographic conflict representation.

Figures

Figures reproduced from arXiv: 2607.00018 by Poli Nemkova.

Figure 1
Figure 1. Figure 1: Media attention ratio across 22 conflict zones, measured as GDELT English-language news articles per [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages · 3 internal anchors

  1. [1]

    and Dementieva, Daryna

    Karamolegkou, Antonia and Borah, Angana and Cho, Eunjung and Choudhury, Sagnik Ray and Galletti, Martina and Gupta, Pranav and Ignat, Oana and Kargupta, Priyanka and Kotonya, Neema and Lamba, Hemank and Lee, Sun-Joo and Mangla, Arushi and Mondal, Ishani and Moudakir, Fatima Zahra and Nazar, Deniz and Nemkova, Poli and Pisarevskaya, Dina and Rizwan, Naquee...

    work page doi:10.18653/v1/2026.eacl-long.238 2026

  2. [2]

    International Studies Review , volume=

    Learning from the past and stepping into the future: Toward a new generation of conflict prediction , author=. International Studies Review , volume=

    work page

  3. [3]

    Big Data

    Forecasting civil wars: Theory and structure in an age of "Big Data" and machine learning , author=. Journal of Conflict Resolution , volume=

    work page

  4. [4]

    Advances in Neural Information Processing Systems , volume=

    Language models are few-shot learners , author=. Advances in Neural Information Processing Systems , volume=

    work page

  5. [5]

    Introducing the

    Sundberg, Ralph and Melander, Erik , journal=. Introducing the

    work page

  6. [6]

    Harvard Humanitarian Initiative , year=

    Responsible use of artificial intelligence in the humanitarian sector , author=. Harvard Humanitarian Initiative , year=

    work page

  7. [7]

    Language Models (Mostly) Know What They Know

    Language models (mostly) know what they know , author=. arXiv preprint arXiv:2207.05221 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    AAAI workshop , year=

    Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data , author=. AAAI workshop , year=

    work page

  9. [9]

    ISA annual convention , volume=

    Gdelt: Global data on events, location, and tone, 1979--2012 , author=. ISA annual convention , volume=. 2013 , organization=

    work page 1979

  10. [10]

    GPT-4 Technical Report

    GPT-4 Technical Report , author=. arXiv preprint arXiv:2303.08774 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    The Llama 3 Herd of Models

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    International Journal of Forecasting , volume =

    Rost, Nicolas and Ronco, Michele , title =. International Journal of Forecasting , volume =. 2026 , doi =

    work page 2026

  13. [13]

    2022 , url =

    Project Jetson: Predictive Analytics for Forced Displacement in. 2022 , url =

    work page 2022

  14. [14]

    News Droughts, News Floods, and

    Eisensee, Thomas and Str. News Droughts, News Floods, and. The Quarterly Journal of Economics , volume =. 2007 , doi =

    work page 2007

  15. [15]

    Proceedings of the 41st International Conference on Machine Learning , series =

    Manvi, Rohan and Khanna, Samar and Burke, Marshall and Lobell, David and Ermon, Stefano , title =. Proceedings of the 41st International Conference on Machine Learning , series =. 2024 , url =

    work page 2024

  16. [16]

    Proceedings of the 4th Workshop on Ethics in Natural Language Processing , year =

    Wu, Shijie and Dredze, Mark , title =. Proceedings of the 4th Workshop on Ethics in Natural Language Processing , year =

    work page

  17. [17]

    Political Analysis , volume =

    Muchlinski, David and Siroky, David and He, Jingrui and Kocher, Matthew , title =. Political Analysis , volume =. 2016 , doi =

    work page 2016

  18. [18]

    Combining Transformer-Based Actor Embeddings with

    Croicu, Mihai and. Combining Transformer-Based Actor Embeddings with. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , year =

    work page

  19. [19]

    Journal of Peace Research , volume =

    Galtung, Johan and Ruge, Mari Holmboe , title =. Journal of Peace Research , volume =. 1965 , doi =

    work page 1965

  20. [20]

    , title =

    Entman, Robert M. , title =. Journal of Communication , volume =. 1993 , doi =

    work page 1993

  21. [21]

    and Gebru, Timnit and McMillan-Major, Angelina and Shmitchell, Shmargaret , title =

    Bender, Emily M. and Gebru, Timnit and McMillan-Major, Angelina and Shmitchell, Shmargaret , title =. Proceedings of the 2021. 2021 , doi =

    work page 2021

  22. [22]

    Documenting Large Webtext Corpora:

    Dodge, Jesse and Sap, Maarten and Marasovi. Documenting Large Webtext Corpora:. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages =. 2021 , doi =

    work page 2021

  23. [23]

    Datasheets for Datasets , journal =

    Gebru, Timnit and Morgenstern, Jamie and Vecchione, Briana and Vaughan, Jennifer Wortman and Wallach, Hanna and. Datasheets for Datasets , journal =. 2021 , doi =

    work page 2021

  24. [24]

    Introducing

    Raleigh, Clionadh and Linke, Andrew and Hegre, H. Introducing. Journal of Peace Research , volume =. 2010 , doi =

    work page 2010

  25. [25]

    arXiv preprint arXiv:2510.22823 , year=

    Cross-Lingual Stability and Bias in Instruction-Tuned Language Models for Humanitarian NLP , author=. arXiv preprint arXiv:2510.22823 , year=

    work page arXiv

  26. [26]

    Proceedings of

    Nemkova, Apollinaire Poli and others , title =. Proceedings of

    work page

  27. [27]

    Do Large Language Models Know Conflict?

    Nemkova, Apollinaire Poli and Lingareddy, Sarath Chandra and. Do Large Language Models Know Conflict?. arXiv preprint arXiv:2505.09852 , year =

    work page arXiv

  28. [28]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    work page doi:10.1145/322234.322243 1981

  29. [29]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    work page 1972

  30. [30]

    Publications Manual , year = "1983", publisher =

    work page 1983

  31. [31]

    Dan Gusfield , title =. 1997

    work page 1997

  32. [32]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    work page 2015

  33. [33]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

    work page