{"id":"bf50fbf3-b357-43bf-8ada-35cfeb82f69b","arxiv_id":"2505.09852","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLMs show limited parametric knowledge for conflict forecasting; adding retrieved context from GDELT and ACLED improves GPT-4's predictions modestly but does not help LLaMA-2.","lead":"This paper tests whether large language models can forecast violent conflict from memory alone, and whether adding recent news and event data helps. It finds that retrieval-augmented context improves GPT-4 but not LLaMA-2, and that fine-grained escalation predictions stay weak.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central RAG-vs-parametric comparison is not yet trustworthy: neither the retrieval date filter nor the model training cutoff is specified, and identical Ethiopia metrics across experiments suggest silent pipeline failure; temporal leakage alone could produce the reported gains.","rationale":"The reader identified temporal integrity as the weakest assumption, and the manuscript text supports that concern. The central claim is empirical and modest, but it depends on a causal retrieval pipeline that is never specified or evidenced. In good faith, the paper could be a useful contribution if the RAG inputs are actually restricted to the three months before each forecast; however, the absence of a stated date filter, the lack of any training-cutoff analysis for GPT-4, and the exact duplication of all Ethiopia metrics across experiments collectively make the comparison unsecured. The identical Ethiopia numbers are especially important because they indicate a silent failure mode that the authors did not notice or did not report; if the same failure affected other countries, the reported RAG gains could be artifacts. No statistical testing is reported either, so many small deltas in Tables 1 and 2 could be noise; that reinforces, rather than replaces, the leakage concern. I would not reject the paper outright, because the concern is testable and fixable, but I would require the audit described above before accepting the central claim. If the audit shows strict temporal filtering and the Ethiopia duplication is explained as a one-off retrieval miss, the conditional acceptance could be upgraded; as written, the claim is unverified.","tokens_in":9558,"tokens_out":5295,"duration_ms":56739,"concrete_test":"Re-run the RAG pipeline for one sampled country-month (e.g., Sudan, October 2023, and Ethiopia, March 2021) with timestamp logging on retrieved articles. Inspect the top-k article dates: if any article has publication date on or after the forecast month, leakage is confirmed. Then re-run all of Experiment 2 with a strict filter `publication_date <= first day of forecast month` and recompute Table 1 and Table 2. If the RAG-vs-parametric deltas shrink or reverse on Sudan, Israel, or Iran, the headline claim is unsupported. Independently, print the actual context string fed to the model for Ethiopia and confirm it differs from the parametric prompt; if it is identical, fix the retrieval failure and rerun before drawing any conclusions.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim—that RAG improves GPT-4's conflict forecasts—requires that non-parametric inputs are strictly causal: retrieved GDELT articles and GPT-3.5 summaries must contain no information from the forecast month or later. Section 3.2.1 says the context covers 'the previous three months' but nowhere specifies a date filter on the FAISS retrieval, and Section 3.1 says GDELT was queried 'between 2020 and 2024' as one corpus. If the embedding search over this whole window has no `date <= forecast_month` constraint, articles from the ground-truth month are semantically close to the query and leak the outcome, inflating exactly the RAG improvements the paper reports. The parametric condition has a symmetric problem: Section 3.2 asks the model to forecast 2020–2024 months without establishing whether GPT-4's training cutoff precedes each forecast month; where it does not, zero-shot 'forecasts' are memory retrieval, not prediction. A second red flag is that every Ethiopia metric in Table 1 is identical across Exp 1 and Exp 2 (0.2712 accuracy, 0.2091 macro-F1, MAE 446.59), which suggests the RAG branch silently fell back to the parametric prompt or failed for that country; if the pipeline can fail this way without being detected, the reported cross-country gains are not trustworthy as-is. The limitations section acknowledges small N and noisy summaries but does not mention temporal leakage, so the concern is unaddressed rather than dismissed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two ways of using LLMs for one-month-ahead conflict forecasting: parametric zero-shot prompting (Experiment 1) and retrieval-augmented prompting with GDELT-derived news summaries and ACLED indicators (Experiment 2). Using GPT-4 and LLaMA-2 across five countries over 2020-2024, it reports that RAG improves several GPT-4 metrics in some countries while LLaMA-2 shows smaller and inconsistent gains. The central claim is modest: non-parametric augmentation helps conflict forecasting only when the model can integrate retrieved context.","tokens_in":9852,"tokens_out":6113,"duration_ms":62351,"significance":"The question is timely, and the two-condition design is a natural way to separate parametric memory from external grounding. The use of real ACLED and GDELT data gives the study ecological validity, and the authors state that prompts and code will be released, which would support replication. However, the empirical evidence as presented is not yet sufficient to establish the central claim: temporal leakage is not ruled out, several reported rows are identical across the two conditions for reasons the paper does not explain, and no uncertainty quantification or baselines are provided. If these issues are addressed, the comparison would be a useful contribution to the growing literature on LLM-based event forecasting.","major_comments":[{"comment":"The RAG condition is described as providing 'the previous three months' of context, but the paper never states that the FAISS retrieval index is filtered so that only articles dated before the forecast month are returned; Section 3.1 says GDELT was queried 'between 2020 and 2024' as a single corpus. Without an explicit date constraint, retrieved articles from the ground-truth month are semantically close to the query and would leak the outcome into the prompt, inflating exactly the RAG improvements claimed in Section 4. Please specify the date filter on retrieval and, ideally, rerun the experiments with a strict date-before-forecast constraint.","section":"Section 3.2.1 and Section 3.1"},{"comment":"The parametric condition assumes that the models' pretraining excludes the forecast months, but no training cutoffs are reported for GPT-4 or LLaMA-2 relative to the 2020-2024 evaluation window. For months before a model's knowledge cutoff, zero-shot 'forecasts' can be memory retrieval rather than prediction. Please state the training cutoffs, restrict the evaluation to months after the later cutoff, or demonstrate that the results are robust when pre-cutoff months are removed.","section":"Section 3.2"},{"comment":"All GPT-4 metrics for Ethiopia are identical between Experiment 1 and Experiment 2 in Table 1 (e.g., 0.2712 accuracy, 0.2091 macro-F1, MAE 446.59, and identical binned-regression values). This is exactly what one would expect if the RAG branch silently fell back to the parametric prompt or failed to complete. The paper does not acknowledge or explain this, and identical rows also appear for Israel in Table 2 (e.g., class-categorical F1 macro 0.2710 and binned-regression accuracy 0.3443 in both experiments). The cross-condition comparison is not trustworthy until these rows are explained.","section":"Table 1 and Table 2"},{"comment":"The Results and Discussion narrative contradicts Table 1: the text says 'macro F1 scores in Ethiopia and Israel improved (from 0.2091 to 0.2710 and from 0.1471 to 0.2710)', but Table 1 shows Ethiopia's macro-F1 unchanged at 0.2091 in both experiments, and the second value (0.2710) is Israel's Experiment 2 value, not a gain for either country from the stated starting point. This makes it impossible to tell which of the reported gains are real.","section":"Section 4 and Section 5"},{"comment":"No confidence intervals, significance tests, or comparison baselines are reported. With 59 country-months, class imbalance, and often marginal differences between conditions, the claim that RAG 'consistently' improves GPT-4 is not supported as presented. At minimum, report a majority-class or persistence baseline and paired significance tests or bootstrap intervals, and disclose the number of repeated generations at temperature 0.2.","section":"Section 4 and Section 3.3"}],"minor_comments":[{"comment":"The paper should define how the slope thresholds and quantile bins for the 'Class (From Fatalities)' and binned-regression tasks are computed, and state whether they are fit on training months only; currently the reader cannot tell whether the evaluation uses full-period ground truth to construct labels.","section":"Section 3.3"},{"comment":"The table captions should report the number of observations per country and the number of repeated generations or standard errors; the current tables show only point estimates.","section":"Table 1 and Table 2 captions"},{"comment":"Several citations are incomplete: 'Yu et al.' has no year or venue, and the TimeRAG citation by 'Yang et al.' is not matched to a complete reference entry.","section":"Section 2"},{"comment":"The name 'LLama' is written inconsistently with 'LLaMA' in the tables and text; please standardize.","section":"Throughout"},{"comment":"The abstract mentions ACLED as a source of structured context, but the method section should clarify that ACLED is used only for weekly fatality counts and ground truth, not for retrieved text passages.","section":"Section 3.1"},{"comment":"The GitHub link is anonymous; before publication, please replace it with a stable repository and add a data availability statement describing the exact GDELT query dates and any filtering steps.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The identical rows in Table 1 and Table 2 suggest a possible unnoticed pipeline failure, and I would ask the authors to provide execution logs or rerun output for the RAG branch. I also recommend verifying that the retrieval implementation actually applies a temporal filter; if it does not, the comparison should be redone. The contribution is modest but within scope for cs.CL, and the issues are fixable within the manuscript, which supports major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe headline: this paper asks a genuinely useful question—do LLMs forecast conflict from parametric memory alone, and does RAG actually help?—but the headline comparison is not yet trustworthy. Two things stand out: the retrieval step never states a date filter for the FAISS search, so articles from the forecast month could leak into the context, and every Ethiopia metric in Table 1 is identical between Experiments 1 and 2, which strongly suggests the RAG branch silently fell back to the parametric pipeline for that country. These are load-bearing issues for the paper's central claim that RAG improves GPT-4.\n\nWhat is new: a systematic side-by-side of parametric vs. RAG for conflict forecasting across two models, five countries, and 59 monthly points. The setup is a reasonable extension of prior work (Mirai, Chang et al., Conflict-RAG), and the paper is honest about its small N and noisy summaries. The qualitative finding that GPT-4 integrates external context better than LLaMA-2 is plausible and consistent with other LLM knowledge-conflict work.\n\nWhere the soft spots are, in order: (1) Temporal integrity is unspecified. Section 3.1 says GDELT was queried \"between 2020 and 2024\" as a corpus; Section 3.2.1 says the context covers \"the previous three months\" but never says the retrieval itself is restricted to months before the forecast date. Without that constraint, the retrieved summaries can contain the answer, and the RAG gains in Table 1 are uninterpretable. (2) The identical Ethiopia numbers are a red flag that the pipeline can fail silently; the paper never mentions it. (3) There are no baselines—majority class, persistence, or simple statistical models—so we don't know whether LLM performance means anything at all. (4) No confidence intervals or significance tests; a difference could be noise. (5) The code and data are promised on an anonymous GitHub but not linked or verified.\n\nThese are not manufactured complaints; they are evident from the paper itself. The central claim is left standing only if the leakage concern is resolved. That makes the paper appropriate for peer review, but only as a major-revision candidate. If the authors can specify the temporal filters, diagnose the Ethiopia fallback, add baselines, and release artifacts, this could become a useful empirical benchmark for LLM conflict early warning.\n\nMy recommendation: engage with it, but be clear that the evaluation needs to be redone with temporal integrity and baseline comparisons before the results can be trusted.","headline":"Useful question, untrustworthy comparison: the RAG-vs-parametric result hinges on an unspecified retrieval date filter and silent Ethiopia fallback, so treat the headline gains as provisional.","tokens_in":10396,"tokens_out":2554,"would_cite":false,"duration_ms":25352,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that conflict forecasting from LLM pretrained knowledge alone captures broad trends but misses fine-grained ones, and that retrieval-augmented context improves results only when the model can integrate it.","keywords":["conflict forecasting","large language models","parametric knowledge","retrieval-augmented generation","zero-shot prompting","GDELT","ACLED","early warning systems"],"falsifier":"Re-run the RAG condition with a strict retrieval cutoff that provably excludes all articles dated after the forecast month, and compare against the paper's pipeline; if the improved macro-F1 and binned-regression scores vanish, the reported RAG gains are inflated by future leakage, and if they persist, the gains are genuine.","tokens_in":9366,"feed_emoji":"📈","tokens_out":6819,"duration_ms":59994,"temperature":0.7,"pith_summary":"The paper asks whether large language models can forecast violent conflict from what they already know, and whether adding fresh external context helps. It compares zero-shot predictions from GPT-4 and LLaMA-2 with a retrieval-augmented condition that supplies summaries of recent news, event tone, and fatality counts. The core claim is that parametric knowledge alone captures broad conflict dynamics but misses fine-grained trends and numeric estimates, while retrieval augmentation helps only when the model can integrate the retrieved context. On most tasks and countries, GPT-4 improves with RAG; LLaMA-2 shows minimal or inconsistent gains. If true, this means early-warning systems should pair LLMs with curated real-time conflict data rather than rely on pretrained knowledge.","feed_headline":"Adding current news data lifts GPT-4 conflict forecasts, not LLaMA-2","feed_subtitle":"Adding three months of news and fatality context beats zero-shot GPT-4, but not LLaMA-2.","key_machinery":"The machinery is a two-condition evaluation comparing zero-shot parametric forecasting with a retrieval-augmented generation pipeline. In the RAG condition, semantic search retrieves relevant news excerpts, a summarizer condenses them, and the prompt is enriched with the average GDELT tone, the average Goldstein scale score (a numeric proxy for cooperation-conflict in an event), and weekly ACLED fatalities over the past twelve weeks. The model then produces a four-label trend (Escalate, De-escalate, Peace/No Conflict, Stable Conflict) and a fatality estimate, scored against ACLED ground truth. This controlled contrast is what separates the contribution of external context from the model's internal knowledge.","core_discovery":"The central discovery is that non-parametric knowledge does not automatically improve conflict forecasting. In the paper's experiments, zero-shot LLMs recognize peaceful or stable dynamics but are weak at escalation and de-escalation classes and at fatality counts. When the models receive three months of retrieved news summaries, average article tone, Goldstein scale scores, and weekly fatalities, GPT-4's macro-F1 and binned regression improve across most countries, while LLaMA-2 mostly does not benefit and sometimes degrades. The authors interpret this as evidence that effective non-parametric forecasting requires both retrieval and sufficient model capacity to interpret and incorporate the retrieved information.","pith_inferences":["A natural next test is to control the retrieval window explicitly: if the pipeline is rerun with provably pre-forecast-only articles, and scores persist, the RAG gains are real; if they drop, the reported effects partly reflect future leakage.","The paper's label-from-fatality mapping may already encode the outcome; a sharper test would ask for direct ordinal fatality predictions and then map to labels, isolating whether the model actually forecasts conflict rather than reproducing recent levels.","The model-specific gap hints that retrieval-augmented conflict forecasting is gated by context adherence: instruction-tuned models that follow the supplied numbers benefit, while smaller chat models may ignore or override them, connecting to the knowledge-conflict phenomenon surveyed in the paper."],"forward_implications":["For high-stakes early warning, zero-shot LLM outputs should be treated as directional priors, not calibrated forecasts.","Retrieval augmentation is worth deploying for models with strong instruction-following like GPT-4, but model choice matters more than the mere presence of retrieval.","Binned intensity classes improve with context while raw fatality mean absolute error does not, so evaluation metrics that collapse fatality counts into quantile bins can overstate RAG's practical value.","Open-weight models such as LLaMA-2 may still be usable in low-resource settings when paired with retrieval, but only in select regions and task formulations."],"supporting_citations":[{"why":"Supplies the GDELT event data source that provides article URLs, tone scores, and Goldstein scale values for retrieval.","marker":"(Leetaru and Schrodt, 2013)"},{"why":"Provides the four-label temporal state system (Escalate, De-escalate, Peace/No Conflict, Stable Conflict) used for classification.","marker":"(Croicu and von der Maase, 2025)"},{"why":"Establishes the ViEWS early-warning baseline that motivates the forecasting task and the use of conflict and governance indicators.","marker":"(Hegre et al., 2019)"},{"why":"Shows that newspaper text features can predict conflict onset, supporting the use of news-derived context in forecasting.","marker":"(Mueller and Rauh, 2018)"},{"why":"Demonstrates that some LLMs bias toward parametric knowledge even when RAG provides external context, framing the model-specific differences observed here.","marker":"(Wu et al.)"},{"why":"Surveys knowledge conflicts between parametric and retrieved information, motivating the parametric-versus-RAG comparison.","marker":"(Xu et al., 2024)"},{"why":"Provides evidence on when to trust parametric versus non-parametric memories, informing the expected benefit of retrieval.","marker":"(Mallen et al., 2022)"},{"why":"Introduces TS-RAG, a retrieval-augmented time series forecasting method that the paper's RAG design extends to conflict data.","marker":"(Ning et al., 2025)"},{"why":"Proposes Conflict-RAG for multilingual conflict understanding, a direct predecessor to the retrieval approach used here.","marker":"(Wood and Joshi, 2024)"}],"fun_headline_variants":["RAG improves GPT-4 conflict forecasts, not LLaMA-2","News data boosts GPT-4 conflict predictions, LLaMA-2 no gain","Conflict forecasting: retrieval helps GPT-4, not LLaMA-2","GPT-4 benefits from news context in conflict forecasting, LLaMA-2 doesn't","Retrieval-augmented conflict forecasting: GPT-4 up, LLaMA-2 flat"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the data pipeline is temporally clean: retrieved articles and summaries cover only the three months before each forecast, and the models' pretraining did not already contain the outcome being predicted, but the paper never states the retrieval date filters or model training cutoffs.","fun_headline_variants_meta":{"raw":{"variants":["RAG improves GPT-4 conflict forecasts, not LLaMA-2","News data boosts GPT-4 conflict predictions, LLaMA-2 no gain","Conflict forecasting: retrieval helps GPT-4, not LLaMA-2","GPT-4 benefits from news context in conflict forecasting, LLaMA-2 doesn't","Retrieval-augmented conflict forecasting: GPT-4 up, LLaMA-2 flat"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1424,"prompt_tokens":915,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":399}},"tokens_in":531,"tokens_out":509,"duration_ms":4535,"temperature":1.0,"reasoning_tokens":399,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:22:43.114955+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the RAG condition with a strict retrieval cutoff that provably excludes all articles dated after the forecast month, and compare against the paper's pipeline; if the improved macro-F1 and binned-regression scores vanish, the reported RAG gains are inflated by future leakage, and if they persist, the gains are genuine.","supporting_citations":[{"cited_title":"From Newswire to Nexus: Using text-based actor embeddings and transformer networks to forecast conflict dynamics","cited_arxiv_id":"2501.03928","evidence_quote":"Provides the four-label temporal state system (Escalate, De-escalate, Peace/No Conflict, Stable Conflict) used for classification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the ViEWS early-warning baseline that motivates the forecasting task and the use of conflict and governance indicators."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that newspaper text features can predict conflict onset, supporting the use of news-derived context in forecasting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Proposes Conflict-RAG for multilingual conflict understanding, a direct predecessor to the retrieval approach used here."}],"review_version":1}