{"id":"b9ebc6d4-d601-491d-8397-96d27c8ee7ca","arxiv_id":"2411.14042","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"WORLDREP is a new expert-validated dataset of 44,706 news articles with LLM-generated country-pair relationship scores, reporting higher expert agreement than GDELT.","lead":"This paper introduces WORLDREP, a dataset of nearly 45,000 news articles in which large language models score the relationship between each pair of countries mentioned, and experts validate a subset of the labels. The authors report that models trained on these labels match expert judgments far better than models trained on GDELT, and they provide a zero-shot LLM benchmark for forecasting future country relationships.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 77.4% vs 30.6% agreement gap in Table 2 is inflated by GDELT's lack of an 'Unknown' category, making the headline GDELT comparison structurally unfair.","rationale":"The reader identified the expert subset as the weakest assumption. I agree that the 1,030 articles are non-random and inter-annotator agreement is unreported, but the most load-bearing concern is the headline comparison itself: it is computed in a way that guarantees GDELT a lower score because it lacks the 'Unknown' category. This is an internal inconsistency in the evaluation protocol, not just a generalization risk. If the gap in Table 2 mostly vanishes when unknown samples are excluded or when GDELT is allowed to abstain, the central claim of 'higher-quality labels' is not established by the reported evidence. The paper's Section 3.2 already provides a fairer two-class comparison, and that result may still support the claim, but the paper should not advertise the aggregate number as the headline validation. The forecast benchmark in Section 4 is also circular (ground truth from the same LLM labeling system), but it is downstream of the label-quality claim; fixing the Table 2 comparison is the first-order issue. Given the dataset and code are released and the flaws are re-analyzable, a conditional accept remains appropriate, so I keep the reader's verdict.","tokens_in":23052,"tokens_out":6932,"duration_ms":64993,"concrete_test":"Recompute Table 2 agreement restricted to the expert labels that are Conflict/Cooperation (excluding Unknown) for both WORLDREP and GDELT, and also compute a version where GDELT is allowed to abstain on expert-Unknown pairs and is scored as a random guess. If WORLDREP's advantage over GDELT is much smaller than the reported 46.8-point gap, the headline comparison is an artifact of the asymmetric label space. Additionally, recompute both datasets' agreement under alternative score thresholds (e.g., 0.1/0.9 bands instead of 0.25/0.75) to test sensitivity to the binning choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative evidence for WORLDREP's superior label quality is Table 2, where WORLDREP achieves 77.4% overall agreement with experts versus GDELT's 30.6%. This comparison is not apples-to-apples: GDELT has no 'Unknown' label, so every expert 'Unknown' pair is necessarily counted as a GDELT error, while WORLDREP can score 73.7% on those same pairs. The paper acknowledges GDELT lacks 'Unknown' in the caption but still presents the aggregate as a direct quality comparison. A fair comparison would either restrict to pairs where experts chose Conflict/Cooperation, or give GDELT an abstention option. The paper's own Section 3.2 does the former and still shows a gap, but the headline number in the abstract and Section 3.1 is the misleading aggregate. The threshold binning (0.0-0.25 cooperation, 0.75-1.0 conflict, 0.25-0.75 unknown) is also arbitrary and not justified against the expert distribution, and changing the boundaries could shift both datasets' agreement rates. This is a more immediate threat to the central claim than the representativeness of the 1,030-article expert subset, because it affects the validity of the comparison even on the annotated subset.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WORLDREP, a dataset of 44,706 news articles annotated with pairwise country-relationship scores (cooperation-to-conflict, plus an 'Unknown' category) generated by GPT-4 using a two-stage scratchpad with verification and self-correction. The authors validate label quality against domain-expert annotations on 1,030 articles, report that WORLDREP agrees with experts at 77.4% overall versus GDELT's 30.6%, and show that classifiers trained on WORLDREP labels outperform those trained on GDELT labels when evaluated on expert-labeled test data. They then propose a zero-shot LLM benchmark for forecasting future country-pair relationships, reporting 45-61% accuracy across nine LLMs.","tokens_in":23325,"tokens_out":2785,"duration_ms":27710,"significance":"If the validation claims hold, WORLDREP would be a useful resource for text-based international event prediction, addressing two real limitations of GDELT: multilateral country extraction and a nuanced relationship label with an explicit 'Unknown' class. The paper ships the dataset, expert labels, and automation code, and the expert-annotation comparison in Section 3.2 is a non-circular, appropriately designed transfer experiment. The self-correcting scratchpad's measured improvement in country-extraction F1 (0.825 to 0.963 against expert labels, Table 3) is a concrete and credible contribution. However, the headline quality comparison in Table 2 is structurally unfair to GDELT, the expert gold standard is small and non-random, and the future-event benchmark in Section 4.1 uses the paper's own LLM labels as ground truth, so the forecasting numbers should not be read as externally validated accuracy.","major_comments":[{"comment":"The aggregate agreement comparison of 77.4% versus 30.6% is structurally unfair because GDELT has no 'Unknown' label: every expert-labeled 'Unknown' pair is automatically counted as a GDELT error, while WORLDREP can score 73.7% on those same pairs. The authors acknowledge GDELT lacks 'Unknown' in the caption but still present the aggregate as the headline result and repeat it in the abstract. The fairer two-class comparison in Section 3.2 (restricting to pairs experts labeled Conflict or Cooperation) still shows a gap, but the paper should report that restricted agreement as the primary quality comparison, or give GDELT an explicit abstention option, and clearly delabel the aggregate table.","section":"Section 3.1, Table 2"},{"comment":"The expert validation uses only 1,030 articles selected because GDELT had fully annotated samples, which is not a random sample of the 44,706-article corpus, and the labels come from three graduate students supervised by one professor with no inter-annotator agreement reported. As stated, the 77.4% agreement rate cannot be shown to generalize to the full dataset. The authors should report per-annotator agreement (e.g., Cohen's or Fleiss' kappa), describe how disagreements were adjudicated, and either add a random-sample expert evaluation or explicitly present the current numbers as a non-random, limited pilot rather than a corpus-wide validation.","section":"Section 3.1 and Appendix B.1"},{"comment":"The threshold binning (0.0-0.25 cooperation, 0.25-0.75 unknown, 0.75-1.0 conflict) is introduced without justification or sensitivity analysis. Since both Table 2 and the downstream classification experiments depend on these boundaries, and since changing the boundaries could shift agreement rates for both WORLDREP and GDELT, the authors should either justify the thresholds from the expert label distribution or report how agreement changes under reasonable alternative boundaries.","section":"Section 3.1, score-to-class thresholds"},{"comment":"The future prediction test labels are produced by the same automated LLM labeling pipeline that generated WORLDREP, so the 45-61% accuracy figures in Table 5 partly measure how well one LLM predicts another LLM's labels rather than how well the models forecast real future international relationships. This is a load-bearing issue for the benchmark claim. The authors should validate a sample of the May 2024 target labels against expert or independent event ground truth, or at minimum report agreement between the labeling model and the prediction models as a caveat, and avoid presenting the raw accuracy as evidence of real-world forecasting ability.","section":"Section 4.1, future event prediction benchmark"}],"minor_comments":[{"comment":"The dataset start date is given as February 18, 2015 in Table 1 but as February 2013 in Appendix B.5.2; these dates should be reconciled.","section":"Table 1 vs Appendix B.5.2"},{"comment":"The main text says 'several graduates' while Appendix B.1 says exactly three graduate students; use a single consistent description.","section":"Section 3.1 vs Appendix B.1"},{"comment":"The prompt in Figure 10 lists Countries: 'CHN', 'PRK' but Step 1 asks about the relationship between 'FRA' and 'RUS', which appears to be a copy-paste error from the previous figure.","section":"Figure 10"},{"comment":"The claim that the May 2024 test period is 'beyond the knowledge cutoff dates of all these models' should be substantiated per model, since some of the listed models have version-specific and sometimes later cutoff dates.","section":"Section 4.1"},{"comment":"No confidence intervals, standard deviations, or significance tests are reported for the F1/accuracy improvements; given the small expert test set, these would help establish that the gains are not noise.","section":"Tables 3 and 4"},{"comment":"The abstract claims the labels are 'rigorously validated by domain experts' on the full dataset, but the validation is on a non-random 1,030-article subset; the wording should be tempered to match the actual scope.","section":"Abstract and Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The dataset and code release are valuable, and the expert transfer experiment (Table 4) is the strongest part of the paper. The main risk is that the headline comparison in Table 2 and the forecasting benchmark in Section 4.1 overstate the evidence in ways that could mislead downstream users. Both issues are fixable with additional analysis and reporting, so I do not recommend rejection, but the revision needs to confront them directly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the WORLDREP dataset is a real contribution. You get 44,706 articles, 147,931 labeled country pairs, the self-correcting scratchpad pipeline, and public code and data. The country extraction F1 improvement from 0.825 to 0.963 is concrete evidence the correction steps help. The classifier transfer experiment in Table 4 also gives non-circular evidence that the labels carry signal: models trained on WORLDREP beat models trained on GDELT when both are evaluated on expert labels. That part holds up.\n\nThe soft spots are real but fixable. First, the headline 77.4% vs 30.6% agreement in Table 2 is structurally inflated, exactly as the stress-test note says. GDELT has no 'Unknown' category, so every expert 'Unknown' pair is automatically a GDELT error while WORLDREP can score points on those same pairs. The caption admits this, but the abstract and Section 3.1 present the aggregate as a direct quality comparison. The fairer comparison in Section 3.2, restricted to conflict/cooperation pairs, still shows a large gap, so the conclusion likely holds, but the headline number should be reworked. The score-to-class thresholds are arbitrary too; a sensitivity analysis would help.\n\nSecond, the expert validation rests on 1,030 non-randomly selected articles and no inter-annotator agreement is reported. The public expert labels help, but the paper should include IAA and ideally a random sample. This is a moderate weakness, not fatal.\n\nThird, the forecasting benchmark in Section 4 uses the automated LLM labeling system as ground truth. The paper openly replaces expert labeling with the automated pipeline, so the 45-61% accuracy numbers partly measure how well one LLM predicts another LLM. The paper does not acknowledge this circularity. A small expert-validated sample of the May 2024 target labels would make the benchmark interpretable.\n\nOverall, the central claim that WORLDREP provides better relationship labels than GDELT is plausible and supported by the non-circular part of the evidence. The dataset and code are the main value, and they are worth serious attention. I would send this to peer review, with the request to fix the GDELT comparison, add IAA, and validate a sample of the forecasting labels.","headline":"WORLDREP is a real dataset contribution, but the headline GDELT comparison is inflated and the forecasting benchmark is partially circular.","tokens_in":23888,"tokens_out":2790,"would_cite":true,"duration_ms":25886,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WORLDREP, an LLM-built dataset of 44,706 news articles, labels country-pair relations that match political-science experts 77.4% of the time, versus GDELT's 30.6%.","keywords":["WORLDREP","international event prediction","LLM-based data annotation","self-correcting scratchpad","GDELT","relationship labeling","geopolitical forecasting","dataset quality evaluation"],"falsifier":"Have a second, independent panel of international-relations experts label a random sample of WORLDREP articles, and report per-label agreement and inter-annotator agreement; if the overall agreement with the original labels falls well below 77.4%, or the expert panel itself disagrees, the headline quality claim fails.","tokens_in":22840,"feed_emoji":"🌍","tokens_out":4904,"duration_ms":42700,"temperature":0.7,"pith_summary":"The paper introduces WORLDREP, a dataset of 44,706 news articles in which country pairs are labeled with a continuous conflict–cooperation score plus an 'unknown' category, generated by large language models guided by a self-correcting scratchpad. The central claim is that these labels are substantially more reliable than the widely used GDELT database, and the paper supports this by comparing both label sets against judgments made by political-science experts on 1,030 articles. On that comparison, WORLDREP agrees with the experts 77.4% of the time overall—84.8% on conflict, 70.6% on cooperation, and 73.7% on unknown—whereas GDELT agrees only 30.6% of the time. If this holds, WORLDREP gives researchers a more trustworthy training signal and benchmark for predicting future international relations from news text, and its automated pipeline means the labeling can be extended to newer articles without the usual expert-labeling bottleneck.","feed_headline":"LLM-labeled world-event dataset beats GDELT on expert agreement","feed_subtitle":"WORLDREP hits 77% agreement with political scientists on country-pair relations; GDELT scores 31%.","key_machinery":"The central mechanism is a structured scratchpad prompt: each annotation is produced in steps—extract, verify, explain disagreements, correct—so the model can catch its own mistakes before the label is recorded. Country extraction and relationship scoring each use this pattern, and each relationship pair is scored five times with a majority vote on 'unknown' and averaging of the remaining scores. This yields a continuous 0-to-1 score (0 = full cooperation, 1 = full conflict) that is thresholded into cooperation, conflict, and unknown classes. The paper credits this self-correcting, ensembled scoring for both the higher agreement with experts and the improved country-extraction F1.","core_discovery":"The paper's discovery is that an LLM-based annotation pipeline with built-in verification and correction can produce relationship labels that align with domain-expert judgment far more closely than GDELT's rule-based labels, while also capturing multilateral relations and an 'unknown' category that GDELT lacks. The claim is established through a direct comparison on a shared test set: expert-labeled relationships in 1,030 articles, where WORLDREP attains 77.4% overall agreement versus GDELT's 30.6%. The paper further shows that classifiers trained on WORLDREP labels and evaluated on expert labels reach an average accuracy of 0.854, versus 0.469 for classifiers trained on GDELT labels, and that the self-correcting extraction step raises country-identification F1 from 0.825 to 0.963. The authors also provide a zero-shot benchmark in which frontier LLMs predict future country-pair relations from recent news, with top accuracy around 61%.","pith_inferences":["A natural next validation step, not performed in the paper, is to measure inter-annotator agreement among the experts; without it, the 77.4% agreement rate conflates label quality with expert consensus.","The 1,030-article expert subset was chosen for having full GDELT annotations, so it may over-represent clear-cut bilateral events; the quality gap on the full, more multilateral corpus could be different.","The same scratchpad-plus-ensemble recipe could be transferred to other annotation domains that need graded, multi-subject labels, such as economic or public-health event tracking.","If the 45–61% zero-shot accuracy on the future-prediction benchmark is representative, an immediate research target is context retrieval: feeding models the most relevant past articles rather than a fixed recent window."],"forward_implications":["WORLDREP can serve as a training set for models that forecast whether a country pair will be in conflict, cooperation, or an indeterminate state at a future date.","The automated pipeline makes it feasible to refresh the dataset continuously, replacing the expensive expert-labeling bottleneck with LLM scoring that has been validated against experts.","The 'unknown' category lets models avoid forced binary predictions, which should reduce spurious conflict or cooperation classifications on ambiguous news.","Because labels are continuous, downstream systems can treat relationship intensity as a graded signal rather than a binary category.","The published expert-labeled test set gives the community a stable ground truth for comparing future event-prediction models."],"supporting_citations":[{"why":"Supplies the GDELT event database that serves both as the source of news articles for WORLDREP and as the baseline whose label quality the paper argues is inferior.","marker":"Leetaru and Schrodt, 2013"},{"why":"Provides the prior task setup of treating each news article as a single event with an occurrence time, which WORLDREP follows in its data collection and prediction framing.","marker":"Shi et al., 2024"},{"why":"Introduces the scratchpad technique that the paper adapts into verification-and-correction prompts for extraction and labeling.","marker":"Nye et al., 2021"},{"why":"Represents the AutoCast-style QA benchmark for future world events that WORLDREP positions itself against and extends with continuous relationship scoring.","marker":"Zou et al., 2022"},{"why":"One of the pretrained encoders used in the document classification experiment that compares models trained on WORLDREP versus GDELT labels.","marker":"Devlin et al., 2018"}],"fun_headline_variants":["WORLDREP LLM labels beat GDELT 77% vs 31% on expert agreement","LLM-labeled event dataset hits 77% expert agreement, GDELT at 31%","WORLDREP: LLM labels align with experts at 77%, GDELT at 31%","LLM-based event labels outperform GDELT on expert agreement: 77% vs 31%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison treats the labels that one professor and three graduate students assigned to 1,030 selected articles as an unbiased gold standard for all 44,706 articles, without reporting inter-annotator agreement.","fun_headline_variants_meta":{"raw":{"variants":["WORLDREP LLM labels beat GDELT 77% vs 31% on expert agreement","LLM-labeled event dataset hits 77% expert agreement, GDELT at 31%","WORLDREP: LLM labels align with experts at 77%, GDELT at 31%","LLM-based event labels outperform GDELT on expert agreement: 77% vs 31%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":3057,"prompt_tokens":894,"completion_tokens":2163,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":2058}},"tokens_in":510,"tokens_out":2163,"duration_ms":14201,"temperature":1.0,"reasoning_tokens":2058,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:36:07.590391+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a second, independent panel of international-relations experts label a random sample of WORLDREP articles, and report per-label agreement and inter-annotator agreement; if the overall agreement with the original labels falls well below 77.4%, or the expert panel itself disagrees, the headline quality claim fails.","supporting_citations":[],"review_version":1}