REVIEW 5 major objections 12 minor 28 references
Using LLMs to create analytical datasets: A case study of reconstructing the historical memory of Colombia
T0 review · 5 major / 12 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read By feeding 235,000 scanned Spanish-language news articles to a large language model, this paper builds a 78,685-event dataset of violence in Colombia that is more granular than official records and finds no statistically significant link be
desk verdict Useful, transparent pipeline for LLM-based event extraction, but the dataset's representativeness is unproven and the regression tables contain a copy-paste error. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the prompt-and-filter pipeline: a 32-question featurization prompt sent to GPT-4o mini, an inclusion question that keeps only articles about a single specific violent event, a geocoding step, and a supervised random-forest deduplication model (trained on 2,245 manually labeled articles) that groups duplicate reports into cliques and keeps the most informative article. The prompt design matters because yes/no questions such as 'Was FARC involved?' outperform open-ended attacker identification; temperature is lowered for consistency; and multi-event articles are excluded because the model struggles to separate them.
What would settle it
Re-run the pipeline on a random sample of the excluded multi-event articles with human extraction of each event, then re-estimate the coca-eradication fixed-effects regression; if the coefficient becomes statistically significant or changes sign, the paper's null result does not hold for full conflict coverage.
Extended reading notes
Core claim
The central claim is that a large language model, guided by a carefully engineered 32-question prompt and a deduplication step, can convert a huge archive of newspaper scans into a structured event-level dataset with enough accuracy to support descriptive and policy analysis. Applied to Colombia, the pipeline yields 78,685 violent events, with manual review of 514 articles showing 89.7% accuracy on victim counts, 98.6% on location, and 86.6% on violence type. The resulting data complements official records: for matched events it supplies names, motives, attack methods, and community context; for unmatched events it adds incidents official records miss. The authors use the dataset to estimate
Load-bearing premise
The manually measured extraction accuracy on 514 articles (victim counts 89.7%, locations 98.6%) is assumed to hold across the full corpus, even though articles describing multiple events were filtered out because the model handled them poorly.
Editorial extensions
If this is right
- The LLM pipeline can be reused on other large text corpora, not just conflict news, whenever structured analytical datasets are needed.
- The dataset provides granular event details—victim names, attack methods, group involvement—that official records often reduce to a single count.
- The null finding on coca eradication and violence challenges the earlier result that aerial spraying increases homicides, since that earlier result relied on official homicide statistics that may under-report violence.
- Mapping attacker groups and locations over time can help identify corridors where post-accord violence persists, informing resource allocation and anti-narcotic policy.
- Because the dataset is event-level and covers 2000–2022, it enables longitudinal analyses of conflict dynamics that official records, ending in 2012, cannot.
Reading between the lines
- The decision to discard articles that narrate multiple distinct events likely removes the most complex incidents—massacres, coordinated attacks—so the 78,685-event dataset may understate both overall violence and the concentration of violence in single episodes. Testing this would require human extraction on a sample of the excluded articles.
- The 89.7% victim-count accuracy measured on 514 articles may not hold on the full corpus, because error-prone articles were disproportionately removed by the multi-event filter; if accuracy degrades, the descriptive trends and regression estimates could shift.
- The null eradication result is conditional on newspaper coverage: if the press under-reports violence in areas where eradication is intense, the regression would be attenuated toward zero even if a true relationship exists.
- The same pipeline could be applied to other under-documented conflicts where newspaper archives exist but official statistics are missing, with the caveat that prompt validation must be repeated per language and genre.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a pipeline using GPT-4o-mini to automatically extract structured data on violent events from 235,000 scanned Spanish newspaper articles about Colombia, yielding a dataset of 78,685 events after filtering and deduplication. The authors manually validate a 514-article sample, compare the dataset to the Center for Historical Memory's registry, and use the resulting data for descriptive analysis and a fixed-effects regression of coca eradication on violence. They report no statistically significant relationship between eradication and violence, and position the work as a template for LLM-based dataset construction in the social sciences and humanities.
Significance. If validated, the dataset would be a valuable new resource for conflict research and historical memory, offering far more granular detail than official records (e.g., victim names, attack methods, group involvement). The paper demonstrates a reproducible pipeline and makes a concrete policy analysis contribution. However, the strength of these contributions is conditional on addressing the selection bias from filtering multi-event articles, the calibration of the deduplication threshold, and the relatively modest extraction accuracies for several key variables. The paper's claim that LLMs enable previously infeasible text mining at scale is plausible and well-motivated.
major comments (5)
- [Materials and Methods: Data quality evaluation and filtering] The pipeline filters out 59,380 articles that GPT identified as discussing multiple distinct events, under the assumption that such events would be reported independently elsewhere. However, the accuracy evaluation is based only on 514 retained articles; no validation is performed on the filtered set. The paper itself notes that GPT errors were concentrated when articles mentioned multiple events, victim types, or locations (pp. 3–4). Consequently, the reported accuracies (Table S.2) likely overstate performance on the excluded articles, and the retained dataset may be systematically biased toward simpler, single-event reports. This could distort descriptive patterns and the eradication regression if the excluded events are non-randomly distributed by department or year. The authors should validate on a random sample of the filtered articles or otherwise demonstrate that the exclusion do
- [Materials and Methods: Data deduplication] The duplicate classification cutoff threshold is set to 0.95 'that yields an overall duplicate rate close to the rate estimated from manual labeling.' This is circular: the final event count is partially calibrated to the manual duplicate rate (24.2%). Since the model has 72% specificity, the threshold choice materially affects the number of retained events. The paper should report sensitivity of the key results (descriptive patterns and the regression) to the threshold and justify the threshold independently of the target duplicate rate.
- [Table S.2 and Materials and Methods] Extraction accuracies are 89.7% for victim count, 86.6% for violence type, and 84.6% for victim type. For a dataset intended to support policy analysis, this level of measurement error is nontrivial. Measurement error in both event counts and violence-type classifications can bias regression coefficients (often toward zero) and distort descriptive conclusions. The claim that GPT accuracy is 'comparable to human parsing' is not supported by any human benchmark in the paper. The authors should either provide a human-parsing comparison or discuss the potential impact of measurement error on their findings, possibly with a misclassification sensitivity analysis.
- [Materials and Methods: Policy analysis] The regression equation includes lagged dependent variables and department and year fixed effects. With only 33 departments and 23 years, the fixed-effects estimator with lagged dependent variables is known to suffer from Nickell bias. Moreover, the dependent variable is derived from the LLM-extracted dataset, which is subject to the selection and measurement issues above. The finding of 'no statistically significant relationship' may be an artifact of these biases rather than a true null effect. The authors should acknowledge this limitation and, if possible, compare results using official data (e.g., CHM or police reports) for a subset of years.
- [Data and materials availability] The paper states that 'code and event summaries will be made available upon publication,' but the compiled dataset of 78,685 events is not released. For a dataset paper, the actual data should be made publicly available (with appropriate confidentiality safeguards) to enable verification and independent analyses. This is standard practice for papers whose primary contribution is a new dataset.
minor comments (12)
- [Tables S.4 and S.5] These tables appear to have identical coefficients and standard errors for murders and armed conflict; this is likely a copy-paste error and needs correction.
- [Materials and Methods: Policy analysis] The regression equation uses '𝛴ⅈ' and 'departmentok' instead of proper summation and department fixed effects notation.
- [Abstract] The abstract says 'over 200,000 violence-related newspaper articles,' but the pipeline starts with 235,000 and includes 107,445 after the inclusion criterion. Please clarify the numbers.
- [Materials and Methods: Data quality evaluation] The statement 'we found 18 (3.5%) articles where GPT either misidentified the group or did not correctly detect the groups' conflicts with the earlier 96% accuracy (4% error). Clarify the relationship.
- [Materials and Methods: Data deduplication] The model metrics (83% accuracy, 95% sensitivity, 72% specificity, 83% AUC, 91% F-measure) are reported on a 'balanced test set'; define the balancing procedure and include a confusion matrix.
- [Materials and Methods: Data] The original data source 'Cinep-Peace program' is not cited; provide a reference or description of how the articles were collected and their coverage.
- [Throughout] Minor typos: 'share size' should be 'large size'; in ref 26, 'Tirras' should be 'Tierras'.
- [Abstract and Results] The paper refers to 'over 107,445' but uses exact numbers elsewhere; use consistent phrasing.
- [Introduction/Policy Analysis] The policy analysis references 'two prior studies' but citations 9 and 10 are for other topics; ensure the correct references are cited.
- [Supplementary Materials] The robustness checks mentioned in the Materials and Methods are not reported; include them in the supplement.
- [Materials and Methods: Prompt engineering] The iterative prompt testing on 100 articles is mentioned but no details on the iteration process; consider adding a summary table.
- [Materials and Methods: Data processing] The paper should specify the exact GPT model (e.g., gpt-4o-mini-2024-07-18) and the temperature setting used for reproducibility.
Circularity Check
Mild calibration feedback in deduplication threshold; central findings remain independent.
-
fitted input called prediction
[Supplementary Materials, Materials and Methods, Data deduplication]
"We set the duplicate classification model cutoff threshold as 0.95, that yields an overall duplicate rate close to the rate estimated from manual labeling."
The deduplication threshold is a fitted parameter, explicitly chosen to reproduce the manually estimated duplicate rate (about 24.2% in the labeled sample). Since the final event count (78,685) is the direct complement of the duplicate-removal rate (27%), the reported number of events is partially calibrated to the manual labels by construction, rather than being an independent prediction of the pipeline. This means any inference that depends on the aggregate number of events inherits the calibration target. However, the paper's central substantive claims (descriptive trends, the eradication-violence regression) depend on which specific article pairs are merged, not solely on the aggregate rate, so the circularity is partial and does not force the main conclusions.
full rationale
The paper's central claims—that the GPT-derived dataset provides complementary, more granular information than official records and that coca eradication shows no statistically significant relationship with violence—are not circular. The dataset is constructed from newspaper text via OCR, prompt-based LLM featurization, filtering, and a supervised deduplication model trained on hand-labeled pairs; accuracy is independently assessed on a random 514-article sample, and the comparison with CHM's Sievcac database is an external benchmark. The regression uses UNODC eradication data as an external regressor and includes robustness checks. The only mild feedback loop is in deduplication: the classification threshold (0.95) is deliberately set to reproduce the manually estimated duplicate rate, so the aggregate event count is anchored to the labeled-sample estimate rather than being a free prediction. This is a calibration choice, not a derivation: the pairwise merge decisions remain model outputs, and the paper's substantive conclusions do not reduce to the threshold value. The skeptic's concern about filtering 59,380 multi-event articles is an external-validity/correctness risk (selection bias), not a circularity, because the paper does not use the filtered articles to define its conclusions and explicitly acknowledges the assumption. No self-citations or imported uniqueness theorems are load-bearing; the reference list contains no works by the authors.
Assumptions & free parameters
free parameters (4)
- Deduplication model cutoff threshold =
0.95
- Article length filter =
500 characters
- N-gram size in deduplication =
n=2
- Information score weights in clique selection =
location weighed twice as much as other factors
assumptions (4)
- domain assumption Newspaper articles in the Cinep-Peace archive are representative of violence events in Colombia and contain sufficient factual information.
- domain assumption GPT-4o-mini's extraction accuracy measured on 514 articles generalizes to the full corpus of 107,445 articles.
- ad hoc to paper Articles that discuss multiple distinct events would have been reported independently, so filtering them out does not introduce systematic bias.
- domain assumption The CHM database provides a valid external benchmark for comparison, despite having different collection protocols and temporal coverage.
Cite this review
Pith. "Pith review of Using LLMs to create analytical datasets: A case study of reconstructing the historical memory of Colombia." pith.science (2026). https://pith.science/paper/BBIBH2LZ
@misc{pith2026250904523,
author = {Pith},
title = {Pith review of: Using LLMs to create analytical datasets: A case study of reconstructing the historical memory of Colombia},
year = {2026},
howpublished = {\url{https://pith.science/paper/BBIBH2LZ}},
note = {Machine review of arXiv:2509.04523}
}
read the original abstract
Colombia has been submerged in decades of armed conflict, yet until recently, the systematic documentation of violence was not a priority for the Colombian government. This has resulted in a lack of publicly available conflict information and, consequently, a lack of historical accounts. This study contributes to Colombia's historical memory by utilizing GPT, a large language model (LLM), to read and answer questions about over 200,000 violence-related newspaper articles in Spanish. We use the resulting dataset to conduct both descriptive analysis and a study of the relationship between violence and the eradication of coca crops, offering an example of policy analyses that such data can support. Our study demonstrates how LLMs have opened new research opportunities by enabling examinations of large text corpora at a previously infeasible depth.
Figures
Reference graph
Works this paper leans on
-
[1]
Ochos Años por la Paz: Los Avances y Rezagos de la Implementación del Acuerdo
C.V. Rico, N.G. Sánchez, C.S. Quiroga, D.C. Moreno, L.F.D. González, M.F. Padilla, L.G. Triana, “Ochos Años por la Paz: Los Avances y Rezagos de la Implementación del Acuerdo” (Fundación Ideas para La Paz (FIP), UK Government, Del Capitolio Al Territorio, 2024)
work page 2024
-
[2]
UN News (2021); https://news.un.org/es/story/2021/04/1491312
United Nations, Preocupa el aumento de la violencia en territorios controlados por grupos armados ilegales y organizaciones criminales en Colombia. UN News (2021); https://news.un.org/es/story/2021/04/1491312
work page 2021
-
[3]
S. Torrado, La Cruz Roja alerta de la peor situación humanitaria en Colombia desde la firma del acuerdo de paz con las extintas FARC. El País (2025); https://elpais.com/america- 7 colombia/2025-03-27/la-cruz-roja-alerta-de-la-peor-situacion-humanitaria-en-colombia-desde-la- firma-del-acuerdo-de-paz-con-las-extintas-farc.html
work page 2025
-
[4]
United Nations Office on Drug and Crime (UNODC), “World Drug Report 2023”. (UNODC, 2023); http://www.unodc.org/unodc/en/data-and-analysis/world-drug-report-2023.html
work page 2023
-
[5]
J.S. Holmes, S.A.G. De Piñeres, K.M. Curtin, Drugs, Violence, and Development in Colombia: a Department-Level analysis. Latin American Politics and Society 48, 157–184 (2006)
work page 2006
-
[6]
Felbab-Brown, The Coca Connection: Conflict and drugs in Colombia and Peru
V. Felbab-Brown, The Coca Connection: Conflict and drugs in Colombia and Peru. The Journal of Conflict Studies 25, (2005)
work page 2005
-
[7]
E. Dávalos, L.F. Morales, Diffusion of crime control benefits: forced eradication and coca crops in Colombia. Spatial Economic Analysis 18, 292–317 (2022)
work page 2022
-
[8]
D.G. Fisher, A.A. Meitus, Uprooting or sowing violence?: Coca eradication and guerrilla violence in Colombia. Studies in Conflict and Terrorism 40, 790–807 (2016)
work page 2016
Show all 28 references
-
[9]
Rozo, On the unintended consequences of anti-drug eradication programs in producing countries
S.V. Rozo, On the unintended consequences of anti-drug eradication programs in producing countries. Association for Public Policy Analysis and Management (APPAM) (2013)
2013
-
[10]
So what if ChatGPT wrote it?
Y.K. Dwivedi, N. Kshetri, L. Hughes, E.L. Slade, A. Jeyaraj, A.K. Kar, A.M. Baabdullah, A. Koohang, V. Raghavan, M. Ahuja, H. Albanna, M.A. Albashrawi, A.S. Al-Busaidi, J. Balakrishnan, Y. Barlette, S. Basu, I. Bose, L. Brooks, D. Buhalis, L. Carter, S. Chowdhury, T. Crick, S....
2023
-
[11]
Internet of Things and Cyber-Physical Systems 3, 121– 154 (2023)
P.P Ray, ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems 3, 121– 154 (2023)
2023
- [12]
-
[13]
Kahle, H
D. Kahle, H. Wickham, ggmap: Spatial Visualization with ggplot2. The R Journal, 5, 144– 161 (2013)
2013
- [14]
-
[15]
Crime event Localization and Deduplication
F. Rollo, L. Po, “Crime event Localization and Deduplication” in International Semantic Web Conference (Cham: Springer International Publishing, 2020) pp. 361–377
2020
-
[16]
Zhang, A practical algorithm for efficiently deduplicating highly similar news in large news corpora
W. Zhang, A practical algorithm for efficiently deduplicating highly similar news in large news corpora. International Conference on Software Engineering DOI: 10.5121/csit.2023.131214 (2023)
2023
-
[17]
Mastering the Media Hype: Methods for Deduplication of Conflict Events from News Reports
V. Zavarella, J. Piskorski, C. Ignat, H. Tanev, M. Atkinson, “Mastering the Media Hype: Methods for Deduplication of Conflict Events from News Reports” in Proceedings of International Joint Conference on Artificial Intelligence (Publisher, 2020) pp. 29–34
2020
-
[18]
Silcock, L
E. Silcock, L. D’Amico-Wong, J. Yang, M. Dell, Noise-Robust De-Duplication at scale. National Bureau of Economic Research DOI: 10.3386/w30726 (2022)
2022 doi
-
[19]
On training classifiers for linking event templates
J. Piskorski, F. Saric, V. Zavarella, M. Atkinson, “On training classifiers for linking event templates” in Proceedings of the Workshop Events and Stories in the News 2018 (2018) pp. 68– 78
2018
-
[20]
Event extraction alone is not enough
J. Huang, L. Jiang, C. Moller, R. Uzbeck, “Event extraction alone is not enough” in Text2Story@ ECIR (2024) pp. 105–114
2024
-
[21]
Cox, The regression analysis of binary sequences
D.R. Cox, The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B (Statistical Methodology) 20, 215–232 (1958)
1958
-
[22]
Cortes, V
C. Cortes, V. Vapnik, Support-vector networks. Machine Learning 20, 273–297 (1995)
1995
-
[23]
Haykin, Neural networks: a comprehensive foundation (Prentice Hall PTR, 1994)
S. Haykin, Neural networks: a comprehensive foundation (Prentice Hall PTR, 1994)
1994
-
[24]
Random decision forests
N.T.K. Ho, “Random decision forests” in Proceedings of 3rd International Conference on Document Analysis and Recognition (IEEE, 1995) pp. 278–282. 9
1995
-
[25]
XGBoost: A Scalable Tree Boosting System
T. Chen, C. Guestrin, “XGBoost: A Scalable Tree Boosting System” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016) pp. 785–794
2016
-
[26]
Ley de Víctimas y Restitución de Tirras Agencia Presidencial para la Acción Social y la Cooperación Internacional
Republica de Colombia, “Ley de Víctimas y Restitución de Tirras Agencia Presidencial para la Acción Social y la Cooperación Internacional”. (Centro de Memoria Historica, 2011); www.centrodememoriahistorica.gov.co/descargas/ley_victimas/ley_victimas_completa_web.pdf
2011
-
[27]
El Conflicto en Cifras: Base de Datos, Observatorio de Memoria y Conflicto
“El Conflicto en Cifras: Base de Datos, Observatorio de Memoria y Conflicto”. (Centro Nacional de Memoria Historica, 2024); https://micrositios.centrodememoriahistorica.gov.co/observatorio/portal-de-datos/base-de-datos/
2024
-
[28]
Sistema Integrado de Monitoreo de Cultivos Ilícitos (SIMCI), Monitoreo de los territorios con presencia de cultivos de coca 2022
Oficina de las Naciones Unidas contra la Droga y el Delito (UNODC)-Sistema Integrado de Monitoreo de Cultivos Ilícitos (SIMCI), “Sistema Integrado de Monitoreo de Cultivos Ilícitos (SIMCI), Monitoreo de los territorios con presencia de cultivos de coca 2022” (UNODC-SIMCI, 2023...
2022
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.