REVIEW 3 major objections 5 minor 22 references
Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A two-stage LLM pipeline extracts auditable causal evidence from long humanitarian reports.
desk verdict The extraction pipeline is a genuine contribution, but the headline LoE convergence score is underdetermined because a third of the extracted records are excluded from polarity normalization with no sensitivity analysis. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the two-stage extraction pipeline with query conditioning and snippet grounding. Stage 1 receives a report $x$ and a query $q$ (an intervention class, e.g., cash assistance) and extracts candidate intervention–outcome pairs $a,o$ plus a supporting snippet $z$; Stage 2, given the candidates and snippets, predicts direction $\delta\in\{\mathrm{INC},\mathrm{DEC},\mathrm{NO}\}$ and strength $\sigma\in\{\mathrm{WEAK},\mathrm{MOD},\mathrm{STRONG}\}$, producing records $(a,o,\delta,\sigma,z)$ with $z$ providing auditability. For triangulation, records are placed in cells $c=(d,s)$ by disaster type and source type; strength is mapped to ordinal weights $w(\mathrm{WEAK})=0$, $w(\mathrm{MOD})=1$, $w(\mathrm{STRONG})=2$, and each cell yields smoothed probabilities via Laplace smoothing ($\alpha=0.1$). Cells are averaged equally, and the Level-of-Evidence score is $\mathrm{LoE}=(\max(P_P,P_Z,P_N)-1/3)/(1-1/3)\in[0,1]$, measuring how much the cell-averaged evidence converges on one directional signal.
What would settle it
Have two independent experts re-annotate a random subset of the 100 evaluation reports and compute inter-annotator agreement on direction and strength labels; if agreement is low (e.g., $\kappa<0.6$), the reported 90.73% and 94.15% weighted F1 scores are not trustworthy. Alternatively, recompute the LoE for cash assistance with $\alpha=0$, with alternative polarity assignments for ambiguous outcomes such as food insecurity, and with cells weighted by report count rather than equally; if LoE drops sharply, the 0.865 convergence score is an artifact of the smoothing, polarity labels, or equal-cell-weighting design.
Extended reading notes
Core claim
The central discovery is that the over-extraction problem that plagues naive LLM relation extraction on humanitarian reports can be controlled by query-conditioning the first stage, and that grounding the second-stage relation classification in short supporting snippets improves faithfulness and auditability. The paper claims these design choices, not model size, drive accuracy: across Qwen-Plus, GPT-4o-mini, and DeepSeek-V3, query conditioning consistently yields the largest gains, and the best closed-source configuration reaches 90.73% weighted F1. It further claims that a small open-weight student (Llama-3.1-8B-Instruct) fine-tuned with LoRA on teacher outputs reaches 94.15% weighted F1, exceeding the closed-source teacher, while the distilled model remains two-stage and snippet-grounded. For synthesis, the paper's key claim is that context-preserving triangulation—aggregating within disaster×source cells, smoothing, then equally weighting cells—reveals cross-context convergence rather than pooling raw counts; cash assistance shows strong positive convergence for food-related outcomes (LoE = 0.865, $P_P=0.91$) over 19 non-empty cells, with stable long-horizon trajectories.
Load-bearing premise
Everything hinges on the 100 test reports that experts labeled by hand; the paper gives no measure of whether different experts would label them the same way, so both the F1 scores and the triangulation results could be built on uncertain ground.
Editorial extensions
If this is right
- Query conditioning reduces over-extraction: on Qwen-Plus, weighted F1 rises from 61.44% in the two-step baseline to 89.78% with conditioning, and to 90.73% with snippet grounding added.
- Every extracted relation carries a supporting snippet, so a human reviewer can check the evidence behind each direction and strength label without rereading the full report.
- A small open-weight model fine-tuned on teacher outputs reaches 94.15% weighted F1, which would allow near-free deployment on much larger report corpora than the 8,029 used here.
- The triangulation score is designed so that high-volume contexts do not dominate: equal cell weighting across 19 non-empty disaster×source cells yields $P_P=0.91$ and LoE=0.865 for cash assistance on food outcomes.
- Cumulative temporal trajectories can monitor whether evidence convergence changes over time, e.g., peaking around 2017 and mildly softening in 2018–2024.
Reading between the lines
- The same two-stage pipeline could be pointed at other intervention classes (WASH, shelter, vaccination) and other outcome families; the paper only demonstrates cash assistance, so whether the 90%+ F1 transfers is untested.
- A natural stress test is to vary the design choices the paper fixes—Laplace $\alpha$, strength weights, and the set of polarity labels—and see how much LoE moves; the paper does not report a sensitivity analysis, and LoE=0.865 could be partly an artifact of those choices.
- An independent re-annotation of a subset of the 100 reports with inter-annotator agreement would tell whether the reported F1 gap between closed-source and distilled student is real; the paper reports no such agreement.
- If equal cell weighting is meant to downweight high-volume contexts, it also downweights high-quality evidence from major sources; weighting cells by source reliability or report recency would be a logical alternative the paper leaves unexplored.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops and evaluates a two-stage LLM extraction pipeline for ReliefWeb humanitarian reports, with query-conditioned extraction and snippet-grounded relation classification, and a cell-based triangulation method that computes a Level-of-Evidence (LoE) score. On an expert-annotated sample of 100 reports (220 relations), the best closed-source configuration (Qwen-Plus, two-step QC+SG) achieves a weighted F1 of 90.73%, and LoRA-finetuned Llama-3.1-8B achieves 94.15%. Applied to cash-assistance reports, the triangulation yields LoE=0.865 for food-related outcomes with a positive global probability P_P=0.91.
Significance. The extraction contribution is well positioned, and the evaluation design has real strengths: the benchmark is independently expert-annotated, the 100 evaluation reports are excluded from the distillation data, and the ablations (base vs QC vs QC+SG) are clean. The cost-efficiency comparison and the LoRA distillation are practically useful. If the extraction results survive a robustness analysis of the annotation reliability and the triangulation score is recomputed with a clearly specified treatment of rare outcomes and a sensitivity analysis, the work would be a valuable demonstration of auditable evidence extraction at scale. The triangulation method is transparently specified, but the headline convergence score is not yet robust because the treatment of rare outcomes is unspecified and the aggregation contains choices that can materially change the result; the limitations section acknowledges this but does not quantify it.
major comments (3)
- [Section 3.2, Appendix D, Eqs. (2)–(5)] The triangulation case study labels polarity only for outcome strings with frequency ≥3, which cover 13,554 of 19,568 records (69.27%). The manuscript never states how the remaining 6,014 records (5,080 rare outcome vocabularies, 30.73%) are treated in Eqs. (2)–(5). If they are dropped, LoE=0.865 is computed on a subset of the extracted evidence; if they are included without polarity normalization, positively and negatively framed rare outcomes are not compared on the same scale. Either way, the headline convergence is not a summary of the extracted evidence base, and the stress-test concern is valid. Please specify the inclusion rule for rare outcomes, report cell-level and global results on both the full and frequent-outcome subsets, and add a sensitivity analysis over the frequency threshold and the treatment of unlabeled records.
- [Eqs. (2)–(4)] The strength weighting w(WEAK)=0 means weak evidence never contributes to the cell counts. For example, a cell with 100 WEAK positive records and 1 MOD negative record has C_P=0 and C_N=1, and after Laplace smoothing (α=0.1) the cell probability is P_N=0.846. Since Eq. (4) gives every non-empty cell equal weight, sparse cells of this kind can move the global LoE substantially. The Limitations concede sensitivity to weighting and smoothing, but no sensitivity analysis or per-cell counts are reported. Please report the cell count matrix and vary α and the strength weights, or justify w(WEAK)=0 with an explicit audit-trail rationale that shows the robustness of the LoE=0.865 claim.
- [Section 3, Table 2, Appendix C] The central extraction claim rests on 100 expert-annotated reports containing 220 relations, yet no inter-annotator agreement is reported and no confidence intervals are given for the weighted F1 scores. With only 220 relations, differences such as 90.73 vs 89.17 (Qwen-Plus vs DeepSeek-V3 in the two-step QC+SG condition) may not be statistically distinguishable. The authors should report an agreement measure (e.g., Cohen's kappa or Krippendorff's alpha) and bootstrap or exact binomial confidence intervals for the headline F1 numbers, particularly because the Limitations note that performance estimates may be sensitive to sampling and annotation guidelines.
minor comments (5)
- [Section 4.3] The statement '87 positive vs. 65 negative outcome strings with frequency ≥3' should clarify whether these are unique outcome vocabularies or occurrence counts, since the two readings have different implications for the coverage of the polarity lexicon.
- [Table 2] The weights '0.30/0.30/0.20/0.20' should be mapped explicitly to the four scored components (intervention, outcome, direction, strength) in the table caption, and the partial-credit scheme for strength should be stated there as well.
- [Section 3.2 and Appendix D] The main text says polarity is labeled 'for frequent outcomes, e.g., frequency ≥3', but Appendix D presents this as a fixed design decision; please state clearly in the main text that the reported triangulation uses this threshold.
- [Fig. 2] The figure is cited in Section 4.3 but does not appear in the provided text; ensure the final version includes the figure with labeled axes, a legend, and a panel or annotation showing the number of contributing disaster×source cells over time.
- [Appendix E] Several citations in Appendix E (e.g., composed image/video retrieval works) are only loosely connected to the presented experiments; consider trimming these references or moving them to a clearly separated future-directions discussion to keep the related-work focus.
Circularity Check
No significant circularity: extraction is evaluated on independent expert labels and triangulation is a closed-form descriptive aggregation.
full rationale
The paper's derivation chain is self-contained. The headline extraction numbers are measured against 100 expert-annotated reports (Section 3; Appendix C), and the LoRA student is trained on teacher outputs from a disjoint split with the 100 evaluation reports explicitly excluded (Section 3.1; Appendix F), so the 94.15% weighted-F1 result is not fitted to the evaluation labels. The triangulation score LoE is computed by explicit closed-form equations (2)-(5) from extracted records: strength-weighted cell counts, Laplace smoothing, equal cell averaging, and a normalized maximum-probability transform. No equation inverts or is fitted to the reported convergence; LoE=0.865 is a deterministic function of the cell-averaged PP=0.91, not a prediction validated against a target. The only self-citation (Zhang et al., 2026c) appears in Related Work as context and is not load-bearing. The acknowledged limitations about annotation-set size and sensitivity to polarity normalization, weighting, and smoothing are robustness and validity risks rather than circularity: Appendix D's decision to label only frequent outcomes is an explicit preprocessing choice, and its potential effect on the headline LoE is a measurement concern, not a self-referential derivation. Overall, the extraction claims are externally grounded and the aggregation is a transparent summary statistic.
Assumptions & free parameters
free parameters (5)
- Laplace smoothing alpha =
0.1
- Strength weights w(sigma) =
WEAK=0, MOD=1, STRONG=2
- Weighted-F1 component weights =
0.30, 0.30, 0.20, 0.20
- BERTScore threshold =
0.8
- Polarity labeling frequency threshold =
frequency >= 3
assumptions (5)
- domain assumption The 100-report, 220-relation expert-annotated set is an accurate and consistent ground truth for intervention-outcome relations.
- domain assumption LLM-extracted direction and strength labels correspond to causal evidence in the reports.
- domain assumption The filtered corpus (four disaster types, five source types, English-only, keyword 'cash assistance') is representative enough for cross-context triangulation.
- ad hoc to paper Laplace smoothing with alpha=0.1 and equal cell weighting is a valid aggregation for the Level-of-Evidence score.
- domain assumption Manual polarity labels for outcome terms with frequency >=3 correctly map outcome framing to direction.
invented entities (1)
-
Level-of-Evidence (LoE) score
Cite this review
Pith. "Pith review of Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study." pith.science (2026). https://pith.science/paper/XK4YWFZR
@misc{pith2026260804576,
author = {Pith},
title = {Pith review of: Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/XK4YWFZR}},
note = {Machine review of arXiv:2608.04576}
}
abstract
Humanitarian reports are long, noisy, and multi-topic, making it difficult to consolidate decision-relevant causal evidence. We present a ReliefWeb study (2000-2024) and a two-stage Large Language Model (LLM) pipeline that extracts structured intervention-outcome records with direction and strength attributes. Query-conditioned extraction restricts output to a specified intervention class, reducing retrieval-induced over-extraction, while snippet grounding links each relation to supporting text for auditability and classification. In an expert-annotated dataset of 100 reports, the best closed-source LLM achieved a weighted F1 score of 90.73% with strong cost-efficiency, while Llama-3.1-8B with supervised fine-tuning reached 94.15% weighted F1 score. We further propose context-preserving triangulation that aggregates strength-weighted evidence within disaster$\times$source cells, applies Laplace smoothing and equally weights cells to quantify cross-context convergence via a Level-of-Evidence score. Applied to cash assistance, food-related outcomes show strong positive convergence (LoE=0.865) and stable long-horizon trajectories.
Figures
Reference graph
Works this paper leans on
-
[6]
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang
The llama 3 herd of models.Preprint, arXiv:2407.21783. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang
- [8]
-
[9]
Faithful-First Reasoning, Planning, and Acting for Multimodal LLMs
Faithact: Faithfulness planning and acting in mllms. arXiv preprint arXiv:2511.08409. Zixu Li, Yupeng Hu, Zhiwei Chen, Qinlei Huang, Guozhi Qiu, Zhiheng Fu, and Meng Liu. 2026a. Retrack: Evidence-driven dual-stream directional anchor calibration network for composed video re- trieval.Proceedings of the AAAI Conference on Arti- ficial Intelligence, 40(28):...
-
[10]
Peiyang Liu, Ziqiang Cui, Di Liang, and Wei Ye
Mmfinereason: Closing the multimodal reasoning gap via open data-centric methods.Preprint, arXiv:2601.21821. Peiyang Liu, Ziqiang Cui, Di Liang, and Wei Ye. 2025a. Who stole your data? a method for detecting unau- thorized rag theft.Preprint, arXiv:2510.07728. Peiyang Liu, Xi Wang, Ziqiang Cui, and Wei Ye. 2025b. Queries are not alone: Clustering text emb...
-
[11]
Retrieval-based unsu- pervised noisy label detection on text data. InPro- ceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, page 4099–4104, New York, NY , USA. Associa- tion for Computing Machinery. Zheng Liu, Hao Liang, Bozhou Li, Wentao Xiong, Chong Chen, Conghui He, Wentao Zhang, and Bin Cui. 2025c. Sy...
-
[12]
Yunbo Long, Yuhan Liu, Liming Xu, and Alexandra Brintrup
Emo- mas: Emotion-aware multi-agent system for high- stakes edge-deployable negotiation with bayesian or- chestration.Preprint, arXiv:2604.07003. Yunbo Long, Yuhan Liu, Liming Xu, and Alexandra Brintrup
-
[13]
Kexin Ma, Ruochun Jin, Wang Haotian, Wang Xi, Huan Chen, Yuhua Tang, and Qian Wang
Emodebt: Bayesian-optimized emo- tional intelligence for strategic agent-to-agent debt recovery.Preprint, arXiv:2503.21080. Kexin Ma, Ruochun Jin, Wang Haotian, Wang Xi, Huan Chen, Yuhua Tang, and Qian Wang
-
[14]
Context- driven index trimming: A data quality perspective to enhancing precision of RALMs. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 4886–4901, Miami, Florida, USA. Asso- ciation for Computational Linguistics. Xiaoxu Ma, Xiangbo Zhang, and Zhenyu Weng
work page 2024
Show all 22 references
-
[15]
Preprint, arXiv:2601.09833
Stable and explainable personality trait evaluation in large language models with internal activations. Preprint, arXiv:2601.09833. Potsawee Manakul, Adian Liusie, and Mark Gales
-
[16]
InProceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 9004–9017, Singapore
SelfCheckGPT: Zero-resource black-box hallucina- tion detection for generative large language models. InProceedings of the 2023 Conference on Empiri- cal Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computa- tional Linguistics. Cheng Niu,...
2023
-
[17]
https://apidoc.reliefweb.int/
Reliefweb api documentation. https://apidoc.reliefweb.int/. Service pro- vided by UN OCHA; accessed 2025-12-20. Roberta Rocca, Nicolò Tamagnone, Selim Fekih, Xi- mena Contla, and Navid Rekabsaz
2025
-
[18]
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun
Stable: Effi- cient hybrid nearest neighbor search via magnitude- uniformity and cardinality-robustness.Preprint, arXiv:2604.01617. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun
-
[19]
InThe Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11,
Making retrieval-augmented language models robust to irrelevant context. InThe Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11,
2024
-
[20]
Qianchi Zhang, Hainan Zhang, Liang Pang, Yongxin Tong, Hongwei Zheng, and Zhiming Zheng
OpenReview.net. Qianchi Zhang, Hainan Zhang, Liang Pang, Yongxin Tong, Hongwei Zheng, and Zhiming Zheng. 2026a. Less is more: Compact clue selection for efficient retrieval-augmented generation reasoning. InPro- ceedings of the ACM Web Conference 2026, WWW ’26, page 1971–1982,...
2026
-
[22]
cash assistance
OpenRe- view.net. Wenyuan Zhang, Xinghua Zhang, Haiyang Yu, Shuaiyi Nie, Bingli Wu, Juwei Yue, Tingwen Liu, and Yong- bin Li. 2026b. Expseek: Self-triggered experience seeking for web agents.Preprint, arXiv:2601.08605. Yuanjun Zhang, Fuzel Ahamed Shaik, Suvojit Acharjee, Fahad...
2000 arXiv
-
[2015]
Enhanced Response Capacity Project 2014–2015
Operational guidance and toolkit for multipurpose cash grants. Enhanced Response Capacity Project 2014–2015. Zhiwei Chen, Yupeng Hu, Zhiheng Fu, Zixu Li, Jiale Huang, Qinlei Huang, and Yinwei Wei
2014
-
[2020]
In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,
Bertscore: Evalu- ating text generation with BERT. In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,
2020
-
[2022]
InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,
Lora: Low-rank adaptation of large language models. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,
2022
-
[2023]
Qwen technical report.Preprint, arXiv:2309.16609. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, R...
-
[2024]
InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,
Self-rag: Learning to retrieve, generate, and critique through self-reflection. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,
2024
-
[2025]
Preprint, arXiv:2412.19437
Deepseek-v3 technical report. Preprint, arXiv:2412.19437. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace
-
[2026]
Emsdialog: Synthetic multi- person emergency medical service dialogue genera- tion from electronic patient care reports via multi-llm agents.Preprint, arXiv:2604.07549. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.