{"id":"932cf3f6-c4b3-4a6a-bedb-896e08597951","arxiv_id":"2608.04576","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A query-conditioned, snippet-grounded LLM pipeline extracts causal evidence from ReliefWeb reports and triangulates it into a Level-of-Evidence score, reporting high F1 and strong positive convergence for cash assistance on food outcomes.","lead":"This paper builds a two-stage large language model pipeline that pulls structured intervention-outcome claims from humanitarian reports, then aggregates them across disaster and source types into a level-of-evidence score. In a cash assistance case study, food-related outcomes show strong positive convergence with an evidence score of 0.865.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LoE=0.865 rests on a triangulation that appears to drop or fail to normalize the 30.7% of extracted records with rare outcome strings; re-running with all outcomes labeled and varied smoothing/cell thresholds could change the headline convergence.","rationale":"The reader's weakest assumption (annotation reliability) is a legitimate concern for the extraction F1 numbers, and I do not dispute it; our partial agreement reflects that the reader's rationale also flags triangulation parameter sensitivity, but not the same mechanism. In my reading, the more decisive vulnerability is in the triangulation claim itself, which is a headline result. The method as described either drops or fails to normalize the 30.7% of extracted records with rare outcome strings, and the strength weighting with w(WEAK)=0 combined with equal cell weighting and Laplace smoothing lets a small number of moderate/strong records in sparse cells dominate the average. The manuscript's own Limitations section acknowledges that the aggregation depends on design choices, yet no sensitivity analysis is provided. This does not invalidate the extraction pipeline or the overall evaluation design; it means the evidence-synthesis contribution is not yet established as claimed. The missing piece is a re-analysis, not a new method, so the appropriate verdict remains CONDITIONAL: accept only if the robustness check supports the headline LoE, or soften the claim accordingly. I therefore keep the verdict unchanged.","tokens_in":13311,"tokens_out":12072,"duration_ms":132261,"concrete_test":"Recompute Eqs. (2)–(5) on all extracted food-outcome records and report PP and LoE under three perturbations: (a) polarity-label all 5,080 rare outcome vocabularies (LLM-judged or manually curated) instead of only the frequency>=3 subset; (b) vary alpha over {0.01, 0.05, 0.1, 0.5} and the minimum per-cell count over {1, 5, 20}; (c) replace equal cell weighting with raw pooled strength-weighted counts. If PP and LoE stay above 0.8 under every variant, the convergence claim is robust; if PP drops below about 0.7 or the dominant direction changes, LoE=0.865 is an artifact of the frequent-outcome subset and smoothing choices, and the triangulation conclusion should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2's LoE computation (Eqs. 2–5) grounds the headline 'strong positive convergence (LoE=0.865)' for cash assistance on food outcomes. Appendix D restricts polarity normalization to outcome strings occurring at least 3 times; these cover 13,554 of 19,568 records (69.27%). The remaining 6,014 records (5,080 rare outcome vocabularies) are not polarity-labeled, and the manuscript does not state how they enter the P/Z/N aggregation: if they are dropped, the LoE describes only the frequent-outcome subset; if included unnormalized, negatively framed rare outcomes would be inverted relative to the positive-improvement convention. Either way, the score is not a summary of the extracted evidence base. Aggregation compounds this: w(WEAK)=0 in Eq. (2), so weak evidence never contributes; a cell with 100 WEAK positive records and one MOD negative record is scored as 0.846 negative after Laplace smoothing (alpha=0.1), and equal cell weighting lets such sparse cells move the global LoE. The Limitations paragraph concedes sensitivity to polarity normalization, weighting, and smoothing, but no sensitivity analysis or per-cell counts are reported. The convergence claim is therefore underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops and evaluates a two-stage LLM extraction pipeline for ReliefWeb humanitarian reports, with query-conditioned extraction and snippet-grounded relation classification, and a cell-based triangulation method that computes a Level-of-Evidence (LoE) score. On an expert-annotated sample of 100 reports (220 relations), the best closed-source configuration (Qwen-Plus, two-step QC+SG) achieves a weighted F1 of 90.73%, and LoRA-finetuned Llama-3.1-8B achieves 94.15%. Applied to cash-assistance reports, the triangulation yields LoE=0.865 for food-related outcomes with a positive global probability P_P=0.91.","tokens_in":13587,"tokens_out":7686,"duration_ms":77931,"significance":"The extraction contribution is well positioned, and the evaluation design has real strengths: the benchmark is independently expert-annotated, the 100 evaluation reports are excluded from the distillation data, and the ablations (base vs QC vs QC+SG) are clean. The cost-efficiency comparison and the LoRA distillation are practically useful. If the extraction results survive a robustness analysis of the annotation reliability and the triangulation score is recomputed with a clearly specified treatment of rare outcomes and a sensitivity analysis, the work would be a valuable demonstration of auditable evidence extraction at scale. The triangulation method is transparently specified, but the headline convergence score is not yet robust because the treatment of rare outcomes is unspecified and the aggregation contains choices that can materially change the result; the limitations section acknowledges this but does not quantify it.","major_comments":[{"comment":"The triangulation case study labels polarity only for outcome strings with frequency ≥3, which cover 13,554 of 19,568 records (69.27%). The manuscript never states how the remaining 6,014 records (5,080 rare outcome vocabularies, 30.73%) are treated in Eqs. (2)–(5). If they are dropped, LoE=0.865 is computed on a subset of the extracted evidence; if they are included without polarity normalization, positively and negatively framed rare outcomes are not compared on the same scale. Either way, the headline convergence is not a summary of the extracted evidence base, and the stress-test concern is valid. Please specify the inclusion rule for rare outcomes, report cell-level and global results on both the full and frequent-outcome subsets, and add a sensitivity analysis over the frequency threshold and the treatment of unlabeled records.","section":"Section 3.2, Appendix D, Eqs. (2)–(5)"},{"comment":"The strength weighting w(WEAK)=0 means weak evidence never contributes to the cell counts. For example, a cell with 100 WEAK positive records and 1 MOD negative record has C_P=0 and C_N=1, and after Laplace smoothing (α=0.1) the cell probability is P_N=0.846. Since Eq. (4) gives every non-empty cell equal weight, sparse cells of this kind can move the global LoE substantially. The Limitations concede sensitivity to weighting and smoothing, but no sensitivity analysis or per-cell counts are reported. Please report the cell count matrix and vary α and the strength weights, or justify w(WEAK)=0 with an explicit audit-trail rationale that shows the robustness of the LoE=0.865 claim.","section":"Eqs. (2)–(4)"},{"comment":"The central extraction claim rests on 100 expert-annotated reports containing 220 relations, yet no inter-annotator agreement is reported and no confidence intervals are given for the weighted F1 scores. With only 220 relations, differences such as 90.73 vs 89.17 (Qwen-Plus vs DeepSeek-V3 in the two-step QC+SG condition) may not be statistically distinguishable. The authors should report an agreement measure (e.g., Cohen's kappa or Krippendorff's alpha) and bootstrap or exact binomial confidence intervals for the headline F1 numbers, particularly because the Limitations note that performance estimates may be sensitive to sampling and annotation guidelines.","section":"Section 3, Table 2, Appendix C"}],"minor_comments":[{"comment":"The statement '87 positive vs. 65 negative outcome strings with frequency ≥3' should clarify whether these are unique outcome vocabularies or occurrence counts, since the two readings have different implications for the coverage of the polarity lexicon.","section":"Section 4.3"},{"comment":"The weights '0.30/0.30/0.20/0.20' should be mapped explicitly to the four scored components (intervention, outcome, direction, strength) in the table caption, and the partial-credit scheme for strength should be stated there as well.","section":"Table 2"},{"comment":"The main text says polarity is labeled 'for frequent outcomes, e.g., frequency ≥3', but Appendix D presents this as a fixed design decision; please state clearly in the main text that the reported triangulation uses this threshold.","section":"Section 3.2 and Appendix D"},{"comment":"The figure is cited in Section 4.3 but does not appear in the provided text; ensure the final version includes the figure with labeled axes, a legend, and a panel or annotation showing the number of contributing disaster×source cells over time.","section":"Fig. 2"},{"comment":"Several citations in Appendix E (e.g., composed image/video retrieval works) are only loosely connected to the presented experiments; consider trimming these references or moving them to a clearly separated future-directions discussion to keep the related-work focus.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a competent applied NLP paper with a transparent extraction pipeline and independent evaluation, but the triangulation headline needs a sensitivity analysis and an explicit treatment of rare outcomes before the convergence claim can be accepted. The evaluation set is small, so the reported F1 differences should be accompanied by uncertainty estimates. The paper fits the journal scope; with these fixes, a revised version would be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the extraction half of this paper is genuinely useful — query-conditioned two-stage prompting with snippet grounding is a sensible way to tame over-extraction on long, noisy humanitarian reports, and the independent expert-annotated evaluation, small as it is, supports the main F1 claims. The triangulation half, and specifically the headline LoE=0.865, does not hold up as stated. Polarity is labeled only for outcome strings appearing at least 3 times, covering 69.27% of the 19,568 records; the manuscript never says what happens to the remaining 30.73%. Dropped, the LoE describes only a frequent-outcome subset; included without normalization, negatively framed rare outcomes would be inverted. Either way, the convergence score is not a summary of the extracted evidence. The stress-test example is on point: with w(WEAK)=0, a cell with 100 weak positive records and one moderate negative record scores 0.846 negative after Laplace smoothing, and equal cell weighting lets sparse cells move the global LoE. The Limitations concede sensitivity to these choices, but no sensitivity analysis or per-cell counts are reported, so the convergence claim is underdetermined.\n\nCredit where due: the two-stage query-conditioned + snippet-grounded pipeline is a clean combination, and the ablations support it — QC gives large gains, SG improves faithfulness. The LoRA distillation is careful (disjoint eval split, cross-family student). The evaluation uses an independently annotated set, which is the right design; the absence of inter-annotator agreement and confidence intervals is a real but fixable gap. No code or data are released, which will hamper reproducibility.\n\nWho this is for: NLP researchers working on evidence extraction from messy text, and humanitarian analysts interested in semi-automated evidence synthesis. The extraction contribution deserves a serious referee. The triangulation section needs substantial revision: define how rare outcomes enter the aggregation, report per-cell counts, and run a sensitivity analysis over alpha, the strength weights, and the frequency threshold. As is, I would not trust the LoE=0.865 headline.","headline":"The extraction pipeline is a genuine contribution, but the headline LoE convergence score is underdetermined because a third of the extracted records are excluded from polarity normalization with no sensitivity analysis.","tokens_in":14110,"tokens_out":3608,"would_cite":false,"duration_ms":35690,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage LLM pipeline extracts auditable causal evidence from long humanitarian reports.","keywords":["causal evidence extraction","humanitarian reports","ReliefWeb","large language models","query-conditioned extraction","snippet grounding","evidence triangulation","cash assistance"],"falsifier":"Have two independent experts re-annotate a random subset of the 100 evaluation reports and compute inter-annotator agreement on direction and strength labels; if agreement is low (e.g., $\\kappa<0.6$), the reported 90.73% and 94.15% weighted F1 scores are not trustworthy. Alternatively, recompute the LoE for cash assistance with $\\alpha=0$, with alternative polarity assignments for ambiguous outcomes such as food insecurity, and with cells weighted by report count rather than equally; if LoE drops sharply, the 0.865 convergence score is an artifact of the smoothing, polarity labels, or equal-cell-weighting design.","tokens_in":13086,"feed_emoji":"📄","tokens_out":9970,"duration_ms":86913,"temperature":0.7,"pith_summary":"This paper claims that long, noisy, multi-topic humanitarian reports can be turned into structured, auditable causal evidence at scale. The proposed two-stage LLM pipeline first extracts candidate intervention–outcome pairs restricted to a query-defined intervention class, then classifies each pair's direction (increase, no change, decrease) and strength (weak, moderate, strong) using the supporting text snippet that grounds the claim. On 100 expert-annotated ReliefWeb reports covering 220 relations, the paper reports weighted F1 of 90.73% for the best closed-source model (Qwen-Plus with two-step query-conditioned, snippet-grounded extraction) and 94.15% for a LoRA fine-tuned Llama-3.1-8B student distilled from it. To synthesize across contexts, the paper introduces a cell-based triangulation score that aggregates strength-weighted evidence within disaster×source cells, applies Laplace smoothing, and averages cells equally to produce a Level-of-Evidence score; applied to cash assistance and food-related outcomes, it yields LoE = 0.865. If correct, this gives humanitarian decision-makers a scalable way to consolidate what works where from thousands of field reports, with every extracted relation traceable to a quoted snippet.","feed_headline":"Fine-tuned 8B LLM extracts crisis evidence at 94% F1","feed_subtitle":"Two-stage query-conditioned, snippet-grounded pipeline also finds strong convergence for cash assistance on food outcomes.","key_machinery":"The load-bearing mechanism is the two-stage extraction pipeline with query conditioning and snippet grounding. Stage 1 receives a report $x$ and a query $q$ (an intervention class, e.g., cash assistance) and extracts candidate intervention–outcome pairs $a,o$ plus a supporting snippet $z$; Stage 2, given the candidates and snippets, predicts direction $\\delta\\in\\{\\mathrm{INC},\\mathrm{DEC},\\mathrm{NO}\\}$ and strength $\\sigma\\in\\{\\mathrm{WEAK},\\mathrm{MOD},\\mathrm{STRONG}\\}$, producing records $(a,o,\\delta,\\sigma,z)$ with $z$ providing auditability. For triangulation, records are placed in cells $c=(d,s)$ by disaster type and source type; strength is mapped to ordinal weights $w(\\mathrm{WEAK})=0$, $w(\\mathrm{MOD})=1$, $w(\\mathrm{STRONG})=2$, and each cell yields smoothed probabilities via Laplace smoothing ($\\alpha=0.1$). Cells are averaged equally, and the Level-of-Evidence score is $\\mathrm{LoE}=(\\max(P_P,P_Z,P_N)-1/3)/(1-1/3)\\in[0,1]$, measuring how much the cell-averaged evidence converges on one directional signal.","core_discovery":"The central discovery is that the over-extraction problem that plagues naive LLM relation extraction on humanitarian reports can be controlled by query-conditioning the first stage, and that grounding the second-stage relation classification in short supporting snippets improves faithfulness and auditability. The paper claims these design choices, not model size, drive accuracy: across Qwen-Plus, GPT-4o-mini, and DeepSeek-V3, query conditioning consistently yields the largest gains, and the best closed-source configuration reaches 90.73% weighted F1. It further claims that a small open-weight student (Llama-3.1-8B-Instruct) fine-tuned with LoRA on teacher outputs reaches 94.15% weighted F1, exceeding the closed-source teacher, while the distilled model remains two-stage and snippet-grounded. For synthesis, the paper's key claim is that context-preserving triangulation—aggregating within disaster×source cells, smoothing, then equally weighting cells—reveals cross-context convergence rather than pooling raw counts; cash assistance shows strong positive convergence for food-related outcomes (LoE = 0.865, $P_P=0.91$) over 19 non-empty cells, with stable long-horizon trajectories.","pith_inferences":["The same two-stage pipeline could be pointed at other intervention classes (WASH, shelter, vaccination) and other outcome families; the paper only demonstrates cash assistance, so whether the 90%+ F1 transfers is untested.","A natural stress test is to vary the design choices the paper fixes—Laplace $\\alpha$, strength weights, and the set of polarity labels—and see how much LoE moves; the paper does not report a sensitivity analysis, and LoE=0.865 could be partly an artifact of those choices.","An independent re-annotation of a subset of the 100 reports with inter-annotator agreement would tell whether the reported F1 gap between closed-source and distilled student is real; the paper reports no such agreement.","If equal cell weighting is meant to downweight high-volume contexts, it also downweights high-quality evidence from major sources; weighting cells by source reliability or report recency would be a logical alternative the paper leaves unexplored."],"forward_implications":["Query conditioning reduces over-extraction: on Qwen-Plus, weighted F1 rises from 61.44% in the two-step baseline to 89.78% with conditioning, and to 90.73% with snippet grounding added.","Every extracted relation carries a supporting snippet, so a human reviewer can check the evidence behind each direction and strength label without rereading the full report.","A small open-weight model fine-tuned on teacher outputs reaches 94.15% weighted F1, which would allow near-free deployment on much larger report corpora than the 8,029 used here.","The triangulation score is designed so that high-volume contexts do not dominate: equal cell weighting across 19 non-empty disaster×source cells yields $P_P=0.91$ and LoE=0.865 for cash assistance on food outcomes.","Cumulative temporal trajectories can monitor whether evidence convergence changes over time, e.g., peaking around 2017 and mildly softening in 2018–2024."],"supporting_citations":[{"why":"Supplies the corpus and the disaster-type and source-type metadata used to define triangulation cells.","marker":"ReliefWeb, 2025"},{"why":"Provides the baseline extraction strategy and medical-domain comparison used in the ablations.","marker":"Shi et al., 2025"},{"why":"Provides BERTScore, threshold 0.8, used for semantic set matching in evaluation.","marker":"Zhang et al., 2020"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method used to build the distilled student.","marker":"Hu et al., 2022"},{"why":"Provides the open-weight Llama-3.1-8B-Instruct model used as the student.","marker":"Grattafiori et al., 2024"},{"why":"Cites triangulation as the methodological basis for aggregating evidence across contexts.","marker":"Jick, 1979"},{"why":"Systematic review of cash-based humanitarian approaches that motivates focusing on cash assistance.","marker":"Doocy and Tappis, 2017"},{"why":"Supports the auditability requirement that leads to snippet grounding.","marker":"DeYoung et al., 2020"}],"fun_headline_variants":["Fine-tuned 8B LLM hits 94% F1 extracting crisis evidence","Query-conditioned LLM pipeline extracts crisis evidence at 94% F1","Small open LLM beats big models on crisis evidence at 94% F1","Distilled 8B LLM extracts crisis evidence at 94% F1","Two-stage snippet-grounded LLM pipeline achieves 94% F1 on crisis evidence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything hinges on the 100 test reports that experts labeled by hand; the paper gives no measure of whether different experts would label them the same way, so both the F1 scores and the triangulation results could be built on uncertain ground.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned 8B LLM hits 94% F1 extracting crisis evidence","Query-conditioned LLM pipeline extracts crisis evidence at 94% F1","Small open LLM beats big models on crisis evidence at 94% F1","Distilled 8B LLM extracts crisis evidence at 94% F1","Two-stage snippet-grounded LLM pipeline achieves 94% F1 on crisis evidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001043,"raw_usage":{"total_tokens":4413,"prompt_tokens":1003,"completion_tokens":3410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":3303}},"tokens_in":619,"tokens_out":3410,"duration_ms":23316,"temperature":1.0,"reasoning_tokens":3303,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:35:33.618692+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent experts re-annotate a random subset of the 100 evaluation reports and compute inter-annotator agreement on direction and strength labels; if agreement is low (e.g., $\\kappa<0.6$), the reported 90.73% and 94.15% weighted F1 scores are not trustworthy. Alternatively, recompute the LoE for cash assistance with $\\alpha=0$, with alternative polarity assignments for ambiguous outcomes such as food insecurity, and with cells weighted by report count rather than equally; if LoE drops sharply, the 0.865 convergence score is an artifact of the smoothing, polarity labels, or equal-cell-weighting design.","supporting_citations":[{"cited_title":"https://apidoc.reliefweb.int/","cited_arxiv_id":null,"evidence_quote":"Supplies the corpus and the disaster-type and source-type metadata used to define triangulation cells."}],"review_version":1}