REVIEW 4 major objections 6 minor 37 references
Detecting Actionable Requests and Offers on Social Media During Crises Using LLMs
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims that query-specific few-shot retrieval—pulling class-relevant labeled examples from an embedding database into the prompt—improves multi-label classification of crisis requests and offers, and of actionability, over…
desk verdict Useful taxonomy, plausible dynamic few-shot method, but the headline QSF gains need a leakage check before I'd trust them. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is QSF Learning, a retrieval-augmented few-shot prompting loop: for each input post, an embedding is computed with OpenAI's text-embedding-3-small model; the $k/2$ most similar labeled examples are retrieved by cosine similarity from a training-set embedding database, $k/2$ random examples are added for variability, and the combined examples are inserted into the prompt before the model outputs a structured JSON tuple. The other load-bearing component is the taxonomy itself: 1,093 elements organized into three branches (supplies, actions, emergency personnel), with 11 labels at depth 2 per branch defining the classification target space. Together they let the model adapt its label choices to each query without any fine-tuning.
What would settle it
Take a freshly collected, independently annotated crisis tweet set from a disaster not among those used to generate the synthetic data, run the QSF pipeline against the best static baseline on Actions and Personnel, and report inter-annotator agreement on the labels; if the QSF advantage over BL5 shrinks to a few points or disappears, the paper's central claim that retrieval-specific examples drive the gains is not supported.
Extended reading notes
Core claim
The paper claims that dynamically retrieving labeled examples—specifically, the $k/2$ nearest labeled posts by embedding similarity plus $k/2$ randomly chosen labeled posts—and appending them to an LLM prompt consistently outperforms static baselines (BL1–BL5) for multi-label, multi-class classification of requests and offers into supplies, actions, and personnel, and for predicting a binary actionability flag. The strongest evidence is on Actions, where F1 rises from 0.62 to 0.77 for Mistral on synthetic data and from 0.42 to 0.57 for Llama 3 on real-world data, and on Personnel, where Llama 3.1 rises from 0.27 to 0.47 on real-world data. The paper also reports that static prompting plateaus quickly, and that QSF Learning narrows the gap between smaller open models and a stronger paid model.
Load-bearing premise
The load-bearing premise is that the labeled test data—GPT-4o-generated synthetic tweets with light human correction and a 300-tweet manually annotated Hurricane Sandy sample—are trustworthy proxies for real crisis streams; if synthetic posts are systematically easier or more stereotyped than authentic tweets, the measured QSF improvements may not transfer to live disaster response.
Editorial extensions
If this is right
- If the central claim holds, disaster-response teams can deploy a classifier for a new crisis by reusing labeled examples from past events, without collecting new training data or fine-tuning a model.
- The largest predicted gains are on Actions and Personnel, the categories that previous request-and-offer systems handled least well, so operational triage could surface more rescue, medical, and debris-clearance posts.
- Retrieval-based prompting may reduce the performance gap between small open models and large proprietary ones, making the approach feasible when API access is limited or costly.
- The mixed actionability results mean QSF Learning is not uniformly better on the binary actionability decision; its benefit concentrates on granular multi-label tasks, so a practical system would still need to decide which task to optimize.
- Because examples are drawn only from the training split, the reported improvements are not an artifact of test-set leakage, assuming the retrieval database is assembled correctly.
Reading between the lines
- A testable extension the paper leaves implicit is varying the ratio of nearest to random examples: if the random half stabilizes the prompt and prevents class bias, changing the ratio should shift the precision-recall tradeoff, but the paper reports no such ablation.
- The taxonomy's imbalance—946 supply elements versus 90 actions and 57 personnel—suggests that the lower F1 on actions and personnel may partly reflect sparser label coverage rather than intrinsic difficulty; a balanced taxonomy experiment could isolate that.
- If these gains carry over to live streams, a practical system could rank incoming posts by predicted actionability and auto-generate structured requests for supplies, personnel, and actions, feeding directly into humanitarian coordination dashboards.
- Because the synthetic test data was itself generated by an LLM, a stronger validation would compare QSF Learning on independently sourced, professionally annotated crisis corpora across multiple languages and disaster types.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper targets fine-grained identification of requests and offers in crisis-related social media. It introduces a top-down taxonomy with three branches (supplies, actions, emergency personnel; 1,093 elements) and frames the task as multi-label classification over 11 labels per branch plus a binary actionability judgment. The main methodological contribution is Query-Specific Few-shot Learning (QSF): for each input, k/2 nearest labeled examples are retrieved from an embedding database built with text-embedding-3-small, combined with k/2 random examples, and appended to the prompt. The authors compare QSF with five static prompting baselines (BL1-BL5) using Llama 3, Llama 3.1, Gemma 2, Mistral 7B, and GPT-4o mini on 673 synthetic GPT-4o-generated tweets and 200 manually re-annotated Hurricane Sandy tweets. The paper reports micro-F1 for the multi-label tasks and macro-F1 for actionability, and concludes that QSF outperforms the baselines, with the largest gains on Actions and Personnel.
Significance. If the central claim were cleanly established, this would be a useful contribution: a reusable fine-grained taxonomy, a low-annotation retrieval-augmented prompting method, and an evaluation across several open LLMs. The multi-label framing and public release of the dataset and taxonomy are positive features, and the limitations section is candid about the synthetic-data and small-real-data risks. However, the current evidence is conditional: the synthetic split permits a plausible retrieval-leakage path, the real-world test set is small and lacks reliability measures, and the actionability results are mixed. These issues are fixable with additional analyses, so the work is promising but not yet conclusive.
major comments (4)
- [Synthetic Data Generation / Evaluation metrics] The reported QSF gains may be inflated by retrieval leakage. The synthetic corpus is deduplicated at a cosine-similarity threshold of 0.925 using text-embedding-3-small and then randomly split 50/50 into training and evaluation sets. QSF retrieves the k/2 nearest training examples per test message in the same embedding space, so any test post that is a near-duplicate of a training post below the 0.925 cutoff can have its labels copied into the prompt, whereas the static few-shot baselines cannot exploit query-specific proximity. The manuscript does not report the train/test nearest-neighbor similarity distribution, does not use a batch-disjoint or cluster-disjoint split, and does not state k. Because the largest gains are on the sparse Actions and Personnel tasks, this concern is load-bearing for the central claim. Please add a similarity analysis and an ablation that removes near-duplicate test posts or performs cluster-disjoint splitting.
- [Real-World Data / Tables 3-4] The real-world evaluation cannot independently support the generalization claim. The test set is 200 Hurricane Sandy tweets re-annotated by the authors, with no inter-annotator agreement reported and no description of adjudication. The text is internally inconsistent about the sample size (300 sampled tweets vs 107 training + 200 test = 307 labeled posts). With n=200 and sparse labels (e.g., Personnel F1 near 0.2-0.3 in baselines), the reported QSF improvements may rest on very few instances; no confidence intervals, bootstraps, or significance tests are given. Please report IAA, reconcile the counts, and supply interval estimates or per-class support.
- [Results and Discussion / Tables 3-4] The claim that QSF outperforms baseline prompting strategies is too broad for the actionability task, which is part of the abstract and conclusion. On synthetic data, Gemma 2 QSF scores 0.49 vs BL5 0.78, and Mistral QSF 0.67 vs BL5 0.72; on real data, Llama 3 QSF 0.47 vs BL5 0.45 is essentially tied. The text acknowledges mixed results but the overall claim is not qualified accordingly. Please either scope the central claim to the multi-label taxonomy tasks or provide evidence for a consistent actionability benefit.
- [Synthetic Data Generation] The synthetic ground truth is generated by GPT-4o and only 40% of it was human-reviewed, with fewer than 10% corrections; the evaluator GPT-4o mini is from the same model family. This raises the risk that the synthetic benchmark is systematically easier or more stereotyped than real streams, and that QSF's retrieval of nearest neighbors propagates generation artifacts rather than learning generalizable structure. Please report the per-class correction counts, the label-noise distribution, and at least one analysis that compares QSF on the synthetic labels with QSF on the human-corrected subset (or on real labels) to separate retrieval benefit from data-generation bias.
minor comments (6)
- [Real-World Data] Reconcile the 300 vs 307 discrepancy in the number of annotated tweets.
- [QSF Learning] The value of k is never stated; please report k and any sensitivity analysis over it.
- [Figure 6] The caption/text mentions BL 3 through 6 although only BL3-BL5 and QSF exist; correct the label.
- [References] Reference [22] contains a stray 'asdfdfsdfas' string; clean up the bibliography entry.
- [Tables 3-4] For GPT-4o mini only QSF scores are shown; add a note that no baseline comparison was run, since the text otherwise implies all models were tested on all prompts.
- [Task Definition] At depth 2 the paper says 11 labels per set while the taxonomy has 33 Level-2 elements; clarify that these are per-branch depth-2 labels.
Circularity Check
No significant circularity: QSF Learning's reported gains come from direct F1 comparisons against static baselines, not from a derivation that reduces to its own inputs or from a load-bearing self-citation.
full rationale
This paper is an empirical LLM prompting study, not a formal derivation. It builds a taxonomy, creates synthetic crisis tweets, and compares Query-Specific Few-shot Learning (QSF) against five static prompts (BL1-BL5) using micro/macro F1 on held-out splits. No equation defines the QSF output in terms of the evaluation labels, and no parameter is fitted to the test set: the training split supplies only in-context exemplars for both the static few-shot baselines and QSF, and the reported gains (e.g., Actions Mistral 0.62 to 0.77 on synthetic data; Personnel Llama 3.1 0.27 to 0.47 on real data) are empirical table values rather than quantities forced by construction. The paper does cite prior work by the same authors in several places: [13] motivates class-specific shot selection, [17]/[25] supply the Hurricane Sandy corpus, and [33] grounds the actionability definition. None of these citations is used to prove the QSF-versus-baseline result; the experiments are self-contained against external LLMs and an existing crisis-tweet dataset, so no prediction reduces to a self-citation chain. The acknowledged limitations (GPT-4o-generated synthetic data, a small 200-tweet real-world test, and the possibility that synthetic data do not fully capture real noise) are external-validity and annotation-quality concerns, not logical circularity. A possible near-duplicate retrieval leak between the synthetic training and test splits is a legitimate experimental confound worth investigating, but it is not a definitional reduction: the retrieved labels are ground-truth context provided to the LLM, and the model could fail to exploit them. Overall, the central claim is independently evaluated and no circular step is present.
Assumptions & free parameters
free parameters (4)
- k (number of few-shot examples in QSF) =
not reported
- Synthetic duplicate-removal threshold =
0.925
- Data generation prompt mix =
80% Prompt 4, 20% Prompt 3
- Synthetic data generation temperature =
0.8
assumptions (4)
- domain assumption GPT-4o-generated synthetic tweets are representative of real social media crisis posts
- domain assumption The 1,093-element taxonomy derived from 20 humanitarian documents is complete and correctly organized
- domain assumption Manual annotations of the 300 Hurricane Sandy tweets are accurate
- domain assumption OpenAI text-embedding-3-small captures similarity relevant for selecting useful few-shot examples
Cite this review
Pith. "Pith review of Detecting Actionable Requests and Offers on Social Media During Crises Using LLMs." pith.science (2026). https://pith.science/paper/4SMZSPMI
@misc{pith2026250416144,
author = {Pith},
title = {Pith review of: Detecting Actionable Requests and Offers on Social Media During Crises Using LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/4SMZSPMI}},
note = {Machine review of arXiv:2504.16144}
}
read the original abstract
Natural disasters often result in a surge of social media activity, including requests for assistance, offers of help, sentiments, and general updates. To enable humanitarian organizations to respond more efficiently, we propose a fine-grained hierarchical taxonomy to systematically organize crisis-related information about requests and offers into three critical dimensions: supplies, emergency personnel, and actions. Leveraging the capabilities of Large Language Models (LLMs), we introduce Query-Specific Few-shot Learning (QSF Learning) that retrieves class-specific labeled examples from an embedding database to enhance the model's performance in detecting and classifying posts. Beyond classification, we assess the actionability of messages to prioritize posts requiring immediate attention. Extensive experiments demonstrate that our approach outperforms baseline prompting strategies, effectively identifying and prioritizing actionable requests and offers.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Resource mapping during a natural disaster: a case study on the 2015 nepal earthquake
Moumita Basu, Saptarshi Ghosh, Arnab Jana, Somprakash Bandyopadhyay, and Ravikant Singh. Resource mapping during a natural disaster: a case study on the 2015 nepal earthquake. International journal of disaster risk reduction, 24:24–31, 2017
work page 2015
-
[2]
Moumita Basu, Anurag Shandilya, Prannay Khosla, Kripabandhu Ghosh, and Saptarshi Ghosh. Extracting resource needs and availabilities from microblogs for aiding post-disaster relief operations.IEEE Transactions on Computational Social Systems, 6(3):604–618, 2019
work page 2019
-
[3]
Utilizing microblogs for optimized real-time resource allocation in post-disaster scenarios
Moumita Basu, Sipra Das Bit, and Saptarshi Ghosh. Utilizing microblogs for optimized real-time resource allocation in post-disaster scenarios. Social Network Analysis and Mining, 12:1–20, 2022
work page 2022
-
[4]
Social media after nepal earthquake, April 27 2015
Carla Bleiker. Social media after nepal earthquake, April 27 2015. URL https://www.dw.com/en/ nepal-searching-for-missing-loved-ones-with-google/a-18411530
work page 2015
- [5]
-
[6]
A. Devaraj, D. Murthy, and A. Dontula. Machine-learning methods for identifying social media-based requests for urgent help during hurricanes. International Journal of Disaster Risk Reduction, 51:Art. no. 101757, 2020. doi: 10.1016/j.ijdrr.2020.101757
arXiv 2020
-
[7]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186. Association for Computational Linguistics, 2019. doi: 10.1865...
-
[8]
J. Durham, S. Chowdhury, and A. Alzarrad. Unveiling key themes and establishing a hierarchical taxonomy of disaster-related tweets: A text mining approach for enhanced emergency management planning. Information, 14(7):385, 2023. doi: 10.3390/info14070385. URL https://doi.org/10.3390/info14070385
Show all 37 references
-
[9]
Utilizing microblogs for assisting post-disaster relief operations via matching resource needs and availabilities
Ritam Dutt, Moumita Basu, Kripabandhu Ghosh, and Saptarshi Ghosh. Utilizing microblogs for assisting post-disaster relief operations via matching resource needs and availabilities. Information Processing & Management, 56(5):1680–1697, 2019
2019
-
[10]
Grattafiori, A
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, and Z. Ma. The llama 3 herd of models, July 2024. URL htt...
2024
-
[11]
The signals and noise: actionable information in improvised social media channels during a disaster
Xingsheng He, Di Lu, Drew Margolin, Mengdi Wang, Salma El Idrissi, and Yu-Ru Lin. The signals and noise: actionable information in improvised social media channels during a disaster. In Proceedings of the 2017 ACM on web science conference, pages 33–42, 2017
2017
-
[12]
Howard and S
J. Howard and S. Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 328–339. Association for Computational Linguistics, 2018. doi: 10.18...
2018 doi
-
[13]
Evaluating robustness of llms on crisis- related microblogs across events, information types, and linguistic features
Muhammad Imran, Abdul Wahab Ziaullah, Kai Chen, and Ferda Ofli. Evaluating robustness of llms on crisis- related microblogs across events, information types, and linguistic features. arXiv preprint arXiv:2412.10413, 2025
2025 arXiv
-
[14]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas...
2023 arXiv
-
[15]
Decomposed prompting: A modular approach for solving complex tasks
Tushar Khot, Harsh Trivedi, Ashish Sabharwal, and Peter Clark. Decomposed prompting: A modular approach for solving complex tasks. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. CoRe Paper – Social Media for Crisis Management Proceedin...
2023
-
[16]
Large language models are zero-shot reasoners
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. URL https://arxiv.org/abs/ 2205.11916
2022 arXiv
-
[17]
Crema: Crisis response through computational identification and matching of cross-lingual requests and offers shared on social media
Rabindra Lamsal, MariaRodriguez Read, Shanika Karunasekera, and Muhammad Imran. Crema: Crisis response through computational identification and matching of cross-lingual requests and offers shared on social media. IEEE Transactions on Computational Social Systems, 2024
2024
-
[18]
Retrieval-augmented generation for knowledge-intensive nlp tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. InAdvances in Neur...
2020
-
[19]
V. V. Mihunov, N. S. N. Lam, L. Zou, Z. Wang, and K. Wang. Use of twitter in disaster rescue: lessons learned from hurricane harvey. International Journal of Digital Earth, 2020
2020
-
[20]
T. H. Nazer, F. Morstatter, H. Dani, and H. Liu. Finding requests in social media for disaster relief. In2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 1410–1413. IEEE, 2016. doi: 10.1109/ASONAM.2016.7752424
2016
-
[21]
Response planning and coordination, 2023
OCHA. Response planning and coordination, 2023. URL https://www.unocha.org. Retrieved from OCHA Official Website
2023
-
[22]
Turkey/Syria: Earthquakes - feb 2023, 2023
OCHA. Turkey/Syria: Earthquakes - feb 2023, 2023. URL https://reliefweb.int/disaster/ eq-2023-000015-tur . asdfdfsdfas
2023
-
[23]
S. Olawale. Social media and crisis management: A review and analysis of existing studies. LA ¨U Sosyal Bilimler Dergisi, 9(2):199–215, 2018
2018
-
[24]
Pennington, R
J. Pennington, R. Socher, and C. D. Manning. GloVe: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, 2014
2014
-
[25]
Emergency-relief coordination on social media: Automatically matching resource requests and offers
Hemant Purohit, Carlos Castillo, Fernando Diaz, Amit Sheth, and Patrick Meier. Emergency-relief coordination on social media: Automatically matching resource requests and offers. First Monday, 19(1), 2014. doi: 10.5210/fm.v19i1.4848. URL https://doi.org/10.5210/fm.v19i1.4848
2014 doi
-
[26]
Seeberger and K
P. Seeberger and K. Riedhammer. Enhancing crisis-related tweet classification with entity-masked language modeling and multi-task learning. In Proceedings of the Workshop on Natural Language Processing for Positive Impact (NLP4PI), pages 70–78, 2022. URL https://aclanthology.o...
2022
-
[27]
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le ...
2025 arXiv
-
[28]
Rweetminer: Automatic identification and categorization of help requests on twitter during disasters
Irfan Ullah, Sharifullah Khan, Muhammad Imran, and Young-Koo Lee. Rweetminer: Automatic identification and categorization of help requests on twitter during disasters. Expert Systems with Applications, 176:114787, 2021
2021
-
[29]
Emergency response and guidelines, 2023
UNHCR. Emergency response and guidelines, 2023. URL https://www.unhcr.org. Retrieved from UNHCR Official Website
2023
-
[30]
Humanitarian action reports and guidelines, 2023
UNICEF. Humanitarian action reports and guidelines, 2023. URL https://www.unicef.org. Retrieved from UNICEF Official Website
2023
-
[31]
Vaswani, N
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017
2017
-
[32]
Multilingual serviceability model for detecting and ranking help requests on social media during disasters
Fedor Vitiugin and Hemant Purohit. Multilingual serviceability model for detecting and ranking help requests on social media during disasters. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pages 1571–1584, 2024
2024
-
[33]
H. Zade, A. Shah, M. Imran, and F. O. Ostermann. From situational awareness to actionability: Towards improving the utility of social media data for crisis response. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):1–18, 2018
2018
-
[34]
A topic model based framework for identifying the distribution of demand for relief supplies using social media data
Ting Zhang, Shi Shen, Changxiu Cheng, Kai Su, and Xiangxue Zhang. A topic model based framework for identifying the distribution of demand for relief supplies using social media data. International Journal of Geographical Information Science, 35(11):2216–2237, 2021
2021
-
[35]
L. Zhou, X. Wu, Z. Xu, and H. Fujita. VictimFinder: Harvesting rescue requests in disaster response from social media with BERT. Computers, Environment and Urban Systems , 95:101824, 2022. doi: 10.1016/j.compenvurbsys.2022.101824
2022
-
[36]
The mystery of in-context learning: A comprehensive survey on interpretation and analysis
Yang Zhou, Jing Li, Yifan Xiang, Hao Yan, Lin Gui, and Yulan He. The mystery of in-context learning: A comprehensive survey on interpretation and analysis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14365–14378. Association...
2024
-
[37]
Monitoring critical infrastructure facilities during disasters using large language models
Abdul Wahab Ziaullah, Ferda Ofli, and Muhammad Imran. Monitoring critical infrastructure facilities during disasters using large language models. In Proceedings of the International ISCRAM Conference, 2024. doi: https://doi.org/10.59297/755e8b64. CoRe Paper – Social Media for ...
2024 doi
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.