{"id":"23e8299a-3c55-492b-8e1f-f31b89b994b6","arxiv_id":"2504.16144","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Query-specific retrieval of few-shot examples improves LLM classification of actionable disaster requests and offers on social media compared to static prompting.","lead":"A team from QCRI built a fine-grained taxonomy of disaster relief requests and offers, then used retrieval-augmented few-shot prompting to classify social media posts into supplies, actions, and personnel categories. The method beats static prompting baselines in tests on synthetic and a small real-world set of Hurricane Sandy tweets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"QSF gains may be inflated by near-duplicate training examples retrieved for synthetic test posts; the paper reports no train/test similarity separation or batch-disjoint split, so the central comparison is not yet clean.","rationale":"The reader's weakest assumption focused on whether synthetic and real data are representative of genuine crisis streams. My concern is sharper: within the synthetic data, the random split plus embedding retrieval can leak near-duplicate labels into QSF prompts, attacking internal validity rather than external representativeness. I credit the paper's real-world results and its transparency about limitations, but 200 manually re-annotated tweets with no reported inter-annotator agreement cannot independently support the general claim, and the actionability results contradict the 'outperforms' framing in several cells. A simple similarity-gap or batch-disjoint re-run would settle the issue, so this is not a rejection of the method, only of the current evidence as sufficient for the central claim.","tokens_in":17216,"tokens_out":5389,"duration_ms":51244,"concrete_test":"Compute for every synthetic test instance its maximum cosine similarity to any training instance using text-embedding-3-small; report the distribution and the breakdown of QSF vs BL5 F1 for test instances above and below a similarity threshold (e.g., 0.85). Additionally, re-run the full comparison with a batch-disjoint split (train on generation runs 1-5, test on runs 6-10) so no test post can have a near-duplicate in the retrieval DB. If QSF's margin over BL5 collapses or shrinks materially in the disjoint split, the reported synthetic gains are inflated by retrieval leakage.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The paper's strongest evidence for QSF Learning comes from the synthetic split, but the construction of that split can produce a subtle retrieval leak. In 'Synthetic Data Generation', 2,000 GPT-4o examples are deduplicated only at cosine similarity > 0.925, leaving 1,346 examples; in 'Evaluation Metrics' these are randomly split 50/50 into train and test. QSF then retrieves the k/2 nearest training examples for each test message using the same text-embedding-3-small embeddings. A generated test post that is a near-duplicate of a training post (same event, template, hashtags, or phrasing below the 0.925 cutoff) will therefore have its label effectively copied into the prompt. Static baselines BL1-BL5 do not get this benefit, so part of the reported F1 gap, especially on sparse tasks like Actions and Personnel, may be an artifact of retrieval leakage rather than of query-specific learning. The paper does not report the train/test nearest-neighbor similarity distribution, the value of k, or any batch-disjoint split. The real-world test (200 Hurricane Sandy tweets, no inter-annotator agreement) is too small and unvalidated to independently establish the claimed margin, and the Actionability results are explicitly mixed (e.g., Gemma 2 synthetic QSF 0.49 vs BL5 0.78). The central claim 'QSF outperforms static baselines across synthetic and real-world data' is therefore not yet supported by clean evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper targets fine-grained identification of requests and offers in crisis-related social media. It introduces a top-down taxonomy with three branches (supplies, actions, emergency personnel; 1,093 elements) and frames the task as multi-label classification over 11 labels per branch plus a binary actionability judgment. The main methodological contribution is Query-Specific Few-shot Learning (QSF): for each input, k/2 nearest labeled examples are retrieved from an embedding database built with text-embedding-3-small, combined with k/2 random examples, and appended to the prompt. The authors compare QSF with five static prompting baselines (BL1-BL5) using Llama 3, Llama 3.1, Gemma 2, Mistral 7B, and GPT-4o mini on 673 synthetic GPT-4o-generated tweets and 200 manually re-annotated Hurricane Sandy tweets. The paper reports micro-F1 for the multi-label tasks and macro-F1 for actionability, and concludes that QSF outperforms the baselines, with the largest gains on Actions and Personnel.","tokens_in":17527,"tokens_out":7618,"duration_ms":68062,"significance":"If the central claim were cleanly established, this would be a useful contribution: a reusable fine-grained taxonomy, a low-annotation retrieval-augmented prompting method, and an evaluation across several open LLMs. The multi-label framing and public release of the dataset and taxonomy are positive features, and the limitations section is candid about the synthetic-data and small-real-data risks. However, the current evidence is conditional: the synthetic split permits a plausible retrieval-leakage path, the real-world test set is small and lacks reliability measures, and the actionability results are mixed. These issues are fixable with additional analyses, so the work is promising but not yet conclusive.","major_comments":[{"comment":"The reported QSF gains may be inflated by retrieval leakage. The synthetic corpus is deduplicated at a cosine-similarity threshold of 0.925 using text-embedding-3-small and then randomly split 50/50 into training and evaluation sets. QSF retrieves the k/2 nearest training examples per test message in the same embedding space, so any test post that is a near-duplicate of a training post below the 0.925 cutoff can have its labels copied into the prompt, whereas the static few-shot baselines cannot exploit query-specific proximity. The manuscript does not report the train/test nearest-neighbor similarity distribution, does not use a batch-disjoint or cluster-disjoint split, and does not state k. Because the largest gains are on the sparse Actions and Personnel tasks, this concern is load-bearing for the central claim. Please add a similarity analysis and an ablation that removes near-duplicate test posts or performs cluster-disjoint splitting.","section":"Synthetic Data Generation / Evaluation metrics"},{"comment":"The real-world evaluation cannot independently support the generalization claim. The test set is 200 Hurricane Sandy tweets re-annotated by the authors, with no inter-annotator agreement reported and no description of adjudication. The text is internally inconsistent about the sample size (300 sampled tweets vs 107 training + 200 test = 307 labeled posts). With n=200 and sparse labels (e.g., Personnel F1 near 0.2-0.3 in baselines), the reported QSF improvements may rest on very few instances; no confidence intervals, bootstraps, or significance tests are given. Please report IAA, reconcile the counts, and supply interval estimates or per-class support.","section":"Real-World Data / Tables 3-4"},{"comment":"The claim that QSF outperforms baseline prompting strategies is too broad for the actionability task, which is part of the abstract and conclusion. On synthetic data, Gemma 2 QSF scores 0.49 vs BL5 0.78, and Mistral QSF 0.67 vs BL5 0.72; on real data, Llama 3 QSF 0.47 vs BL5 0.45 is essentially tied. The text acknowledges mixed results but the overall claim is not qualified accordingly. Please either scope the central claim to the multi-label taxonomy tasks or provide evidence for a consistent actionability benefit.","section":"Results and Discussion / Tables 3-4"},{"comment":"The synthetic ground truth is generated by GPT-4o and only 40% of it was human-reviewed, with fewer than 10% corrections; the evaluator GPT-4o mini is from the same model family. This raises the risk that the synthetic benchmark is systematically easier or more stereotyped than real streams, and that QSF's retrieval of nearest neighbors propagates generation artifacts rather than learning generalizable structure. Please report the per-class correction counts, the label-noise distribution, and at least one analysis that compares QSF on the synthetic labels with QSF on the human-corrected subset (or on real labels) to separate retrieval benefit from data-generation bias.","section":"Synthetic Data Generation"}],"minor_comments":[{"comment":"Reconcile the 300 vs 307 discrepancy in the number of annotated tweets.","section":"Real-World Data"},{"comment":"The value of k is never stated; please report k and any sensitivity analysis over it.","section":"QSF Learning"},{"comment":"The caption/text mentions BL 3 through 6 although only BL3-BL5 and QSF exist; correct the label.","section":"Figure 6"},{"comment":"Reference [22] contains a stray 'asdfdfsdfas' string; clean up the bibliography entry.","section":"References"},{"comment":"For GPT-4o mini only QSF scores are shown; add a note that no baseline comparison was run, since the text otherwise implies all models were tested on all prompts.","section":"Tables 3-4"},{"comment":"At depth 2 the paper says 11 labels per set while the taxonomy has 33 Level-2 elements; clarify that these are per-branch depth-2 labels.","section":"Task Definition"}],"recommendation":"major_revision","confidential_remarks":"I believe the paper is salvageable within the scope of revision: the central methodological idea is reasonable and the resource contribution is real. The key condition is the retrieval-leakage analysis; if the QSF gains collapse under near-duplicate removal or cluster-disjoint splitting, the main claim should be weakened or the method repositioned. The actionability overclaim should also be corrected. The fit with ISCRAM is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a solid applied paper from a group that knows crisis NLP. The new artifacts are a 1,093-element request/offer taxonomy across supplies, actions, and personnel, and a retrieval-based few-shot prompting method (QSF). The paper reports consistent F1 gains over static baselines across several open LLMs on both synthetic and real tweets. I believe the effects are probably real, but the headline comparison has a clean-evidence problem that the paper does not address.\n\nThe taxonomy is the strongest contribution. It is top-down, grounded in humanitarian guidance, and fills real gaps (e.g., dust masks after Beirut, baby formula after Harvey). The multi-label framing is appropriate; a single tweet can be both request and offer. Releasing the resources is a plus, and the human review of 40% of the synthetic labels is more than many papers do.\n\nThe soft spot is the evaluation, exactly where the stress-test lands. The synthetic set is 2,000 GPT-4o posts deduplicated at cosine similarity >0.925, then split 50/50. QSF retrieves nearest-neighbor examples from the training half using the same embedding model. Nothing in the paper shows that test posts are not near-duplicates of training posts just below the dedup cutoff. If they are, the retrieved examples can hand the model the label. The paper does not report k, the train/test similarity distribution, or any batch-disjoint split. So part of the synthetic QSF margin, especially on sparse labels like Personnel, could be leakage. This is not fatal—the real-world 200-tweet evaluation also shows QSF gains on Actions and Personnel (e.g., Llama 3.1 Personnel from 0.27 to 0.47), which cannot come from synthetic leakage—but it means the paper's central claim is not yet cleanly supported.\n\nOther gaps: the real test set is small and has no reported inter-annotator agreement; no significance tests or error bars; and the Actionability results are explicitly mixed (Gemma 2 synthetic QSF 0.49 vs BL5 0.78), which the paper mentions but then somewhat overclaims in the conclusion. Effect sizes on Supplies are often tiny.\n\nWho should read this: anyone working in crisis NLP or LLM-based few-shot classification. It deserves a serious referee, not a desk reject. I would send it out with a request to add a leakage analysis (report nearest-neighbor similarity between test and retrieved training items, run a disjoint split, report k), significance testing, and at least an IAA number for the real set. With those, the empirical claim would be in shape.","headline":"Useful taxonomy, plausible dynamic few-shot method, but the headline QSF gains need a leakage check before I'd trust them.","tokens_in":18039,"tokens_out":2949,"would_cite":true,"duration_ms":26031,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that query-specific few-shot retrieval—pulling class-relevant labeled examples from an embedding database into the prompt—improves multi-label classification of crisis requests and offers, and of actionability, over…","keywords":["large language models","crisis informatics","disaster response","social media","few-shot prompting","retrieval-augmented generation","taxonomy","multi-label classification"],"falsifier":"Take a freshly collected, independently annotated crisis tweet set from a disaster not among those used to generate the synthetic data, run the QSF pipeline against the best static baseline on Actions and Personnel, and report inter-annotator agreement on the labels; if the QSF advantage over BL5 shrinks to a few points or disappears, the paper's central claim that retrieval-specific examples drive the gains is not supported.","tokens_in":1637,"feed_emoji":"🆘","tokens_out":1757,"duration_ms":45058,"temperature":0.7,"pith_summary":"The paper aims to show that during disasters, social media posts requesting or offering help can be classified into fine-grained, actionable categories without retraining a model per event. To do this, it introduces a hierarchical taxonomy of supplies, emergency personnel, and actions, and a prompting method called Query-Specific Few-shot Learning (QSF Learning) that retrieves labeled examples similar to each incoming post. The central claim is that this retrieval-based prompting beats five static prompting baselines across models of different sizes, with the largest gains on the hardest tasks: identifying requested or offered actions and personnel. A sympathetic reader would care because the method promises faster, cheaper crisis-response triage than building new labeled datasets for every disaster.","feed_headline":"When prompts fetch similar tweets, crisis triage F1 jumps","feed_subtitle":"Pulling nearest labeled examples into the prompt lifts F1 for requests and offers across five LLMs.","key_machinery":"The central mechanism is QSF Learning, a retrieval-augmented few-shot prompting loop: for each input post, an embedding is computed with OpenAI's text-embedding-3-small model; the $k/2$ most similar labeled examples are retrieved by cosine similarity from a training-set embedding database, $k/2$ random examples are added for variability, and the combined examples are inserted into the prompt before the model outputs a structured JSON tuple. The other load-bearing component is the taxonomy itself: 1,093 elements organized into three branches (supplies, actions, emergency personnel), with 11 labels at depth 2 per branch defining the classification target space. Together they let the model adapt its label choices to each query without any fine-tuning.","core_discovery":"The paper claims that dynamically retrieving labeled examples—specifically, the $k/2$ nearest labeled posts by embedding similarity plus $k/2$ randomly chosen labeled posts—and appending them to an LLM prompt consistently outperforms static baselines (BL1–BL5) for multi-label, multi-class classification of requests and offers into supplies, actions, and personnel, and for predicting a binary actionability flag. The strongest evidence is on Actions, where F1 rises from 0.62 to 0.77 for Mistral on synthetic data and from 0.42 to 0.57 for Llama 3 on real-world data, and on Personnel, where Llama 3.1 rises from 0.27 to 0.47 on real-world data. The paper also reports that static prompting plateaus quickly, and that QSF Learning narrows the gap between smaller open models and a stronger paid model.","pith_inferences":["A testable extension the paper leaves implicit is varying the ratio of nearest to random examples: if the random half stabilizes the prompt and prevents class bias, changing the ratio should shift the precision-recall tradeoff, but the paper reports no such ablation.","The taxonomy's imbalance—946 supply elements versus 90 actions and 57 personnel—suggests that the lower F1 on actions and personnel may partly reflect sparser label coverage rather than intrinsic difficulty; a balanced taxonomy experiment could isolate that.","If these gains carry over to live streams, a practical system could rank incoming posts by predicted actionability and auto-generate structured requests for supplies, personnel, and actions, feeding directly into humanitarian coordination dashboards.","Because the synthetic test data was itself generated by an LLM, a stronger validation would compare QSF Learning on independently sourced, professionally annotated crisis corpora across multiple languages and disaster types."],"forward_implications":["If the central claim holds, disaster-response teams can deploy a classifier for a new crisis by reusing labeled examples from past events, without collecting new training data or fine-tuning a model.","The largest predicted gains are on Actions and Personnel, the categories that previous request-and-offer systems handled least well, so operational triage could surface more rescue, medical, and debris-clearance posts.","Retrieval-based prompting may reduce the performance gap between small open models and large proprietary ones, making the approach feasible when API access is limited or costly.","The mixed actionability results mean QSF Learning is not uniformly better on the binary actionability decision; its benefit concentrates on granular multi-label tasks, so a practical system would still need to decide which task to optimize.","Because examples are drawn only from the training split, the reported improvements are not an artifact of test-set leakage, assuming the retrieval database is assembled correctly."],"supporting_citations":[{"why":"Supplies the Retrieval-Augmented Generation idea that QSF Learning builds on for retrieving and appending relevant examples.","marker":"[18]"},{"why":"Provides the original Hurricane Sandy request-and-offer dataset that the real-world evaluation samples from.","marker":"[25]"},{"why":"Provides the improved and re-annotated version of the real-world crisis dataset used for testing.","marker":"[17]"},{"why":"Defines actionability as posts containing sufficient context about time, location, urgency, or specific needs, which the paper adopts.","marker":"[33]"},{"why":"Shows that adding equal numbers of few-shot examples per class can hurt performance, motivating the query-specific retrieval design.","marker":"[13]"},{"why":"Establishes few-shot prompting as an effective technique for LLMs, the foundation of the BL3–BL5 baselines and QSF Learning.","marker":"[5]"},{"why":"Provides the Llama 3 and Llama 3.1 models used as evaluation targets in the experiments.","marker":"[10]"},{"why":"Provides the Gemma 2 model used as another evaluation target.","marker":"[27]"}],"fun_headline_variants":["Similar-example prompts lift F1 for crisis requests","Nearest-neighbor prompts boost crisis triage F1","Retrieval-augmented LLM prompts improve crisis F1","Pulling similar posts into prompts raises F1","Crisis F1 gains from retrieval-based examples"],"cache_read_input_tokens":20096,"weakest_assumption_plain":"The load-bearing premise is that the labeled test data—GPT-4o-generated synthetic tweets with light human correction and a 300-tweet manually annotated Hurricane Sandy sample—are trustworthy proxies for real crisis streams; if synthetic posts are systematically easier or more stereotyped than authentic tweets, the measured QSF improvements may not transfer to live disaster response.","fun_headline_variants_meta":{"raw":{"variants":["Similar-example prompts lift F1 for crisis requests","Nearest-neighbor prompts boost crisis triage F1","Retrieval-augmented LLM prompts improve crisis F1","Pulling similar posts into prompts raises F1","Crisis F1 gains from retrieval-based examples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1429,"prompt_tokens":854,"completion_tokens":575,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":497}},"tokens_in":470,"tokens_out":575,"duration_ms":5132,"temperature":1.0,"reasoning_tokens":497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:19:18.930847+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a freshly collected, independently annotated crisis tweet set from a disaster not among those used to generate the synthetic data, run the QSF pipeline against the best static baseline on Actions and Personnel, and report inter-annotator agreement on the labels; if the QSF advantage over BL5 shrinks to a few points or disappears, the paper's central claim that retrieval-specific examples drive the gains is not supported.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","cited_arxiv_id":null,"evidence_quote":"Supplies the Retrieval-Augmented Generation idea that QSF Learning builds on for retrieving and appending relevant examples."},{"cited_title":"Emergency-relief coordination on social media: Automatically matching resource requests and offers","cited_arxiv_id":null,"evidence_quote":"Provides the original Hurricane Sandy request-and-offer dataset that the real-world evaluation samples from."},{"cited_title":"Crema: Crisis response through computational identification and matching of cross-lingual requests and offers shared on social media","cited_arxiv_id":null,"evidence_quote":"Provides the improved and re-annotated version of the real-world crisis dataset used for testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines actionability as posts containing sufficient context about time, location, urgency, or specific needs, which the paper adopts."},{"cited_title":"Evaluating Robustness of LLMs on Crisis-Related Microblogs across Events, Information Types, and Linguistic Features","cited_arxiv_id":"2412.10413","evidence_quote":"Shows that adding equal numbers of few-shot examples per class can hurt performance, motivating the query-specific retrieval design."},{"cited_title":"Brown, B","cited_arxiv_id":null,"evidence_quote":"Establishes few-shot prompting as an effective technique for LLMs, the foundation of the BL3–BL5 baselines and QSF Learning."},{"cited_title":"Grattafiori, A","cited_arxiv_id":null,"evidence_quote":"Provides the Llama 3 and Llama 3.1 models used as evaluation targets in the experiments."}],"review_version":1}