{"id":"41d98366-f8f6-4dd3-9b90-4861acf61348","arxiv_id":"2412.10413","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A systematic evaluation shows GPT-4o and GPT-4 outperform open-source LLMs on crisis tweet classification, with flood events and urgent-need messages as consistent failure points.","lead":"This paper benchmarks six large language models on classifying disaster-related tweets from 19 real-world events, testing zero-shot and few-shot settings and linguistic feature effects. It finds that GPT-4 and GPT-4o generalize best, while flood data, urgent request categories, and typos remain weak points for most models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Flood-related weakness is confounded with non-native English: two of four flood events are non-native, and the paper's own native/non-native analysis shows a large performance drop, so the flood finding needs a within-language-group re-analysis before it can be accepted.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the flood result is confounded with non-native English. This is the most serious issue because the flood finding is one of the paper's headline claims, and the paper itself provides direct evidence for a non-native-English penalty. The proposed test is straightforward with the existing predictions and would settle whether the flood effect is real or an artifact. I agree with the CONDITIONAL verdict: the paper is a valuable empirical benchmark with clear prompts, deterministic temperature-zero inference, and useful class-level error analyses, but the flood claim and several other comparative findings need statistical support. A minor additional inconsistency is that the Introduction states all GPTs achieve F1 below 0.60 on the RUN class, while Table 4 reports RUN F1 values of 0.70-0.85 for GPT-4 and GPT-4o; this should be reconciled, but it does not change the primary concern. Since the main concern matches the reader's weakest assumption and the recommended verdict already captures the needed revisions, no verdict change is warranted.","tokens_in":16135,"tokens_out":6622,"duration_ms":64454,"concrete_test":"Using the paper's predicted labels, compute macro-F1 for flood vs non-flood events separately within (a) native-English events (Midwestern US Floods, Maryland Floods vs US hurricanes/wildfires) and (b) non-native-English events (Kerala, Sri Lanka vs Ecuador/Italy/Pakistan/Puebla earthquakes, Cyclone Idai, Greece wildfires) for each model and shot setting. Add bootstrap 95% CIs over events or a mixed-effects logistic regression with disaster type and native-language status as crossed predictors. If the native-English flood-vs-non-flood contrast is not significantly negative, the 'flood challenge' conclusion should be reattributed to non-native English rather than disaster type.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 groups 19 events into four disaster types and concludes that flood data are the hardest (Figure 2). Section 4.3 shows that all models perform worse on tweets from non-native-English-speaking countries (Figure 5). The flood group contains Kerala 2018 and Sri Lanka 2017 (non-native) alongside Midwestern US 2019 and Maryland 2018 (native). Because the non-native penalty is large and the flood group is only four events, the aggregate flood deficit may be driven by the two non-native flood events rather than by flood semantics. No stratified analysis is reported: Figure 9 gives event-level scores but the paper never compares flood vs non-flood within native-English or within non-native-English subsets. No confidence intervals or significance tests are provided, so it is unknown whether the flood-vs-others gap exceeds event-level noise. The abstract's headline claim that 'most LLMs face challenges in processing flood-related data' therefore rests on an uncontrolled comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates six LLMs (GPT-3.5, GPT-4, GPT-4o, Llama-2 13B, Llama-3 8B, Mistral 7B) on the HumAID crisis microblog dataset across 19 disasters, using zero- and few-shot classification into nine information classes. It reports macro-F1 by disaster type, information type, native/non-native English countries, and linguistic features, and provides event-wise benchmarks. The headline findings are that proprietary models, especially GPT-4/GPT-4o, generalize better; that floods are the hardest disaster type; that urgent-needs tweets are hard for all models; that few-shot examples yield little gain; and that certain linguistic features (e.g., typos, special characters) hurt performance.","tokens_in":16295,"tokens_out":7510,"duration_ms":62851,"significance":"The study is a useful, clearly-scoped empirical benchmark in an underexplored application area. Its strengths are the use of a public dataset, a fixed prompt template with temperature zero, and coverage of both proprietary and open-source models across multiple shot settings. The finding that GPT-4o does not beat a fine-tuned RoBERTa baseline even with 10 shots is an informative result for practitioners. However, the central comparative claims are undermined by the language/flood confound and by the absence of uncertainty quantification, so the paper currently overstates what the data establish.","major_comments":[{"comment":"The conclusion that 'most LLMs face challenges in processing flood-related data' (Abstract) is confounded with the native/non-native English split. The flood group comprises Kerala 2018 and Sri Lanka 2017 (non-native) and Midwestern US 2019 and Maryland 2018 (native). Section 4.3 shows all models perform worse on non-native English tweets, and Figure 9 shows the lowest event-level F1s are for Kerala and Sri Lanka for nearly every model. The paper never compares flood vs. non-flood within the same language group, so the flood deficit could be entirely attributable to non-native English. Please provide the stratified analysis (e.g., native flood vs. native other, non-native flood vs. non-native other) or downgrade the claim.","section":"§4.1, §4.3, Figs. 2 and 5"},{"comment":"The Introduction states 'with all GPTs achieving F1 below 0.60' for requests or urgent needs, but Table 4 shows GPT-4 and GPT-4o never fall below 0.66 and 0.70, respectively, in any shot setting; only some GPT-3.5 settings are below 0.60. This is a factual contradiction between the text and the appendix results. The body's 'below 0.75' is also not true for GPT-4o in 5- and 10-shot (0.81, 0.85). Please correct the text and make the claim match the data.","section":"§1 and §4.2/Table 4"},{"comment":"All comparative statements—e.g., 'GPT-4 and GPT-4o offer better generalizability', 'all models consistently show high performance for earthquakes', 'floods consistently remain the lowest'—are based on point estimates without confidence intervals or significance tests. With only 4-7 events per disaster type, the differences could be noise. Provide bootstrap CIs for event-level F1 or a model with random effects for event, and state whether the observed gaps are statistically reliable. This is needed to support the few-shot 'no improvement' claim as well.","section":"§4, Figs. 2, 5, 9, Table 3"},{"comment":"The hashtag-positioning analysis is reported only as radar charts and qualitative descriptions ('hashtags placed in the middle of messages frequently result in higher error rates'). No numerical F1 values or inferential tests are given for the three positions, and the charts appear to show small and inconsistent differences. Please report the underlying F1 values in a table and test whether position has a reliable effect, or soften the claim.","section":"§4.4, Fig. 8"},{"comment":"The logistic regression treats individual tweets as independent even though they are nested within events and classes, so the standard errors are likely too small; clustering by event (or a mixed-effects model) is needed before the coefficient significances can be trusted. Moreover, the Conclusion's statement that 'message length, typographical errors, and the presence of special characters' are 'significant challenges' is not supported by the reported p-values: typos have p=0.113 (Mistral) and p=0.077 (GPT-4o), and special characters have p=0.091 and p=0.241. Align the Conclusion with the regression results or re-run the analysis with appropriate inference.","section":"§4.4, Tables 1-2, §6"}],"minor_comments":[{"comment":"The acronym ORI is defined for 'other relevant information' but that class is dropped; either remove the definition or state it is excluded before the definition.","section":"§3.1"},{"comment":"The abbreviation RUE is used for 'requests or urgent needs' in the discussion of Figure 3(b) and the confusion matrix text, but the defined acronym is RUN; please use RUN consistently.","section":"§4.2"},{"comment":"The caption does not define LM2, LM3, and MST; add a legend or define them as Llama-2 13B, Llama-3 8B, and Mistral 7B, respectively.","section":"Figure 8"},{"comment":"Use 'zero-shot' instead of '0-shot' for consistency with the rest of the paper.","section":"Table 3"},{"comment":"The prompt template is shown, but the exact additional instructions used for Llama-2 and Mistral 'to control for verbosity' are not provided; include the full prompts in an appendix for reproducibility.","section":"§3.2"},{"comment":"The text says Figure 9 shows results 'across various few-shot settings' but the heatmap includes zero-shot; change to 'across all shot settings'.","section":"§4.5"}],"recommendation":"major_revision","confidential_remarks":"The authors are also the creators of the HumAID dataset and the RoBERTa baseline, which is a natural extension of their prior work. This is not a problem per se, but external validation of the LLM comparisons would strengthen the benchmark. The main risk is the unqualified flood claim, which is likely to be cited. The manuscript fits a venue interested in empirical LLM evaluation and crisis informatics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's contribution is breadth: six LLMs, 19 events, nine classes, four shot settings, plus a linguistic-feature analysis that includes logistic regression on correctness and hashtag-positioning experiments. No prior paper covers this ground on crisis microblogs. The evaluation is straightforward and the reported numbers look consistent with the confusion matrices and appendix tables. The findings that few-shot examples give little benefit, that the RUN class is hard for everyone, and that proprietary models beat open models are useful for practitioners choosing models for disaster monitoring.\n\nThe main soft spot is the flood claim in Section 4.1. Two of the four flood events (Kerala, Sri Lanka) are from non-native English-speaking countries, and Section 4.3 shows all models drop sharply on non-native English tweets. The paper never stratifies flood vs. non-flood within native and non-native groups, so the aggregate flood deficit could be a language confound rather than flood semantics. That is an addressable issue, not a fatal one, but the abstract language ('most LLMs face challenges in processing flood-related data') overstates what the analysis supports.\n\nOther weaknesses are minor: no confidence intervals or significance tests on the main F1 comparisons; the hashtag-positioning result is qualitative (radar charts, no test); and no code or prompt templates are released, so full reproduction is hard. The logistic regression is a nice addition, though its tweet-level observations ignore event clustering.\n\nThe paper is honest about its limitations and uses a public dataset (HumAID) with established labels. The authors are the dataset creators, but that is not a problem here because they are not fitting the target result; they are measuring LLMs on a fixed benchmark.\n\nI would send this to peer review. It is a benchmark paper, not a method paper, but it fills a clear gap in crisis informatics and has enough detail to be checked. A referee should ask for the stratified flood analysis and some error bars, and for code release. If those come back, this becomes a standard citation for LLM evaluation on disaster tweets.","headline":"A solid, useful benchmark of six LLMs on crisis microblogs, with a real confound in the flood finding that needs a stratified re-analysis.","tokens_in":16833,"tokens_out":2636,"would_cite":true,"duration_ms":25478,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that GPT-4o and GPT-4 generalize best across disaster types and information types among six tested LLMs, yet all six models underperform on flood-related tweets and on requests-or-urgent-needs messages, and few-shot…","keywords":["large language models","crisis informatics","disaster response","microblog classification","few-shot learning","robustness","HumAID","social media analysis"],"falsifier":"Compare GPT-4o's zero-shot macro-F1 on native-English flood events (Maryland floods 2018, Midwestern US floods 2019) against non-native-English flood events (Kerala 2018, Sri Lanka 2017) and against non-flood events from the same language groups, using the per-event results in the paper's Figure 9; if native-English floods score close to other native-English events rather than at the low flood level, the 'flood weakness' claim reduces to the language effect.","tokens_in":15941,"feed_emoji":"🌊","tokens_out":9898,"duration_ms":79908,"temperature":0.7,"pith_summary":"This paper asks whether off-the-shelf large language models can automatically sort crisis-related tweets into humanitarian information categories well enough to be useful during real disasters. It benchmarks three proprietary and three open-source LLMs on 77,196 tweets from 19 real-world disasters, testing both zero-shot and few-shot classification. The authors aim to show that proprietary models, especially GPT-4o and GPT-4, generalize reasonably across disasters and information types, but that all models share weak points: flood-related tweets, non-native English, and the requests-or-urgent-needs class. They also argue that adding few-shot examples brings little consistent gain, which matters for real-time disaster response where labeled examples are scarce. The paper's value is a detailed performance map that tells emergency-response systems where to trust LLM output and where to add human review or extra filtering.","feed_headline":"All six LLMs lag on flood tweets and urgent requests","feed_subtitle":"Benchmark on 77K disaster tweets finds GPT-4o leads overall, but few-shot examples barely help.","key_machinery":"The central object is the HumAID dataset, containing 77,196 human-annotated crisis tweets from 19 disasters across 11 countries, with a test split of 15,160 tweets labeled into ten information categories (one dropped, leaving nine). The argument is carried by a fixed-prompt, temperature-zero classification protocol applied to six LLMs in zero-shot, one-shot, three-shot, five-shot, and ten-shot settings, with macro-F1 as the headline metric. This protocol isolates the models' pretrained generalization ability rather than their ability to absorb task-specific training, and it gives a common yardstick for comparing disaster types, information types, native versus non-native English, and linguistic features. The few-shot condition adds class-specific example tweets to the prompt, which is the main manipulated variable for probing whether in-context examples help.","core_discovery":"The paper's central discovery is a performance map of six LLMs on crisis microblogs from the HumAID dataset. GPT-4o and GPT-4 achieve macro-F1 scores around 0.75-0.76 in the zero-shot setting on the full test split, outperforming GPT-3.5 (0.66) and clearly beating the open-source models Llama-2 13B (0.56), Mistral 7B (0.63), and Llama-3 8B (0.53). Across disaster types, earthquakes are handled best and floods worst, with the paper reporting that all models score lowest on flood events. Across information types, requests or urgent needs (RUN) is the persistent weak spot, which the authors describe as below 0.60 F1 for GPT models, and the open-source models also struggle with caution-and-advice messages. The paper further finds that few-shot prompting does not reliably improve performance and sometimes hurts it, and that non-native English tweets plus linguistic features such as typos, long character counts, and mid-sentence hashtags degrade accuracy.","pith_inferences":["The paper's flood finding may be partly a language-effect artifact: two of the four flood events (Kerala and Sri Lanka) come from non-English-speaking countries, and the paper itself shows all models do worse on non-native English; a controlled comparison of floods across native and non-native English would test whether 'flood' is the real cause or a proxy for language difficulty.","The minimal few-shot gains hint that in-domain labeled examples are not the bottleneck; retrieval-augmented generation that pulls event-specific context, or targeted instruction tuning, could be more effective for the requests-or-urgent-needs class than simple shots, though the paper does not test these.","The mid-sentence hashtag result suggests a cheap preprocessing intervention: re-locating or stripping mid-sentence hashtags before classification could recover lost accuracy, which is an extension the paper does not evaluate.","The paper's benchmark could be extended to multimodal crisis data, such as tweets with images, using vision-language models, a direction the authors themselves list as future work."],"forward_implications":["Emergency-response deployments should not count on few-shot examples to close performance gaps, since the paper finds minimal or negative returns from adding shots for most models.","Flood-related social media content should be flagged as a high-risk input, because all six models score lowest on it and downstream triage systems will need extra filtering or human review for flood events.","The persistent confusion between requests-or-urgent-needs and rescue-or-donation messages means that LLM-based classification alone is not ready to prioritize urgent requests without additional disambiguation.","The native versus non-native English analysis implies that LLM performance will be systematically lower for crisis tweets from non-native English-speaking regions, which are common in large-scale disasters.","Because proprietary models outperform open-source models on every task and setting tested, organizations relying on open-source 7B-13B models should expect a performance ceiling in this application."],"supporting_citations":[{"why":"It supplies the HumAID dataset (77,196 tweets, 19 disasters) and the supervised RoBERTa baseline (F1=0.78) that the LLM results are compared against.","marker":"[3]"},{"why":"It defines the GPT-4 model used as a proprietary baseline in all settings.","marker":"[1]"},{"why":"It defines the GPT-4o model that achieves the best overall LLM performance in the paper.","marker":"[25]"},{"why":"It underlies the GPT-3.5 model and the few-shot learning paradigm that the shot-setting experiments rely on.","marker":"[5]"},{"why":"It defines the Llama-2 13B open-source model evaluated.","marker":"[29]"},{"why":"It defines the Llama-3 8B open-source model evaluated.","marker":"[10]"},{"why":"It defines the Mistral 7B open-source model evaluated.","marker":"[18]"}],"fun_headline_variants":["GPT-4o tops crisis tweet test, floods trip all models","Few-shot barely helps LLMs on disaster tweets, floods worst","LLMs weakest on flood tweets and urgent requests in new benchmark","Crisis microblog LLM test: GPT-4o leads, typos and floods hurt","All LLMs miss urgent requests; GPT-4o still best on disaster posts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison across disaster types treats 'flood' as the explanatory factor, but two of the four flood events come from countries where English is not the native language, and the paper's own analysis shows all models perform worse on non-native English data, so the flood-specific weakness could be a language confound rather than a property of flood content.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o tops crisis tweet test, floods trip all models","Few-shot barely helps LLMs on disaster tweets, floods worst","LLMs weakest on flood tweets and urgent requests in new benchmark","Crisis microblog LLM test: GPT-4o leads, typos and floods hurt","All LLMs miss urgent requests; GPT-4o still best on disaster posts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1512,"prompt_tokens":985,"completion_tokens":527,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":429}},"tokens_in":601,"tokens_out":527,"duration_ms":5425,"temperature":1.0,"reasoning_tokens":429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:13:53.258494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare GPT-4o's zero-shot macro-F1 on native-English flood events (Maryland floods 2018, Midwestern US floods 2019) against non-native-English flood events (Kerala 2018, Sri Lanka 2017) and against non-flood events from the same language groups, using the per-event results in the paper's Figure 9; if native-English floods score close to other native-English events rather than at the low flood level, the 'flood weakness' claim reduces to the language effect.","supporting_citations":[{"cited_title":"Gpt-4o system card","cited_arxiv_id":null,"evidence_quote":"It defines the GPT-4o model that achieves the best overall LLM performance in the paper."}],"review_version":1}