{"id":"15744dd9-5900-4835-bea8-18130778ecec","arxiv_id":"2605.20646","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"DisImpact introduces a two-stage MLLM framework to classify disaster-related social media posts into ten impact categories and compute a unified physi-social impact index validated against FEMA and NASA ground-truth data.","lead":"DisImpact is a two-stage framework that uses a multimodal large language model to classify social media posts about natural disasters into ten physical and social impact categories and then builds a weekly unified impact index from category prominence and engagement levels. This could enable faster, comparable measurement of both immediate damage and longer-term community effects for improved disaster response.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"MLLM classification accuracy lacks direct validation, leaving impact indices vulnerable to systematic errors","rationale":"The reader's weakest assumption correctly isolates the classification step as upstream of every subsequent claim. Because the abstract supplies no classification metrics and the validation is performed only on the derived indices, this remains the single most load-bearing unverified link.","tokens_in":1779,"tokens_out":288,"duration_ms":19985,"concrete_test":"Draw a stratified random sample of 400 posts (balanced across the ten categories and multiple disasters), obtain independent expert annotations, and compute macro-F1 and Cohen's kappa between MLLM outputs and human labels; if macro-F1 falls below 0.75 or kappa below 0.65, the index construction is compromised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the MLLM reliably maps posts to the ten specified physi-social categories so that relative prominence and engagement intensity produce meaningful indices. The reported lead-lag correlations with FEMA Public Assistance and NASA FIRMS data test only the final aggregated time series; they do not rule out the possibility that classification noise or social-media sampling bias preserves temporal structure while distorting category weights. No precision, recall, or human-agreement figures for the classification stage are referenced in the provided abstract, so the mapping from raw posts to the ten categories remains the least secured step.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces DisImpact, a two-stage framework using a Multimodal Large Language Model (MLLM) to classify social media posts into ten physi-social disaster impact categories (covering physical and social domains), followed by construction of a weekly impact index that integrates relative category prominence with public engagement intensity. This produces a unified scale for comparing impacts across categories and aggregating to higher-level patterns. The indices are validated via lead-lag correlations against FEMA Public Assistance data and NASA FIRMS fire detections, with additional temporal and spatial analyses showing physical impacts peaking during events and localized, while social impacts emerge later and spread more broadly. The work claims to be the first comprehensive quantification of both dimensions using multimodal social media data.","tokens_in":1901,"tokens_out":585,"duration_ms":26610,"significance":"If the MLLM classification proves reliable, the framework provides a timely, scalable method for real-time physi-social disaster impact assessment that complements traditional surveys and reports. The unified index enables direct cross-category and cross-domain comparisons, and the reported lead-lag correlations with authoritative ground-truth sources offer a concrete test of external validity. The temporal-spatial findings could support improved disaster response and resource allocation. The approach's strength lies in its direct use of user-generated multimodal content rather than fitted models, though significance depends on securing the classification stage.","major_comments":[{"comment":"Abstract and framework description: the validation consists solely of lead-lag correlations on the final aggregated indices against FEMA Public Assistance and NASA FIRMS data, but no precision, recall, F1, or human-agreement figures are reported for the MLLM classification of posts into the ten specified categories. This step is load-bearing for the central claim, as classification noise or sampling bias in social media could preserve aggregate temporal structure while distorting the relative weights that define the impact indices.","section":null},{"comment":"Validation procedure (as described in the abstract): the claim of 'strong validity across both social and physical impact dimensions' rests on 'consistent lead-lag correlations,' yet the manuscript provides no details on the exact index construction formula, handling of missing data or platform-specific sampling rates, statistical tests for correlation significance, or controls for post-hoc category selection. Without these, it is unclear whether the observed correlations confirm the intended physi-social mapping or merely reflect broad disaster timing.","section":null}],"minor_comments":[{"comment":"The abstract refers to 'multimodal data from multiple social media platforms' without specifying which platforms or how modality fusion is performed in the MLLM; clarifying this would improve reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript is submitted to cs.SI yet centers on disaster management applications; the editor may wish to consider fit with journals focused on AI for social good or crisis informatics. No obvious citation anomalies noted from the provided text."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback on our manuscript. We have addressed each major comment below with point-by-point responses. Revisions have been made to strengthen the presentation of the classification evaluation and the validation methodology.","responses":[{"response":"We agree that direct performance metrics for the MLLM classification stage are essential to substantiate the reliability of the framework. The original manuscript emphasized validation of the downstream aggregated indices against ground-truth sources, but we acknowledge that this leaves the classification quality implicit. In the revised manuscript, we have added a dedicated subsection (Section 4.1) reporting precision, recall, and F1 scores for each of the ten categories, along with Cohen's kappa and percentage agreement from a human evaluation study conducted on a stratified sample of 500 posts. These results indicate strong agreement overall and support the use of the classifications for index construction.","revision_made":"yes","referee_comment":"[—] Abstract and framework description: the validation consists solely of lead-lag correlations on the final aggregated indices against FEMA Public Assistance and NASA FIRMS data, but no precision, recall, F1, or human-agreement figures are reported for the MLLM classification of posts into the ten specified categories. This step is load-bearing for the central claim, as classification noise or sampling bias in social media could preserve aggregate temporal structure while distorting the relative weights that define the impact indices."},{"response":"We thank the referee for highlighting these methodological details. The full manuscript (Section 3.2) already contains the exact formula for the weekly impact index, defined as a weighted sum of normalized category prominence (post frequency within category) multiplied by engagement intensity (sum of likes, shares, and comments, normalized by total posts that week). We have now expanded the Methods and Validation sections to explicitly describe: (i) linear interpolation for missing weekly data points with sensitivity checks, (ii) platform-specific normalization to account for differing sampling rates across Twitter, Instagram, and TikTok, (iii) both Pearson and Spearman rank correlations with bootstrap-derived p-values and confidence intervals, and (iv) a post-hoc robustness check via leave-one-category-out analysis and alternative category groupings. These additions demonstrate that the lead-lag patterns align with the intended physical-versus-social distinctions rather than generic event timing. The abstract has been updated to reference the expanded validation details.","revision_made":"yes","referee_comment":"[—] Validation procedure (as described in the abstract): the claim of 'strong validity across both social and physical impact dimensions' rests on 'consistent lead-lag correlations,' yet the manuscript provides no details on the exact index construction formula, handling of missing data or platform-specific sampling rates, statistical tests for correlation significance, or controls for post-hoc category selection. Without these, it is unclear whether the observed correlations confirm the intended physi-social mapping or merely reflect broad disaster timing."}],"tokens_in":1564,"tokens_out":618,"duration_ms":33173,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main contribution is a two-stage setup that feeds multimodal social media posts into an MLLM, sorts them into ten physi-social categories, and then rolls those labels into a weekly impact index that blends category share with engagement strength. This produces a single scale that can be compared across events and split into physical versus social views. They show the index lines up in timing with FEMA aid payouts and NASA fire data, and they add some temporal and spatial breakdowns that line up with common sense about when and where effects show up.","headline":"DisImpact builds a social-media index for physical and social disaster effects but leaves the MLLM classification step without direct accuracy checks.","tokens_in":2401,"tokens_out":178,"would_cite":false,"duration_ms":18134,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/ArithmeticFromLogic.lean","rs_theorem":null,"paper_passage":"two-stage framework that systematically quantifies the physi-social impacts of disasters via a Multimodal Large Language Model (MLLM). The social media posts are first classified into ten disaster impact categories... construct a disaster impact index that integrates the relative prominence of each category with the intensity of public engagement"}],"headline":"Disaster-impact indexing via MLLM classification and smoothed engagement weights","alignment":"orthogonal","rationale":"The paper's core machinery (MLLM single-label classification of posts into 10 physi-social categories, additive smoothing of proportions Pt(c), arctan-based intensity weight wt, and lead-lag Spearman validation against FEMA/NASA ground truth) is a standard empirical pipeline in disaster informatics. It contains none of the RS structural signatures: no J-cost or cosh-cost functional, no golden-ratio ladder, no 8-tick periodicity, no parameter-free derivation of constants, and no recognition-cost forcing from a bare distinction. The domain (sociology of disasters) is one RS claims to cover, yet the paper neither invokes nor parallels any RS theorem (e.g., reality_from_one_distinction, J-uniqueness, or Alexander-duality D=3 forcing). Hence orthogonal.","tokens_in":56129,"confidence":"high","tokens_out":305,"duration_ms":10815,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A two-stage AI framework classifies social media posts to build a unified index measuring both physical destruction and social fallout from natural disasters.","keywords":["natural disasters","social media","impact quantification","multimodal LLM","physical and social impacts","disaster index","temporal analysis","spatial analysis"],"falsifier":"If the constructed indices showed no consistent lead-lag correlation with FEMA Public Assistance amounts or NASA FIRMS fire detections when tested on an independent set of disasters, the claim of validity would be refuted.","tokens_in":2686,"feed_emoji":"📊","tokens_out":714,"duration_ms":36262,"temperature":0.7,"pith_summary":"The paper introduces DisImpact, which first routes multimodal social media posts through a large language model to assign them to one of ten categories that span physical damage and social consequences. It then folds the share of posts in each category together with how intensely the public engages with them into a single weekly index. This matters because it gives a comparable, real-time scale for impacts that conventional surveys and official tallies often capture too late or too narrowly. The resulting measures line up with FEMA aid records and satellite fire detections through consistent lead-lag patterns, while also showing that physical effects concentrate in time and place during the event itself whereas social effects surface later and spread farther.","feed_headline":"AI turns social media into unified index of disaster physical and social toll","feed_subtitle":"Framework classifies posts into ten categories and shows physical damage peaks early and locally while social effects spread later and wider","key_machinery":"The disaster impact index, formed by combining the relative share of posts in each of ten classified categories with the intensity of public engagement measured on a weekly basis.","core_discovery":"The paper establishes that sorting social media posts into ten physi-social impact categories via a multimodal large language model and then weighting them by relative prominence and public engagement produces an index that tracks authoritative physical-impact records and exposes distinct temporal and spatial signatures: physical impacts peak during the disaster and stay localized, while social impacts emerge afterward and diffuse more widely.","pith_inferences":["Emergency agencies could monitor the index in near real time to detect emerging social needs before official tallies are available.","The same pipeline might be applied to disasters in other countries or languages to test whether the physical-versus-social timing pattern generalizes.","Longer-term studies could check whether spikes in the social-impact component predict measurable differences in community recovery speed or mental-health outcomes.","Planners might use the index to balance immediate physical-repair spending against programs that address the later, more diffuse social consequences."],"forward_implications":["Physical impacts reach their highest levels during the disaster and remain concentrated in the directly affected geographic areas.","Social impacts appear with a delay and spread across wider regions and longer time windows than physical damage.","The single index permits direct numerical comparison among any pair of impact categories and supports flexible aggregation into domain-level or overall trends.","Lead-lag alignment with government aid data and satellite observations holds across both the social and physical dimensions of the index."],"fun_headline_variants":["DisImpact indexes physi-social disaster effects from social media posts via MLLM","Ten categories sort social posts into physical and social disaster index","Physical impacts peak locally while social ones emerge later and spread wider","MLLM framework builds index from ten impact categories to compare disaster effects","Social media yields index tracking physical records with distinct social patterns"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The multimodal large language model classifies social media posts into the ten impact categories accurately enough that the resulting index reflects genuine physi-social effects rather than model errors or sampling biases in the posts themselves.","fun_headline_variants_meta":{"raw":{"variants":["DisImpact indexes physi-social disaster effects from social media posts via MLLM","Ten categories sort social posts into physical and social disaster index","Physical impacts peak locally while social ones emerge later and spread wider","MLLM framework builds index from ten impact categories to compare disaster effects","Social media yields index tracking physical records with distinct social patterns"]},"model":"grok-4.3","cost_usd":0.007165,"raw_usage":{"total_tokens":3254,"prompt_tokens":722,"num_sources_used":0,"completion_tokens":86,"cost_in_usd_ticks":71653000,"prompt_tokens_details":{"text_tokens":722,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2446,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":722,"tokens_out":86,"duration_ms":26076,"temperature":1.0,"reasoning_tokens":2446,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-21T02:35:25.807264+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If the constructed indices showed no consistent lead-lag correlation with FEMA Public Assistance amounts or NASA FIRMS fire detections when tested on an independent set of disasters, the claim of validity would be refuted.","supporting_citations":[],"review_version":1}