{"id":"5ef38304-a3e6-43c5-8ce1-22891a4ef525","arxiv_id":"2508.00650","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"Evac-Cast, an XGBoost model with SHAP interpretation, predicts evacuation rates with about 4% error and identifies hazard intensity as the dominant feature across hurricanes and wildfires.","lead":"This paper describes Evac-Cast, a machine learning framework that forecasts tract-level evacuation rates for hurricanes and wildfires using over 20 public data features. It reports mean absolute errors of 4.5% for hurricanes and 3.5% for wildfires, with hazard intensity as the most influential predictor.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on the validity of mobility-derived evacuation rates as ground truth, which the abstract does not establish; without independent label validation, the reported MAEs and SHAP rankings may not describe actual evacuation behavior.","rationale":"The central claim has two pillars: (i) the model's predicted evacuation rates are accurate (MAE 4.5%/3.5%), and (ii) SHAP analysis reveals consistent feature importance led by hazard intensity. Both pillars depend on the definition of the target variable 'evacuation rate.' The abstract gives no information about the provenance of these rates. The most likely source is mobile-device mobility data, which is well known to suffer from biases during disasters: network outages, differential phone ownership, and voluntary movement unrelated to hazard. If the labels are biased, the XGBoost model is trained to reproduce those biases, and the reported MAE is a measure of fit to the label, not to reality. The SHAP hierarchy, in turn, reflects the label's noise structure. This is more fundamental than event representativeness: even if the four events are perfect exemplars, invalid labels would invalidate the framework. A single independent validation of the mobility-derived rates against survey or shelter data would resolve this concern. If such validation already exists in the full paper, the concern is moot; but on the abstract alone it is unresolved. The reader's UNVERDICTED verdict is therefore appropriate; I do not propose changing it, but I flag this as the key assumption to verify.","tokens_in":765,"tokens_out":4514,"duration_ms":57109,"concrete_test":"Obtain the evacuation-rate labels used in the paper for Hurricane Ian (2022) at the tract level. Independently collect post-storm survey-based evacuation rates for the same tracts, e.g., from the FEMA household surveys or published academic studies. Compute a paired comparison: correlation, mean absolute error, and a Bland-Altman analysis between the mobility-derived rates and the survey rates. If the mean absolute error between the two exceeds 5 percentage points, or if the correlation is below 0.7, the reported MAE of 4.5% cannot be interpreted as accuracy against true evacuation behavior, and the central claim should be revised or the targets recalibrated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract defines tract-level evacuation rates as the prediction target but does not specify how these rates were measured. If, as is common, they come from mobile-device mobility data, the labels inherit systematic biases: devices may be detected moving out of a tract for reasons unrelated to evacuation (work commutes, tourism, voluntary departures before official orders), and evacuees may be missed when cellular networks fail during a disaster or when residents lack smartphones. The reported MAE of 4.5%/3.5% is relative to this unvalidated label. If the label is biased, the XGBoost model is fitting those biases, and the SHAP claim that hazard intensity dominates importance is an artifact of the label's noise structure rather than a statement about human behavior. The paper must show that the mobility-derived rates agree with an independent measure of evacuation, such as post-event surveys or traffic/shelter counts, in at least one event. Without this, the central claim that Evac-Cast predicts evacuation behavior is unsupported. The reader's secondary concern about representativeness of the four events is less immediate; even if the events are representative, invalid labels would sink the framework. This is a load-bearing gap because the entire interpretability and accuracy story rests on the meaning of the target variable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Evac-Cast, an interpretable machine-learning framework that predicts tract-level evacuation rates using XGBoost with more than 20 features across four dimensions (hazard intensity, community vulnerability, evacuation readiness, built environment). The authors report mean absolute errors of 4.5% for two hurricanes (Ian 2022, Milton 2024) and 3.5% for two wildfires (Kincade 2019, Palisades–Eaton 2025), and a SHAP-based feature hierarchy led by hazard intensity that is consistent across hazards. The claimed contribution is a survey-free, macro-level, interpretable tool for evacuation forecasting.","tokens_in":1018,"tokens_out":1637,"duration_ms":22502,"significance":"If the results hold, Evac-Cast would be a practically valuable decision-support tool for emergency management, filling a gap between costly survey-based behavioral studies and purely aggregate disaster models. The paper's strengths include the explicit focus on interpretability via SHAP, the use of multi-source data, the concrete error numbers on four named events, and the falsifiable claim that macro-level proxies can substitute for psychosocial survey variables. However, the significance currently rests on unverified assumptions about the target variable and the representativeness of the four events; the abstract alone does not establish that the reported accuracy and feature hierarchy generalize beyond these cases.","major_comments":[{"comment":"The abstract defines the prediction target as 'tract-level evacuation rates' but never states how these rates were measured. If they are derived from mobile-device mobility data, the labels can systematically misclassify non-evacuees (commuters, tourists, voluntary movers) and miss evacuees when networks fail. Since the reported MAEs (4.5% and 3.5%) and the SHAP feature hierarchy are computed relative to this target, the entire interpretability and accuracy story is conditional on the label's validity. The paper must provide evidence, ideally in at least one event, that the mobility-based rates agree with an independent measure of evacuation (e.g., post-event surveys, traffic counts, shelter registrations). Without such validation, the central claim that Evac-Cast predicts evacuation behavior is unsupported.","section":"Abstract, target definition"},{"comment":"The abstract reports point MAEs but gives no information on cross-validation, hyperparameter selection, error bars, or baseline comparisons. With only four events, the reported errors could reflect overfitting to event-specific characteristics or random chance in a small sample. The authors should report leave-one-event-out cross-validation, dispersion of errors across tracts and events, and comparisons against simple baselines (e.g., event-constant rates, demographic-only models). This is not a minor omission: it is necessary to assess whether the claimed accuracy is statistically meaningful.","section":"Abstract, model evaluation"},{"comment":"The paper opens with a claim that Evac-Cast predicts evacuation for hurricanes and wildfires generally, yet the evidence base is four named events. Two hurricanes and two wildfires cannot plausibly span the diversity of hazards, geographic regions, demographic compositions, and emergency-management protocols. The authors should either restrict the claim to a defined event envelope or provide explicit reasoning, with data, for why these four events are representative. A statement of limitations in the abstract would strengthen rather than weaken the paper.","section":"Abstract, generalizability claim"}],"minor_comments":[{"comment":"The term 'survey-free' is ambiguous: it likely means the model does not use survey-collected psychosocial variables, but the underlying mobility data may still require opt-in consent and aggregation. Please clarify what 'survey-free' means operationally.","section":"Abstract, 'survey-free' phrase"},{"comment":"The abstract does not specify how the features are constructed or which dimensions they belong to. A brief enumeration of feature types would help readers judge the plausibility of the macro-level proxy claim.","section":"Abstract, 'more than 20 features'"},{"comment":"The claim of consistency across hazards should be quantified: are the SHAP value orders identical, or merely similar? Reporting rank correlations or overlap metrics would make the claim falsifiable.","section":"Abstract, 'consistent feature importance hierarchy'"}],"recommendation":"major_revision","confidential_remarks":"This is an abstract-only review, so the assessment is necessarily provisional. The major concern is the unvalidated dependent variable; if the full text lacks independent label validation, the paper's central claim should be rejected or substantially downgraded. The representativeness concern is secondary but also load-bearing given the small number of events. I would encourage the editor to verify that the full manuscript contains cross-validation and baseline comparisons before sending for a full technical review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the abstract only, so this is a first-pass view. The interesting piece is the cross-hazard design: one XGBoost shared across two hurricanes and two wildfires, with a common feature set, and a consistent SHAP hierarchy led by hazard intensity. That is genuinely useful for an applied field that mostly works single-hazard. The survey-free pitch also matters if it holds up. Credit where due: the MAEs (4.5% and 3.5%) are plausible and the feature set covers the four dimensions you would want.\n\nThe soft spot is exactly where the stress-test note lands. The target—tract-level evacuation rate—is almost certainly derived from mobile-device mobility data, and the abstract says nothing about validating those labels against an independent measure. That is not a minor caveat. If the labels are biased (commuters counted as evacuees, evacuees missed when cell towers fail), then the model is fitting the bias, and the SHAP findings become statements about the noise structure rather than about evacuation behavior. The full text may well include label validation, but the abstract does not, and that is the load-bearing wall.\n\nTwo lesser concerns: no baselines are reported, so I have no sense whether the model beats a simple persistence or historical-average predictor. And four events is thin for the generalization claim, though I would treat that as a secondary issue. Error bars and cross-validation details are also absent, but again, those may appear in the full text.\n\nFor a reader, the value is clear: emergency managers want an interpretable, quick tool, and this framework is aimed at that. The paper deserves a serious referee, but the referee should require independent label validation for at least one event, baseline comparisons, and some form of event-level cross-validation. If those are in the current manuscript, the paper is fine. If they are absent, the central claim is unsupported. I would send it out, with those demands.","headline":"Evac-Cast is a plausible cross-hazard evacuation forecasting framework, but the central claim hinges on unvalidated mobility-derived labels; worth sending to referees who will insist on that validation.","tokens_in":1478,"tokens_out":1498,"would_cite":false,"duration_ms":20328,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Evac-Cast: an interpretable machine-learning framework that predicts tract-level evacuation rates within 4.5% error for hurricanes and 3.5% for wildfires, without survey data.","keywords":["evacuation forecasting","XGBoost","SHAP","hurricane evacuation","wildfire evacuation","tract-level prediction","interpretable machine learning","disaster management"],"falsifier":"Run Evac-Cast on a fifth disaster and benchmark its mobility-derived labels against household surveys of who actually evacuated; agreement would confirm the reported errors, while systematic disagreement would show the 4.5% and 3.5% figures measure label self-consistency rather than true evacuation accuracy.","tokens_in":590,"feed_emoji":"🚨","tokens_out":5658,"duration_ms":59588,"temperature":0.7,"pith_summary":"The paper introduces Evac-Cast, an interpretable machine-learning framework that forecasts how many people will evacuate from each census tract during a hurricane or wildfire. It claims that an XGBoost model using more than 20 features—spanning hazard intensity, community vulnerability, evacuation readiness, and built environment—achieves mean absolute errors of 4.5% for hurricanes and 3.5% for wildfires. The authors also report that SHAP analysis ranks hazard intensity as the most important predictor, and that the model works without explicit psychosocial survey variables. If correct, this would give emergency managers a fast, survey-free, high-resolution evacuation forecast built from data already available during a disaster.","feed_headline":"Survey-free ML model predicts evacuation rates within 4.5%","feed_subtitle":"The model covers hurricanes and wildfires using only hazard data, so managers get guidance without waiting for surveys.","key_machinery":"The mechanism is a gradient-boosted decision-tree model, XGBoost, trained on tract-level features grouped into four dimensions: hazard intensity, community vulnerability, evacuation readiness, and built environment. SHAP (SHapley Additive exPlanations) values are used to decompose each prediction and identify which features drive evacuation rates. The central work these tools do is to turn heterogeneous geospatial and demographic data into per-tract evacuation-rate predictions and to make those predictions interpretable by yielding a stable feature-importance ranking across events.","core_discovery":"On the paper's own terms, the central discovery is that tract-level evacuation behavior during hurricanes and wildfires can be predicted accurately from macro-level data alone, without individual-level surveys. Using evacuation rates derived from mobile-device mobility data as training targets, Evac-Cast's XGBoost model reaches mean absolute errors of 4.5% for hurricanes (Ian 2022 and Milton 2024) and 3.5% for wildfires (Kincade 2019 and Palisades–Eaton 2025). SHAP feature attribution shows a consistent importance hierarchy across both hazard types, led by hazard intensity and followed by the other three feature dimensions, which the authors interpret as evidence that behaviorally relevant signals are encoded in readily available macro-level proxies. The paper presents this as a step toward transparent, data-driven decision support for emergency management.","pith_inferences":["Beyond the paper, the same four-feature-dimension architecture could be tested on flood or earthquake evacuation, where mobility data and hazard-intensity maps are increasingly available.","A testable extension is to compare Evac-Cast's predictions against survey-reported evacuation intentions or post-disaster recall surveys; agreement would strengthen the claim that macro proxies encode genuine behavioral signals.","The paper's reliance on four high-profile events leaves open whether the 4.5% and 3.5% errors hold for smaller, slower-onset, or less-publicized disasters; an out-of-sample test on a fifth event would clarify the boundary."],"forward_implications":["Emergency managers could receive near-real-time evacuation forecasts for individual census tracts as a storm or fire approaches, without waiting for survey data.","The consistent dominance of hazard intensity in the SHAP rankings suggests that a single class of predictors—how dangerous the hazard is locally—carries most of the signal across different disaster types.","Because the model performs well without psychosocial survey variables, agencies without survey infrastructure can still obtain behaviorally meaningful evacuation estimates.","The framework transfers across hurricanes and wildfires with comparable error, pointing toward a unified evacuation-forecasting tool rather than hazard-specific models."],"supporting_citations":[],"fun_headline_variants":["Evac-Cast: survey-free ML predicts evacuations within 4.5%","Survey-free AI forecasts hurricane and wildfire evacuations","ML predicts evacuation rates for hurricanes and wildfires without surveys","Interpretable ML forecasts evacuations, no surveys needed","Evac-Cast: AI predicts evacuations for fires and storms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that evacuation rates measured from mobile-device mobility data reflect who actually evacuated, and that the four studied disasters stand in for all hurricanes and wildfires.","fun_headline_variants_meta":{"raw":{"variants":["Evac-Cast: survey-free ML predicts evacuations within 4.5%","Survey-free AI forecasts hurricane and wildfire evacuations","ML predicts evacuation rates for hurricanes and wildfires without surveys","Interpretable ML forecasts evacuations, no surveys needed","Evac-Cast: AI predicts evacuations for fires and storms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001139,"raw_usage":{"total_tokens":4713,"prompt_tokens":916,"completion_tokens":3797,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":3712}},"tokens_in":532,"tokens_out":3797,"duration_ms":29629,"temperature":1.0,"reasoning_tokens":3712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T06:00:32.959090+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Evac-Cast on a fifth disaster and benchmark its mobility-derived labels against household surveys of who actually evacuated; agreement would confirm the reported errors, while systematic disagreement would show the 4.5% and 3.5% figures measure label self-consistency rather than true evacuation accuracy.","supporting_citations":[],"review_version":1}