REVIEW 3 major objections 4 minor 33 references
Challenges and Research Directions from the Operational Use of a Machine Learning Damage Assessment System via Small Uncrewed Aerial Systems at Hurricanes Debby and Helene
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper reports the first known operational deployment of an sUAS-based machine learning damage assessment system at Hurricanes Debby and Helene, and identifies four challenges that delayed, degraded, or blocked its data products.
desk verdict The paper's real value is the operational field documentation of the first ML-based sUAS damage assessment deployment; the quantitative degradation claims are suggestive but under-supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a six-step inference pipeline: sUAS imagery is stitched into a georectified orthomosaic, overlaid with a priori building polygons from a public building footprint dataset, and passed to an Attention UNet - a segmentation network that emphasizes relevant image regions - trained on a benchmark dataset of georectified sUAS imagery to assign labels from the standard Joint Damage Scale to each building polygon; the labels are then converted to GEOJSON/KML files, summary statistics, and, after Hurricane Debby, CSV/PDF reports with street addresses. The same pipeline was run on 436 orthophotos from crewed aircraft in Pennsylvania. The orthomosaic, the polygons, the model, and the output formats are the objects whose real-world variability generates the four challenges.
What would settle it
A ground survey recording actual damage states for the buildings in the Dekle Beach and Mayo, Florida orthomosaics, compared against the model's labels on the full-resolution, aligned imagery, would settle whether the resolution- and misalignment-driven disagreements reflect true operational degradation or pre-existing model error; if the model is already inaccurate on clean inputs, the challenge claims lose their evidential support.
Extended reading notes
Core claim
The paper's central claim is that the first real-world, operationally used sUAS-based ML damage assessment system was deployed in the 2024 hurricane season, and that four concrete operational challenges - variation in spatial resolution, non-uniform spatial misalignment, limited wireless connectivity, and insufficient data product formats - each prevented, degraded, or delayed the delivery of its outputs. As evidence, it cites an 82.6% label disagreement between full-resolution and downsampled orthomosaics at Dekle Beach, a 39.1% label disagreement between aligned and misaligned building polygons, roughly one-day delays in every mission from data transmission, and the need to add CSV and PDF outputs because GIS specialists in emergency operations centers were task-saturated. The paper explicitly frames its contribution as an analysis of deployment, not a study of the accuracy or correctness of the system.
Load-bearing premise
The load-bearing premise is that the damage-labeling model is accurate enough that the observed 82.6% and 39.1% label disagreement rates can be read as evidence of operational degradation; the paper provides no ground-truth damage labels or accuracy estimates to anchor that reading.
Editorial extensions
If this is right
- Future ML damage assessment models should be trained and tested across ground sample distances spanning at least 1.65-25.3 cm/px, since operational imagery cannot be assumed to match curated training resolutions.
- Spatial misalignment between orthomosaics and building polygons must be corrected automatically before inference, because manual correction delays data-to-decision times and misalignment alone produced a 39.1% label disagreement rate.
- Field deployments should co-locate GPU compute with sUAS teams, because wireless transfer of gigabyte-scale imagery was the single largest source of delay in all missions.
- Data products should include plain formats such as CSV/PDF with street addresses, since GIS expertise is scarce and task-saturated in emergency operations centers.
- The three Florida orthomosaics and 436 Pennsylvania orthophotos provide a first operational benchmark for future sUAS-based ML damage assessment deployments.
Reading between the lines
- A testable extension the paper leaves implicit: the 82.6% and 39.1% disagreement rates only measure operational degradation if the base model is reasonably accurate on clean inputs, so future deployments should report ground-truth accuracy alongside disagreement rates.
- Because the same model ran on crewed 9-11 cm/px orthophotos, the resolution and misalignment challenges are likely not specific to sUAS and may affect any aerial ML damage assessment pipeline.
- The connectivity and resolution challenges are coupled: Dekle Beach imagery was downsampled specifically to transmit faster, so solving the wireless bottleneck may also soften the resolution-variation problem.
- A practical threshold the paper does not state: future work could identify a minimum usable ground sample distance for reliable damage labels, which would let responders decide what to transmit first when bandwidth is limited.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports on the operational deployment of a small uncrewed aerial system (sUAS) equipped with a machine learning damage assessment pipeline during Hurricanes Debby and Helene in Florida, and on the same model applied to crewed aircraft imagery in Pennsylvania. It claims this is the first known operational use of such a system where the outputs were used by emergency managers, and it identifies four challenges that prevented, degraded, or delayed delivery of ML data products: wide variation in spatial resolution, non-uniform spatial misalignment, limited wireless connectivity, and insufficient data product formats. The paper presents a mission table with ground sample distances, illustrative figures, qualitative operational observations, and label disagreement rates of 82.6% (resolution variation) and 39.1% (misalignment) as quantitative evidence, while explicitly disclaiming that the work is not a study of model accuracy.
Significance. If the claims hold, the paper is significant because concrete field reports of how data transmission constraints, GIS specialist workload, and data-format mismatches affect real disaster response are rare and valuable for both the robotics and machine learning communities. The qualitative observations—wireless-transfer delays, task-saturated GIS specialists, manual misalignment correction, and the need for CSV/PDF outputs—are credible and do not depend on the model-noise assumption. The paper is also honest in stating upfront that it is not an accuracy study. However, the quantitative degradation claims rest on small, uncontrolled comparisons without baseline model stability or sample sizes, so the strength of those specific sub-claims is not yet established.
major comments (3)
- [Sec. IV-A/B] The 82.6% and 39.1% label disagreement rates are interpreted as evidence that spatial resolution and spatial misalignment degraded model outputs. Because the paper states in Sec. I that it is 'not a study of the accuracy or correctness' and provides no ground-truth labels, no repeatability experiments, and no control condition, these rates could reflect the deployed Attention UNet's intrinsic sensitivity to small input shifts rather than the operational challenges themselves. Please add a baseline: for example, repeated inference on identical input, small random translations of aligned building polygons, or resampling the full-resolution orthomosaic to a matched GSD without changing other content, and report the disagreement rate under that control. Without this, the quantitative 'degraded' claims are underdetermined.
- [Sec. IV-A/B and Fig. 4] The sample sizes behind the quantitative claims are not reported. The 6-of-7 label disagreement in Fig. 4 is a seven-building sample, and no N is given for the 82.6% or 39.1% rates. Please report the number of buildings compared for each rate, the per-class label counts, and, where feasible, a confidence interval or a permutation test comparing the observed disagreement against the control condition from the previous comment. Without these, the reader cannot assess the statistical weight of the numbers.
- [Sec. IV-A] The statement that the downsampled Dekle Beach orthomosaic produced model outputs that were 'deemed invalid by the authors and not transmitted' is used as evidence that resolution variation prevented delivery of a data product. The criteria for 'invalid' are not defined. Please specify what made the outputs invalid (e.g., a threshold on label disagreement, visual inspection of predictions, or comparison with the full-resolution outputs) and how that decision was made, so that the claim can be evaluated and replicated.
minor comments (4)
- [Sec. IV-A and Fig. 3 caption] The Fig. 3 caption says 'the imagery on the left is 15x lower spatial resolution than the left,' which should read 'than the right.' In addition, the caption gives the Dekle Beach GSD as 23.5 cm/px while Table I lists 25.3 cm/px and the text says 25.3 cm/px; please reconcile these values.
- [Sec. II] The 'first known deployment' claim is hedged with 'suggests,' but the criteria for what counts as an operational use by emergency managers and the scope of the literature search are not stated. A brief definition and search description would make the claim falsifiable.
- [Sec. IV-C] The claim that transmission delays were 'approximately one day' in all instances is asserted without supportive timestamp data. A small table of mission-to-inference timestamps or a log excerpt would strengthen this claim.
- [Sec. IV-D] The phrase 'limited resources to visualize and comprehend such formats' is vague; please specify whether the limitation was software, hardware, or trained personnel, as this affects the recommendation about data product formats.
Circularity Check
No significant circularity: the paper is an empirical operational case study; its four challenges are field observations, and the label-disagreement evidence is an interpretive step subject to validity caveats, not a definitional reduction.
full rationale
The paper does not derive a quantity from a fitted parameter or invoke a uniqueness theorem. Its central contribution, four operational challenges, rests on direct field observations (GSD statistics, transmission delays, GIS specialist task saturation, format changes) and on computed disagreement rates between model outputs under different inputs. The disagreement rates (82.6% in Sec. IV-A and 39.1% in Sec. IV-B) are not predictions forced by construction: they are observed outputs of an Attention UNet on altered inputs, and interpreting them as degradation assumes model reliability, which the paper explicitly disclaims in Sec. I: 'it is not a study of the accuracy or correctness of the sUAS-based ML damage assessment system.' That is a completeness or validity caveat, not circularity. The self-citations to CRASAR-U-DROIDs [24] and the misalignment-adjustment process [23] supply components of the deployed system, but the operational findings do not reduce to those citations; the same challenges would be observable regardless of which model was used. The absence of a baseline disagreement rate for the model is a missing-support concern that should be weighed as correctness risk, but no step exhibits the reduction-by-construction required by the circularity standard.
Assumptions & free parameters
assumptions (3)
- domain assumption The Attention UNet trained on CRASAR-U-DROIDs produces labels reliable enough to be used as evidence of operational degradation.
- domain assumption Label disagreement between two runs of the same model is a valid proxy for model performance degradation.
- domain assumption The literature review in Sec. II is complete enough to support the 'first deployment' claim.
Cite this review
Pith. "Pith review of Challenges and Research Directions from the Operational Use of a Machine Learning Damage Assessment System via Small Uncrewed Aerial Systems at Hurricanes Debby and Helene." pith.science (2026). https://pith.science/paper/HTPYFGWP
@misc{pith2026250615890,
author = {Pith},
title = {Pith review of: Challenges and Research Directions from the Operational Use of a Machine Learning Damage Assessment System via Small Uncrewed Aerial Systems at Hurricanes Debby and Helene},
year = {2026},
howpublished = {\url{https://pith.science/paper/HTPYFGWP}},
note = {Machine review of arXiv:2506.15890}
}
read the original abstract
This paper details four principal challenges encountered with machine learning (ML) damage assessment using small uncrewed aerial systems (sUAS) at Hurricanes Debby and Helene that prevented, degraded, or delayed the delivery of data products during operations and suggests three research directions for future real-world deployments. The presence of these challenges is not surprising given that a review of the literature considering both datasets and proposed ML models suggests this is the first sUAS-based ML system for disaster damage assessment actually deployed as a part of real-world operations. The sUAS-based ML system was applied by the State of Florida to Hurricanes Helene (2 orthomosaics, 3.0 gigapixels collected over 2 sorties by a Wintra WingtraOne sUAS) and Debby (1 orthomosaic, 0.59 gigapixels collected via 1 sortie by a Wintra WingtraOne sUAS) in Florida. The same model was applied to crewed aerial imagery of inland flood damage resulting from post-tropical remnants of Hurricane Debby in Pennsylvania (436 orthophotos, 136.5 gigapixels), providing further insights into the advantages and limitations of sUAS for disaster response. The four challenges (variationin spatial resolution of input imagery, spatial misalignment between imagery and geospatial data, wireless connectivity, and data product format) lead to three recommendations that specify research needed to improve ML model capabilities to accommodate the wide variation of potential spatial resolutions used in practice, handle spatial misalignment, and minimize the dependency on wireless connectivity. These recommendations are expected to improve the effective operational use of sUAS and sUAS-based ML damage assessment systems for disaster response.
Figures
Reference graph
Works this paper leans on
-
[1]
2024 atlantic hurricane season,
“2024 atlantic hurricane season,” https://www.nhc.noaa.gov/data/tcr/ index.php?season=2024&basin=atl, accessed: 2025-03-18
work page 2024
-
[2]
Microsoft us building footprints,
“Microsoft us building footprints,” https://github.com/Microsoft/USBuildingFootprints, 2021
work page 2021
-
[3]
J. Adams, L. Hart, J. McBride, D. Merrick, and R. Murphy, “Use of small unmanned aerial systems for tactical response during kilauea volcano lower east rift zone event,” in 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) . IEEE, 2018, pp. 1–2
work page 2018
-
[4]
Big data and disaster management: a systematic review and agenda for future research,
S. Akter and S. F. Wamba, “Big data and disaster management: a systematic review and agenda for future research,” Annals of Operations Research, vol. 283, no. 1, pp. 939–959, 2019. [Online]. Available: https://doi.org/10.1007/s10479-017-2584-2
-
[5]
S. Al Shafian and D. Hu, “Integrating machine learning and remote sensing in disaster management: A decadal review of post-disaster building damage assessment,” Buildings, vol. 14, no. 8, p. 2344, 2024
work page 2024
-
[6]
Disaster and pandemic management using machine learning: a survey,
V . Chamola, V . Hassija, S. Gupta, A. Goyal, M. Guizani, and B. Sik- dar, “Disaster and pandemic management using machine learning: a survey,” IEEE Internet of Things Journal , vol. 8, no. 21, pp. 16 047– 16 071, 2020
work page 2020
-
[7]
Dorianet: A visual dataset from hurricane dorian for post-disaster building damage assessment,
C.-S. Cheng, A. H. Behzadan, and A. Noshadravan, “Dorianet: A visual dataset from hurricane dorian for post-disaster building damage assessment,” DesignSafe-CI, 2021
work page 2021
-
[8]
N. R. Council, D. on Earth, L. Studies, B. on Earth Sciences, Resources, M. S. Committee, C. on Planning for Catastrophe, A. B. for Improving Geospatial Data, Tools, and Infrastructure, Successful response starts with a map: improving geospatial support for disaster management. National Academies Press, 2007
work page 2007
Show all 33 references
-
[9]
Applications of drone in disaster management: A scoping review,
S. M. S. M. Daud, M. Y . P. M. Yusof, C. C. Heo, L. S. Khoo, M. K. C. Singh, M. S. Mahmood, and H. Nawawi, “Applications of drone in disaster management: A scoping review,” Science & Justice , vol. 62, no. 1, pp. 30–42, 2022
2022
-
[10]
Quantitative data analysis: Crasar small unmanned aerial systems at hurricane harvey,
O. Fernandes, R. Murphy, J. Adams, and D. Merrick, “Quantitative data analysis: Crasar small unmanned aerial systems at hurricane harvey,” in 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) . IEEE, 2018, pp. 1–6
2018
-
[11]
Quantitative data analysis: Small unmanned aerial systems at hurricane michael,
O. Fernandes, R. Murphy, D. Merrick, J. Adams, L. Hart, and J. Broder, “Quantitative data analysis: Small unmanned aerial systems at hurricane michael,” in 2019 IEEE international symposium on safety, security, and rescue robotics (SSRR) . IEEE, 2019, pp. 116– 117
2019
-
[12]
Computer vision-based post-disaster needs assessment from low altitude aerial imagery,
R. A. Garc ´ıa Franceschini, “Computer vision-based post-disaster needs assessment from low altitude aerial imagery,” Ph.D. dissertation, Massachusetts Institute of Technology, 2021
2021
-
[13]
xbd: A dataset for as- sessing building damage from satellite imagery,
R. Gupta, R. Hosfelt, S. Sajeev, N. Patel, B. Goodman, J. Doshi, E. Heim, H. Choset, and M. Gaston, “xbd: A dataset for as- sessing building damage from satellite imagery,” arXiv preprint arXiv:1911.09296, 2019
1911 arXiv
-
[14]
Drones for flood monitoring, mapping and detection: A bibliometric review,
U. Iqbal, M. Z. B. Riaz, J. Zhao, J. Barthelemy, and P. Perez, “Drones for flood monitoring, mapping and detection: A bibliometric review,” Drones, vol. 7, no. 1, p. 32, 2023
2023
-
[15]
Uav-based structural damage mapping: A review,
N. Kerle, F. Nex, M. Gerke, D. Duarte, and A. Vetrivel, “Uav-based structural damage mapping: A review,” ISPRS international journal of geo-information, vol. 9, no. 1, p. 14, 2019
2019
-
[16]
Advances in ai and drone-based natural disaster management: A survey,
M. Krichen and M. S. Abdalzaher, “Advances in ai and drone-based natural disaster management: A survey,” in 2023 20th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA). IEEE, 2023, pp. 1–6
2023
-
[17]
Machine learning for emergency management: A survey and future outlook,
C. Kyrkou, P. Kolios, T. Theocharides, and M. Polycarpou, “Machine learning for emergency management: A survey and future outlook,” Proceedings of the IEEE , vol. 111, no. 1, pp. 19–41, 2022
2022
-
[18]
Machine learning in disaster management: recent developments in methods and applications,
V . Linardos, M. Drakaki, P. Tzionas, and Y . L. Karnavas, “Machine learning in disaster management: recent developments in methods and applications,” Machine Learning and Knowledge Extraction , vol. 4, no. 2, 2022
2022
-
[19]
Data collection tools for post-disaster damage assessment of building and lifeline infrastructure systems,
J.-M. Lozano and I. Tien, “Data collection tools for post-disaster damage assessment of building and lifeline infrastructure systems,” International Journal of Disaster Risk Reduction , vol. 94, p. 103819, 2023. [Online]. Available: https://www.sciencedirect.com/science/articl...
2023
-
[20]
Effect of label noise in semantic segmentation of high resolution aerial images and height data,
A. Maiti, S. Oude Elberink, and G. V osselman, “Effect of label noise in semantic segmentation of high resolution aerial images and height data,” ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences , vol. 2, pp. 275–282, 2022
2022
-
[21]
Quantitative data analysis: Crasar small unmanned aerial systems at hurricane ian,
T. Manzini, R. Murphy, and D. Merrick, “Quantitative data analysis: Crasar small unmanned aerial systems at hurricane ian,” in 2023 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). IEEE, 2023, pp. 7–12
2023
-
[22]
Wireless network demands of data products from small uncrewed aerial systems at hur- ricane ian,
T. Manzini, R. Murphy, D. Merrick, and J. Adams, “Wireless network demands of data products from small uncrewed aerial systems at hur- ricane ian,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2023, pp. 9941–9946
2023
-
[23]
Non-uniform spatial alignment errors in suas imagery from wide-area disasters,
T. Manzini, P. Perali, R. Karnik, M. Godbole, H. Abdullah, and R. Murphy, “Non-uniform spatial alignment errors in suas imagery from wide-area disasters,” arXiv preprint arXiv:2405.06593 , 2024
2024 arXiv
-
[24]
Crasar-u- droids: A large scale benchmark dataset for building alignment and damage assessment in georectified suas imagery,
T. Manzini, P. Perali, R. Karnik, and R. Murphy, “Crasar-u- droids: A large scale benchmark dataset for building alignment and damage assessment in georectified suas imagery,” arXiv preprint arXiv:2407.17673, 2024
2024 arXiv
-
[25]
Now you see it, now you don’t: Damage label agreement in drone & satellite post-disaster imagery,
T. Manzini, P. Perali, J. Tripathi, and R. Murphy, “Now you see it, now you don’t: Damage label agreement in drone & satellite post-disaster imagery,” arXiv preprint arXiv:2505.08117 , 2025
2025 arXiv
-
[26]
Drones4good: Supporting disaster relief through remote sensing and ai,
N. Merkle, R. Bahmanyar, C. Henry, S. M. Azimi, X. Yuan, S. Schopferer, V . Gstaiger, S. Auer, A. Schneibel, M. Wieland et al., “Drones4good: Supporting disaster relief through remote sensing and ai,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, ...
2023
-
[27]
Crew roles and operational protocols for rotary-wing micro-uavs in close urban environments,
R. R. Murphy, K. S. Pratt, and J. L. Burke, “Crew roles and operational protocols for rotary-wing micro-uavs in close urban environments,” in Proceedings of the 3rd ACM/IEEE international conference on Human robot interaction, 2008, pp. 73–80
2008
-
[28]
Cooperative use of unmanned sea surface and micro aerial vehicles at hurricane wilma,
R. R. Murphy, E. Steimle, C. Griffin, C. Cullins, M. Hall, and K. Pratt, “Cooperative use of unmanned sea surface and micro aerial vehicles at hurricane wilma,” Journal of Field Robotics , vol. 25, no. 3, pp. 164–180, 2008
2008
-
[29]
Robot-assisted bridge inspection after hurricane ike,
R. R. Murphy, E. Steimle, M. Hall, M. Lindemuth, D. Trejo, S. Hurlebaus, Z. Medina-Cetina, and D. Slocum, “Robot-assisted bridge inspection after hurricane ike,” in 2009 IEEE International Workshop on Safety, Security & Rescue Robotics (SSRR 2009). IEEE, 2009, pp. 1–5
2009
-
[30]
Attention u-net: Learning where to look for the pancreas,
O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y . Hammerla, B. Kainz et al., “Attention u-net: Learning where to look for the pancreas,” arXiv preprint arXiv:1804.03999, 2018
2018 arXiv
-
[31]
Analysis of interior rubble void spaces at champlain towers south collapse,
A. Rao, R. Murphy, D. Merrick, and H. Choset, “Analysis of interior rubble void spaces at champlain towers south collapse,” in 2022 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). IEEE, 2022, pp. 379–384
2022
-
[32]
Scale-mae: A scale-aware masked autoencoder for multiscale geospatial repre- sentation learning,
C. J. Reed, R. Gupta, S. Li, S. Brockman, C. Funk, B. Clipp, K. Keutzer, S. Candido, M. Uyttendaele, and T. Darrell, “Scale-mae: A scale-aware masked autoencoder for multiscale geospatial repre- sentation learning,” in Proceedings of the IEEE/CVF International Conference on Co...
2023
-
[33]
Correcting rural building annotations in openstreetmap using convolutional neural networks,
J. E. Vargas-Mu ˜noz, S. Lobry, A. X. Falc ˜ao, and D. Tuia, “Correcting rural building annotations in openstreetmap using convolutional neural networks,” ISPRS journal of photogrammetry and remote sensing, vol. 147, pp. 283–293, 2019
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.