Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix

Axel Brando; Caroline K\"onig; Jaume Abella; Unai Gurbindo

REVIEW 3 major objections 5 minor 19 references

Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read InstructPix2Pix weather edits improve object detection in fog, rain, and night—with model-dependent limits.

desk verdict A clean empirical study of InstructPix2Pix weather augmentation; the idea is promising for two-stage detectors, but the headline gains rest on single training runs and may not clear training noise. read the letter →

arxiv 2505.08228 v2 pith:5VBHXQMD submitted 2025-05-13 cs.CV cs.AI

Unai Gurbindo , Axel Brando , Jaume Abella , Caroline König This is my paper · ORCID

classification cs.CVcs.AI

keywords adverseweatherobjectdetectiondataaugmentationInstructPix2PixdiffusionmodelsautonomousdrivingFasterR-CNNYOLOv10

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that diffusion-model image editing can act as a practical data augmentation for object detectors operating in adverse weather. The authors use InstructPix2Pix with sequential text prompts to transform clear-weather driving images into foggy, rainy, snowy, or nighttime versions, then train Faster R-CNN and YOLOv10 models on the augmented data. In a controlled CARLA simulator setting, the augmentation consistently improves mAP50 under fog, night, and rain for all three detectors tested. On real-world data, the same approach improves Faster R-CNN under adverse conditions but does not clearly help the YOLO models, a limitation the paper connects to the YOLO models' lower baseline stability. The central contribution is evidence that text-guided diffusion edits can narrow the weather-induced performance gap when combined with a filtering step that removes hallucinated images.

What carries the argument

The central object is the InstructPix2Pix diffusion model, which performs localized image editing from paired image and text inputs. The method works by applying sequential text prompts, such as "What would it look like if it were raining a lot?" followed by "Add raindrops on the camera lens," with deliberately low guidance scales to preserve semantic content, and then a manual filtering step removes hallucinated outputs. The evaluation machinery is bootstrap resampling: 1,000 samples of the test set per weather condition are drawn with replacement, and the mean and standard deviation of mAP50 are reported.

What would settle it

Take a random sample of the augmented images produced by the pipeline and compare each one against its source image to check whether every originally annotated object still exists at the same location with the same class. If a substantial fraction of augmented images show moved, merged, or missing objects, then the reported mAP gains could be partially an artifact of invalid labels rather than true robustness improvement; a second check would be to train on the unfiltered augmented set and compare with the filtered set to measure how much the manual filtering step contributes.

Watch

Extended reading notes

Core claim

The central claim is that augmenting clear-weather training images with InstructPix2Pix weather edits reduces the object-detection performance gap under fog, night, rain, and snow, provided that a manual filtering step removes augmented images where the model hallucinated missing or distorted objects. The evidence is a systematic comparison of mAP50 between a Basic dataset and an Augmented dataset across two frameworks. In the simulated CARLA framework, the augmentation improves all three detectors under every adverse condition tested, with the largest gains often in fog and night. In the real-world framework using BDD100K for training and ACDC for testing, Faster R-CNN shows improved robustness under adverse weather—for example, roughly a 0.07 mAP50 gain for pedestrians in night and rain—while YOLOv10-M and YOLOv10-N show no clear benefit, and the paper attributes this to the need for further investigation into their training process.

Load-bearing premise

The method assumes that InstructPix2Pix edits preserve the location, identity, and class of every annotated object, so that the original bounding boxes remain valid on the augmented images; if an edit shifts, merges, or erases an object, the detector is trained on mislabeled data.

Editorial extensions

If this is right

In simulated environments, augmenting clear-weather training with fog, night, and rain edits narrows the mAP50 gap for Faster R-CNN, YOLOv10-M, and YOLOv10-N, with the largest gains typically in fog and night conditions.
On real-world data, Faster R-CNN benefits from the augmentation across fog, night, rain, and snow, reducing the gap between default and adverse-condition performance.
For YOLOv10 models, the same augmentation does not yield a clear robustness benefit on real-world data, indicating that the effectiveness of diffusion-based weather augmentation depends on the detector architecture and training stability.
The inclusion of a manual filtering step to discard hallucinated images is a necessary part of the pipeline, as unfiltered augmentations could introduce mislabeled training data.
The approach transfers from simulated to real images, suggesting that the prompting strategy itself is not tied to the source domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The method could be tested as a general robustness tool for other perception tasks, such as semantic segmentation or multi-object tracking, where weather shifts also degrade performance.
The manual filtering step could likely be automated with a learned hallucination detector; the paper itself points toward multimodal language models as a possible filter, which would make the pipeline fully automated.
For the YOLO models, the lack of benefit on real data may reflect training configuration rather than a fundamental limitation of the augmentation, so a fair test would involve tuning learning rate, epochs, or augmentation ratios specifically for YOLO on the augmented set.
Combining InstructPix2Pix weather edits with standard photometric augmentations like brightness, contrast, and noise might produce compounding gains, but the paper does not test this interaction.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The paper tests a straightforward idea: use InstructPix2Pix with hand-crafted prompts to turn clear-weather images into fog, rain, night, and snow versions, then train detectors on the mix. That's genuinely new as an application, and the experiments are more careful than most augmentation papers—separate train/test, two frameworks (CARLA and BDD100K→ACDC), bootstrap resampling of the test sets, and per-class AP alongside mAP. They're also honest in the discussion: YOLOv10 models don't benefit in the real-world framework, and they flag the manual filtering as a limitation.

The soft spot is exactly what the stress-test note says. The bootstrap only resamples test images, so the reported means and stds capture test-set composition, not training stochasticity. Everything is one training run per model/dataset. Some headline gains are small: Faster R-CNN simulated rain +0.008, real-world fog +0.020, rain +0.027, snow +0.022. Those are within a standard deviation of the bootstrap spread, and typical detector training has seed-to-seed variation at that scale. The paper even concedes models are not deterministic, but never quantifies it. So the abstract's 'significantly enhance' is not supported. The proper conclusion is 'may improve robustness for two-stage detectors; the effect on one-stage YOLO is unclear,' which is what the body actually says.

There are two other issues worth naming, both secondary. No comparison to prior weather-augmentation methods (style transfer, other generative augmentations, or simply adding real adverse-weather data), so we don't know how this stacks up. And the ACDC test labels for traffic signs/lights are derived from semantic segmentation masks, which is a plausible stopgap but a weaker ground truth than manual box annotations.

None of this is fatal. The method is transparent, reproducible in structure, and the design is good enough that a serious referee could guide it into a solid contribution. I'd want multi-seed training, a proper significance test, and at least one baseline augmentation before the claims are publishable as stated. But it deserves peer review, not a desk reject.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes a data-augmentation pipeline in which clear-weather driving images are edited with InstructPix2Pix using hand-crafted text prompts to simulate rain, fog, night, and snow, and the edited images are added to the training set of object detectors. The pipeline is evaluated in two frameworks: simulated CARLA images (Faster R-CNN, YOLOv10-M, YOLOv10-N) and real-world BDD100K training with ACDC testing. The reported results show mAP50 improvements under adverse conditions for all models in the simulated framework and for Faster R-CNN in the real-world framework, while YOLOv10 models show no clear real-world benefit. The paper concludes that the augmentation strategy significantly enhances robustness.

Significance. The empirical setup is well structured in several respects: train and test sets are separated by scenario, evaluation is reported per weather condition, and a bootstrap resampling of test images is used to quantify sampling variability. The manual filtering of hallucinated augmentations is a practical safeguard, and the authors honestly acknowledge the limited YOLO benefit in the real-world framework. However, the central claim of significant robustness enhancement is currently not fully supported because the only uncertainty quantified is test-set composition, not training stochasticity, and the abstract and contribution statements overstate the uniformity of the benefit. With additional seed variability analysis and a more precise significance assessment, the contribution could become a useful benchmark for diffusion-based weather augmentation in object detection.

major comments (3)

[V.C, Tables I and VII] The bootstrap procedure resamples only test images, so the reported standard deviations do not reflect the variability of training the detector. Each model appears to be trained once per dataset (Section V.B), yet Section V.C states that models are not entirely deterministic. Consequently, small gains such as +0.020 for real-world fog or +0.027 for real-world rain in Table VII cannot be distinguished from seed-to-seed training noise. Please report results over multiple training seeds and provide a significance test (e.g., paired comparison across seeds) or explicitly restrict the claims to test-set sampling variability.
[Abstract and Section VI] The abstract and conclusion claim that the data augmentation strategy 'significantly enhances' robustness, but the evidence is mixed. In the real-world framework, YOLOv10-M and YOLOv10-N show no improvement on most adverse conditions (Tables VIII and IX: night and snow are often worse with Augmented than Basic), and the paper itself acknowledges this in Section V.D. The claims should be softened to reflect that the benefit is model-dependent and, in the real-world case, observed mainly for Faster R-CNN.
[IV.A, Algorithm 1 Step 3] The validity of the augmented training data depends on the assumption that InstructPix2Pix edits preserve the location, identity, and class of every annotated object, so that original bounding boxes remain correct on edited images. The only safeguard is the manual filtering step for 'hallucinations', but its criteria are not specified, no count of removed images is reported, and no quantitative verification of label preservation is provided. Please add a validation procedure (for example, comparing detections or semantic labels on original vs edited images) and report filtering statistics; otherwise silent mislabels may be introduced into the training set and bias the comparison.

minor comments (5)

[V.D] The text describing the default-condition results does not match Table II: it says Faster R-CNN and YOLO Nano achieve better results with Basic, but Table II shows YOLOv10-M with Augmented (0.550) outperforming Basic (0.536). Please correct the description to cover all models consistently.
[III.B] Traffic sign and traffic light bounding boxes in ACDC are derived from semantic segmentation masks using regionprops. This procedure may produce boxes that do not tightly enclose the actual objects (e.g., merged or fragmented regions). Please add a note on the expected accuracy of these derived labels, since they are used to compute AP for those classes on the real-world test set.
[V.A, Real-World Datasets] The real-world training set description is internally inconsistent: 7,083 images split 70/30 would give about 4,958 training and 2,125 validation images, but the text later describes the Augmented dataset as containing 2,500 default training and 1,043 default validation images. Please clarify the actual composition and size of the Basic and Augmented training sets.
[V.A, Simulated Datasets] The final augmented-image counts after filtering (4,270 fog, 4,484 rain, 4,780 night) are described together with an intended 20% per condition allocation of 673 images per condition. Please explain how the post-filtering counts relate to the stated dataset composition.
[References] Reference [6] is titled 'ISO 26262. Road Vehicles — Safety of the Intended Functionality,' which appears to be a bibliographic error: the safety-of-the-intended-functionality standard is ISO 21448. Please verify and correct the citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central conclusion is a direct empirical comparison and does not derive its result from its own inputs by construction.

full rationale

The paper's core claim is an empirical comparison: object detectors are trained on a clear-weather Basic dataset versus an Augmented dataset containing InstructPix2Pix weather edits, and both are evaluated on held-out test sets under each weather condition (Sections V.A-V.D). No equation defines the target mAP result from input parameters, and no fitted parameter is renamed as a prediction. The prompt formulations are inspired by the external InstructPix2Pix paper [4], and the guidance scales and inference steps are fixed design choices, not fit to the test outcomes. The manual filtering step removes hallucinated or extreme augmentations, which is a data-quality safeguard rather than a mechanism that forces the measured improvements. The bootstrap procedure in Section V.C resamples test images and therefore quantifies test-set composition uncertainty, not training-seed variability; this is a statistical validity concern, not circularity. The only apparent self-citation is reference [8] (Fernandez et al., some of whose authors may overlap with the present paper) used in Section II.A to describe ISO/PAS 8800 development processes; this is background context and is not load-bearing for the experimental conclusion. Consequently, no specific step in the paper reduces to its own inputs by definition, and the honest finding is that the derivation chain is self-contained as an empirical study.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions, most importantly that the generative edits preserve object labels and that the derived ACDC annotations are reliable. The hand-chosen augmentation hyperparameters (guidance scales, inference steps, and mixing ratios) are free parameters that affect all reported results.

free parameters (7)

Guidance scale for rain prompt 1 = 1.45 (sim), 1.35 (real)
Controls prompt adherence for the 'raining a lot' edit; chosen by visual inspection, not by optimizing detection mAP.
Guidance scale for rain prompt 2 = 1.65 (sim), 2.0 (real)
Second sequential prompt for lens raindrops; chosen by visual inspection.
Guidance scale for fog prompt = 1.9 (sim and real)
Chosen by visual inspection to create dense fog while limiting hallucinations.
Guidance scale for night prompts = 1.5 then 1.75 (both frameworks)
Two sequential prompts for night and extra darkness; chosen by visual inspection.
Guidance scale for snow prompts = 1.25 then 1.5 (real only)
Two sequential prompts for snow; chosen by visual inspection.
Inference steps per prompt = 100, 125, 150, 200, or 250 depending on prompt
Higher steps improve image quality at compute cost; chosen by hand.
Proportion of augmented images in training set = 60% (sim), 50% (real)
Chosen by hand; directly affects the trade-off between default and adverse-weather performance.

assumptions (4)

domain assumption InstructPix2Pix generated weather images retain spatial and semantic consistency so that original bounding boxes remain valid
Stated in Section I and relied on by Algorithm 1; manual filtering only partially enforces this assumption.
domain assumption ACDC-derived bounding boxes for traffic lights and signs, obtained from semantic segmentation masks, are accurate enough for evaluation
Section III.B; these labels are not human-verified and may introduce noise into the mAP measurements.
standard math Bootstrap resampling with replacement yields valid estimates of mAP variability
Section V.C; standard statistical practice, though the test sets are not i.i.d. samples of real driving conditions.
domain assumption Pretrained COCO weights and the chosen fine-tuning procedure transfer to the BDD100K and ACDC domains
Section V.B; standard practice but not validated against other initialization schemes.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix." pith.science (2026). https://pith.science/paper/5VBHXQMD

@misc{pith2026250508228,
  author       = {Pith},
  title        = {Pith review of: Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VBHXQMD}},
  note         = {Machine review of arXiv:2505.08228}
}

read the original abstract

Enhancing the robustness of object detection systems under adverse weather conditions is crucial for the advancement of autonomous driving technology. This study presents a novel approach leveraging the diffusion model Instruct Pix2Pix to develop prompting methodologies that generate realistic datasets with weather-based augmentations aiming to mitigate the impact of adverse weather on the perception capabilities of state-of-the-art object detection models, including Faster R-CNN and YOLOv10. Experiments were conducted in two environments, in the CARLA simulator where an initial evaluation of the proposed data augmentation was provided, and then on the real-world image data sets BDD100K and ACDC demonstrating the effectiveness of the approach in real environments. The key contributions of this work are twofold: (1) identifying and quantifying the performance gap in object detection models under challenging weather conditions, and (2) demonstrating how tailored data augmentation strategies can significantly enhance the robustness of these models. This research establishes a solid foundation for improving the reliability of perception systems in demanding environmental scenarios, and provides a pathway for future advancements in autonomous driving.

Figures

Figures reproduced from arXiv: 2505.08228 by the authors.

**Figure 2.** Example of data augmentation in a simulated data framework to [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Example of hallucinations generated and corrected after applying data [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Example of hallucinations generated and corrected after applying data [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 9 canonical work pages

[1]

Autonomous driving system: a comprehensive sur- vey,

J. Zhao et al., “Autonomous driving system: a comprehensive sur- vey,” Expert Systems with Applications , vol. 242, p. 122836, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0957417423033389

work page 2024
[2]

Faster R-CNN: towards real- time object detection with region proposal networks,

S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: towards real- time object detection with region proposal networks,” 2016. [Online]. Available: https://arxiv.org/abs/1506.01497

arXiv 2016
[3]

YOLOv10: real-time end-to-end object detection,

A. Wang et al., “YOLOv10: real-time end-to-end object detection,” 2024. [Online]. Available: https://arxiv.org/abs/2405.14458

arXiv 2024
[4]

InstructPix2Pix: learning to follow image editing instructions,

T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: learning to follow image editing instructions,” 2023. [Online]. Available: https: //arxiv.org/abs/2211.09800

arXiv 2023
[5]

Road Vehicles — Functional Safety (Edition 2) , 2018

International Standards Organization, ISO 26262. Road Vehicles — Functional Safety (Edition 2) , 2018

work page 2018
[6]

Road Vehicles — Safety of the Intended Functionality , 2022

——, ISO 26262. Road Vehicles — Safety of the Intended Functionality , 2022

work page 2022
[7]

Road Vehicles — Safety and Artificial Intelligence, 2024

——, ISO/PAS 8800. Road Vehicles — Safety and Artificial Intelligence, 2024

work page 2024
[8]

AI-FSM: Towards Functional Safety Management for Artificial Intelligence-based Critical Systems,

J. Fern ´andez et al., “AI-FSM: Towards Functional Safety Management for Artificial Intelligence-based Critical Systems,” inCARS@EDCC2024 Workshop - Critical Automotive applications: Robustness & Safety , Leuven, Belgium, Apr. 2024. [Online]. Available: https://hal.science/ hal-04769949

work page 2024

Show all 19 references

[9]

A comprehensive review of YOLO architectures in computer vision: from YOLOv1 to YOLOv8 and YOLO-NAS,

J. Terven, D.-M. C ´ordova-Esparza, and J.-A. Romero-Gonz ´alez, “A comprehensive review of YOLO architectures in computer vision: from YOLOv1 to YOLOv8 and YOLO-NAS,” Machine Learning and Knowl- edge Extraction, vol. 5, no. 4, pp. 1680–1716, 2023. [Online]. Available: http://...

2023 doi
[10]

Diffusion models: a comprehensive survey of methods and applications,

L. Yang et al., “Diffusion models: a comprehensive survey of methods and applications,” 2024. [Online]. Available: https://arxiv.org/abs/2209. 00796

2024
[11]

Denoising diffusion probabilistic models,

J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,”
[12]

High- resolution image synthesis with latent diffusion models,

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” 2022. [Online]. Available: https://arxiv.org/abs/2112.10752

2022 arXiv
[13]

CARLA: an open urban driving simulator,

A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: an open urban driving simulator,” 2017. [Online]. Available: https://arxiv.org/abs/1711.03938

2017 arXiv
[14]

Proceedings of the 10th International Conference on Vehicle Technology and Intelligent Transport Systems – VEHITS,

M. Chaar, J. Raiyn, and G. Weidl, “Proceedings of the 10th International Conference on Vehicle Technology and Intelligent Transport Systems – VEHITS,” INSTICC, SciTePress, 2024, pp. 267–275

2024
[15]

BDD100K: a diverse driving dataset for heterogeneous multitask learning,

F. Yu et al., “BDD100K: a diverse driving dataset for heterogeneous multitask learning,” 2020. [Online]. Available: https://arxiv.org/abs/1805. 04687

2020
[16]

ACDC: the adverse conditions dataset with cor- respondences for robust semantic driving scene perception,

C. Sakaridis et al., “ACDC: the adverse conditions dataset with cor- respondences for robust semantic driving scene perception,” 2024. [Online]. Available: https://arxiv.org/abs/2104.13395

2024
[17]

scikit-image: image processing in Python,

S. van der Walt et al., “scikit-image: image processing in Python,” PeerJ, vol. 2, p. e453, 2014. [Online]. Available: https://doi.org/10.7717/peerj. 453

2014 doi
[18]

Finetuned multimodal language models are high-quality image-text data filters,

W. Wang et al., “Finetuned multimodal language models are high-quality image-text data filters,” 2024. [Online]. Available: https://arxiv.org/abs/ 2403.02677

2024 arXiv
[2020]

Available: https://arxiv.org/abs/2006.11239

[Online]. Available: https://arxiv.org/abs/2006.11239

2006 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Autonomous driving system: a comprehensive sur- vey,

J. Zhao et al., “Autonomous driving system: a comprehensive sur- vey,” Expert Systems with Applications , vol. 242, p. 122836, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0957417423033389

work page 2024

[2] [2]

Faster R-CNN: towards real- time object detection with region proposal networks,

S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: towards real- time object detection with region proposal networks,” 2016. [Online]. Available: https://arxiv.org/abs/1506.01497

arXiv 2016

[3] [3]

YOLOv10: real-time end-to-end object detection,

A. Wang et al., “YOLOv10: real-time end-to-end object detection,” 2024. [Online]. Available: https://arxiv.org/abs/2405.14458

arXiv 2024

[4] [4]

InstructPix2Pix: learning to follow image editing instructions,

T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: learning to follow image editing instructions,” 2023. [Online]. Available: https: //arxiv.org/abs/2211.09800

arXiv 2023

[5] [5]

Road Vehicles — Functional Safety (Edition 2) , 2018

International Standards Organization, ISO 26262. Road Vehicles — Functional Safety (Edition 2) , 2018

work page 2018

[6] [6]

Road Vehicles — Safety of the Intended Functionality , 2022

——, ISO 26262. Road Vehicles — Safety of the Intended Functionality , 2022

work page 2022

[7] [7]

Road Vehicles — Safety and Artificial Intelligence, 2024

——, ISO/PAS 8800. Road Vehicles — Safety and Artificial Intelligence, 2024

work page 2024

[8] [8]

AI-FSM: Towards Functional Safety Management for Artificial Intelligence-based Critical Systems,

J. Fern ´andez et al., “AI-FSM: Towards Functional Safety Management for Artificial Intelligence-based Critical Systems,” inCARS@EDCC2024 Workshop - Critical Automotive applications: Robustness & Safety , Leuven, Belgium, Apr. 2024. [Online]. Available: https://hal.science/ hal-04769949

work page 2024