{"id":"05642a41-d360-42b1-8322-170ad0a3271b","arxiv_id":"2505.08228","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Training on images with artificially added weather using InstructPix2Pix improves Faster R-CNN under adverse conditions but not YOLOv10 in real-world tests.","lead":"This paper tests whether adding weather effects to training images with a text-guided image editor can make self-driving car object detectors more robust in rain, fog, snow, and night. It helps a two-stage detector in real-world tests, but shows little or no benefit for YOLO models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Observed mAP gains may be within seed-to-seed training noise; bootstrap CIs only resample test images, not training runs.","rationale":"I considered the reader's identified assumption about label preservation; it is a real limitation, and the manual filtering step is indeed under-specified. However, even if labels were perfectly preserved, the empirical claim would still be insecure because each reported mAP is a point estimate from one training run, with confidence intervals that ignore training variance. The paper's own bootstrap methodology acknowledges model nondeterminism but does not measure it, making small improvements such as the real-world Faster R-CNN gains (+0.020 to +0.027) potentially indistinguishable from noise. A repeated-seed retraining test would settle this directly. Since the reader already issued a CONDITIONAL verdict emphasizing missing significance testing, my concern strengthens that condition but does not change the overall verdict: the paper is a useful empirical study whose central claims require additional evidence before being accepted at face value.","tokens_in":11734,"tokens_out":6981,"duration_ms":78451,"concrete_test":"Retrain each model (Faster R-CNN, YOLOv10-M, YOLOv10-N) on the Basic and Augmented datasets for both frameworks at least 5 times with different random seeds while holding all hyperparameters fixed, then report the mean and 95% CI of the Augmented-minus-Basic mAP50 difference per weather condition across seeds. If a condition's difference is not positive in a majority of seeds or its CI crosses zero, that condition should not be cited as an improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All reported improvements come from a single training run per model and dataset (Section V.B), while the bootstrap procedure in Section V.C resamples only test images, providing uncertainty over test-set composition, not over training stochasticity. The paper explicitly states that models are not entirely deterministic (Section V.C) yet never quantifies seed-to-seed variation. Several key gains are small: Faster R-CNN real-world fog +0.020, rain +0.027, snow +0.022, and simulated Faster R-CNN rain +0.008 (Tables VII and I). With 10-100 training epochs and typical detector stochasticity, these differences are well within plausible training noise. Thus the central claim that InstructPix2Pix augmentation robustly improves adverse-weather mAP50 is not yet statistically supported; the conclusion that the approach can 'significantly enhance' robustness overreaches the evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a data-augmentation pipeline in which clear-weather driving images are edited with InstructPix2Pix using hand-crafted text prompts to simulate rain, fog, night, and snow, and the edited images are added to the training set of object detectors. The pipeline is evaluated in two frameworks: simulated CARLA images (Faster R-CNN, YOLOv10-M, YOLOv10-N) and real-world BDD100K training with ACDC testing. The reported results show mAP50 improvements under adverse conditions for all models in the simulated framework and for Faster R-CNN in the real-world framework, while YOLOv10 models show no clear real-world benefit. The paper concludes that the augmentation strategy significantly enhances robustness.","tokens_in":11912,"tokens_out":5625,"duration_ms":52252,"significance":"The empirical setup is well structured in several respects: train and test sets are separated by scenario, evaluation is reported per weather condition, and a bootstrap resampling of test images is used to quantify sampling variability. The manual filtering of hallucinated augmentations is a practical safeguard, and the authors honestly acknowledge the limited YOLO benefit in the real-world framework. However, the central claim of significant robustness enhancement is currently not fully supported because the only uncertainty quantified is test-set composition, not training stochasticity, and the abstract and contribution statements overstate the uniformity of the benefit. With additional seed variability analysis and a more precise significance assessment, the contribution could become a useful benchmark for diffusion-based weather augmentation in object detection.","major_comments":[{"comment":"The bootstrap procedure resamples only test images, so the reported standard deviations do not reflect the variability of training the detector. Each model appears to be trained once per dataset (Section V.B), yet Section V.C states that models are not entirely deterministic. Consequently, small gains such as +0.020 for real-world fog or +0.027 for real-world rain in Table VII cannot be distinguished from seed-to-seed training noise. Please report results over multiple training seeds and provide a significance test (e.g., paired comparison across seeds) or explicitly restrict the claims to test-set sampling variability.","section":"V.C, Tables I and VII"},{"comment":"The abstract and conclusion claim that the data augmentation strategy 'significantly enhances' robustness, but the evidence is mixed. In the real-world framework, YOLOv10-M and YOLOv10-N show no improvement on most adverse conditions (Tables VIII and IX: night and snow are often worse with Augmented than Basic), and the paper itself acknowledges this in Section V.D. The claims should be softened to reflect that the benefit is model-dependent and, in the real-world case, observed mainly for Faster R-CNN.","section":"Abstract and Section VI"},{"comment":"The validity of the augmented training data depends on the assumption that InstructPix2Pix edits preserve the location, identity, and class of every annotated object, so that original bounding boxes remain correct on edited images. The only safeguard is the manual filtering step for 'hallucinations', but its criteria are not specified, no count of removed images is reported, and no quantitative verification of label preservation is provided. Please add a validation procedure (for example, comparing detections or semantic labels on original vs edited images) and report filtering statistics; otherwise silent mislabels may be introduced into the training set and bias the comparison.","section":"IV.A, Algorithm 1 Step 3"}],"minor_comments":[{"comment":"The text describing the default-condition results does not match Table II: it says Faster R-CNN and YOLO Nano achieve better results with Basic, but Table II shows YOLOv10-M with Augmented (0.550) outperforming Basic (0.536). Please correct the description to cover all models consistently.","section":"V.D"},{"comment":"Traffic sign and traffic light bounding boxes in ACDC are derived from semantic segmentation masks using regionprops. This procedure may produce boxes that do not tightly enclose the actual objects (e.g., merged or fragmented regions). Please add a note on the expected accuracy of these derived labels, since they are used to compute AP for those classes on the real-world test set.","section":"III.B"},{"comment":"The real-world training set description is internally inconsistent: 7,083 images split 70/30 would give about 4,958 training and 2,125 validation images, but the text later describes the Augmented dataset as containing 2,500 default training and 1,043 default validation images. Please clarify the actual composition and size of the Basic and Augmented training sets.","section":"V.A, Real-World Datasets"},{"comment":"The final augmented-image counts after filtering (4,270 fog, 4,484 rain, 4,780 night) are described together with an intended 20% per condition allocation of 673 images per condition. Please explain how the post-filtering counts relate to the stated dataset composition.","section":"V.A, Simulated Datasets"},{"comment":"Reference [6] is titled 'ISO 26262. Road Vehicles — Safety of the Intended Functionality,' which appears to be a bibliographic error: the safety-of-the-intended-functionality standard is ISO 21448. Please verify and correct the citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's main weakness is statistical: single training runs and test-only bootstrap resampling do not support the strength of the stated conclusions. This is fixable with additional experiments. I would also ask the authors to tighten the abstract after the YOLO real-world results. The manuscript otherwise fits the journal's scope and the empirical comparison is useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper tests a straightforward idea: use InstructPix2Pix with hand-crafted prompts to turn clear-weather images into fog, rain, night, and snow versions, then train detectors on the mix. That's genuinely new as an application, and the experiments are more careful than most augmentation papers—separate train/test, two frameworks (CARLA and BDD100K→ACDC), bootstrap resampling of the test sets, and per-class AP alongside mAP. They're also honest in the discussion: YOLOv10 models don't benefit in the real-world framework, and they flag the manual filtering as a limitation.\n\nThe soft spot is exactly what the stress-test note says. The bootstrap only resamples test images, so the reported means and stds capture test-set composition, not training stochasticity. Everything is one training run per model/dataset. Some headline gains are small: Faster R-CNN simulated rain +0.008, real-world fog +0.020, rain +0.027, snow +0.022. Those are within a standard deviation of the bootstrap spread, and typical detector training has seed-to-seed variation at that scale. The paper even concedes models are not deterministic, but never quantifies it. So the abstract's 'significantly enhance' is not supported. The proper conclusion is 'may improve robustness for two-stage detectors; the effect on one-stage YOLO is unclear,' which is what the body actually says.\n\nThere are two other issues worth naming, both secondary. No comparison to prior weather-augmentation methods (style transfer, other generative augmentations, or simply adding real adverse-weather data), so we don't know how this stacks up. And the ACDC test labels for traffic signs/lights are derived from semantic segmentation masks, which is a plausible stopgap but a weaker ground truth than manual box annotations.\n\nNone of this is fatal. The method is transparent, reproducible in structure, and the design is good enough that a serious referee could guide it into a solid contribution. I'd want multi-seed training, a proper significance test, and at least one baseline augmentation before the claims are publishable as stated. But it deserves peer review, not a desk reject.","headline":"A clean empirical study of InstructPix2Pix weather augmentation; the idea is promising for two-stage detectors, but the headline gains rest on single training runs and may not clear training noise.","tokens_in":12440,"tokens_out":2610,"would_cite":false,"duration_ms":26566,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InstructPix2Pix weather edits improve object detection in fog, rain, and night—with model-dependent limits.","keywords":["adverse weather","object detection","data augmentation","InstructPix2Pix","diffusion models","autonomous driving","Faster R-CNN","YOLOv10"],"falsifier":"Take a random sample of the augmented images produced by the pipeline and compare each one against its source image to check whether every originally annotated object still exists at the same location with the same class. If a substantial fraction of augmented images show moved, merged, or missing objects, then the reported mAP gains could be partially an artifact of invalid labels rather than true robustness improvement; a second check would be to train on the unfiltered augmented set and compare with the filtered set to measure how much the manual filtering step contributes.","tokens_in":11537,"feed_emoji":"🌧️","tokens_out":2902,"duration_ms":29208,"temperature":0.7,"pith_summary":"This paper tries to establish that diffusion-model image editing can act as a practical data augmentation for object detectors operating in adverse weather. The authors use InstructPix2Pix with sequential text prompts to transform clear-weather driving images into foggy, rainy, snowy, or nighttime versions, then train Faster R-CNN and YOLOv10 models on the augmented data. In a controlled CARLA simulator setting, the augmentation consistently improves mAP50 under fog, night, and rain for all three detectors tested. On real-world data, the same approach improves Faster R-CNN under adverse conditions but does not clearly help the YOLO models, a limitation the paper connects to the YOLO models' lower baseline stability. The central contribution is evidence that text-guided diffusion edits can narrow the weather-induced performance gap when combined with a filtering step that removes hallucinated images.","feed_headline":"Diffusion weather edits lift detection in fog and rain","feed_subtitle":"InstructPix2Pix augmentation improves Faster R-CNN and YOLO in simulation; real-world gains depend on the model.","key_machinery":"The central object is the InstructPix2Pix diffusion model, which performs localized image editing from paired image and text inputs. The method works by applying sequential text prompts, such as \"What would it look like if it were raining a lot?\" followed by \"Add raindrops on the camera lens,\" with deliberately low guidance scales to preserve semantic content, and then a manual filtering step removes hallucinated outputs. The evaluation machinery is bootstrap resampling: 1,000 samples of the test set per weather condition are drawn with replacement, and the mean and standard deviation of mAP50 are reported.","core_discovery":"The central claim is that augmenting clear-weather training images with InstructPix2Pix weather edits reduces the object-detection performance gap under fog, night, rain, and snow, provided that a manual filtering step removes augmented images where the model hallucinated missing or distorted objects. The evidence is a systematic comparison of mAP50 between a Basic dataset and an Augmented dataset across two frameworks. In the simulated CARLA framework, the augmentation improves all three detectors under every adverse condition tested, with the largest gains often in fog and night. In the real-world framework using BDD100K for training and ACDC for testing, Faster R-CNN shows improved robustness under adverse weather—for example, roughly a 0.07 mAP50 gain for pedestrians in night and rain—while YOLOv10-M and YOLOv10-N show no clear benefit, and the paper attributes this to the need for further investigation into their training process.","pith_inferences":["The method could be tested as a general robustness tool for other perception tasks, such as semantic segmentation or multi-object tracking, where weather shifts also degrade performance.","The manual filtering step could likely be automated with a learned hallucination detector; the paper itself points toward multimodal language models as a possible filter, which would make the pipeline fully automated.","For the YOLO models, the lack of benefit on real data may reflect training configuration rather than a fundamental limitation of the augmentation, so a fair test would involve tuning learning rate, epochs, or augmentation ratios specifically for YOLO on the augmented set.","Combining InstructPix2Pix weather edits with standard photometric augmentations like brightness, contrast, and noise might produce compounding gains, but the paper does not test this interaction."],"forward_implications":["In simulated environments, augmenting clear-weather training with fog, night, and rain edits narrows the mAP50 gap for Faster R-CNN, YOLOv10-M, and YOLOv10-N, with the largest gains typically in fog and night conditions.","On real-world data, Faster R-CNN benefits from the augmentation across fog, night, rain, and snow, reducing the gap between default and adverse-condition performance.","For YOLOv10 models, the same augmentation does not yield a clear robustness benefit on real-world data, indicating that the effectiveness of diffusion-based weather augmentation depends on the detector architecture and training stability.","The inclusion of a manual filtering step to discard hallucinated images is a necessary part of the pipeline, as unfiltered augmentations could introduce mislabeled training data.","The approach transfers from simulated to real images, suggesting that the prompting strategy itself is not tied to the source domain."],"supporting_citations":[{"why":"Supplies the InstructPix2Pix diffusion model and the prompting approach the whole augmentation pipeline is built on.","marker":"[4]"},{"why":"Provides the CARLA simulator used to generate the simulated road scenarios and their weather-rendered test conditions.","marker":"[13]"},{"why":"Supplies the BDD100K real-world training and validation images in clear daytime conditions.","marker":"[15]"},{"why":"Supplies the ACDC adverse-weather test set used to evaluate real-world performance.","marker":"[16]"},{"why":"Defines the Faster R-CNN detector that is one of the three object detection models evaluated.","marker":"[2]"},{"why":"Defines the YOLOv10 detector, in its Nano and Medium variants, that is evaluated alongside Faster R-CNN.","marker":"[3]"},{"why":"Provides the ghost-box filtering method used to obtain valid bounding box annotations from CARLA.","marker":"[14]"},{"why":"Provides the regionprops function used to generate bounding boxes for traffic signs and lights from ACDC semantic masks.","marker":"[17]"}],"fun_headline_variants":["Diffusion aug boosts object detection in fog, rain, and night","Weather-augmented training sharpens autonomous driving perception","Diffusion weather edits improve detection, but real-world gains are model-dependent","InstructPix2Pix weather aug lifts Faster R-CNN, but YOLO lags","Simulation-tested diffusion aug for weather-robust detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that InstructPix2Pix edits preserve the location, identity, and class of every annotated object, so that the original bounding boxes remain valid on the augmented images; if an edit shifts, merges, or erases an object, the detector is trained on mislabeled data.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion aug boosts object detection in fog, rain, and night","Weather-augmented training sharpens autonomous driving perception","Diffusion weather edits improve detection, but real-world gains are model-dependent","InstructPix2Pix weather aug lifts Faster R-CNN, but YOLO lags","Simulation-tested diffusion aug for weather-robust detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000703,"raw_usage":{"total_tokens":3157,"prompt_tokens":916,"completion_tokens":2241,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":2145}},"tokens_in":532,"tokens_out":2241,"duration_ms":15445,"temperature":1.0,"reasoning_tokens":2145,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:59:51.316362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the augmented images produced by the pipeline and compare each one against its source image to check whether every originally annotated object still exists at the same location with the same class. If a substantial fraction of augmented images show moved, merged, or missing objects, then the reported mAP gains could be partially an artifact of invalid labels rather than true robustness improvement; a second check would be to train on the unfiltered augmented set and compare with the filtered set to measure how much the manual filtering step contributes.","supporting_citations":[{"cited_title":"BDD100K: a diverse driving dataset for heterogeneous multitask learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the BDD100K real-world training and validation images in clear daytime conditions."},{"cited_title":"Proceedings of the 10th International Conference on Vehicle Technology and Intelligent Transport Systems – VEHITS,","cited_arxiv_id":null,"evidence_quote":"Provides the ghost-box filtering method used to obtain valid bounding box annotations from CARLA."}],"review_version":1}