{"id":"df0520d7-6d5d-420c-a338-684d81257c4b","arxiv_id":"1908.04034","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The proposed 3D CNN outperforms the classical Bossu detector for surveillance rain detection in matched scenes, but does not generalize to visually different regions of interest.","lead":"This paper introduces a 215-hour public dataset of surveillance footage labeled with rainfall, and tests two algorithms for detecting rain from the video. A 3D convolutional network beats an older hand-crafted method on similar scenes, but fails when the camera view changes to a different surface.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract claim 'outperforms ... on all metrics, for both traffic crossings' is contradicted by Table 5: on Crossing2-asphalt, C3D-FCN has F1=0 and MCC=0 while Bossu-EM has F1=0.136 and MCC=0.008.","rationale":"The reader's strongest_claim correctly identifies the contradiction between the abstract and Table 5. My independent reading confirms this: on Crossing2-asphalt, C3D-FCN has F1=0 and MCC=0 while Bossu-EM has F1=0.136 and MCC=0.008, so the claimed superiority on all metrics is unsupported. This is a load-bearing concern because it undermines the headline contribution as written. However, the paper's underlying empirical contribution is a new public dataset and a benchmark of two methods, and the body text is appropriately nuanced about the failure on textured regions. The reader's weakest_assumption instead focused on the distance between the cameras and the laser disdrometer (970 m and 1230 m). That is a valid concern about label quality, but it is less direct than the internal contradiction in the reported results. The ground-truth distance would affect all methods equally and would not by itself explain the qualitative failure pattern, whereas the Table 5 contradiction is definitive evidence that the abstract overclaims. Therefore I recommend keeping the reader's CONDITIONAL verdict: the paper should be revised to align the abstract with the experimental results and to discuss the label-distance limitation, but the dataset and the comparison on similar-texture regions remain a useful contribution.","tokens_in":11778,"tokens_out":3649,"duration_ms":36639,"concrete_test":"Recompute the per-metric comparison from Table 5 for Crossing2-asphalt using the given confusion matrices. For C3D-FCN (TP=0, TN=1,069,231, FP=0, FN=102,717) and Bossu-EM (TP=224,853, TN=6,335,804, FP=2,257,136, FN=591,994), verify that F1 and MCC for Bossu-EM exceed those of C3D-FCN. If confirmed, the abstract's 'all metrics, for both traffic crossings' statement is false and should be revised to state that the improvement holds on Crossing1 and Crossing2-brick but not on Crossing2-asphalt.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central advertised claim, stated in the abstract, is that the proposed 3D CNN 'outperforms the previous state-of-the-art method by a large margin on all metrics, for both of the traffic crossings.' Table 5 directly contradicts this. On Crossing2-asphalt, C3D-FCN predicts no rain for every frame: TP=0, FP=0, TN=1,069,231, FN=102,717, giving F1=0.0 and MCC=0.0. Bossu-EM, by contrast, has TP=224,853, TN=6,335,804, FP=2,257,136, FN=591,994, giving F1=0.136 and MCC=0.008. Thus Bossu-EM is better than C3D-FCN on two of the three reported metrics, and the 'large margin on all metrics' claim fails. The body of the paper acknowledges the failure in Section 6 ('the C3D CNN evaluating the entire frame predicts no rain') and Section 7 ('our 3D CNN failed to function' on the different-texture region), so the abstract is internally inconsistent with the paper's own results. This matters because the headline claim is the main advertised contribution; if the central claim is instead the more nuanced conclusion in Section 7, the abstract and contribution statements must be corrected.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents the AAU VIRADA dataset, 215 hours of surveillance video from two traffic crossings with rainfall ground truth from a laser disdrometer and a tipping-bucket rain gauge, and compares a reimplementation of the Bossu et al. rain detection method with a 3D convolutional network (C3D) converted to a fully convolutional network. The authors report that the proposed C3D method outperforms Bossu et al. on all metrics for both crossings, and also analyze how the choice of region of interest affects generalization. The dataset and implementations are released publicly.","tokens_in":12101,"tokens_out":2990,"duration_ms":28800,"significance":"If the central comparative claim were supported, this would be a useful benchmark and baseline for rain detection from general-purpose surveillance cameras, and the public dataset plus open-source implementations would be a valuable community resource. The paper's strengths are the scale of the dataset, the explicit use of MCC as a primary metric for imbalanced data, and the public release of code and data. However, the headline claim is contradicted by the paper's own Table 5, and the evaluation protocol contains circularity and label-quality issues that currently undermine the reported comparisons.","major_comments":[{"comment":"The abstract states that the proposed 3D CNN 'outperforms the previous state-of-the-art method by a large margin on all metrics, for both of the traffic crossings,' but Table 5 contradicts this. On Crossing2-asphalt, C3D-FCN achieves F1=0.0 and MCC=0.0, while Bossu-EM achieves F1=0.136 and MCC=0.008, so Bossu is better on two of the three metrics. The body text acknowledges this failure in Section 6 ('the C3D CNN evaluating the entire frame predicts no rain'), so the abstract and contribution statements must be revised to match the actual results, which support only the more nuanced conclusion in Section 7.","section":"Abstract and Section 6"},{"comment":"The Bossu parameters are selected by a search on six video snippets from Crossing1 and then the method is evaluated on the full Crossing1-trn and Crossing1-val splits. If those snippets are part of the evaluation sets, the hyperparameters are tuned on the test data, which invalidates the Crossing1 comparison as an estimate of generalization. The authors should clarify whether the tuning snippets are included in the reported splits and should evaluate Bossu on a held-out set not used for parameter selection.","section":"Section 5.1 and Table 3"},{"comment":"Table 5 reports C3D-FCN and C3D-Center results on Crossing1-trn, which is the training split used to fit the network. Reporting performance on the training set is not evidence of generalization and inflates the apparent superiority of the C3D method. The comparison should focus on Crossing1-val and the Crossing2-test splits, with the training row either removed or clearly labeled as training fit rather than as a benchmark result.","section":"Section 5.2 and Table 5"},{"comment":"The ground-truth rain labels come from a laser disdrometer located 1230 m (Crossing1) and 970 m (Crossing2) from the cameras, and per-minute gauge readings are converted to per-frame labels. Rainfall is spatially variable at these distances, so the labels may not match the rain condition at the camera scene. This label noise affects every reported metric and should be quantified, for example by comparing the disdrometer and tipping-bucket records, by analyzing the sensitivity of results to label temporal smoothing, or by restricting evaluation to periods of spatially homogeneous rain.","section":"Section 3.2 and Table 1"}],"minor_comments":[{"comment":"There are several typographical errors, including 'at-glace' in the Figure 1 caption, 'millenium' in Section 2, 'trough' in Section 3.1, and 'Kolomogrov' in Section 4.1; these should be corrected.","section":"Throughout"},{"comment":"In the text, C3D-Center on Crossing2-asphalt is reported as having an MCC of -0.08, while Table 5 reports -0.0837; the rounding should be consistent.","section":"Section 6"},{"comment":"The paper says the network is trained with a temporal stride of 8 frames and that the method 'analyse 16 frames at a time'; it would be clearer to state explicitly how the 16-frame clips are formed relative to the stride and the per-frame label conversion.","section":"Section 5.2"},{"comment":"The phrase 'outperforms the Bossu algorithm on all of the Crossing1 videos' is imprecise because the training split is included; it should be clarified that the strong result is on Crossing1-val or qualified as training-set performance.","section":"Section 6"},{"comment":"The table would benefit from a column showing the number of predictions for each method, since the C3D methods produce far fewer predictions than the per-frame Bossu methods; this is explained in the text but should be visible in the table.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The dataset and code release are genuine contributions, but the paper's central advertised claim is contradicted by its own table, and the evaluation protocol has circularity and label-quality issues. The authors should be asked to correct the abstract and present results only on held-out data, and to address the ground-truth distance issue explicitly. This is fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it: the AAU VIRADA dataset (215 hours of surveillance video with rain labels from two crossings) is a genuine contribution, and the open-source reimplementation of Bossu et al. is useful for benchmarking. The second thing: the abstract says the proposed 3D CNN outperforms Bossu by a large margin on all metrics for both crossings, but Table 5 shows the opposite on Crossing2-asphalt—the C3D-FCN predicts no rain at all (F1=0, MCC=0) while Bossu gets F1=0.136. The body admits this in Section 6, so the abstract and the results do not match.\n\nWhat the paper does well: the dataset fills a real gap, and the authors are honest in the conclusion about the model failing on a different texture. The comparison on brick versus asphalt is informative: the C3D generalizes to similar textures but not to a very different one. The open implementations let others reproduce the baseline without reimplementing from scratch.\n\nSoft spots, in order. First, the abstract overclaim is not a minor wording issue—it misrepresents the central result and will mislead anyone who reads only the abstract. Second, the ground-truth rain labels come from a laser disdrometer 970 m and 1230 m away from the two camera sites. Rain is spatially patchy, and per-minute labels at that distance can easily mismatch the camera view. The authors never quantify this uncertainty. Third, the Bossu parameters were tuned on six snippets from Crossing1 and then evaluated on the full Crossing1 split; the C3D is also evaluated on its own training split. Those train-set numbers (MCC 0.94, etc.) are not evidence of generalization. The unseen Crossing2 results are the only real test, and they are mixed.\n\nThe rest of the paper is methodologically okay. The metrics are reported with confusion matrices, MCC is used as primary metric, and the failure analysis is candid. The citation pattern is fine.\n\nWho this is for: anyone needing a rain-detection dataset or a baseline for surveillance video; also useful for deraining pipelines that need a rain-presence gate. It deserves a serious referee, but the abstract and the interpretation of the train-set results must be fixed before publication. I would accept it for review with a request for major revision.","headline":"Genuine dataset and baseline, but the abstract overclaims on the paper's own numbers.","tokens_in":12639,"tokens_out":2987,"would_cite":true,"duration_ms":27697,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a fully-convolutional 3D CNN detects rainfall from surveillance cameras only on scenes resembling its training data, failing entirely on an asphalt surface.","keywords":["rain detection","surveillance cameras","3D CNN","C3D","rainfall measurement","VIRADA dataset","computer vision","fully convolutional network"],"falsifier":"If a disdrometer were placed directly beside one of the cameras, and the current model's predictions disagreed with that local ground truth on a large fraction of rain minutes, the assumption tying distant gauge data to the camera scene would be falsified. Alternatively, manually labeling rain visibility on Crossing2-asphalt and showing that the C3D-FCN predicts no rain for clips with clearly visible streaks would falsify the claim that the network detects rain in that setting.","tokens_in":11583,"feed_emoji":"🌧️","tokens_out":5741,"duration_ms":52857,"temperature":0.7,"pith_summary":"The paper asks whether ordinary surveillance cameras, not weather-beaten special-purpose setups, can serve as rain detectors. To answer it, the authors introduce the AAU Visual Rain Dataset (VIRADA), 215 hours of video from two Danish traffic crossings with rainfall labels from a laser disdrometer. They reimplement the previous state-of-the-art, Bossu et al.'s histogram-of-orientation-of-streaks algorithm, and compare it with a fully-convolutional 3D CNN (C3D) adapted from action recognition. On the crossing used for training, the CNN clearly outperforms Bossu's method, and it retains that edge on a similarly textured region of another crossing. On a differently textured asphalt surface, however, the CNN predicts no rain at all, showing that texture generalization, not rain detection per se, is the unresolved obstacle.","feed_headline":"Street-camera CNN detects rain - but only on familiar surfaces","feed_subtitle":"A 215-hour benchmark shows a 3D CNN outperforms streak counting, then fails on asphalt.","key_machinery":"The central object is the modified C3D network: a 3D convolutional architecture that takes 16-frame clips as input and ends in two 2D convolutional layers plus global averaging, converting it into a fully-convolutional network that scores every 112×112 crop of the frame. It is trained from scratch as a binary rain classifier. The comparison baseline is Bossu's pipeline, which extracts candidate rain streaks via the photometric constraint, builds a histogram of streak orientations, fits a Gaussian-uniform mixture with expectation-maximization, and uses a Kalman filter to smooth detections. The C3D's temporal receptive field is what lets it in principle see rain streaks moving across frames, while the Bossu method explicitly models streak orientation.","core_discovery":"On its own terms, the paper establishes that a 3D CNN trained on 16-frame clips of one traffic crossing can detect rainfall in surveillance footage from that crossing with high accuracy (MCC 0.94 on training, 0.58 on validation), comfortably beating the hand-crafted Bossu detector whose MCC stays near zero. The same network, applied as a fully-convolutional network with global averaging, also outperforms Bossu on a brick-textured region of a second crossing but fails completely on an asphalt region of that crossing, where it outputs 'no rain' for every frame. The authors interpret this as evidence that the CNN is learning scene texture rather than a generic rain signature, and conclude that rain detection for general-purpose surveillance cameras is not yet solved. The paper's contribution therefore is a public benchmark (VIRADA), an open-source reimplementation of the baseline, and a careful demonstration of where the data-driven method wins and where it breaks.","pith_inferences":["The failure on asphalt hints that the network may be short-cutting the task by recognizing static scene appearance rather than temporal rain dynamics; training on a larger variety of surfaces (or adding synthetic rain) could be a testable extension.","Because the camera and disdrometer are separated by about a kilometer, some per-minute label mismatches are inevitable; a denser network of local sensors could sharpen both training and evaluation.","The same fully-convolutional architecture could be repurposed to estimate rain rate rather than binary presence, using camera geometry approaches, if labeled rainfall volume were available."],"forward_implications":["Deploying this detector as a preprocessing stage for rain-removal algorithms is feasible only at sites whose surfaces resemble the training data, or after fine-tuning on the target scene.","The VIRADA dataset gives the community a shared 215-hour benchmark with laser-disdrometer labels, so future rain-detection methods can be compared on identical surveillance footage.","For general-purpose camera networks, the practical implication is that a site-specific region of interest (e.g., a brick wall) is a safer choice than a textured surface like asphalt.","The comparison quantifies how far hand-crafted streak-orientation methods lag behind learned features on this task."],"supporting_citations":[{"why":"Provides the hand-crafted rain detection baseline that the paper reimplements and compares against.","marker":"[6]"},{"why":"Supplies the original C3D architecture that the paper adapts into a fully-convolutional rain classifier.","marker":"[33]"},{"why":"Describes the laser disdrometer used as the ground-truth rainfall measurement device.","marker":"[24]"},{"why":"Establishes the photometric constraint of rain streaks, which underpins the baseline detector's candidate segmentation.","marker":"[13]"},{"why":"Earlier camera-based rain gauge work whose geometry-based approach is contrasted with the data-driven method.","marker":"[1]"},{"why":"Motivates the need for rain detection in traffic surveillance as a prerequisite for rain removal.","marker":"[3]"}],"fun_headline_variants":["Rain CNN aces known crossing, fails on new asphalt","Rain detection CNN fails on unfamiliar surfaces","VIRADA benchmark: Rain CNN works on trained crossings only","CNN rain detector blind to new asphalt","VIRADA dataset exposes rain CNN's surface bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground truth assumes that the laser disdrometer, situated roughly one kilometer from each camera, records the same rain that is falling at the camera scene, so per-minute labels are accurate for every frame of video.","fun_headline_variants_meta":{"raw":{"variants":["Rain CNN aces known crossing, fails on new asphalt","Rain detection CNN fails on unfamiliar surfaces","VIRADA benchmark: Rain CNN works on trained crossings only","CNN rain detector blind to new asphalt","VIRADA dataset exposes rain CNN's surface bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000746,"raw_usage":{"total_tokens":3328,"prompt_tokens":949,"completion_tokens":2379,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":2308}},"tokens_in":565,"tokens_out":2379,"duration_ms":17318,"temperature":1.0,"reasoning_tokens":2308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:53:07.820012+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a disdrometer were placed directly beside one of the cameras, and the current model's predictions disagreed with that local ground truth on a large fraction of rain minutes, the assumption tying distant gauge data to the camera scene would be falsified. Alternatively, manually labeling rain visibility on Crossing2-asphalt and showing that the C3D-FCN predicts no rain for clips with clearly visible streaks would falsify the claim that the network detects rain in that setting.","supporting_citations":[{"cited_title":"Rain or snow detection in image sequences through use of a histogram of orientation of streaks","cited_arxiv_id":null,"evidence_quote":"Provides the hand-crafted rain detection baseline that the paper reimplements and compares against."},{"cited_title":"Learning spatiotemporal features with 3d convolutional networks","cited_arxiv_id":null,"evidence_quote":"Supplies the original C3D architecture that the paper adapts into a fully-convolutional rain classifier."},{"cited_title":"An optical disdrometer for measuring size and velocity of hydrometeors","cited_arxiv_id":null,"evidence_quote":"Describes the laser disdrometer used as the ground-truth rainfall measurement device."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the photometric constraint of rain streaks, which underpins the baseline detector's candidate segmentation."},{"cited_title":"Toward the camera rain gauge","cited_arxiv_id":null,"evidence_quote":"Earlier camera-based rain gauge work whose geometry-based approach is contrasted with the data-driven method."},{"cited_title":"Rain removal in traffic surveillance: Does it matter? IEEE Transactions on Intelligent Transportation Systems, 2018","cited_arxiv_id":null,"evidence_quote":"Motivates the need for rain detection in traffic surveillance as a prerequisite for rain removal."}],"review_version":1}