{"id":"aefa9138-ab2a-4eb8-9471-a1f4bb39543a","arxiv_id":"2505.02613","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"A video-based multi-module framework for lane-wise highway anomaly detection is presented with a new 73,139-sample dataset, but the main performance claim rests on in-sample evaluation.","lead":"A team from Purdue built a lane-wise highway anomaly detection system from surveillance video and released a dataset of 73,139 labeled lane samples. The paper's main performance claim is undermined by an evaluation where the machine-learning detector was tested on the same anomaly samples used to train it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline F1 is not an out-of-sample estimate: the ML module is trained on the verified anomaly samples that reappear in the test set, and the DL threshold is selected using test performance.","rationale":"The task is to find the most load-bearing concern about the central claim. I considered the reader's formal weakest_assumption (accuracy of lane-wise vision features: the occupancy estimator, YOLOv5x-CBAM, Deep SORT, and lane learning). That is a genuine deployment risk, but it is not the sharpest problem: even granting perfect features, the reported evaluation is contaminated. The text itself documents that verified anomalies are used to train the ML module and then reappear in the test set, and the DL threshold is chosen by comparing variants on the same test set (Table IV). Both are direct violations of the condition that test performance estimates generalization. The metric inconsistency is a concrete, checkable symptom: from a balanced 341/341 test set, Re=1.0, Acc=0.9787, FPR=0.024, and Pre=0.843 cannot all be true simultaneously. Solving for FP from precision with TP=341 gives FP ≈ 63, which implies FPR ≈ 0.19, not 0.024. The numbers in Table III are therefore not mutually consistent, further undermining the headline. The proposed test—cross-validation over anomaly clips with thresholds fixed before testing—would settle whether any of the reported performance survives without train/test leakage. Because the central empirical claim fails this minimal validity check, the REJECT verdict stands.","tokens_in":9312,"tokens_out":5919,"duration_ms":71785,"concrete_test":"Held-out-clip evaluation: split the 43/46 expert-validated anomaly clips by video ID (e.g., 80% train / 20% test); retrain the Isolation Forest ML module only on normal lane samples plus anomaly samples from training clips; fix the DL threshold percentile before inspecting test labels (or use 95% as originally specified); evaluate the full fusion on the held-out clips and on normal samples from held-out time periods. Repeat 5-fold and report the full confusion matrix, including FP/FN counts. If the held-out F1 remains near 0.9149 and all metrics are mutually consistent, the leakage objection is refuted; if it drops materially (e.g., below 0.85), the reported Table III result is an artifact of train/test contamination.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim of Table III—0.9787 accuracy, 0.9149 F1, outperforming SOTA—requires an honest out-of-sample evaluation. The manuscript's own pipeline description contradicts this. In §III ('Anomalies Labeling'), after Isolation Forest preselects and experts validate 341 anomaly samples, the text states: 'The verified anomalies are then used to train the ML-based anomaly detection model.' Section VI-A then builds the test set from '341 anomalous samples from 43 videos' drawn from the same validated pool and evaluates all modules on it. The Isolation-Forest ML module is therefore scored on data it was fitted on. Separately, §IV-A and Table IV select the DL reconstruction threshold (99%, time-dependent) by comparing 90/95/99 percentiles on this same test set, so the DL contribution is also post-hoc tuned. The Fig. 1 caption ('The anomalies labeling is only implemented during training stage') does not fix this: train/test overlap and test-set threshold selection remain. A further inconsistency supports the concern: on the stated 341-anomaly/341-normal test set, Re=1.0, Acc=0.9787, FPR=0.024, and Pre=0.843 imply incompatible numbers of false positives; the reported metrics cannot all derive from one confusion matrix. The headline performance numbers are thus not a valid estimate of out-of-sample detection quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a modular, video-only framework for lane-wise highway traffic anomaly detection. Lane-wise vehicle count, occupancy, and truck percentage are extracted from surveillance video; anomalies are labeled through an Isolation Forest preselection followed by manual and expert validation. Detection combines a VQ-VAE on CWT spectrograms with time-of-day thresholds, a rule-based jam/slow detector, and an Isolation Forest on occupancy and truck-percentage features. The authors introduce a dataset of 73,139 lane-wise samples with 341 anomaly samples and report a full-framework F1 of 0.9149, claiming state-of-the-art performance. The paper also describes a web platform and includes qualitative wavelet and UMAP analyses.","tokens_in":9662,"tokens_out":10885,"duration_ms":131039,"significance":"The dataset is a potentially useful contribution: it provides lane-level traffic features from highway surveillance video with expert-validated anomaly types, which is rare in the literature. The modular design combining deep, rule-based, and machine-learning detectors is pragmatic and interpretable. However, the central empirical claim is not supported by the current evaluation. The ML module is trained on the same verified anomaly samples used in the test set, the DL threshold is selected using test performance, and the reported confusion-matrix metrics are internally inconsistent. If the evaluation were redone with proper train/test separation, validation-based threshold selection, and corrected metrics, the framework and dataset could be assessed fairly; as written, the quantitative results do not provide a valid estimate of out-of-sample detection quality.","major_comments":[{"comment":"The ML-based module is trained on the verified anomaly samples and then evaluated on the same samples. Section III states, \"The verified anomalies are then used to train the ML-based anomaly detection model,\" and Section VI-A states that the test set contains \"341 anomalous samples from 43 videos\" drawn from the same validated pool of 341 anomalies. Since the ML module is an Isolation Forest fit on those samples, the ML-based contribution in Table III and the full-framework F1 of 0.9149 are in-sample estimates and do not measure detection of unseen anomalies. The test set must be disjoint from the training anomalies, for example through a video-level or temporal split, and the ML module must not receive the labeled anomalies on which it was fitted.","section":"Section III; Section VI-A"},{"comment":"The DL threshold is selected using test performance. Table IV compares 90%, 95%, and 99% percentiles, with and without time-of-day grouping, and reports the configuration that performs best on the test set (99% time-dependent). This means the DL module's operating point is post-hoc tuned on the same test data used to report Table III. The DL-based F1 and the claimed benefit of time-dependent thresholds are therefore not honest out-of-sample results. Thresholds should be chosen on a validation split before any evaluation on the held-out test set.","section":"Section IV-A; Table IV"},{"comment":"The confusion-matrix-based metrics in Table III are mutually inconsistent for the stated test set of 341 anomalies and 341 normal samples. With Re=1.0 and FNR=0, all 341 anomalies are true positives; then Pre=0.8431 would require approximately 63 false positives, which would yield FPR around 0.186 and Acc around 0.907, not the reported 0.024 and 0.9787. Conversely, Acc=0.9787 with FNR=0 implies about 14 false positives, which would give Pre around 0.959. The DL-only row is likewise inconsistent: Re=0.9767 and FPR=0.045 imply roughly 333 true positives and 15 false positives, giving Acc around 0.9645, not 0.5279. Please report the full confusion matrix and recompute all metrics from it.","section":"Table III"},{"comment":"The state-of-the-art comparison is not reproducible from the reported numbers. Table II lists \"Reconstruction Ours\" with Pre=0.8077, Re=0.9767, F1=0.8841, while Table III reports the DL-based module with Pre=0.7368, Re=0.9767, F1=0.84 and the full framework with F1=0.9149. If \"Ours\" in Table II is the DL module, the rows should match; if it is the full framework, the F1 should be 0.9149. Please clarify which configuration is compared and why the numbers differ.","section":"Table II vs Table III"},{"comment":"The baseline methods are evaluated under unequal conditions. The text states that for each baseline \"the input consists of our lane-wise vehicle count data, with data from all lanes in each video concatenated into a single sequence\" and that the original configurations are preserved. The proposed modules, in contrast, use CWT spectrograms, occupancy, and truck percentage, and the proposed thresholds are tuned on the test set. This comparison does not establish superiority over state-of-the-art methods. The baselines should receive the same feature sets and be given hyperparameter selection on validation data, with the final evaluation on a held-out test set.","section":"Section VI-B"},{"comment":"The normal test samples are drawn from the validation set used for VQ-VAE early stopping. The paper states that 20% of anomaly-free sequences serve as validation data and that the 341 normal test samples are \"randomly selected from the validation set.\" These normal sequences were therefore used during model development, so the DL evaluation is not fully out-of-sample even for normal traffic. A proper evaluation requires a held-out test set with normal samples disjoint from both the training and validation sets.","section":"Section VI-A"}],"minor_comments":[{"comment":"The caption states \"The anomalies labeling is only implemented during training stage,\" but Section VI-A uses the anomaly labels in the test set; please clarify the intended data flow and whether the same 341 anomalies appear in both training and test.","section":"Figure 1 caption"},{"comment":"The table heading contains a typo: \"stragies\" should be \"strategies,\" and the heading says \"ML-based module\" while the surrounding text describes DL-based threshold strategies; please correct the inconsistency.","section":"Table IV"},{"comment":"The symbol rendered as \"/exclamati⌢n-triangle\" is not a standard notation and appears malformed; please replace it with an appropriate marker defined in the table notes.","section":"Table I"},{"comment":"The term \"Variable Autoencoder\" should be \"Variational Autoencoder,\" and the inconsistent spacing in \"VQ-V AE\" should be fixed throughout the manuscript.","section":"Section IV-A"},{"comment":"The sentence \"We proposed a lane-wise traffic anomaly data collection and ground truth labeling pipeline\" should be \"We propose,\" since the work is being presented in the present tense.","section":"Section III"},{"comment":"Reference [17] lists \"X. Xtraffic et al.\" as the first author, which appears to be a placeholder or an error; please verify the citation.","section":"References"},{"comment":"The dataset link is given only as \"Lane-wise-Traffic-AD\"; please provide a complete, working URL.","section":"Abstract / Dataset availability"}],"recommendation":"major_revision","confidential_remarks":"The paper's dataset and modular system are potentially valuable, but the empirical evaluation needs to be completely reworked before the claims can be taken seriously. I would be open to a resubmission with a proper held-out test set, validation-based threshold selection, and corrected, consistent metrics. As it stands, the quantitative results should not be cited as evidence of out-of-sample performance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of arXiv:2505.02613. The dataset is the real story: 73,139 lane-wise samples from five cameras over six months, with four expert-validated anomaly types (lane blockage, foreign object, sustained congestion, camera shift). That's a useful resource for the ITS community, and the feature set (count, occupancy, truck percentage) is sensible. The modular three-branch detection idea is plausible, but the evaluation does not support the headline F1 of 0.9149.\n\nThe load-bearing problem is leakage. In Section III the verified anomalies are explicitly used to train the ML-based model, and in Section VI-A the test set is built from those same 341 anomalous samples. So the ML module's recall of 1.0 is a fitted result, not a detection result. The DL branch is also post-hoc tuned: Table IV picks the 99% time-dependent threshold because it performs best on that same test set. On top of that, the baselines get a different input format (concatenated counts only) while the proposed method uses richer features, so the SOTA comparison is not apples-to-apples.\n\nThere's also an internal inconsistency in Table III. With 341 anomalies and 341 normals, Re=1.0 means TP=341, FN=0. Then Pre=0.843 forces FP≈63, and FPR=0.024 would force TN+FP≈2,650. Those cannot come from one confusion matrix. The Acc likewise doesn't match. This could be a reporting typo, but as written the numbers are mutually incompatible, and the authors need to fix that.\n\nThe circularity burden is real but somewhat mitigated: the labels came from an Isolation Forest with contamination 0.3 followed by expert manual review, and the ML detector is another Isolation Forest with contamination 0.1. The expert validation breaks the pure algorithmic circularity, but the detector is still a relative of the label generator, and the train/test overlap is the bigger issue.\n\nOn the positive side, the dataset construction shows thought, and the interpretability angle (wavelet patterns, rule-based thresholds) is worth something. The occupancy estimator is crude (bounding-box heights relative to ROI), but that's a modeling choice, not a fatal flaw.\n\nBottom line: the dataset deserves peer review and eventual release; the detection framework's performance claims need a reworked evaluation protocol. If I were refereeing, I'd ask for an honest out-of-sample split (e.g., by video or time period), fixed pre-registered thresholds, and a corrected confusion matrix.\n\nRecommendation: send it to peer review, but with a request for major revision. The dataset alone is enough to justify referee time.","headline":"The dataset is a genuine new resource for lane-wise highway anomaly detection, but the reported F1 is not out-of-sample and the metrics in Table III are internally inconsistent.","tokens_in":10148,"tokens_out":3446,"would_cite":true,"duration_ms":36911,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a video-only, lane-wise framework detects highway traffic anomalies with 0.9787 accuracy and 0.9149 F1-score, outperforming state-of-the-art baselines.","keywords":["lane-wise anomaly detection","highway surveillance","video-only traffic monitoring","VQ-VAE","Continuous Wavelet Transform","Isolation Forest","occupancy estimation","truck percentage"],"falsifier":"Take a different set of highway surveillance cameras with ground-truth lane occupancy from loop detectors or manual frame-by-frame counts, run the identical feature-extraction pipeline, and compare the estimated occupancy and counts against ground truth; large mismatches would invalidate the features that the whole framework relies on.","tokens_in":9129,"feed_emoji":"🚗","tokens_out":4219,"duration_ms":47633,"temperature":0.7,"pith_summary":"This paper tries to establish that lane-wise time series of vehicle count, occupancy, and truck percentage, extracted purely from roadside surveillance video, are enough to detect highway anomalies without expensive sensors or detailed road modeling. The authors build a three-branch detector: a deep reconstruction model, a rule-based traffic-state classifier, and an Isolation Forest on occupancy plus truck percentage. They report that the fused system reaches 0.9787 accuracy and 0.9149 F1-score on a balanced test set, exceeding four compared methods. A sympathetic reader would take the core claim as: interpretable lane-level features from ordinary cameras can support practical, scalable highway anomaly detection.","feed_headline":"Video-only pipeline detects lane anomalies at 97.9% accuracy","feed_subtitle":"Fusing deep learning, rules, and machine learning, it catches blockages, congestion, foreign objects, and camera shifts.","key_machinery":"The central mechanism is the fusion of three anomaly detectors over lane-wise features. The first branch uses normalized Continuous Wavelet Transform spectrograms of per-lane counts fed into a Vector Quantized Variational Autoencoder (VQ-VAE), with adaptive thresholds taken from the 95th or 99th percentile of reconstruction errors within time-of-day groups. The second branch uses the rule-based status classifier on flow rate and occupancy, where occupancy is estimated as $\\bar{O}_T = \\frac{1}{M}\\sum_i h_i/H_{ROI}$, the mean contribution of detected bounding-box heights relative to the learned region height. The third branch vertically stacks occupancy and truck-percentage matrices into $X=[O;T]$ and applies Isolation Forest to identify road-dependent outliers. The key idea is that each module captures a different anomaly signature, and their union covers more anomaly types than any single model.","core_discovery":"The central claim is that combining three complementary detection mechanisms on lane-wise traffic features yields accurate anomaly detection from video alone. The deep-learning branch converts 15-minute lane count sequences into CWT spectrograms and reconstructs them with a VQ-VAE, flagging anomalies when reconstruction error exceeds time-of-day-specific percentiles. The rule-based branch thresholds flow rate and an occupancy estimate to catch jams and slow traffic. The machine-learning branch runs Isolation Forest on vertically stacked occupancy and truck-percentage matrices to capture road-dependent events such as truck-induced slowdowns. The paper reports that this fusion lifts F1-score from 0.851 with the first two branches to 0.9149 with all three, and that it outperforms the tested baselines on the new dataset.","pith_inferences":["The three-branch design is a transferable template: pair a frequency-domain deep learner with interpretable physical thresholds and a statistical outlier detector whenever cheap sensor signals carry correlated evidence.","Because occupancy is proxied by detection-box heights rather than true spatial occupancy, deployment on cameras with different angles or zoom levels will likely need recalibration; the paper does not test this transfer.","The observed wavelet signature difference—normal traffic showing a dominant roughly 9-minute periodicity while anomalies show shorter 2 and 4 minute components—suggests a testable extension: classify anomaly types by their wavelet spectra.","With only 341 anomaly samples drawn from 43 videos, the reported high recall may not generalize to extreme weather or lighting; a multi-site held-out evaluation would be the natural next test."],"forward_implications":["Highway agencies with existing surveillance cameras could deploy lane-wise anomaly detection without loop detectors, radar, or GPS-based road modeling.","The reported fusion results imply that combining a frequency-domain deep model, interpretable traffic-state rules, and an occupancy-plus-truck machine-learning model outperforms any single approach.","Time-of-day-dependent thresholds materially improve deep detection over a global threshold, so traffic normality needs to be conditioned on the clock.","The dataset of 73,139 lane-wise samples with expert-validated anomaly labels gives the community a common test bed for lane-wise highway anomaly detection and a concrete benchmark for future methods."],"supporting_citations":[{"why":"Supplies the learned ROIs, lane centers, and lane identities that define the lane-wise feature extraction.","marker":"[11]"},{"why":"Provides the Isolation Forest algorithm used both in the anomaly labeling pipeline and in the machine-learning detection module.","marker":"[18]"},{"why":"Supplies the YOLOv5x-CBAM detector that generates vehicle counts and truck detection from surveillance video.","marker":"[22]"},{"why":"Provides the Deep SORT tracker that links detections into per-lane trajectories.","marker":"[23]"},{"why":"Provides the VQ-VAE architecture used for reconstruction-based deep anomaly detection.","marker":"[24]"},{"why":"Serves as a generative time-series anomaly detection baseline compared in Table II.","marker":"[20]"},{"why":"Serves as a transformer-based baseline for multivariate time series anomaly detection compared in Table II.","marker":"[21]"},{"why":"Serves as an outlier-weighted autoencoder baseline in the comparative evaluation.","marker":"[25]"}],"fun_headline_variants":["Three-branch video pipeline lifts highway anomaly F1 to 0.915","Video-only lane anomaly detection improved by fusing three branches","New 73k-sample lane-wise highway anomaly benchmark from video","Three-branch fusion catches lane blockages, jams, and camera shifts from video","Deep learning plus rules plus ML detects highway anomalies lane-wise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything depends on the vision pipeline's lane-wise features—counts, occupancy estimated from bounding-box heights, and truck percentage—being accurate proxies for physical traffic; if the detector or lane assignment is wrong, the rule-based and machine-learning modules act on faulty inputs.","fun_headline_variants_meta":{"raw":{"variants":["Three-branch video pipeline lifts highway anomaly F1 to 0.915","Video-only lane anomaly detection improved by fusing three branches","New 73k-sample lane-wise highway anomaly benchmark from video","Three-branch fusion catches lane blockages, jams, and camera shifts from video","Deep learning plus rules plus ML detects highway anomalies lane-wise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00078,"raw_usage":{"total_tokens":3398,"prompt_tokens":848,"completion_tokens":2550,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":2458}},"tokens_in":464,"tokens_out":2550,"duration_ms":22083,"temperature":1.0,"reasoning_tokens":2458,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:47:14.488908+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a different set of highway surveillance cameras with ground-truth lane occupancy from loop detectors or manual frame-by-frame counts, run the identical feature-extraction pipeline, and compare the estimated occupancy and counts against ground truth; large mismatches would invalidate the features that the whole framework relies on.","supporting_citations":[{"cited_title":"Intelligent highway adaptive lane learning system in multiple rois of surveillance camera video,","cited_arxiv_id":null,"evidence_quote":"Supplies the learned ROIs, lane centers, and lane identities that define the lane-wise feature extraction."},{"cited_title":"Isolation-based anomaly detection,","cited_arxiv_id":null,"evidence_quote":"Provides the Isolation Forest algorithm used both in the anomaly labeling pipeline and in the machine-learning detection module."},{"cited_title":"Attention mechanism improves yolov5x for detecting vehicles on surveillance videos,","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLOv5x-CBAM detector that generates vehicle counts and truck detection from surveillance video."},{"cited_title":"Simple online and realtime tracking with a deep association metric,","cited_arxiv_id":null,"evidence_quote":"Provides the Deep SORT tracker that links detections into per-lane trajectories."},{"cited_title":"Explainable time series anomaly detection using masked latent generative modeling,","cited_arxiv_id":null,"evidence_quote":"Serves as a generative time-series anomaly detection baseline compared in Table II."}],"review_version":1}