{"id":"2b43922a-7828-4094-8341-2dc4c3936c39","arxiv_id":"2506.07014","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Under a standardized evaluation on the public MMDAP dataset, a simple random forest method (88% accuracy, AUC 0.85) outperforms re-implementations of three published vehicle-dynamics drowsiness detectors, whose reported metrics are largely not reproducible.","lead":"This paper re-implements three published driver-drowsiness detection methods and a simple random forest baseline on a public driving dataset under a common evaluation protocol. It reports that previously published accuracies of up to 98% do not reproduce, and that the random forest baseline reaches 88% accuracy.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The C2 split may leak the same driver's windows into train and test; without participant-level grouping, RF's 88%/0.85 could be inflated and the ranking unreliable.","rationale":"The paper's contribution is an empirical comparison under a supposedly standard protocol. For the conclusion that prior methods are not reproducible and that RF is a strong baseline, the test set must be clean. The only description of the split is an 8:1:1 ratio; no participant grouping is mentioned. With 82 participants and continuous recordings, cross-subject generalization is the relevant target, and window-level random splitting is a known leakage route. The reader identified this same assumption, and I agree it is the weakest link. Other concerns, such as unequal hyperparameter tuning (VI-A) and feature bias toward RF (VI-B), are acknowledged and would not by themselves overturn the central claim; leakage would. The proposed test can settle the issue because the code and dataset are public: if participant-stratified splitting preserves RF's accuracy and ranking, the concern is resolved; if not, the headline numbers and conclusion must be revised.","tokens_in":10957,"tokens_out":7040,"duration_ms":70057,"concrete_test":"Inspect the public repository's data-splitting code; if the 8:1:1 split is a plain random split over windows without grouping by participant, rerun the C2 evaluation using a participant-stratified split (e.g., GroupShuffleSplit) so all windows of a participant stay in one fold. Compare RF's accuracy/AUC and the ranking of all four methods to Table V; if RF drops substantially or no longer ranks first, leakage is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RF is best under the common configuration C2 (Table V) rests on the validity of the 8:1:1 train/validation/test split described in Sect. IV-C2. The paper states only 'we split the dataset at a 8:1:1 ratio' and does not specify whether the split is stratified by the 82 participants or applied to individual 3-second windows (with 50% overlap) pooled across sessions. If the split is at window level, the same participant's data, including overlapping windows sharing 1.5 seconds, appear in both training and testing. Vehicle dynamics and EEG-derived drowsiness states are strongly autocorrelated within a session, so this violates the i.i.d. assumption and inflates test metrics. The paper itself criticizes prior studies for data leakage (Sect. I) and lists individual differences as a threat (Sect. VI-D), yet the split design does not address this. Because RF uses far more features than the other methods, it is the most likely to exploit driver-specific memorization, so the reported 88% accuracy and 0.85 AUC may not reflect generalization to unseen drivers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a common framework for vehicle-dynamics-based driver drowsiness detection (DDD) using the public MMDAP dataset, re-implements three published lightweight methods (SvmA, SvmW, Lstm) and a proposed random-forest method (RF), and compares them under a common configuration C2 (8:1:1 split, 3-second windows, 50% overlap, 60 Hz). The central claim is that the RF method achieves the best performance (accuracy 88%, AUC 0.85), while the three re-implementations fall far below their originally reported values, suggesting that previously reported DDD results are not reproducible and that the field needs standardized evaluation.","tokens_in":11156,"tokens_out":7652,"duration_ms":76516,"significance":"If the comparison is valid, the paper provides a useful open benchmark and a reproducible baseline for vehicle-dynamics-based DDD. The authors publicly release their implementation, which is a concrete strength, and they make a good-faith effort to check reproducibility of prior work on a single public dataset. The paper also honestly lists threats to validity, including possible hyperparameter bias and dataset generalizability. However, the significance is limited by the single straight-road simulator dataset, the lack of uncertainty quantification, and the unresolved split protocol; these issues affect whether the reported ranking of RF over the other methods generalizes beyond the specific experimental setup.","major_comments":[{"comment":"The C2 split, which produces the results in Table V, is described only as 'we split the dataset at a 8:1:1 ratio' with no statement about whether the split is participant-level or window-level. Because C2 uses 3-second windows with 50% overlap, a window-level random split puts overlapping 1.5-second segments from the same driver into both training and test sets; vehicle dynamics and EEG-derived labels are strongly autocorrelated within a session, so this violates the independence assumption and can inflate the reported accuracy and AUC. This is a load-bearing issue because the paper's central claim is that RF is best under C2, and the paper itself criticizes prior work for data leakage (Sect. I) and lists individual differences as a threat (Sect. VI-D). Please specify how participants and overlapping windows were assigned to the three folds; if the split was not grouped by participant, rerun the comparison with a participant-level split (e.g., group all windows of a participant into one set) and report the new metrics and ranking.","section":"Section IV-C2, Table V"},{"comment":"The comparison reports single point estimates (RF accuracy 88%, AUC 0.85) from one split, without error bars, confidence intervals, or significance tests. The qualitative claim that RF is 'the highest' among the four methods is therefore a statement about one random realization of the 8:1:1 split. Given that the paper's purpose is to provide a reliable comparison, please report repeated random splits or cross-validation with confidence intervals, and add a paired test (e.g., McNemar's test on the test set) between RF and each baseline.","section":"Section V, Table V"},{"comment":"The RF method description says the top-ranked features are selected with SelectKBest but does not state the number K, and the Optuna tuning step (Sect. III-E) is described without reporting search spaces or selected hyperparameters. Section VI-A explicitly concedes that hyperparameters may not have been equally optimized and that this could make the comparison unfair. Since Table V is the paper's central result, please report K and the hyperparameter configurations for all four methods (or provide a versioned pointer to the code), or temper the comparative conclusion to match the admitted uncertainty.","section":"Section III-D.4, Section VI-A"},{"comment":"The EEG labeling rule classifies the lower 60% and upper 22.2% of 'the level distribution' as awake and drowsy, but the paper does not say whether this distribution is computed per participant, per session, or globally across the dataset. This choice changes the learning task: per-participant quantiles make the labels rank-based within a driver, while a global distribution makes the threshold depend on dataset composition. Please state the grouping used for the quantiles and, if it is per-participant, discuss the implications for generalization to new drivers.","section":"Section III-C"}],"minor_comments":[{"comment":"The abstract contains typographical errors: 'frameworkiscarefullydesigned' is missing spaces and 'onfigurations' should be 'configurations'.","section":"Abstract"},{"comment":"The text says 'It uses an SMV model'; this should be 'SVM'.","section":"Section III-D.1"},{"comment":"Table III lists the SvmW input as θ at 25 Hz, but Sect. IV-B.1 states the sampling rate was set to 60 Hz; please clarify which rate was used for SvmW in C1 and C2.","section":"Section IV-B.1 and Table III"},{"comment":"The paragraph about Lstm repeats itself: 'the recall rate was relatively low' and later 'showed a recall as low as 5%'; consider merging the two sentences.","section":"Section V-A"},{"comment":"The statement that 'it was necessary to use test data to achieve a similar level of performance' for [4] is vague; specify whether this refers to the C1 training-configuration evaluation or another protocol.","section":"Section I"},{"comment":"The figure panels are labeled (a) and (b), but the text refers to 'above' and 'below'; please make the references consistent.","section":"Figure 4"},{"comment":"For archival reproducibility, please include a specific commit hash or versioned DOI for the public repository, in addition to the URL.","section":"Section III-E"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the open code repository may already contain a participant-level split that the paper fails to document. I recommend asking the authors to confirm the split implementation and, if the split is window-level, to rerun the experiments before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the negative results, not for the RF method. The paper does something useful: it reimplements three cited DDD methods on a public dataset under a common protocol, shows their reported numbers do not reproduce, and contributes a simple RF baseline that gets 88% accuracy and 0.85 AUC. The framework and code are public, the C1 vs C2 configurations are clearly distinguished, and the authors are candid about where their reimplementation might not match the originals. That is real evidence and a legitimate benchmark contribution.\n\nThe honest reading of the results is narrower than the abstract suggests. The claimed ranking (RF > SvmA > Lstm > SvmW) rests on the C2 split, and the paper only says the dataset was split 8:1:1. It does not say whether the split is at the participant level or the window level. With 3-second windows and 50% overlap, the same driver's overlapping windows can easily end up on both sides of the split. Driving behavior and EEG-derived drowsiness are autocorrelated within a session, so that can inflate accuracy and favor the model with the most features, which is RF. The stress-test note gets this right. The authors list individual differences as a threat to validity in VI-D but do not address it in the split design. That is the main soft spot.\n\nThe other weaknesses are real but smaller. There are no error bars or significance tests, so the gap between 88% and 65% is not shown to be robust. There is no majority-class baseline. The threats section admits RF had more features and that hyperparameter tuning may not have been equal across methods. Those concessions are honest, but they weaken the 'RF is best' claim.\n\nThe central reproducibility finding, that the 98%, 95%, and 91% numbers do not hold up under a standard train/test split, is credible and worth taking seriously. Even if the participant-level split issue were fixed, the paper would still show that the original methods lose most of their reported performance. The citation pattern is fair, and the public code and dataset make the comparison checkable.\n\nWho it is for: anyone in driver drowsiness detection who needs a sanity check on published numbers, and anyone benchmarking lightweight methods on public vehicle-dynamics data. It deserves a serious referee, mainly to force an explicit split description and, ideally, a participant-level analysis. I would accept it with revisions, not desk reject.","headline":"A genuinely useful reproducibility benchmark for vehicle-dynamics drowsiness detection, undermined mainly by an under-specified data split that could favor the proposed RF baseline.","tokens_in":11714,"tokens_out":1972,"would_cite":true,"duration_ms":20410,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simple random forest model, evaluated alongside three re-implemented published methods on a public dataset, achieves the best drowsiness-detection accuracy (88%) and shows the published methods' claimed performance does not reproduce.","keywords":["driver drowsiness detection","vehicle dynamics","random forest","SVM","LSTM","reproducibility","data leakage","MMDAP dataset"],"falsifier":"Re-run the four methods on a participant-disjoint split of MMDAP (all windows from each driver assigned to only one partition). If RF's accuracy and AUC fall to the level of the other methods, the central claim that RF is best under a fair protocol would be undermined.","tokens_in":10729,"feed_emoji":"🚗","tokens_out":6950,"duration_ms":65898,"temperature":0.7,"pith_summary":"This paper tries to establish that the performance of established vehicle-dynamics-based driver drowsiness detection (DDD) methods is not reproducible when the methods are re-implemented and evaluated under a common, standardized protocol on the public MMDAP dataset, and that in that same protocol a simple random forest (RF) method beats all three re-implementations. Re-implementing SvmA, SvmW, and Lstm, the authors obtain accuracies of 65%, 49%, and 70% in the standard configuration, far below the 98%, 95%, and 91% reported in the original papers; the proposed RF reaches 88% accuracy and 0.85 AUC. The authors attribute the gaps to non-standard evaluation practices in the original studies, including possible data leakage and reporting of training metrics. If true, the paper provides a strong argument for standardizing DDD evaluation and shows that simple methods on standard signals deserve attention as baselines.","feed_headline":"Simple random forest beats published drowsiness detectors in fair retest","feed_subtitle":"Published methods scored 49-70% accuracy in the same setup despite claims of 91-98%; the RF baseline hit 88%.","key_machinery":"The paper's central mechanism is a configurable DDD framework that parameterizes the machine-learning pipeline by data-split, labeling source, and hyperparameters, enabling a common evaluation. The proposed RF method combines all feature sets used by the three prior methods (36 statistical steering features, 8 wavelet energy features, and 15 summary features from five signals) and applies univariate feature selection by ANOVA F-values before training the random forest classifier; ground-truth labels come from EEG spectral ratios, with DRT events as an alternative label source.","core_discovery":"On the public MMDAP dataset, the authors re-implement three published lightweight vehicle-dynamics DDD methods (SvmA, SvmW, Lstm) and propose RF, which aggregates the feature sets of all three methods with an ANOVA F-value-based feature selection, trained with EEG-derived labels. Under configuration C2—an 8:1:1 train/validation/test split, 3-second windows with 50% overlap, 60 Hz sampling, and EEG-based drowsiness labels—RF achieves the highest AUC (0.85) and accuracy (88%) among the four methods. The re-implemented prior methods attain only 65% accuracy (SvmA), 49% accuracy (SvmW), and 70% accuracy (Lstm), whereas the original papers reported 98%, 95%, and 91%, respectively. The authors conclude that the prior results are largely due to non-standard evaluation protocols—likely training accuracy or data leakage—and that RF under a standardized protocol is a competitive and reliable baseline.","pith_inferences":["The paper does not specify whether the 8:1:1 split is participant-disjoint or purely per-window; a per-window random split could let the same driver's data appear in both training and test, inflating accuracy and possibly favoring the RF, so a participant-disjoint re-split is a direct test of the claim.","Because the RF uses strictly more features than the other methods, some of its advantage may come from feature availability rather than the classifier; testing SvmA, SvmW, and Lstm with the same full feature set would isolate that contribution.","The EEG labeling rule discards the middle 17.8% of the drowsiness distribution (upper 22.2% drowsy, lower 60% awake), making the remaining two classes easier to separate, so the reported 88% is specific to that labeling cutoff.","The released framework could serve as a benchmark harness that future DDD papers adopt to report standardized metrics, similar to fixed evaluation protocols in other applied ML fields."],"forward_implications":["The four-way comparison provides a reproducible baseline: a simple random forest on vehicle dynamics yields 88% accuracy and 0.85 AUC on MMDAP under a standardized protocol.","The published accuracy claims for the three re-implemented methods (98%, 95%, 91%) are not achieved under the common configuration, indicating those reported values likely stem from training accuracy or non-standard evaluation practices.","Using a public dataset and standard metrics allows DDD evaluation to be standardized, and the proposed framework supports multiple configurations and feature sets for future comparisons.","EEG-based labeling in a straight-road simulator scenario offers one fair labeling protocol, but the same framework's results may change with other label sources such as DRT events."],"supporting_citations":[{"why":"Supplies the SvmA method's design (SVM with ANFIS feature selection) and the reported 98.12% accuracy that the paper re-implements and fails to reproduce.","marker":"[4]"},{"why":"Supplies the SvmW method's multiwavelet packet energy features and the reported 95% correct rate that the paper re-implements.","marker":"[5]"},{"why":"Supplies the Lstm method's Bi-LSTM with attention architecture and the reported 91.22% accuracy and 0.974 AUC that the paper re-implements.","marker":"[6]"},{"why":"Provides the data-leakage framework and reproducibility-crisis discussion that motivates the paper's concern about inflated DDD metrics.","marker":"[7]"},{"why":"Provides the public MMDAP dataset used for all experiments, including the vehicle-dynamics signals and EEG data used for labeling.","marker":"[11]"},{"why":"Defines the EEG spectral ratio (theta+alpha)/beta that the paper uses to derive drowsiness ground-truth labels.","marker":"[24]"},{"why":"Documents the MMDAP experimental protocol (driving scenarios, DRT events, data collection) used to define windows and labels.","marker":"[25]"},{"why":"Introduces the random forest algorithm on which the proposed RF method is built.","marker":"[26]"}],"fun_headline_variants":["RF hits 88% in fair drowsiness test; prior methods fall to 49-70%","Published drowsiness detectors don't replicate; random forest wins at 88%","Fair benchmark: RF beats 91-98% claims, achieving 88% accuracy","Random forest outperforms published drowsiness models in transparent retest","Drowsiness: prior high scores vanish; RF achieves 88% in fair setup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that splitting the dataset into 8:1:1 training, validation, and test sets prevents data leakage, meaning no overlapping time windows and no data from the same participant appear in both training and test, but the paper does not describe participant-level separation.","fun_headline_variants_meta":{"raw":{"variants":["RF hits 88% in fair drowsiness test; prior methods fall to 49-70%","Published drowsiness detectors don't replicate; random forest wins at 88%","Fair benchmark: RF beats 91-98% claims, achieving 88% accuracy","Random forest outperforms published drowsiness models in transparent retest","Drowsiness: prior high scores vanish; RF achieves 88% in fair setup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1904,"prompt_tokens":979,"completion_tokens":925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":818}},"tokens_in":595,"tokens_out":925,"duration_ms":8298,"temperature":1.0,"reasoning_tokens":818,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:44:33.935006+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four methods on a participant-disjoint split of MMDAP (all windows from each driver assigned to only one partition). If RF's accuracy and AUC fall to the level of the other methods, the central claim that RF is best under a fair protocol would be undermined.","supporting_citations":[{"cited_title":"Driver Drowsiness Detection Based on Steering Wheel Data Applying Adaptive Neuro-Fuzzy Feature Selection,","cited_arxiv_id":null,"evidence_quote":"Supplies the SvmA method's design (SVM with ANFIS feature selection) and the reported 98.12% accuracy that the paper re-implements and fails to reproduce."},{"cited_title":"Detecting Driver’s Drowsi- ness Using Multiwavelet Packet Energy Spectrum,","cited_arxiv_id":null,"evidence_quote":"Supplies the SvmW method's multiwavelet packet energy features and the reported 95% correct rate that the paper re-implements."},{"cited_title":"Driver Distraction Detection Based on Vehicle Dynamics Using Nat- uralistic Driving Data,","cited_arxiv_id":null,"evidence_quote":"Supplies the Lstm method's Bi-LSTM with attention architecture and the reported 91.22% accuracy and 0.974 AUC that the paper re-implements."},{"cited_title":"Multi-modal Data Acquisition Platform for Behavioral Evaluation,","cited_arxiv_id":null,"evidence_quote":"Provides the public MMDAP dataset used for all experiments, including the vehicle-dynamics signals and EEG data used for labeling."},{"cited_title":"Using EEG spectral components to assess algorithms for detecting fatigue [Part 1],","cited_arxiv_id":null,"evidence_quote":"Defines the EEG spectral ratio (theta+alpha)/beta that the paper uses to derive drowsiness ground-truth labels."},{"cited_title":"Estimating Systemic Cognitive States from a Mixture of Physiological and Brain Signals,","cited_arxiv_id":null,"evidence_quote":"Documents the MMDAP experimental protocol (driving scenarios, DRT events, data collection) used to define windows and labels."}],"review_version":1}