{"id":"e1589a4b-f6bc-429f-a45f-c77264108a61","arxiv_id":"2507.13403","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"UL-DD is a 1,400-minute multimodal driver drowsiness dataset with video, biometric, behavioral, and telemetry streams labeled with KSS every four minutes.","lead":"This paper introduces UL-DD, a new multimodal dataset for driver drowsiness detection that combines depth, infrared, and posture video with wristband biosignals, grip pressure, and truck-simulator telemetry from 19 drivers. It aims to support early drowsiness detection by labeling driver state every four minutes with the nine-level Karolinska Sleepiness Scale.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 9-level gradual-change claim is not supported by the reported reliability evidence; the paper's own unweighted Kappa (0.619) and missing label distributions leave the central contribution unverified.","rationale":"The reader's weakest assumption identifies exactly the vulnerability I would stress-test: the KSS self-reports under every label. The paper's own statistics complicate rather than support the 9-level gradual-change claim. Cronbach's alpha is not a validity check for a single-item repeated measure, and the unweighted Kappa of 0.619 is modest. The dataset may still be valuable for binary or three-level drowsiness detection, but the central claim of gradual 9-level annotations requires direct evidence from label distributions. The proposed test would settle this. If the label distribution shows genuine within-session spread, the claim holds; if not, the claim should be softened. Thus the verdict remains CONDITIONAL, with the condition being public label-distribution evidence or a revised claim. This agrees with the reader's analysis, so agreement_with_reader is 'agree.'","tokens_in":17877,"tokens_out":1353,"duration_ms":16992,"concrete_test":"Release or reproduce the label distribution from Labels.csv: for each subject and session, list the ten KSS values and the within-session changes. Then compute (a) the proportion of sessions with at least one monotonic increase of at least 2 KSS levels, and (b) the proportion of sessions with at least 4 distinct KSS levels. If most sessions are confined to a 2-level band, the dataset does not deliver the claimed gradual changes.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that UL-DD captures gradual, 9-level changes in driver drowsiness through KSS self-reports every four minutes. This is load-bearing because the dataset's novelty over existing benchmarks (e.g., DROZY, 3MDAD) rests on 'gradual changes' and '9-level annotations.' The reported validation does not establish this. First, Section 3.7 reports Cronbach's alpha = 0.99, but alpha measures internal consistency of multi-item scales, not the validity of repeated single-item self-reports; a high value here mainly shows that participants rate themselves consistently across the repeated assessments, which is equally consistent with two homogeneous blocks. Second, Section 6.3 reports unweighted Cohen's Kappa = 0.619 for the 9-level scale, i.e., only 'substantial' agreement, despite the claim of reliable gradual labels. The quadratic-weighted Kappa = 0.967 is inflated by the ordinal structure and does not demonstrate that fine 9-level distinctions are trustworthy. Third, the paper never reports the distribution of labels: how many of the ten per-session intervals occupy which KSS levels, and whether within-session increases actually occur. If the labels form two blocks (alert vs. drowsy sessions), the dataset fails to deliver the claimed gradual variation. The reviewer's weakest assumption is therefore supported by the manuscript itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces UL-DD, a multimodal driver drowsiness dataset collected from 19 participants in a fixed-base driving simulator. Each participant was recorded in up to two 40-minute sessions (one alert, one drowsy), with three video streams (ZED 2 depth, IR, and webcam pose), wrist-worn biometrics (Empatica E4: HR, EDA, TEMP, ACC, BVP, IBI; Checkme O2: SpO2, pulse rate, motion), steering-wheel grip pressure, driving telemetry from American Truck Simulator, and derived facial landmarks, facial action units, and pose landmarks. Drowsiness was self-reported with the 9-level Karolinska Sleepiness Scale every four minutes, yielding ten interval labels per session. The technical validation includes SNR analysis, Kruskal-Wallis and mixed-effects tests on biometric signals, inter-rater Kappa, and SVM/Random Forest classification with 5-fold cross-validation. The authors argue that continuous 40-minute sessions and 9-level KSS annotations distinguish UL-DD from existing public drowsiness datasets.","tokens_in":18147,"tokens_out":5760,"duration_ms":59843,"significance":"If the dataset is delivered as described, it is a potentially valuable addition to the field: few public resources combine depth/IR/pose video with wrist-based physiological signals, grip pressure, and simulator telemetry, and the explicit 40-minute continuous sessions are longer than many comparable benchmarks such as DROZY or RLDD. The paper's strengths include clear documentation of the sensor setup, the provision of derived landmark and action-unit features, and a stated intention to release code and tutorials. The main caveat is that the paper's novelty rests on the claim of gradual, 9-level changes in drowsiness, and the reported validity evidence does not yet substantiate that claim; in addition, the classification validation does not demonstrate subject-independent generalization. These issues are addressable with additional analyses and reporting.","major_comments":[{"comment":"The central claim of gradual within-session drowsiness change is not supported by any reported label distribution. Since each subject has only two sessions (one begun alert, one begun at KSS >= 6) and labels are collected every four minutes, the ten KSS values per session could be two nearly homogeneous blocks. Please report per-subject, per-session KSS trajectories, the frequency of each KSS level, and the number of within-session transitions; without this, the 'gradual changes' and '9-level' novelty is unverified.","section":"Sections 3.6, 3.7, and Table 4"},{"comment":"Cronbach's alpha of 0.99 is not an appropriate reliability measure here. Alpha estimates internal consistency among items of a multi-item scale; applying it to ten repeated single-item KSS assessments simply shows that ratings are consistent across time points and is compatible with two constant blocks. Replace this with appropriate analyses such as within-session variance, transition matrices, or test-retest reliability, and do not claim that alpha demonstrates gradual change.","section":"Section 3.7"},{"comment":"The 5-fold cross-validation is ambiguous with respect to subject independence. The text says that raw data from all subjects and sessions were combined before splitting and that folds were 'stratified to maintain the distribution of drowsiness levels across subjects,' which suggests that intervals from the same subject can appear in both training and test folds. If so, the reported accuracies (63-84%) include subject leakage. Please clarify the split at the subject level or rerun the experiments with leave-subjects-out evaluation.","section":"Section 6.4 and Figure 8"},{"comment":"The inter-rater results do not establish that 9-level KSS labels are reliable. Unweighted kappa of 0.619 for the full scale is only 'substantial' agreement, and the quadratic-weighted kappa of 0.967 is expected to be high given the ordinal structure, even when raters differ by one or two levels. Report the full confusion or disagreement matrix between experts and self-reports, per-level agreement, and agreement for the 3-level binning separately, and discuss whether the fine 9-level distinctions are trustworthy.","section":"Section 6.3"},{"comment":"The validation reports only point accuracy without error bars, class distribution, a majority-class baseline, or a chance level. For a three-class problem with possibly imbalanced labels, 63% telemetry accuracy and 80-84% multimodal accuracy are not interpretable without these. Please add confidence intervals, a no-signal baseline such as majority-class prediction, and per-class precision and recall.","section":"Section 6.4 and Figure 8"}],"minor_comments":[{"comment":"The file format descriptions are inconsistent: 'User FL Session.csv' says 136 columns for 68 landmarks plus 'the frame number recorded in the first column,' making 137 columns; likewise, PL (99 + 1) and FAU (30 + 1) need the column counts clarified.","section":"Section 4.1"},{"comment":"The IR camera is described as '1080p' but listed at 640x360 at 60 fps, and the ZED 2 camera is listed at 1344x376 at 60 fps, which is not a standard ZED 2 mode; specify the original and processed resolutions and any cropping or resizing steps.","section":"Section 3.2.2 and Section 3.2.1"},{"comment":"The BVP bullet says 'The TEMP signal was recorded at a frequency of 4 Hz,' which appears to be a copy-paste error, and the Empatica E4's BVP sample rate is typically 64 Hz; verify and correct the stated frequency.","section":"Section 4.1"},{"comment":"The dummy coding (reference level) for the Low/Medium/High bins is not defined, so the signs of T.Low and T.Med cannot be interpreted; state the reference category and reconcile the text's claim that 'EDA decreased with increased drowsiness' with the table's coefficients.","section":"Table 5"},{"comment":"The reporting of non-significant tests is incomplete: the text mentions only 'ACC X' for the Medium-vs-High comparison, and BVP is excluded for Low-vs-High; list the full set of signals and p-values so readers can see which signals failed to reach significance.","section":"Section 6.1.1"},{"comment":"Downsampling all modalities to 4 Hz is applied without justification; for telemetry (60 Hz) and landmark streams (60 fps), this may discard saccadic or steering micro-correction information, so please state why 4 Hz is appropriate or provide a sensitivity analysis.","section":"Section 6.4"},{"comment":"The paper says the dataset is 'available upon request' but also that 'Readers can access all the code and tutorials along with the dataset'; clarify the exact access mechanism, repository URL, and any usage agreement.","section":"Sections 7 and 8"},{"comment":"The 'No of Classes' column lists UL-DD as 9, while the validation experiments use 3-level bins; clarify how the labels are provided in the dataset and how users should treat the 9-level labels relative to the binned evaluation.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper reads as a data-descriptor submission, and the central deliverable is the dataset itself rather than the validation models. The editor may wish to verify that the repository is actually accessible and contains all promised files, since the manuscript currently says 'available upon request' while also claiming code and tutorials are accessible. The major revision should focus on the label-distribution evidence and subject-independent validation, as these directly support the paper's stated novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset itself, if it ships, is a real contribution — the combination of depth/IR/posture video, E4 and O2 Max biosignals, grip pressure, and truck-sim telemetry with KSS every four minutes over 40-minute sessions is not something existing public datasets offer. The protocol description is detailed and the quality checks (SNR, frame drop rate, synchronization) are thoughtful. But the paper's central claim — that UL-DD captures gradual, 9-level changes in drowsiness — is not supported by the evidence it reports.\n\nThe reliability argument is the soft spot. Cronbach's alpha = 0.99 is the wrong tool here: alpha measures internal consistency of a multi-item scale, not the validity of repeated single-item self-reports. The unweighted Kappa of 0.619 for the 9-level scale is 'substantial' but leaves a lot of disagreement; the quadratic-weighted 0.967 is inflated by the ordinal structure. More importantly, the paper never shows the label distribution — how many of the ten per-session intervals actually land at each KSS level, and whether within-session increases occur. Without that, the data could be two homogeneous blocks (alert vs. drowsy) and the 'gradual changes' claim would be hollow.\n\nThe classification validation has a similar problem. Five-fold cross-validation is done on combined data from all subjects and sessions, with stratified folds, but the paper never states that the split is subject-independent. That's a leakage risk. No error bars, no baseline, and the accuracies (e.g., 84% for fusion) are hard to interpret without them.\n\nOn the positive side, the authors are transparent about limitations: small sample, gender imbalance, simulator environment, missing video for some subjects. They provide extracted features for those subjects, which is thoughtful. But the dataset is only 'available upon request,' and the code repository isn't linked in the paper, so reproducibility is currently unverifiable.\n\nThis is a fixable paper. The dataset is potentially valuable, but the label validity and evaluation need to be demonstrated properly. A serious referee should see it, but the revision needs label distributions, a subject-independent benchmark, and a direct data link.\n\nRecommendation: send to peer review, but expect major revision.","headline":"A potentially useful multimodal drowsiness dataset whose core gradual-change claim is not yet supported by the reported validation.","tokens_in":18725,"tokens_out":2353,"would_cite":false,"duration_ms":25229,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UL-DD is a public multimodal dataset that records driver drowsiness as a gradual 9-level process through continuous 40-minute sessions with synchronized video, biometric, and behavioral signals.","keywords":["multimodal driver drowsiness dataset","driver behavior analysis","continuous biosignals drowsiness detection","depth camera","biometric signals","steering behavior telemetry","early fatigue and drowsiness detection","open-access dataset"],"falsifier":"Compute the ten per-session KSS ratings for every subject: if most awake sessions stay below 4 and most drowsy sessions stay above 6, with few intermediate labels, the gradual-change claim fails. A second check is the expert-rated third of the videos: the reported unweighted κ of 0.619 on the 9-level scale means the disagreement pattern should be inspected to see whether errors concentrate at middle levels, which would weaken the ordinal labels.","tokens_in":17711,"feed_emoji":"😴","tokens_out":8710,"duration_ms":90053,"temperature":0.7,"pith_summary":"This paper introduces UL-DD, a public dataset built to record driver drowsiness as a gradual process rather than a binary awake-or-asleep state. It claims that two 40-minute driving-simulator sessions per subject, one started while alert and one started with a Karolinska Sleepiness Scale (KSS) score of at least 6, yield self-reported KSS ratings every four minutes that trace slow transitions across a 9-level scale. The dataset bundles synchronized streams from three video cameras, two wrist-worn sensor bands, steering-wheel grip pressure sensors, and simulator telemetry, together with extracted facial landmarks, facial action units, and pose landmarks. If the claims hold, researchers gain a public benchmark for early and fine-grained drowsiness detection, in contrast to existing public datasets that mostly offer short clips, single modalities, or coarse alert/sleepy classes.","feed_headline":"Drowsiness dataset logs 9 gradual levels over 40-minute drives","feed_subtitle":"Synchronized video, biometrics, grip, and telemetry support earlier, finer-grained detection.","key_machinery":"The carrying object is the dataset itself, specifically its session design: each subject drives a simulated truck for two 40-minute sessions, one from an alert state and one from a KSS ≥ 6 sleepy state, and reports KSS every four minutes to produce ten ordinal labels per session. That design is what allows the paper to claim gradual, 9-level state trajectories instead of discrete alert/drowsy categories. Around it are the synchronized multimodal streams—three video cameras, two wristbands, grip-pressure sensors, and 60 Hz simulator telemetry—and the machine-extracted intermediate features (68 facial landmarks, 30 facial action units, 33 pose landmarks) that let researchers work with the data even when raw video is withheld.","core_discovery":"The central claim is that UL-DD is a multimodal driver-drowsiness resource that captures gradual changes in driver state through continuous 40-minute recording sessions and 9-level KSS annotations. The authors report collecting two sessions per subject (19 subjects total, with 16 completing both sessions), one when the subject was alert and one when the KSS score was at least 6, with self-reports every four minutes during active driving. The dataset combines three video streams, physiological signals from two wrist-worn sensors, grip pressure from both sides of the steering wheel, and simulator telemetry, with all streams temporally aligned to manual session start and end times. The paper further reports technical validation: most biometric signals differ significantly across binned drowsiness levels in mixed-effects models (BVP is the main exception), inter-rater agreement with an expert on the 9-level scale is high when quadratically weighted (κ = 0.967), and an early-fusion model using biometric, behavioral, and facial features reached 84% accuracy in three-level drowsiness classification.","pith_inferences":["Editorial inference: if the KSS trajectories within a session do vary gradually, the dataset could support predicting time-to-drowsiness-onset or continuous alertness scores, tasks that binary benchmarks cannot address.","Editorial inference: the four-minute KSS prompts may themselves briefly raise arousal (for example in EDA or heart rate), so analyses should test for prompt-locked artifacts before treating the physiological signals as purely drowsiness-driven.","Editorial inference: the mixed-effects results in Table 5, where BVP showed no significant drowsiness association and HR did not separate medium from high drowsiness, suggest that pulse rate is the more reliable cardiac index and that modality-fusion weights should reflect that.","Editorial inference: a natural extension is to benchmark models that predict all 9 KSS levels directly and compare their error structure with the 3-level validation reported here; this would reveal whether the intermediate labels carry usable signal."],"forward_implications":["Models trained on UL-DD can be evaluated on 9-level ordinal labels rather than binary alert/sleepy decisions, a direct consequence of the annotation scheme.","In the paper's validation, combining biometric, behavioral, and facial features with early fusion reached 84% accuracy for three-level drowsiness classification, above every individual modality, supporting multimodal fusion as the route to better detection.","Because extracted facial landmarks, action units, and pose landmarks are provided for all subjects, feature-based models can be built and compared even for the subjects whose raw video was withheld.","The per-subject, per-session folder structure with manually synchronized start and end times makes it possible to reproduce the fusion pipeline and to study within-session temporal dynamics at 4 Hz resampling."],"supporting_citations":[{"why":"Defines the DROZY multimodal drowsiness database (EEG, EOG, ECG, EMG, NIR) that UL-DD compares against for multimodal coverage and label granularity.","marker":"[5]"},{"why":"RLDD video-only drowsiness dataset with 10-minute clips and 3 classes; the baseline UL-DD contrasts with its longer continuous sessions and 9-level labels.","marker":"[10]"},{"why":"3MDAD multimodal multi-view driver dataset with RGB, IR, and depth; comparator for the two-class limitation UL-DD aims to surpass.","marker":"[14]"},{"why":"DD-Pose head-pose benchmark from naturalistic drives; cited as a diverse benchmark that UL-DD extends with physiological and behavioral streams.","marker":"[17]"},{"why":"MediaPipe pose estimation framework used to extract the 33 pose landmarks included for every subject.","marker":"[42]"},{"why":"Introduces the Karolinska Sleepiness Scale used for all 9-level drowsiness labels every four minutes.","marker":"[46]"},{"why":"Cronbach's alpha reliability measure used to claim internal consistency (α = 0.99) of the repeated KSS ratings.","marker":"[47]"},{"why":"Dlib used to extract the 68 facial landmarks and the facial action units derived from them.","marker":"[48]"},{"why":"Mixed-effects models used to test whether biometric signals differ across drowsiness levels while accounting for inter-subject variability.","marker":"[65]"}],"fun_headline_variants":["Multimodal drowsiness dataset tracks 9 levels in 40-min sessions","Gradual drowsiness captured: video, biometrics, grip, telemetry","New dataset records drowsiness shifts every 4 minutes in driving sim","UL-DD: 19 drivers, 40-min runs, 9 KSS drowsiness levels","Drowsiness evolution logged via 3 videos, biosignals, and grip data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole dataset's value as a gradual-drowsiness resource depends on the assumption that the KSS ratings participants gave every four minutes were truthful, self-aware, and spread across the scale within each session rather than clustering into one alert block and one sleepy block.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal drowsiness dataset tracks 9 levels in 40-min sessions","Gradual drowsiness captured: video, biometrics, grip, telemetry","New dataset records drowsiness shifts every 4 minutes in driving sim","UL-DD: 19 drivers, 40-min runs, 9 KSS drowsiness levels","Drowsiness evolution logged via 3 videos, biosignals, and grip data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1400,"prompt_tokens":1014,"completion_tokens":386,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":280}},"tokens_in":630,"tokens_out":386,"duration_ms":4540,"temperature":1.0,"reasoning_tokens":280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:42:32.304415+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the ten per-session KSS ratings for every subject: if most awake sessions stay below 4 and most drowsy sessions stay above 6, with few intermediate labels, the gradual-change claim fails. A second check is the expert-rated third of the videos: the reported unweighted κ of 0.619 on the 9-level scale means the disagreement pattern should be inspected to see whether errors concentrate at middle levels, which would weaken the ordinal labels.","supporting_citations":[{"cited_title":"In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp","cited_arxiv_id":null,"evidence_quote":"Defines the DROZY multimodal drowsiness database (EEG, EOG, ECG, EMG, NIR) that UL-DD compares against for multimodal coverage and label granularity."},{"cited_title":"In: Proceedings of the Ieee/cvf 23 Conference on Computer Vision and Pattern Recognition Workshops, pp","cited_arxiv_id":null,"evidence_quote":"RLDD video-only drowsiness dataset with 10-minute clips and 3 classes; the baseline UL-DD contrasts with its longer continuous sessions and 9-level labels."},{"cited_title":"Signal Processing: Image Communication 88, 115960 (2020)","cited_arxiv_id":null,"evidence_quote":"3MDAD multimodal multi-view driver dataset with RGB, IR, and depth; comparator for the two-class limitation UL-DD aims to surpass."},{"cited_title":"In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp","cited_arxiv_id":null,"evidence_quote":"DD-Pose head-pose benchmark from naturalistic drives; cited as a diverse benchmark that UL-DD extends with physiological and behavioral streams."},{"cited_title":"International journal of neuroscience 52(1-2), 29–37 (1990)","cited_arxiv_id":null,"evidence_quote":"Introduces the Karolinska Sleepiness Scale used for all 9-level drowsiness labels every four minutes."},{"cited_title":"International journal of medical education 2, 53 (2011)","cited_arxiv_id":null,"evidence_quote":"Cronbach's alpha reliability measure used to claim internal consistency (α = 0.99) of the repeated KSS ratings."},{"cited_title":"Springer, ??? (2006)","cited_arxiv_id":null,"evidence_quote":"Mixed-effects models used to test whether biometric signals differ across drowsiness levels while accounting for inter-subject variability."}],"review_version":1}