{"id":"0c15ae30-bc9f-4fb5-98dd-07ade82b4aeb","arxiv_id":"2509.01642","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new cognitive-load dataset with driving, gaming, and n-back tasks shows multimodal models beat unimodal ones, but cross-domain transfer remains poor.","lead":"This paper introduces a new multimodal dataset for measuring cognitive load, combining driving data with two video games and a standard n-back task. It trains several deep-learning models and shows that models fail to transfer well from one task to another.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gaming low/high labels are not validated by objective performance, so cross-domain claims may reflect task-phase cues rather than cognitive load; a label-validity re-analysis is needed.","rationale":"The paper's central claims rest on the binary task-load labels. The reader identified the same weakest assumption, and Sections III-C and IV-B confirm the concern is concrete: gaming performance did not decline with level, and Hogwarts lower levels lack performance metrics, leaving only subjective TLX to support the high/low distinction. Several input modalities are strongly task-phase-specific, so high AUROC could reflect phase detection rather than load estimation; Section V-A explicitly concedes this risk. The proposed re-labeling experiment would distinguish 'phase detection' from 'load estimation.' If the design-based labels are not replicable with subjective labels, the headline claims about universal cognitive load estimation and cross-domain generalization are weakened, but the dataset and baselines would remain useful. A conditional rather than outright rejection verdict is therefore appropriate, and I do not propose changing the reader's conditional verdict.","tokens_in":29988,"tokens_out":4068,"duration_ms":51315,"concrete_test":"Retrain the three small architectures on the gaming subset using per-participant NASA-RTLX-based low/high labels (median split per participant across phases; for Overcooked, also a performance-based split) under the same subject-wise 5-fold CV, then evaluate gaming-to-n-back/driving transfer. If design-based labels outperform RTLX-based labels by a large margin (e.g., >0.05 AUROC) or if RTLX-based gaming models fail to transfer, the original labels and cross-domain conclusions are confounded by task structure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claims—that multimodal models estimate cognitive load and that cross-domain performance drops reveal limits of universal load estimation—depend entirely on the binary low/high labels assigned to gaming phases (Section III-C). For Hogwarts Legacy and Overcooked, these labels are defined by task structure (free flight/baseline vs. guided flight/race; education mode vs. time-pressure/two cooks). The paper's own validation fails to confirm the load manipulation: Section IV-B reports that Overcooked performance 'did not decline with increasing level' and that Hogwarts lower levels provided no performance metrics. The only supporting evidence is post-hoc NASA-TLX, which can reflect perceived task demand rather than experienced load and was collected after each level. Since several input modalities—eye movement, head rotation, facial action units—are exactly the signals that differ between game phases (race vs. free flight, one vs. two cooks), a model could achieve high AUROC by detecting phase rather than load. Section V-A concedes this confound explicitly: 'models may exploit task or context specific cues unrelated to experienced load.' Unless label validity is established per participant and per phase, the headline claims about universal cognitive load estimation are not secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces REVELIO, a multimodal cognitive-load dataset that combines an n-back reference task, the existing ADABase driving dataset, and two new video-game scenarios (Overcooked! 2 and Hogwarts Legacy), with synchronized physiological, eye-tracking, facial, and movement signals. The authors train several end-to-end architectures (xLSTM, ConvNeXt, Transformer, LSTM, ResNet) on unimodal and multimodal inputs and evaluate within-domain and cross-domain performance using subject-wise 5-fold cross-validation. They report that multimodal inputs generally outperform unimodal ones, that training on diverse data yields more robust models, and that single-task models transfer poorly to other applications. The main contribution is the dataset plus a broad set of baselines and a data card.","tokens_in":30259,"tokens_out":5275,"duration_ms":66505,"significance":"If the empirical claims hold, this is a useful contribution: it extends cognitive-load benchmarking from laboratory n-back tasks to realistic gaming applications, provides systematic architecture and modality comparisons, uses a leakage-aware evaluation protocol, and reports calibration. The decision to anchor the gaming data to an n-back foundation is methodologically attractive. However, the central empirical claims rest on the validity of the binary low/high task-load labels assigned to the gaming phases, and that validity is not established. Because several reported claims—especially cross-domain generalization and the superiority of multimodal fusion—depend on these labels, the paper's conclusions are proportionate to the evidence only after a label-validity re-analysis. The strengths (subject-wise CV, ECE reporting, reproducible architectural details, a data card) are real and should be credited.","major_comments":[{"comment":"The binary gaming labels are load-bearing. Section III-C1 assigns Overcooked! 2 low load to baseline/education mode and high load to time pressure with two cooks, and Hogwarts Legacy low load to free flight and high load to guided flight/race. However, Section IV-B reports that Overcooked performance 'did not decline with increasing level' (Fig. 9b), and Hogwarts performance data exist only for the race level (Fig. 9c), so the lower-level phases have no objective performance validation. The only supporting evidence is post-hoc NASA-TLX (Fig. 8), with no significance tests or per-participant manipulation checks. Since gaze, head rotation, and facial AUs are exactly the signals that differ between game phases (race vs. free flight, one vs. two cooks), the gaming AUROC values in Table II (e.g., 0.84–0.86 for Eye Comb.) can be achieved by detecting task phase rather than cognitive load. Sect","section":"Section III-C, Section IV-B, Section V-A"},{"comment":"The claim that 'multimodal approaches consistently outperform unimodal baselines' is not consistently supported by the reported numbers. In Table II, for the Gaming subset the full multimodal combination 'Bio., AUs, ET, Move.' achieves 0.73–0.78 AUROC across architectures, whereas the unimodal 'Eye Comb.' achieves 0.85–0.86; the unimodal 'ET Pupil Diameter' also reaches 0.81–0.82. For n-back, 'Eye Comb.' (0.85–0.86) is comparable to or better than the full multimodal combinations (0.81–0.85). No statistical significance tests or effect sizes are reported, so 'consistently outperform' is an overstatement. Please either qualify the claim to specific modality combinations and domains, or provide pairwise statistical comparisons supporting the superiority claim.","section":"Abstract, Section IV-C, Table II"},{"comment":"The cross-domain generalization conclusion is conditioned on label equivalence across domains. Since the gaming labels are not validated (see major comment 1), a model trained on n-back or driving and evaluated on gaming may show low AUROC because the gaming 'high load' label does not correspond to n-back/driving load levels, or because the gaming labels encode phase-specific cues. Conversely, the apparent benefit of training on diverse data could reflect learning domain-specific phase structure rather than load. The conclusion that 'single-task models lack cross-domain generalizability' should be restricted to the label operationalization used here, and ideally re-examined after label validation or with an analysis of which features drive transfer. At minimum, the paper should state that the reported transfer results are for task-phase labels, not necessarily for experienced cognitive l","section":"Section IV-C3, Fig. 12, Table III"}],"minor_comments":[{"comment":"Caption uses 'ConvNext' while the text and Table II use 'ConvNeXt'; please make consistent.","section":"Fig. 10"},{"comment":"The ECE equation defines p_i as 'the accuracy in bin i' but then uses c_i as 'the average confidence in bin i'; the notation should be aligned (typically p_i is the empirical accuracy and c_i is the mean confidence).","section":"Section III-D"},{"comment":"For binary classification the final output should be y ∈ R^{B×1} rather than y ∈ R^{B×C}; as written, C is ambiguous with the input channel dimension.","section":"Section III-D"},{"comment":"The interpretation that two-cook management 'did not achieve expected performance gains, suggesting a ceiling effect' is presented as a post-hoc explanation of a null result. Please label it as a hypothesis, not a finding.","section":"Section IV-B"},{"comment":"There are several typographical issues (e.g., 'Eskofier' with broken encoding in the author block) and inconsistent capitalization of 'Transformer' vs. 'transformer'. A careful proofread is recommended.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main technical concern is construct validity of the gaming labels; this is fixable by adding a manipulation check and re-running the affected analyses or by explicitly reframing the claims as task-phase discrimination. The paper otherwise shows careful methodology and a useful benchmark. I would be willing to accept after the label-validity issue is resolved and the multimodal-superiority claim is made precise. Also note that the dataset is only available under a restrictive EULA; the journal may wish to consider whether this meets its data-availability expectations, though it does not affect my technical assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — you should know this before reading: the paper's real contribution is the dataset, not the modeling. It adds a gaming block (Overcooked, Hogwarts Legacy) to the ADABase driving data plus n-back under a common recording protocol, and it runs a wide, well-documented sweep of architectures and modalities. The headline negative result—single-domain models don't transfer—is credible and worth having, even though the driving/n-back part was already shown by Foltyn et al.\n\nWhat it does well: subject-wise 5-fold CV with no subject leakage; AUROC and ECE reported; training details and runtimes in the supplement; the limitations section (V-A) is unusually straight about construct validity. The multimodal-over-unimodal finding is consistent, though not novel.\n\nThe soft spot is load-bearing: the gaming low/high labels come from task phases, not from demonstrated load. Overcooked performance did not decline with level (Fig. 9b), and Hogwarts lower levels have no performance metrics. The only validation is post-hoc NASA-TLX, which can track perceived demand and is collected after each level. Because eye movement, head rotation, and action units are exactly the signals that distinguish race from free flight or one cook from two, a model can score high AUROC by detecting the phase. The paper admits this in V-A. That means the 'universal cognitive load estimation' claim is not fully secured—the cross-domain drop might partly be a drop in phase-specific cues. This is fixable: per-participant, per-phase label validation, or at least a re-analysis showing the model separates load from structure.\n\nTwo smaller issues: the 40-second window and the final architecture selection appear to have used test-fold performance in the same dataset, which weakens the baseline figures even if the qualitative conclusions hold. And the data are not open; a EULA-gated subset with non-academic restrictions limits benchmark adoption.\n\nMy take: this is a solid dataset paper with an honest negative result, not a breakthrough. It deserves peer review because the benchmark itself is valuable and the flaws are addressable. I would welcome it into the literature after a label-validity re-analysis and clearer separation between model selection and evaluation.","headline":"A solid multimodal benchmark paper whose gaming labels need validation before the cross-domain claims can carry their full weight.","tokens_in":30739,"tokens_out":2439,"would_cite":true,"duration_ms":28364,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Models that read cognitive load from biosignals, eye tracking, and facial expressions in one setting — driving, a memory test, or a video game — lose significant accuracy when moved to another setting, even when signals are fused.","keywords":["cognitive load","task load estimation","multimodal learning","cross-domain generalization","physiological signals","deep learning","dataset","human-computer interaction"],"falsifier":"Train each architecture on one domain alone — n-back, driving, or gaming — with identical windows, sensors, and splits, then test on the other two. The paper reports clear AUROC drops for these transfers; if an n-back-only model matched gaming-trained models on the gaming test set, the lack of cross-domain generalizability would be refuted. A companion check: if removing task-locked signals (gaze targets, pupil luminance) makes the drop vanish, the failure is cue learning, not load estimation; if the drop persists using only task-general signals (ECG, EDA, respiration), the load-induction labe","tokens_in":29904,"feed_emoji":"🧠","tokens_out":9312,"duration_ms":98824,"temperature":0.7,"pith_summary":"This paper tries to establish a sobering fact about automated mental-workload detection: a model trained in one application — a driving simulator, the n-back working-memory test, or a commercial video game — loses accuracy when moved to another, so a 'universal' cognitive-load sensor does not yet exist. To make that claim measurable, it introduces REVELIO, a multimodal dataset that keeps the same physiological and behavioral sensors, the same n-back reference task, and the same low/high annotation logic fixed across three application domains. On this foundation it shows that multimodal models consistently beat unimodal ones, that the most informative signals shift with the task (pupil diameter in n-back, eye movement in driving), and that models trained on all domains together are more accurate and better calibrated on every subset than single-domain models. If the paper is right, the route to universal load estimation is diverse training data and standardized cross-domain benchmarks, not better architectures alone.","feed_headline":"Mental-load AI fails to transfer across tasks, new dataset shows","feed_subtitle":"Even models fusing biosignals, gaze, and facial cues drop sharply on new tasks; diverse training data is the remedy.","key_machinery":"REVELIO, the paper's dataset and evaluation framework. The load-bearing piece is a common protocol: identical Biopac physiological sensors, the same Tobii eye tracker, the same facial-video pipeline, and the same n-back test are recorded across a driving simulator, Overcooked! 2, and Hogwarts Legacy. Each task level is labeled low/high load by design, then verified with performance metrics and post-level NASA-TLX ratings. Subject-wise grouped 5-fold cross-validation plus a full train-on-one/test-on-another matrix, scored by AUROC and Expected Calibration Error, turns 'does it generalize?' into a concrete, reproducible measurement.","core_discovery":"The paper's central claim is that cross-domain generalization, not architecture, is the binding constraint on universal task-load estimation. xLSTM, ConvNeXt, and Transformer encoders, trained with subject-wise cross-validation, reach comparable in-domain accuracy, and multimodal inputs consistently beat unimodal ones across every application subset. But models trained on one domain and tested on another lose accuracy: n-back-trained models do not carry to gaming or driving, and driving-only models excel only in driving. Training on all domains jointly restores accuracy and calibration everywhere. The paper reads the transfer failure as models leaning on task-specific cues for a construct me","pith_inferences":["The n-back anchor could be pushed further than the paper does: if every future domain is recorded with an n-back session under the same sensors, load levels across domains could be aligned quantitatively instead of by design assumption, making transfer a calibrated measurement rather than a model's opinion.","Because gaming performance did not decline with task level in the paper's own data, the gaming labels are probably the noisiest of the three domains; weighting training samples by agreement between design, performance, and NASA-TLX ratings could improve transfer and is a cheap experiment the paper does not run.","A deployment-time extension the paper leaves implicit: instead of always fusing everything, a system could first recognize the domain and then select its best sensors — gaze for driving, pupil for n-back — potentially matching full-fusion accuracy at lower cost, provided the domain itself is detectable.","The binary low/high split may hide the hardest cases: the paper's own limitations note that overload states are collapsed into 'high'; a continuous-regression variant would test whether universal models fail at the extremes or only at the boundary."],"forward_implications":["Deploying a load detector in a new application without training on that application's data should be expected to fail: single-domain training systematically reduces AUROC elsewhere, so system builders should budget for domain-specific data collection.","Fusing modalities is the reliable default: biosignals plus eye tracking, augmented by facial action units, gives the most consistent performance across all subsets, whereas every single modality is weak in at least one domain.","The best signal depends on the task — pupil diameter leads in n-back, eye movement in driving, ECG overall — so modality choice should be validated per deployment context rather than assumed portable.","Calibration matters as much as accuracy: models trained on diverse data stay well-calibrated on unseen subsets, while single-domain models can be confident-but-wrong off-domain, so ECE should be reported alongside AUROC.","The n-back can serve as the common reference that makes future cross-domain comparisons meaningful: any new application recorded with the same sensors and an n-back session can be benchmarked head-to-head against these baselines."],"supporting_citations":[{"why":"Supplies the driving-simulator dataset, recording equipment, and n-back protocol that the new gaming data extends, plus the label-mapping precedent.","marker":"[15]"},{"why":"Provides the NASA-TLX questionnaire used to verify every task-level annotation subjectively.","marker":"[34]"},{"why":"Provides the n-back paradigm used as the scientific reference task anchoring the low/high load labeling.","marker":"[35]"},{"why":"Establishes the windowing and robustness-evaluation protocol (subject-wise splits, train/evaluate across levels) that the paper follows.","marker":"[45]"},{"why":"Gives the inclusive definition of mental workload that guides dataset creation and the annotation categories.","marker":"[1]"},{"why":"Defines the xLSTM encoder, one of the three architecture families compared across domains.","marker":"[53]"},{"why":"Defines the ConvNeXt encoder, the convolutional family that performs best on gaming and n-back subsets.","marker":"[57]"},{"why":"Defines the transformer encoder, the attention-based family with the highest scores on the complete and driving datasets.","marker":"[54]"},{"why":"Precedent for end-to-end deep learning on physiological and visual signals, including input windowing and multimodal channel stacking.","marker":"[51]"}],"fun_headline_variants":["Domain shift defeats universal mental-load AI","n-back models fail on gaming and driving: domain-specific traps","Cross-domain transfer, not architecture, blocks task-load AI","Multimodal helps, but models still can't cross domains","Train on all domains: key to portable mental-load models"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the gaming task levels labeled 'low' and 'high' actually produce different levels of experienced mental load — Overcooked's education mode vs. time-pressured two-cook play, Hogwarts' free flight vs. guided flight and ring race — rather than merely different game content; the paper's own data show gaming performance did not decline with level, and its limitations section concedes models may exploit task-specific cues unrelated to experienced lo","fun_headline_variants_meta":{"raw":{"variants":["Domain shift defeats universal mental-load AI","n-back models fail on gaming and driving: domain-specific traps","Cross-domain transfer, not architecture, blocks task-load AI","Multimodal helps, but models still can't cross domains","Train on all domains: key to portable mental-load models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000563,"raw_usage":{"total_tokens":2504,"prompt_tokens":737,"completion_tokens":1767,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":1688}},"tokens_in":481,"tokens_out":1767,"duration_ms":15494,"temperature":1.0,"reasoning_tokens":1688,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:19:17.381805+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train each architecture on one domain alone — n-back, driving, or gaming — with identical windows, sensors, and splits, then test on the other two. The paper reports clear AUROC drops for these transfers; if an n-back-only model matched gaming-trained models on the gaming test set, the lack of cross-domain generalizability would be refuted. A companion check: if removing task-locked signals (gaze targets, pupil luminance) makes the drop vanish, the failure is cue learning, not load estimation; if the drop persists using only task-general signals (ECG, EDA, respiration), the load-induction labe","supporting_citations":[{"cited_title":"Heart Rate V ariability in Psychology: A Review of HRV Indices and an Analysis Tutorial,","cited_arxiv_id":null,"evidence_quote":"Supplies the driving-simulator dataset, recording equipment, and n-back protocol that the new gaming data extends, plus the label-mapping precedent."},{"cited_title":"Workload Assesment Methodology,","cited_arxiv_id":null,"evidence_quote":"Gives the inclusive definition of mental workload that guides dataset creation and the annotation categories."}],"review_version":1}