{"id":"e212db3f-5ca2-4013-9de3-9b5354a50daf","arxiv_id":"2411.12771","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"In a 19-participant pilot, an MLP using pupil dilation and fixation duration predicted binary low/high NASA-TLX mental-demand scores in a complex VR assembly task with 84% accuracy.","lead":"Researchers tested whether eye tracking could tell when people were working hard in a virtual reality training simulation for cold spray manufacturing. Their machine learning model, trained on pupil size and how long eyes fixated, predicted self-reported mental effort with 84 percent accuracy in a small pilot study.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.84 test accuracy is not interpretable without a participant-independent split; the described sliding-window design risks participant-level label leakage.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: sliding-window samples may share participants between training and test, making the 0.84 accuracy an artifact of participant-specific leakage rather than generalizable cognitive-load detection. This concern is well-supported by the text: Section 2.2 describes windowing but not the split; Section 3 reports test accuracy with no error bars or baseline; and the one-label-per-participant design makes the leakage mechanism concrete. The proposed leave-one-participant-out test directly resolves whether the central claim holds. Since the reader already reached a CONDITIONAL verdict based on this issue, my stress-test does not move the verdict; it reinforces the condition. If the participant-independent check succeeds, the paper's feasibility claim would be credible, though still preliminary. If it fails, the claim should be substantially weakened. No additional equally load-bearing concern was identified; the RF overfitting comment and small sample size are secondary to the leakage issue.","tokens_in":6291,"tokens_out":1373,"duration_ms":14799,"concrete_test":"Perform leave-one-participant-out cross-validation: for each of the 19 participants, train the MLP on all windows from the other 18 participants and test on all windows from the held-out participant, then report the mean accuracy across participants (and per-participant accuracy). If the mean participant-independent accuracy falls near chance or substantially below 0.84, the original test accuracy was inflated by participant leakage. Also report the low/high class split and majority-class baseline accuracy for comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 3 is that the MLP 'evaluated on the test dataset' achieves 0.84 accuracy and precision. The evaluation procedure is under-specified in a way that directly threatens this claim. Section 2.2 describes a 2000-sample sliding window but never states how windows were divided into training and test sets. Each participant provides a single NASA-TLX mental-demand score after the session, which is binarized into low/high and then assigned to every window from that participant. If the same participant contributes windows to both training and test sets, the classifier can exploit participant-specific eye-movement patterns rather than generalizable cognitive-load signatures. With only 19 valid datasets, even a few overlapping participants could substantially inflate test accuracy. The paper also does not report the number of participants in each class, the number of windows per participant, or any confidence intervals, and it does not compare against a majority-class or chance baseline. The authors explicitly call the study preliminary, but the 0.84 figure is presented as evidence of feasibility; without a participant-independent evaluation, it does not support that conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports an ongoing study of eye-tracking-based cognitive load detection in a complex spatiotemporal VR task. Twenty-two participants completed a cold-spray powder-feeder assembly/disassembly simulation in VR; 19 valid eye-tracking datasets and NASA-TLX scores were retained. From each participant's data the authors extracted mean pupil dilation and fixation duration, segmented the signals into 2000-sample sliding windows, and trained MLP and random forest classifiers to predict a binary target created by splitting the NASA-TLX mental-demand scores into low (1-4) and high (5-7). The central reported result is an MLP test accuracy and precision of 0.84 (RF accuracy 0.72), which the authors interpret as demonstrating the feasibility of detecting cognitive load in VR training and motivating future adaptive training systems.","tokens_in":6516,"tokens_out":4510,"duration_ms":43519,"significance":"If the reported evaluation were methodologically airtight, the paper would be a useful empirical data point for adaptive VR training in advanced manufacturing, since it uses a realistic 26-minute assembly task and commercially available VR eye tracking. The authors are appropriately modest in calling the study preliminary and in discussing privacy implications. However, the central claim is not currently supported by the reported methodology: the evaluation protocol permits participant-level label leakage, and the target threshold is chosen from the same sample used for evaluation. Because the contribution is purely empirical, these issues are load-bearing. The paper does not ship code, machine-checked derivations, or preregistered analyses, so the correctness of the evaluation is the only basis for the headline accuracy figure.","major_comments":[{"comment":"The manuscript never states how the 2000-sample sliding windows were partitioned into training and test sets. Every window from a given participant carries the same binarized NASA-TLX score, so if windows from one participant appear in both training and test, the classifier can learn participant-specific pupil or fixation patterns rather than a generalizable cognitive-load signature. The reported MLP accuracy of 0.84 in Section 3 is therefore not interpretable unless the split is participant-independent. Please report a leave-one-participant-out or grouped cross-validation procedure, the number of windows per participant, and the class distribution.","section":"Section 2.2 and Section 3"},{"comment":"The low/high labels are defined by a cutoff (1-4 vs 5-7) that the authors state was chosen after analyzing the distribution of the NASA-TLX mental-demand scores from the same 19 participants. This makes the target itself dependent on the evaluation sample; selecting a cutoff that produces an even split can inflate apparent separability and weakens the claim that the classifier separates objectively defined low and high cognitive load. The threshold should be justified independently, for example from prior literature or a pre-registered criterion, and the resulting class sizes should be reported.","section":"Section 2.2"},{"comment":"The reported metrics are point estimates with no baseline comparison, confidence intervals, or class counts. With only 19 participants and a binary label, a majority-class classifier or a model exploiting participant identity can achieve non-trivial accuracy. Table 1 alone cannot rule out these alternatives. Please provide the confusion matrix, per-class precision and recall, a majority-class or chance baseline, and the variance of metrics across cross-validation folds.","section":"Section 3 and Table 1"},{"comment":"A single post-session NASA-TLX score is used as the label for every window in a participant's session, but the stated goal is real-time adaptive training. The model is therefore trained to predict a session-level construct from short temporal windows, and no time-resolved ground truth is available to validate the real-time claim. The authors should either temper the real-time framing or evaluate on a workload measure that varies within the session.","section":"Section 2.2 and Section 4"}],"minor_comments":[{"comment":"\"Tabel 1\" should be \"Table 1\".","section":"Section 3"},{"comment":"The phrase \"thecold spray\" is missing a space and should read \"the cold spray\".","section":"Section 2"},{"comment":"Please specify the temporal duration of the 2000-sample sliding window given the recording rate stated in Section 2.1, and state the stride or overlap between consecutive windows.","section":"Section 2.2"},{"comment":"Reference [1] contains a typographical error in the author initials; it should be corrected to \"Y. V. Wong\" or the intended spelling.","section":"References"},{"comment":"The sentence \"The MLP's performance was carefully monitored, and the training process was optimized to maintain high precision and recall on both the training and test sets\" is vague; please describe the monitoring procedure and the overfitting checks quantitatively.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short preliminary report with a plausible research question and a realistic VR task, but the headline result is not yet credible because the evaluation protocol is underspecified in a way that permits participant-level leakage, and the target threshold is chosen post hoc. The problems are fixable by re-running the analysis with participant-independent cross-validation and transparent reporting of baselines and class sizes. I would not reject outright, but the 0.84 figure should not be cited until that re-analysis is provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine pilot study, and the authors are appropriately cautious in their framing, but the headline 0.84 accuracy is not something you can repeat from the paper. The stress-test concern lands. The evaluation split for the sliding-window samples is never described, and all windows from one participant share a single NASA-TLX label. If the same participant contributes to both training and test, the classifier can recognize the person rather than the cognitive load, and with only 19 valid datasets even a few overlapping participants could inflate the numbers.\n\nWhat is new: the cold-spray VR assembly task is a new domain for eye-tracking-based cognitive load classification, and the paper compares MLP and RF on fixation duration plus mean pupil dilation. The authors cite prior work reporting accuracies up to 88%, so they are not overclaiming novelty. They also explicitly call the study preliminary and list the low number of participants and limited feature set as limitations. That is the right tone for a pilot.\n\nWhere it gets soft: the missing split description is the load-bearing issue. Related to that, the low/high cutoff on the NASA-TLX mental demand subsection (1-4 vs 5-7) was chosen after inspecting the score distribution of the same 19 participants. The features themselves come from prior literature rather than being derived from the target, so the result is not forced, but the threshold choice should be disclosed and ideally prespecified or validated externally. The paper also reports no majority-class baseline, no confidence intervals, no number of windows per participant, and no per-class N. The claim that RF shows overfitting has no learning curves or train/test gap reported. None of these are fatal in a pilot, but together they mean the 0.84 is a point estimate without a known denominator. There is no code or data shipped, so nothing here is independently reproducible; that is normal for a pilot, but it lowers the evidentiary weight.\n\nWho this is for: people working on adaptive VR training and eye-tracking-based cognitive load detection. If the authors add participant-independent cross-validation (e.g., leave-one-participant-out), report a baseline and error bars, and disclose the threshold choice, this becomes a useful feasibility data point. As it stands, it is a motivated pilot write-up whose central quantitative claim needs verification.\n\nMy recommendation: send it to peer review rather than desk rejecting, but make participant-independent validation a required revision.","headline":"A real pilot study with an honest limitations section, but the 0.84 accuracy is not interpretable as written because the sliding-window evaluation split is underspecified and may leak participant identity.","tokens_in":7042,"tokens_out":2387,"would_cite":false,"duration_ms":24455,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that mean pupil dilation and fixation duration, fed into a five-layer neural network, can separate high from low self-reported mental workload in a 26-minute VR cold-spray assembly task with 84% accuracy.","keywords":["eye tracking","cognitive load","virtual reality training","pupil dilation","fixation duration","NASA-TLX","multi-layer perceptron","machine learning"],"falsifier":"Re-run the same pipeline with a strict participant-independent split (e.g., leave-one-participant-out: train on 18 participants, test on the remaining one, repeat) and check whether the mean accuracy stays near 0.84; if it falls toward the majority-class baseline, the reported accuracy was driven by participant-specific leakage rather than a general cognitive-load signal.","tokens_in":6121,"feed_emoji":"👁️","tokens_out":6377,"duration_ms":52040,"temperature":0.7,"pith_summary":"The paper argues that two eye-tracking signals—mean pupil dilation and fixation duration—can label a user's cognitive load in a complex, interactive VR training session, and it reports an initial demonstration on a cold-spray powder-feeder assembly task. The central result is that a five-layer Multi-Layer Perceptron trained on 2000-sample sliding windows from 19 participants reaches 0.84 accuracy and precision when predicting whether a participant's self-reported mental workload (NASA-TLX mental-demand split into low and high) was high. If the result holds, adaptive VR training could adjust difficulty, pacing, or help in real time based on the learner's cognitive state, instead of treating every user the same. The authors are careful to call this preliminary: the dataset is small, the features are limited to two eye signals, and privacy of gaze data is a stated concern.","feed_headline":"Pupil data picks out mental overload in VR training with 84% accuracy","feed_subtitle":"Two eye signals—pupil dilation and fixation time—separate high from low workload in a 26-minute VR assembly task.","key_machinery":"The load-bearing mechanism is a supervised classifier operating on a sliding window of eye-tracking signals. Raw gaze data are first denoised with a Fast Fourier Transform and normalized; a commercial eye-tracking processing pipeline, using the Velocity-Threshold Identification (I-VT) filter with a 60 ms minimum, converts the signal into fixation events. For each fixation, the feature vector is the fixation duration and the mean pupil dilation across both eyes. A sliding window of 2000 samples chunks the continuous stream into instances, and the binary target is the participant's NASA-TLX mental-demand group (low=1-4, high=5-7), assigned to every window from that participant. The architecture—an MLP with five hidden layers and hyperbolic tangent activations, trained with Adam at learning rate 0.00001 for 500 epochs with batch size 256—maps these windows to a high/low workload class; a Random Forest tuned with GridSearchCV serves as the comparison model.","core_discovery":"The central claim is that eye-tracking features carry enough information about mental workload to separate high from low cognitive load in a complex spatiotemporal VR task, even with a modest number of users. In this study, each participant performed a 12-step disassembly and 11-step assembly of a powder feeder in a cold-spray VR environment, taking about 26 minutes. The authors extracted fixation duration and the mean of both eyes' pupil dilation within each fixation using an I-VT filter, denoised and normalized the signals, and labeled each participant by the mental-demand subscale of NASA-TLX (scores 1-4 low, 5-7 high). A Multi-Layer Perceptron with five tanh hidden layers, trained with Adam at learning rate 0.00001 over 500 epochs on 2000-sample windows, achieved 0.84 accuracy and precision on the test set, with recall 0.94; a tuned Random Forest reached 0.72 accuracy. The authors interpret the MLP's performance as evidence that reliable, non-intrusive cognitive-load detection in dynamic VR training is feasible.","pith_inferences":["The reported 0.84 accuracy may overstate generalizable performance: because the paper does not specify whether the sliding-window test instances come from participants held out of training, the classifier could have learned participant-specific pupil patterns rather than a general workload signature; a leave-one-participant-out evaluation would settle this.","The NASA-TLX mental-demand label is one score per participant, yet the classifier receives many windows from the same participant; modeling the label as constant per participant means the effective sample size for generalization is closer to 19 than to the number of windows, and the confidence intervals around 0.84 would be wide.","If the participant-level generalization holds, the same pipeline could be transferred to other procedural VR training tasks (surgical, maintenance, logistics) that have similar step-by-step assembly structures.","Combining pupil dilation with other gaze events (saccades, blinks) or physiological signals (heart rate, electrodermal activity) could make the workload estimate robust to lighting changes and individual pupil-size baselines."],"forward_implications":["A real-time classifier using only pupil dilation and fixation duration could drive adaptive VR training that changes task difficulty or provides help when cognitive load is high.","The MLP's 0.84 accuracy over the Random Forest's 0.72 supports non-linear models for eye-tracking-based workload prediction in dynamic tasks.","Because the split of NASA-TLX mental-demand into low (1-4) and high (5-7) was balanced, the binary formulation ties the classifier to a practical go/no-go signal for adaptation.","Privacy and data security must be addressed before deployment because gaze data can reveal personal traits and health information.","Expanding features to saccade velocity, saccade amplitude, and blink rate could improve robustness, as the authors note."],"supporting_citations":[{"why":"Establishes the connection between eye-tracking metrics and cognitive load in VR head-mounted displays, motivating the use of pupil dilation and fixation.","marker":"[24]"},{"why":"Provides evidence that pupillometry can measure cognitive load in VR training, supporting the pupil-dilation predictor.","marker":"[14]"},{"why":"Shows fixation-related information reflects perceptual and cognitive load, supporting the fixation-duration predictor.","marker":"[15]"},{"why":"Finds fixation duration surpasses pupil size as a memory-load measure, strengthening the choice of fixation duration.","marker":"[16]"},{"why":"Demonstrates an eye-tracking machine-learning system for task-demand detection, a direct methodological precedent.","marker":"[22]"},{"why":"Presents a machine-learning approach for cognitive workload estimation from eye tracking, another direct precedent.","marker":"[23]"},{"why":"Supplies the NASA-TLX instrument used to construct the high/low mental-workload label.","marker":"[11]"}],"fun_headline_variants":["Eye tracking spots mental overload in VR training with 84% accuracy","Pupil dilation and fixation reveal high cognitive load in VR tasks","MLP model reads eye signals to predict VR training workload at 84%","How eye tracking detects cognitive load in complex VR assembly","84% accuracy: Eye-tracking predicts mental workload in VR training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands or falls on whether the 0.84 test accuracy was computed on windows from participants who were not seen during training; the paper does not describe the train/test split, and every window from one participant shares the same NASA-TLX label, so overlapping participants could let the model exploit individual pupil patterns instead of cognitive load.","fun_headline_variants_meta":{"raw":{"variants":["Eye tracking spots mental overload in VR training with 84% accuracy","Pupil dilation and fixation reveal high cognitive load in VR tasks","MLP model reads eye signals to predict VR training workload at 84%","How eye tracking detects cognitive load in complex VR assembly","84% accuracy: Eye-tracking predicts mental workload in VR training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1319,"prompt_tokens":956,"completion_tokens":363,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":273}},"tokens_in":572,"tokens_out":363,"duration_ms":3585,"temperature":1.0,"reasoning_tokens":273,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:13:06.851594+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same pipeline with a strict participant-independent split (e.g., leave-one-participant-out: train on 18 participants, test on the remaining one, repeat) and check whether the mean accuracy stays near 0.84; if it falls toward the majority-class baseline, the reported accuracy was driven by participant-specific leakage rather than a general cognitive-load signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the connection between eye-tracking metrics and cognitive load in VR head-mounted displays, motivating the use of pupil dilation and fixation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that pupillometry can measure cognitive load in VR training, supporting the pupil-dilation predictor."},{"cited_title":"Liu, K.-A","cited_arxiv_id":null,"evidence_quote":"Shows fixation-related information reflects perceptual and cognitive load, supporting the fixation-duration predictor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Finds fixation duration surpasses pupil size as a memory-load measure, strengthening the choice of fixation duration."},{"cited_title":"Shojaeizadeh, S","cited_arxiv_id":null,"evidence_quote":"Demonstrates an eye-tracking machine-learning system for task-demand detection, a direct methodological precedent."},{"cited_title":"Skaramagkas, E","cited_arxiv_id":null,"evidence_quote":"Presents a machine-learning approach for cognitive workload estimation from eye tracking, another direct precedent."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NASA-TLX instrument used to construct the high/low mental-workload label."}],"review_version":1}