{"id":"68e1eeac-bfb6-4535-b447-b7f437ec9f38","arxiv_id":"2509.05330","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A new VR-based emotion dataset with synchronized eye tracking, body motion, EMG, and GSR from 13 participants, evaluated with classifiers but with questionable validation.","lead":"This paper introduces MVRS, a small multimodal dataset of 13 participants who watched VR videos to elicit five emotions while their gaze, body movement, and physiological signals were recorded. The authors claim the synchronized data support accurate emotion classification, but the evidence is weakened by a tiny sample, potential data leakage, and unvalidated synchronization.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Participant-level leakage likely inflates the reported classification accuracies, so the dataset-quality claim is not yet supported.","rationale":"The reader's weakest_assumption focused on timestamp synchronization and stimulus-based labels, but the rationale also correctly identified participant-level leakage as an issue. I agree with the conditional verdict: the dataset contribution is potentially valuable, but the evidence for its quality is not credible as reported. The single most load-bearing problem is that the classification results—the only quantitative validation of emotion separability—are likely computed with random sample-wise splits over 13 participants, allowing leakage. A concrete GroupKFold rerun would settle this: if accuracies remain high, the dataset-quality claim is largely restored; if they collapse, the paper must be revised to temper claims and re-validate with participant-wise splits, synchronization-error reporting, and per-participant baselines. No ad hominem is intended; this is an experimental-design issue, not an allegation of misconduct. The paper does provide open code and dataset links, which is positive and makes the proposed test feasible.","tokens_in":33085,"tokens_out":2088,"duration_ms":30971,"concrete_test":"Re-run the evaluation pipeline using GroupKFold with participant_id as the grouping variable (or leave-one-participant-out), using the same feature sets, classifiers, and fusion strategies described in the paper. Compare the aggregate accuracy, macro F1, kappa, and MCC against Tables 1-6, especially Table 5 (autoencoder early fusion with XGBoost). If participant-wise accuracy drops substantially (e.g., from ~99% to below ~80%, or Relaxation recall collapses), the published numbers are inflated by sample-level leakage. As a secondary check, inspect whether participant-specific baseline standardization (e.g., subtracting per-participant GSR/EMG means before feature extraction) removes most of the discriminative signal; if it does, the classifiers were exploiting identity information rather than emotional state.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that MVRS is a reliable, high-quality resource rests mainly on the classification results in Tables 1-6, especially the 99.38% accuracy for autoencoder-based early fusion. But the evaluation protocol appears to use random sample-level 5-fold cross-validation after stacking all participants into a single file (Data Processing section: 'stack all participants' data into a single file for each modality'; Classification section: remove 'emotion and participant columns', then 80/20 train/test within folds). With only 13 participants, random splits almost certainly place samples from the same participant in both training and test folds. Because each participant contributes multiple temporally contiguous samples with consistent sensor placement, lighting, electrode impedance, and baseline physiology, the model can memorize participant-specific signatures rather than emotion-specific patterns. The reported accuracies therefore do not establish emotion separability; they may largely reflect identity leakage. The paper does not describe any participant-wise or group-wise split, and the Challenges section admits cross-device timestamp synchronization issues but provides no alignment-error measurement, which further weakens the claim of clean multimodal alignment. The most load-bearing concern is not the synchronization itself but the evaluation protocol: without participant-wise validation, the headline accuracy numbers cannot be interpreted as evidence of dataset quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MVRS, a multimodal VR-based emotion recognition dataset recorded from 13 participants (ages 12–60) under five emotion-eliciting VR video conditions: relaxation, fear, stress, sadness, and joy. Data consist of eye tracking from a webcam mounted inside a modified VR HMD, body motion from Kinect v2, and EMG/GSR from an Arduino-based setup. Features are extracted per modality, fused using two early-fusion schemes and one late-fusion scheme, and evaluated with Random Forest, Decision Tree, and XGBoost classifiers under 5-fold cross-validation. The paper reports very high classification accuracies, up to 99.38%, and concludes that the dataset is a reliable, comprehensive, and high-quality resource for the community.","tokens_in":33356,"tokens_out":3141,"duration_ms":46181,"significance":"If the dataset is delivered as described and if the classification results are valid, MVRS would fill a genuine gap: few publicly available datasets combine body motion, physiological signals, and eye tracking under immersive VR elicitation. The authors provide public code and dataset links on Kaggle, Zenodo, Figshare, and IEEE Data Port, and the protocol is described in unusual detail. However, the central quality claim rests on classification results that currently appear to be inflated by participant-level data leakage. The evaluation protocol does not separate participants between training and test folds, and synchronization accuracy is acknowledged as a challenge but never measured. The dataset may still be a useful contribution after a corrected evaluation, but the present results do not establish emotion separability or 'dataset quality' as claimed.","major_comments":[{"comment":"The evaluation protocol is not participant-independent. The Data Processing section states that 'all participants' data' are stacked into a single file per modality, and the Classification section says the pipeline 'remove[s] the emotion and participant columns' before 5-fold cross-validation with 80/20 splits. With only 13 participants, random sample-level splits will place the same participant's temporally contiguous, sensor-specific samples in both training and test folds. The model can therefore exploit participant-identity signatures rather than emotion-specific patterns. The reported accuracies in Tables 1–6, including the 99.38% headline in Table 5, cannot be interpreted as evidence of emotion separability. Please re-run the evaluation with participant-blocked cross-validation (e.g., leave-one-participant-out or grouped folds) and report per-participant results; if the dataset con","section":"Data Processing / Classification and Metrics"},{"comment":"The central multimodal claim depends on accurate timestamp synchronization across two separate laptops, one for eye tracking and one for Kinect plus Arduino. The Challenges section explicitly admits 'synchronization issues, especially with ensuring consistent timestamps across devices,' but the paper provides no measurement of alignment error, no offset statistics, and no validation using a known event (e.g., the countdown beep). Without an alignment-error analysis, the statement that all modalities are 'timestamp aligned' is unsupported. Please include a quantitative synchronization check and describe how residual misalignment was handled before feature extraction and fusion.","section":"Construction / Challenges and Multimodal Fusion"},{"comment":"Labels are assigned automatically from stimulus timestamps: 'we labeled the data for each row of modalities' raw data accordingly.' The questionnaires are mentioned as validating ground truth, but no self-report or post-stimulus emotion ratings are reported or used. A stimulus label is not necessarily the emotion the participant felt; this is particularly important because the paper's own acknowledgments note that responses depend on participant state, age, gender, culture, and other factors. Please either report the questionnaire results against the stimulus labels or explicitly restrict the dataset claim to 'stimulus-elicited conditions' rather than 'felt emotions.'","section":"Data Collection Protocol and Labeling"},{"comment":"The experimental reporting is internally inconsistent and insufficient for reproducibility. The text says '5-fold cross-validation' with '80 percent train and 20 percent test,' but Figure 21 is described as showing 'classification metrics across 30 folds.' It is not clear whether the tables report the aggregate of five folds, repeated five times, or something else. In addition, the paper does not report the number of feature vectors per participant/class, class sizes after stacking, or confidence intervals. Please clarify the exact splitting protocol and provide sample counts and per-fold variability for all tables.","section":"Classification and Metrics / Figure 21"}],"minor_comments":[{"comment":"Reference [69] is cited as 'Retracted: Human emotion recognition based on galvanic skin response signal feature selection and svm.' Citing a retracted paper as support for a feature set is problematic; please cite the original peer-reviewed version or replace the reference.","section":"References"},{"comment":"Typo: 'VaR's VR Video Player app' should likely be 'Var's VR Video Player' or the correct app name. Please verify.","section":"Introduction / VR HMD Setup"},{"comment":"The text refers to 'relaxation and neutral states' when discussing eye-tracking classification, but the dataset has no neutral class. Please correct the wording.","section":"Data Analysis"},{"comment":"The demographic description says '13 subjects or participant stimuli' and gives only an age range; no table reports sex, age distribution, or per-participant session details. Adding such a table would improve usability.","section":"Dataset Details"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like a dataset description with an illustrative baseline than a rigorously validated benchmark. The public dataset and code are valuable, and the VR elicitation protocol is interesting. However, the classification evaluation is currently the main evidence for dataset quality, and it is not trustworthy because of participant leakage and unverified synchronization. I believe a major revision with a proper participant-independent evaluation and a synchronization-validation section could make this a publishable dataset paper. If the authors cannot provide those, the dataset claim should be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: MVRS is a genuinely new but tiny dataset, and the paper's headline accuracies (up to 99.38%) are almost certainly inflated by participant leakage. The dataset resource is worth a look; the evaluation isn't.\n\nWhat the paper does well: it documents a creative, low-cost hardware setup — a modified VR headset with an internal webcam for eye tracking, Kinect v2, Arduino EMG/GSR — and it releases the data on Kaggle/Zenodo/Figshare/IEEE Data Port. That kind of reproducible data contribution is real. The protocol is described in unusual detail, including the stimulus order and reset videos. The authors also cite the relevant prior VR datasets (VREED, AVDOS-VR) and are honest about synchronization challenges in the Challenges section.\n\nThe soft spots, in order. First and most load-bearing, the classification evaluation. The data processing section says they stack all participants' data into a single file per modality, and the classification section describes removing 'emotion and participant columns' before 5-fold cross-validation. With 13 participants and roughly 65 samples (5 per participant), random sample-level splits will put the same participant in both training and test. The model can memorize person-specific sensor placement, impedance, lighting, and baseline physiology. The reported 99% numbers therefore don't demonstrate emotion separability. The paper never describes a participant-wise or group-wise split. That's a load-bearing flaw for the validity claim, not a minor one.\n\nSecond, synchronization is asserted, not verified. Two laptops, timestamps, but no measurement of alignment error. The authors admit cross-device issues. Without error bounds, claims of clean multimodal alignment are unsupported.\n\nThird, labels are stimulus-based. No self-report beyond the questionnaires is mentioned, and there's no check on whether participants actually felt the target emotion. For a dataset claiming ecological validity, that's a gap.\n\nNone of this kills the dataset as a resource. A 13-participant pilot with this modality mix could still be useful for method development. But the paper overclaims 'reliable, comprehensive, and high-quality' based on an invalid evaluation. The fix is straightforward: participant-wise folds, report per-participant variance, verify synchronization, add self-report labels, and temper the accuracy claims.\n\nWho this is for: affective computing researchers looking for a small, low-cost multimodal VR dataset to prototype on; anyone teaching data collection. Its value is as a pilot dataset, not as a benchmark. The paper itself flags the suspiciously uniform DT results, which shows some awareness, but it doesn't reconsider what that implies for the protocol.\n\nRecommendation: send to peer review, but require the re-evaluation before publication. The dataset contribution deserves referee time; the current validation does not support the conclusions.","headline":"Genuinely new but tiny VR multimodal dataset; the reported 99% accuracies are likely participant-leakage artifacts and the dataset-quality claim needs re-evaluation.","tokens_in":33822,"tokens_out":2111,"would_cite":false,"duration_ms":26968,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces MVRS, a multimodal emotion-recognition dataset built by showing 13 people aged 12 to 60 short VR videos intended to evoke relaxation, fear, stress, sadness, and joy while recording four synchronized streams: one-eye vid","keywords":["emotion recognition","multimodal dataset","virtual reality","eye tracking","body motion","physiological signals","multimodal fusion","affective computing"],"falsifier":"A concrete test: hold out entire participants, not random rows, and retrain. If accuracy drops sharply, the separability is driven by participant-specific patterns rather than emotion. A second test: collect self-reported felt emotion per clip; if a substantial fraction of reports disagree with the assigned labels, the ground truth is stimulus intent rather than felt emotion.","tokens_in":33018,"feed_emoji":"🎭","tokens_out":5142,"duration_ms":60745,"temperature":0.7,"pith_summary":"The paper introduces MVRS, a multimodal emotion-recognition dataset built by showing 13 people aged 12 to 60 short VR videos intended to evoke relaxation, fear, stress, sadness, and joy while recording four synchronized streams: one-eye video from a webcam inside a low-cost headset, skeletal body motion from a Kinect v2, and EMG and GSR signals from an Arduino. The authors want to establish that such synchronized multimodal VR data is a usable, separable resource for automatic emotion recognition, not merely a raw collection. To demonstrate this, they extract per-modality features, fuse them early and late, and classify with Random Forest, Decision Tree, and XGBoost. Reported accuracies reach 99.38 percent in the autoencoder-based early-fusion setting, with XGBoost the strongest classifier overall. The claimed value is that MVRS fills a gap: few emotion datasets combine body motion with physiological signals under immersive stimuli.","feed_headline":"VR plus budget sensors reads five emotions at up to 99%","feed_subtitle":"New MVRS dataset syncs eye, body, EMG and GSR signals; early fusion beats any single channel.","key_machinery":"The load-bearing mechanism is timestamp synchronization across three separate capture systems: a webcam-in-headset eye tracker, a Kinect v2 skeletal tracker, and an Arduino reading EMG and GSR analog pins. Each row in each modality is labeled from the stimulus timeline, so the intended emotion during a clip becomes the ground-truth label. The evaluation hinges on early fusion (concatenating selected features or an autoencoder's learned embedding) versus late fusion (majority voting per modality), with XGBoost as the strongest classifier on the fused feature spaces.","core_discovery":"MVRS's central claim is that synchronized multimodal recording under VR emotional videos yields an emotion-recognition dataset with clear class structure. Thirteen participants watched a fixed 11-minute 39-second sequence of VR clips (relaxation bookends, fear scenes from a horror game and film, stress scenes from Saw and a roller coaster, sadness from A Quiet Place, joy from Mr. Bean), while a low-cost webcam in the headset recorded the right eye, a Kinect v2 recorded skeletal joints, and an Arduino recorded forearm EMG and GSR, all timestamp-aligned. Feature extraction produced statistical, kinematic, spectral, and gaze descriptors; early fusion by feature selection or autoencoder and late","pith_inferences":["The reported near-99 percent figures come from five-fold cross-validation on rows, and the paper removes participant identity before splitting; the same participant's segments may appear in both training and test folds. Leave-one-participant-out evaluation is the natural follow-up and may yield lower, more honest accuracy.","The low-cost webcam-in-headset modification (one eye tracked through a commercial HMD) could lower the barrier for VR emotion datasets, but gaze features may be sensitive to headset slippage and right-eye-only geometry; tests comparing headset-aligned frames against per-frame iris registration would clarify this.","The 12-year-old participant watching horror and torture clips raises consent and distress considerations the paper touches only lightly; a future extension could add post-session emotional state checks or age-segmented stimuli.","Because labels come from the stimulus timeline rather than participant self-report, the dataset is best read as 'responses to intended emotion induction' rather than 'felt emotions'; this distinction matters for clinical or diagnostic uses."],"forward_implications":["Early fusion of body motion, physiological, and eye-tracking features consistently outperforms any single modality, so MVRS supports models that exploit cross-modal emotion cues.","Body motion is the strongest single channel (97.66 percent accuracy with XGBoost), suggesting skeletal tracking alone carries much of the emotion signal in VR settings.","Stress and fear are the most separable classes while relaxation is the hardest, so downstream users should expect class imbalance and use balancing strategies.","The timestamp-aligned, multi-format release (MP4, XEF, CSV/TXT/JSON) is designed for reproducibility of multimodal fusion pipelines."],"supporting_citations":[{"why":"DEAP supplies the standard physiological emotion benchmark whose single-modality reuse motivates MVRS's synchronized multimodal design.","marker":"[2]"},{"why":"VREED demonstrates VR eye-tracking plus physiological emotion elicitation, the closest prior setup MVRS extends with body motion and EMG.","marker":"[3]"},{"why":"Body-gesture emotion-recognition work provides the feature paradigm used for skeletal motion features.","marker":"[4]"},{"why":"WESAD documents wearable stress/affect data with GSR and ECG, serving as a comparison point for physiological-only emotion recognition.","marker":"[42]"},{"why":"AVDOS-VR is a recent VR affective video dataset with physiological signals and continuous ratings, anchoring MVRS in immersive elicitation literature.","marker":"[43]"},{"why":"Russell's circumplex model is used to justify placing the five emotion labels in valence-arousal space.","marker":"[64]"},{"why":"Roller-coaster VR studies support the use of a roller coaster clip as a stress elicitation stimulus.","marker":"[65]"}],"fun_headline_variants":["MVRS: low-cost VR sensors sync eye, body, and skin for emotion AI","Five emotions, one headset, three cheap sensors: MVRS dataset","VR emotional clips + webcam eye, Kinect body, Arduino skin = MVRS","Synchronized multimodal VR dataset for emotion recognition: MVRS","MVRS: low-cost eye, body, EMG, and GSR sync for five emotions"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The whole evaluation assumes that each participant actually felt the emotion the video was chosen to elicit, and that the timestamps from the two laptops and Arduino line up well enough that labels apply to all modalities; neither assumption is directly measured in the paper.","fun_headline_variants_meta":{"raw":{"variants":["MVRS: low-cost VR sensors sync eye, body, and skin for emotion AI","Five emotions, one headset, three cheap sensors: MVRS dataset","VR emotional clips + webcam eye, Kinect body, Arduino skin = MVRS","Synchronized multimodal VR dataset for emotion recognition: MVRS","MVRS: low-cost eye, body, EMG, and GSR sync for five emotions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000709,"raw_usage":{"total_tokens":3002,"prompt_tokens":687,"completion_tokens":2315,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":431,"completion_tokens_details":{"reasoning_tokens":2210}},"tokens_in":431,"tokens_out":2315,"duration_ms":19273,"temperature":1.0,"reasoning_tokens":2210,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:01:38.731346+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: hold out entire participants, not random rows, and retrain. If accuracy drops sharply, the separability is driven by participant-specific patterns rather than emotion. A second test: collect self-reported felt emotion per clip; if a substantial fraction of reports disagree with the assigned labels, the ground truth is stimulus intent rather than felt emotion.","supporting_citations":[{"cited_title":"Real-time Automatic Emotion Recognition from Body Gestures","cited_arxiv_id":"1402.5047","evidence_quote":"Body-gesture emotion-recognition work provides the feature paradigm used for skeletal motion features."},{"cited_title":"Introducing wesad, a multimodal dataset for wearable stress and affect detection","cited_arxiv_id":null,"evidence_quote":"WESAD documents wearable stress/affect data with GSR and ECG, serving as a comparison point for physiological-only emotion recognition."},{"cited_title":"Avdos-vr: Affective video database with physiological signals and continuous ratings collected remotely in vr","cited_arxiv_id":null,"evidence_quote":"AVDOS-VR is a recent VR affective video dataset with physiological signals and continuous ratings, anchoring MVRS in immersive elicitation literature."},{"cited_title":"A circumplex model of affect","cited_arxiv_id":null,"evidence_quote":"Russell's circumplex model is used to justify placing the five emotion labels in valence-arousal space."},{"cited_title":"Neurophysiological and subjective responses to a virtual downhill cycling exercise","cited_arxiv_id":null,"evidence_quote":"Roller-coaster VR studies support the use of a roller coaster clip as a stress elicitation stimulus."}],"review_version":1}