{"id":"e60b0516-f953-44e2-9248-d9a34707c7e4","arxiv_id":"2412.05296","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A nine-participant proof-of-concept shows that EEG during memory recall can be decoded into positive/neutral/negative trajectories, and those trajectories can steer text-to-music and text-to-image generation into personalized music videos.","lead":"This paper builds a system that reads brain-wave recordings during autobiographical memory recall and uses the detected emotional highs and lows to generate a personalized music video. If it holds up, it could turn neural signals into a new kind of personalized media, and the accompanying dataset gives the field a new test bed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The F1=0.9 decoding and r=0.265 reconstruction results may be in-sample artifacts: Section 4.2 trains CEBRA on all participants' EEG and keypress labels and leaves out only the KNN classifier, so held-out embeddings are shaped by their own labels. Retraining per fold is needed.","rationale":"The reader's weakest_assumption identifies the same load-bearing issue: the CEBRA leave-one-out protocol may leak the held-out participant's labels into the encoder, making the F1=0.9 and the subsequent user-study correlation in-sample. This is not a manufactured concern; Section 4.2 explicitly trains on all participants' data and only leaves out the KNN classifier. The central claim of the paper—that EEG-guided generation faithfully reconstructs affect-contextualized memories—depends on the decoded trajectory being a genuine out-of-sample neural readout. If the encoder has seen the held-out participant's keypresses, the trajectory is not an independent neural decode, and the r=0.265 result could be circular. The concern is concrete and testable because the code and dataset are released. I do not recommend rejection: the dataset and task are novel, and a clean retraining protocol could rescue the central claims. The reader already set CONDITIONAL, and this concern confirms that condition rather than moving the verdict. A secondary but non-load-bearing issue is that the preference result (5/9) is reported as 56% without a statistical test, but the CEBRA leakage is the primary threat to the paper's main assertion.","tokens_in":11851,"tokens_out":7443,"duration_ms":68491,"concrete_test":"Retrain the entire affect extractor from scratch for each of the 9 leave-one-out folds, excluding the held-out participant's EEG and keypress sequences from CEBRA training (and from KNN training), then recompute the test-weighted F1. Additionally, regenerate the 'real' videos using these strictly out-of-sample decoded trajectories and recompute the Wilcoxon comparison of cross-correlation with session-1 keypresses. If F1 falls toward chance or r_true no longer exceeds r_perm significantly, the central decoding and reconstruction claims do not survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is that RYM decodes individual affect trajectories from EEG with F1=0.9 and that the decoded trajectories drive faithful reconstruction (r=0.265, p=.012). The evaluation protocol in Section 4.2 does not establish out-of-sample decoding. The text says 'multi-session training with 10 participants' EEG and key-pressed valence sequence' and only then describes leave-one-out for the KNN classifier: 'We trained a k-nearest neighbor (KNN) classifier with 9 participants' identified embeddings to predict valence label at each timepoint and evaluated its predictive performance with the other's embeddings.' If CEBRA's contrastive encoder is trained on all participants, including the held-out participant's EEG and keypress labels, then the held-out participant's embedding geometry is shaped by their own labels, and the KNN evaluation measures label leakage rather than neural generalization. This also contaminates Section 5.4: the 'real' videos are generated from these leaked embeddings, so the higher cross-correlation with session-1 keypresses may reflect memorized training labels rather than EEG-based affect decoding. The inconsistency between Section 3 ('9 participants who completed both sessions') and Section 4.2 ('10 participants') compounds the ambiguity. Until CEBRA is retrained per fold with the held-out participant excluded from all training data, the F1=0.9 and faithful-reconstruction claims are unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RevisitAffectiveMemory, a new task and dataset for affect-contextualized reconstruction of autobiographical memories, and proposes RYM, a three-stage pipeline that decodes affect trajectories from EEG with CEBRA, aligns them with text prompts via an LLM, and generates music and video with MusicGen and Stable Diffusion. The dataset contains EEG, text descriptions, sketches, and music from nine participants who recalled mixed-affect memories while providing real-time keypress valence labels. The authors report F1=0.9 for leave-one-out valence decoding and a user study in which videos generated from decoded trajectories correlate more strongly with participants' session-1 valence reports than videos generated from permuted trajectories (r=0.265 vs 0.095, p=.012).","tokens_in":12073,"tokens_out":5733,"duration_ms":52416,"significance":"If the decoding and generation results hold, the paper would make a useful contribution to affective computing by introducing a multimodal dataset and an end-to-end demonstration of EEG-guided personalized media generation. The real-time keypress protocol for capturing affect dynamics during autobiographical recall is a thoughtful design choice, and making the dataset and code available is valuable. However, the central claims currently rest on an evaluation protocol that is not demonstrably out-of-sample. The F1=0.9 result and the user-study correlation can both be explained by label leakage through the CEBRA embeddings. These concerns are fixable with a properly nested cross-validation, but until then the paper's main quantitative claims are unverified.","major_comments":[{"comment":"The leave-one-out decoding evaluation as described is not out-of-sample for the CEBRA embeddings. Section 4.2 states that CEBRA was trained with '10 participants' EEG and key-pressed valence sequence' and then describes leave-one-out only for the KNN classifier, whose training uses 9 participants' identified embeddings and evaluation uses the remaining participant's embeddings. Because CEBRA's contrastive objective used the held-out participant's keypress labels to shape that participant's embedding, the KNN evaluation can reflect label leakage rather than generalization of neural decoding. Consequently, the F1=0.9 reported in Section 5.1 does not substantiate the abstract's claim of decoding individual affect trajectories. The authors should retrain CEBRA in each fold with the held-out participant excluded from all training data, report per-fold F1 with mean and standard deviation, and provide a per-class breakdown. They should also resolve the inconsistency between the 9 participants reported in Section 3 and the 10 participants named in Section 4.2.","section":"Section 4.2 and Section 5.1"},{"comment":"The user-study result is contaminated by the same protocol issue. The 'true' videos were generated from CEBRA-decoded affect sequences, but those embeddings were produced by a model trained on the same participants' keypress labels, so the higher cross-correlation with session-1 keypresses (r_true=0.265 vs r_perm=0.095, p=.012) may reflect memorized label sequences rather than EEG-based decoding. The authors should regenerate the 'true' videos using per-fold retrained, out-of-sample embeddings, or otherwise demonstrate that the advantage persists when the CEBRA encoder is trained without the test participant. In addition, the definition of 'best cross-correlation coefficients' (e.g., maximum over which time lags?) should be stated, and the appropriate test is a paired Wilcoxon signed-rank test rather than a rank-sum test, since each participant contributes one pair of coefficients from the true and permuted conditions.","section":"Section 5.4"},{"comment":"The mapping from decoded affect to text prompt is underspecified and the wording undercuts the contribution. Section 4.3 says 'we randomly selected words from our word bank and used them constructing prompts.' If word selection is random over the full word bank, then the affect extractor's valence labels do not determine the generated content, in which case 'affect-contextualized' is not supported. If words are sampled uniformly within the decoded positive or negative valence category (with neutral prompts unaltered), this should be stated explicitly, together with how the valence trajectory maps to prompt durations and to the crossfaded music segments described in Section 4.4.","section":"Section 4.3"}],"minor_comments":[{"comment":"Please specify whether the '1' and '3' keypresses are held continuously or are discrete events, and how the resulting sparse label sequence is converted to timepoint-level labels for CEBRA (e.g., forward-fill until the next press).","section":"Section 3.1"},{"comment":"The sentence 'The neighborhood parameters of KNN classifiers were fixed at the square of the number of the input time points' is ambiguous; presumably k = sqrt(number of time points), and the resulting value should be reported.","section":"Section 4.4"},{"comment":"Provide definitions and units for the CLIP and CLAP distances, and clarify the 'Affect Difference' rows; no statistical comparison is reported, so the claim of semantic coherence is descriptive only.","section":"Table 1 and Section 5.3"},{"comment":"Report confidence intervals or statistical tests for the eta-squared effect sizes; with n=9, a descriptive 'larger' comparison is not informative.","section":"Figure 5"},{"comment":"The preference result should be reported with exact counts (5 out of 9 participants), and the analysis should state how 'Both' and 'Neither' responses are handled; 56% is not a strong majority at n=9.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claims are plausible but currently under-supported by the described evaluation protocol. The key fix is nested leave-one-participant-out retraining of the encoder, and the user study should either reuse those out-of-sample embeddings or report the appropriate paired statistical test. The 9-versus-10 participant inconsistency and the 'randomly selected words' wording in Section 4.3 should be resolved in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely new task and dataset, and the pipeline is a reasonable integration of existing parts. But the load-bearing claim—decoding affect trajectories from EEG with F1=0.9—is not yet established. The evaluation section describes multi-session training on 10 participants' EEG and keypresses, then leave-one-out only for the KNN classifier. If CEBRA was fit on the full set, the held-out embeddings are shaped by their own labels, and the F1 is an in-sample fit. The paper also says 9 participants in one place and 10 in another, which does not help. Until CEBRA is retrained per fold, the decoding result is ambiguous.\n\nWhat's new: RevisitAffectiveMemory and the EEG-AffectiveMemory dataset, with essays, sketches, music, and EEG during autobiographical recall. That is a useful resource for affective neuroscience and BCI. The protocol of real-time keypresses during recall is sensible, and the authors are upfront that this is a proof-of-concept with a homogeneous sample. The integration of CEBRA with MusicGen and Stable Diffusion is straightforward but reasonable, and the qualitative examples show the pipeline can produce coherent outputs.\n\nSoft spots beyond the leakage issue: the preference result (5/9 = 56%) is reported without a statistical test, and the user study compares decoded trajectories against randomly permuted ones. If the decoded trajectories are in-sample, the higher cross-correlation (r=0.265) may reflect memorized labels rather than neural decoding. The quantitative CLIP/CLAP distances are descriptive, not inferential. None of these are fatal by themselves, but together they mean the paper's central claims rest on a single ambiguous evaluation.\n\nWho it's for: researchers working on EEG-based affect decoding or affective multimedia generation will want to look at the dataset and task. The paper deserves a serious referee, but the referee should require a re-analysis with proper out-of-sample decoding, per-subject error bars, and a real preference test.\n\nRecommendation: send it to review, but expect heavy revision. The dataset and task are worth preserving; the current analysis does not yet support the abstract's claims.","headline":"New task and dataset, but the F1=0.9 decoding claim is unverified: leave-one-out may only apply to the KNN, not to CEBRA training.","tokens_in":12694,"tokens_out":2558,"would_cite":false,"duration_ms":22331,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that EEG recorded during memory recall can be decoded into per-person affective trajectories with $F_1 = 0.9$, and that these trajectories—converted into text prompts—guide music-video generation that faithfully reflects…","keywords":["affective computing","EEG affect decoding","autobiographical memory","multimodal generation","personalized media","affect dynamics","neural signal interpretation","audiovisual synthesis"],"falsifier":"Retrain the affect encoder with one participant completely excluded (no EEG and no keypress labels from that person) before extracting that participant's trajectory, then recompute the leave-one-out valence $F_1$; if it is much lower than $0.9$, the reported score is an in-sample fit rather than a generalization result. Separately, re-run the session-1 vs session-2 cross-correlation analysis with many random permutations of the decoded trajectory; if the true-condition advantage over permuted trajectories (reported $r = 0.265$ vs $0.095$) does not reliably exceed chance, the alignment between decoded affect and generated content is not established.","tokens_in":11568,"feed_emoji":"🧠","tokens_out":7477,"duration_ms":63527,"temperature":0.7,"pith_summary":"RevisitAffectiveMemory is a new task: take EEG recorded while someone recalls an autobiographical memory, decode how their momentary feeling state changes across the recall, and use that decoded affect trajectory to generate a music video that re-creates the memory's emotional contour. The authors built a nine-participant dataset in which each person wrote an essay, sketched the scene, picked an accompanying song, and pressed keys to report positive/neutral/negative feelings during recall while EEG was recorded. Their RYM framework extracts per-person valence dynamics from the EEG with a contrastive-learning encoder (reported $F_1 = 0.9$ in leave-one-out decoding), converts the dynamics into affect-word prompts, and lets text-to-music and latent-diffusion image generation produce synchronized audiovisual output. In a user study, videos generated from decoded trajectories matched participants' own session-1 valence reports better than videos from permuted trajectories ($r = 0.265$ vs. $r = 0.095$, $p = .012$), and 56% of participants preferred the decoded-trajectory video. The paper positions this as a step toward personalized, affect-aware media creation driven directly by neural signals.","feed_headline":"EEG emotion decoding drives personalized memory videos","feed_subtitle":"Nine participants' recall EEGs were decoded into valence tracks (F1=0.9) and used to generate videos that matched their reported affect.","key_machinery":"The load-bearing component is the affect extractor: a contrastive-learning neural encoder that nonlinearly reduces multi-channel EEG into a low-dimensional latent space in which the auxiliary variable (participant keypress valence) separates the states. The extracted latent valence trajectory is then converted into text prompts through a prompted alignment step using a pre-trained language model plus a fixed emotion-word bank, and these prompts—together with the participant's chosen guiding melody in the case of music—drive a music generation model and a latent diffusion image model to produce the synchronized audiovisual piece.","core_discovery":"The central claim is that the temporal affect state during autobiographical recall is decodable from single-trial EEG at the individual level and that this decoded trajectory carries enough information to steer audiovisual generation so the output is affectively aligned with the original memory. The discovery is an application-level demonstration: a contrastive neural encoder trained on multiple sessions of EEG with keypress valence labels produces latent representations whose leave-one-out valence classification reaches $F_1 = 0.9$, and generated music videos conditioned on these decoded trajectories show significantly higher cross-correlation with participants' own affect ratings than videos conditioned on randomly permuted trajectories. The paper treats this as evidence that affective dynamics, rather than only static emotional labels, can be extracted from neural signals and translated into multimodal content.","pith_inferences":["The affect-text alignment bottleneck suggests that the same decoded EEG trajectory could be expressed in other modalities—for example, haptic or olfactory displays—if their prompt interfaces existed, a direction the paper does not explore.","A direct extension would be to replace the keypress self-reports with stimulus-derived or video-content labels, which would let the decoder be trained on external affective annotations; this would sacrifice personal idiosyncrasy but would allow application to memories no one rates in real time.","The framework does not test whether the decoded trajectories are specific to autobiographical recall or would also be produced by passively watching emotional material; comparing the two settings would clarify what the decoder actually encodes."],"forward_implications":["Decoding affect dynamics from EEG during natural recall works with a small single-session sample (nine participants), suggesting practical applications where per-user calibration on large datasets is not required.","Because generation is driven by a decoded valence trajectory rather than a single emotional label, the framework can render a memory whose feeling state changes over time as a music video with corresponding musical and visual shifts.","The session-1-to-session-2 comparison method—measuring cross-correlation between keypress rating trajectories—offers a general template for validating affect-guided generation without requiring ground-truth media, and the reported preference (56%) supports perceived fidelity.","The framework's modular design lets the affect-text alignment and generation stages be swapped with any text-conditioned generative model, so improvements in music and video synthesis can inherit into the reconstruction pipeline."],"supporting_citations":[{"why":"Supplies the contrastive-learning encoder that maps EEG to latent valence-separated embeddings.","marker":"[41]"},{"why":"Demonstrates the encoder's feasibility for EEG affect decoding and motivates the choice of latent dimensionality for cross-individual valence.","marker":"[53]"},{"why":"Provides the text-conditioned music generation model used to synthesize the audio from affect prompts and a guiding melody.","marker":"[9]"},{"why":"Provides the latent diffusion image model used to generate the video frames conditioned on the affect-aligned prompts.","marker":"[38]"},{"why":"Supports using real-time keypresses as a valid self-report measure of affective state during recall, the training signal for the decoder.","marker":"[48]"},{"why":"Establishes that memory recall elicits measurable short-term affective responses in EEG, grounding the feasibility of the task.","marker":"[4]"},{"why":"Shows that deep learning can classify emotions from EEG during memory recalls, the baseline this work extends to dynamic trajectories.","marker":"[11]"},{"why":"Validates the contrastive encoder in human neural decoding settings, justifying its use on human EEG.","marker":"[33]"}],"fun_headline_variants":["EEG to video: affect decoded at F1=0.9","Brain waves reconstruct your memory videos","Recall affect from EEG, generate video","EEG affect-guided memory video generation","Personalized memory videos from EEG affect"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the leave-one-out evaluation isolates generalization: the affect decoder must not have seen the held-out participant's EEG or keypress labels when producing that participant's embedding.","fun_headline_variants_meta":{"raw":{"variants":["EEG to video: affect decoded at F1=0.9","Brain waves reconstruct your memory videos","Recall affect from EEG, generate video","EEG affect-guided memory video generation","Personalized memory videos from EEG affect"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1193,"prompt_tokens":946,"completion_tokens":247,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":180}},"tokens_in":562,"tokens_out":247,"duration_ms":3068,"temperature":1.0,"reasoning_tokens":180,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:46:37.673073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the affect encoder with one participant completely excluded (no EEG and no keypress labels from that person) before extracting that participant's trajectory, then recompute the leave-one-out valence $F_1$; if it is much lower than $0.9$, the reported score is an in-sample fit rather than a generalization result. Separately, re-run the session-1 vs session-2 cross-correlation analysis with many random permutations of the decoded trajectory; if the true-condition advantage over permuted trajectories (reported $r = 0.265$ vs $0.095$) does not reliably exceed chance, the alignment between decoded affect and generated content is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the contrastive-learning encoder that maps EEG to latent valence-separated embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates the encoder's feasibility for EEG affect decoding and motivates the choice of latent dimensionality for cross-individual valence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports using real-time keypresses as a valid self-report measure of affective state during recall, the training signal for the decoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that memory recall elicits measurable short-term affective responses in EEG, grounding the feasibility of the task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that deep learning can classify emotions from EEG during memory recalls, the baseline this work extends to dynamic trajectories."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Validates the contrastive encoder in human neural decoding settings, justifying its use on human EEG."}],"review_version":1}